Add a Rawx Service

Description

This documentation explains how to add a new Rawx service to your cluster.

It will rely on the following 3 nodes cluster:

+--------------------+  +--------------------+  +--------------------+
|        node1       |  |        node2       |  |        node3       |
|                    |  |                    |  |                    |
|      10.0.0.36     |  |      10.0.0.37     |  |      10.0.0.38     |
+--------------------+  +--------------------+  +--------------------+
|                    |  |                    |  |                    |
|  account           |  |  account           |  |  account           |
|  meta0             |  |  meta0             |  |  meta0             |
|  meta1             |  |  meta1             |  |  meta1             |
|  meta2             |  |  meta2             |  |  meta2             |
|  rawx              |  |  rawx              |  |  rawx              |
|  rdir (rawx node2) |  |  rdir (rawx node3) |  |  rdir (rawx node1) |
|                    |  |                    |  |                    |
+--------------------+  +--------------------+  +--------------------+

Add a new rawx using ansible

This part of the documentation explains how to add a new Rawx service using ansible.

In this example, we will add a new mounted volume on the node1, which will automatically install a new rawx, a new rdir and a new oio-blob-indexer.

First, add a new volume in the host_vars/node1.yml configution file:

host_vars/node1.yml
---
openio_data_mounts:
  - { partition: '/dev/sdb', mountpoint: "/var/lib/oio/sds" }
  - { partition: '/dev/sdb1', mountpoint: "/var/lib/oio/sds1" }
openio_metadata_mounts:
  - { partition: '/dev/sdb', mountpoint: "/var/lib/oio/sds" }

Run the following command to configure your new mounted volume:

oiosds/products/sds/deploy.sh
 # ./deploy.sh

Then, on node1, restart the conscience agent:

# gridinit_cmd restart @conscienceagent

Check that the services have been correctly added:

 # gridinit_cmd status
 KEY                         STATUS      PID GROUP
 OPENIO-account-0            UP        17232 OPENIO,account,0
 OPENIO-beanstalkd-0         UP        16654 OPENIO,beanstalkd,0
 OPENIO-conscienceagent-0    UP         8420 OPENIO,conscienceagent,0
 OPENIO-memcached-0          UP        21677 OPENIO,memcached,0
 OPENIO-meta0-0              UP        21113 OPENIO,meta0,0
 OPENIO-meta1-0              UP        21196 OPENIO,meta1,0
 OPENIO-meta2-0              UP        18261 OPENIO,meta2,0
 OPENIO-oio-blob-indexer-0   UP        19725 OPENIO,oio-blob-indexer,0
 OPENIO-oio-blob-indexer-1   UP         6403 OPENIO,oio-blob-indexer,1
 OPENIO-oio-blob-rebuilder-0 UP        20371 OPENIO,oio-blob-rebuilder,0
 OPENIO-oio-event-agent-0    UP        20629 OPENIO,oio-event-agent,0
 OPENIO-oioproxy-0           UP        16979 OPENIO,oioproxy,0
 OPENIO-oioswift-0           UP        22457 OPENIO,oioswift,0
 OPENIO-rawx-0               UP        19401 OPENIO,rawx,0
 OPENIO-rawx-1               UP         5870 OPENIO,rawx,1
 OPENIO-rdir-0               UP        20127 OPENIO,rdir,0
 OPENIO-rdir-1               UP         6999 OPENIO,rdir,1
 OPENIO-redis-0              UP        15877 OPENIO,redis,0
 OPENIO-redissentinel-0      UP         3636 OPENIO,redissentinel,0
 OPENIO-zookeeper-0          UP         2944 OPENIO,zookeeper,0

Finally, launch the following command to create the assignation of your new rawx:

 # openio rdir bootstrap rawx
 +----------------+----------------+---------------+---------------+
 | Rdir           | Rawx           | Rdir location | Rawx location |
 +----------------+----------------+---------------+---------------+
 | 10.0.0.36:6300 | 10.0.0.38:6200 | node-1.0      | node-3.0      |
 | 10.0.0.38:6300 | 10.0.0.36:6201 | node-3.0      | node-1.1      |
 | 10.0.0.38:6300 | 10.0.0.37:6200 | node-3.0      | node-2.0      |
 | 10.0.0.37:6300 | 10.0.0.36:6200 | node-2.0      | node-1.0      |
 +----------------+----------------+---------------+---------------+

Add a new rawx manually

This part of the documentation explains how to manually add a new Rawx service to your cluster.

Furthermore, to secure your new rawx service, you will have two options:
  • Assign an existing Rdir to the new Rawx.
  • Install a new Rdir service (rdir-2) and assign it to the new Rawx.

Both options will be explained below.

Prerequisites

You must know the IP addresses and ports that will be used for your new services. In this example, we will use the following:

  • The new Rawx will listen on 10.0.0.36:6211
  • The new Rdir will listen on 10.0.0.38:6302

Configure a new Rawx

Create a new directory rawx-2 in /var/lib/oio/sds/OPENIO/

# mkdir /var/lib/oio/sds/OPENIO/rawx-2

Give the rights for this directory to the openio user:

# chown openio:openio /var/lib/oio/sds/OPENIO/rawx-2/

Create a new directory rawx-2 in /etc/oio/sds/OPENIO/.

# mkdir /etc/oio/sds/OPENIO/rawx-2

Give the rights for this directory to the openio user:

# chown openio:openio /etc/oio/sds/OPENIO/rawx-2/

Create a new configuration file (rawx-2-httpd.conf) in your new directory /etc/oio/sds/OPENIO/rawx-2.

The following configuration must be adapted to your new service:

  • Listen
  • PidFile
  • DocumentRoot
  • SetEnv INFO_SERVICES
  • SetEnv HOSTNAME
  • ErrorLog
  • CustomLog
  • grid_docroot
  • grid_dir_run
  • VirtualHost
/etc/oio/sds/OPENIO/rawx-2/rawx-2-httpd.conf
# OpenIO managed
LoadModule mpm_worker_module   /usr/lib64/httpd/modules/mod_mpm_worker.so
LoadModule authz_core_module   /usr/lib64/httpd/modules/mod_authz_core.so
LoadModule dav_module          /usr/lib64/httpd/modules/mod_dav.so
LoadModule mime_module         /usr/lib64/httpd/modules/mod_mime.so
LoadModule dav_rawx_module     /usr/lib64/httpd/modules/mod_dav_rawx.so
LoadModule setenvif_module     /usr/lib64/httpd/modules/mod_setenvif.so
LoadModule alias_module        /usr/lib64/httpd/modules/mod_alias.so
LoadModule env_module          /usr/lib64/httpd/modules/mod_env.so
LoadModule unixd_module        /usr/lib64/httpd/modules/mod_unixd.so
LoadModule log_config_module   /usr/lib64/httpd/modules/mod_log_config.so
LoadModule logio_module        /usr/lib64/httpd/modules/mod_logio.so

Alias / /x/

Listen          10.0.0.36:6211
PidFile         /run/rawx/OPENIO/rawx-2/OPENIO-rawx-2-httpd.pid
ServerRoot      /var/lib/oio/sds/OPENIO/coredump
ServerName      localhost
ServerSignature Off
ServerTokens    Prod
DocumentRoot    /var/lib/oio/sds/OPENIO/rawx-2
TypesConfig     /etc/mime.types

User  openio
Group openio

SetEnv INFO_SERVICES OIO,OPENIO,rawx,2
SetEnv LOG_TYPE access
SetEnv LEVEL INF
SetEnv HOSTNAME node-3.novalocal

SetEnvIf Remote_Addr "^" log-cid-out=1
SetEnvIf Remote_Addr "^" log-cid-in=0
SetEnvIf Request_Method "PUT" log-cid-in=1
SetEnvIf Request_Method "PUT" !log-cid-out
SetEnvIf log-cid-in 0 !log-cid-in

LogFormat "%{%b %d %T}t %{HOSTNAME}e %{INFO_SERVICES}e %{pid}P %{tid}P %{LOG_TYPE}e %{LEVEL}e %{Host}i %a:%{remote}p %m %>s %D %I %{x-oio-chunk-meta-container-id}i %{x-oio-req-id}i %U" log/cid-in
LogFormat "%{%b %d %T}t %{HOSTNAME}e %{INFO_SERVICES}e %{pid}P %{tid}P %{LOG_TYPE}e %{LEVEL}e %{Host}i %a:%{remote}p %m %>s %D %O %{x-oio-chunk-meta-container-id}o %{x-oio-req-id}i %U" log/cid-out
ErrorLog /var/log/oio/sds/OPENIO/rawx-2/rawx-2-httpd-errors.log
SetEnvIf Request_URI "/(stat|info)$" nolog=1

SetEnvIf nolog 1 !log-cid-out
SetEnvIf nolog 1 !log-cid-in

CustomLog /var/log/oio/sds/OPENIO/rawx-2/rawx-2-httpd-access.log log/cid-out env=log-cid-out
CustomLog /var/log/oio/sds/OPENIO/rawx-2/rawx-2-httpd-access.log log/cid-in  env=log-cid-in

<IfModule worker.c>
MaxRequestsPerChild 0
MaxSpareThreads 256
MinSpareThreads 32
ServerLimit 16
StartServers 1
ThreadsPerChild 256
</IfModule>


DavDepthInfinity Off

grid_docroot    /var/lib/oio/sds/OPENIO/rawx-2
# How many hexdigits must be used to name the indirection directories
grid_hash_width  3
# How many levels of directories are used to store chunks
grid_hash_depth 1
# At the end of an upload, perform a fsync() on the chunk file itself
grid_fsync      enabled
# At the end of an upload, perform a fsync() on the directory holding the chunk
grid_fsync_dir  enabled
# Preallocate space for the chunk file (enabled by default)
#grid_fallocate enabled
# Enable compression ('zlib' or 'lzo' or 'off')
grid_compression off
grid_namespace  OPENIO
grid_dir_run    /run/rawx/OPENIO/rawx-2

<Directory />
DAV rawx
AllowOverride None
Require all granted
Options -SymLinksIfOwnerMatch -FollowSymLinks -Includes -Indexes
</Directory>

<VirtualHost 10.0.0.36:6211>
# DO NOT REMOVE (even if empty) !
</VirtualHost>

Create a new directory rawx-2 in /var/log/oio/sds/OPENIO/.

# mkdir /var/log/oio/sds/OPENIO/rawx-2

Give the rights for this directory to the openio user:

# chown openio:openio /var/log/oio/sds/OPENIO/rawx-2/
# chmod 770 /var/log/oio/sds/OPENIO/rawx-2/

Create a new directory rawx-2 in /run/rawx/OPENIO/.

# mkdir /run/rawx/OPENIO/rawx-2

Give the rights for this directory to the openio user:

# chown openio:openio /run/rawx/OPENIO/rawx-2/

Create a new configuration file (OPENIO-rawx-2.conf) in the /etc/gridinit.d/ directory:

/etc/gridinit.d/OPENIO-rawx-2.conf
 # OpenIO managed
 [Service.OPENIO-rawx-2]
 command=/usr/sbin/httpd -D FOREGROUND  -f /etc/oio/sds/OPENIO/rawx-2/rawx-2-httpd.conf
 enabled=true
 start_at_boot=true
 on_die=respawn
 group=OPENIO,rawx,2
 uid=openio
 gid=openio
 env.PATH=/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin

Give the rights for this file to the openio user:

# chown openio:openio /etc/gridinit.d/OPENIO-rawx-2.conf

Create a new configuration file (rawx-2.yml) in the /etc/oio/sds/OPENIO/watch/ directory:

The following configuration must be adapted to your new service:

  • host
  • port
  • type: volume, path
/etc/oio/sds/OPENIO/watch/rawx-2.yml
# OpenIO managed
---
host: 10.0.0.36
port: 6211
type: rawx
location: node-3.0
checks:
 - {type: http, uri: /info}
stats:
 - {type: volume, path: /var/lib/oio/sds/OPENIO/rawx-2}
 - {type: rawx, path: /stat}
 - {type: system}
...

Give the rights for this file to the openio user:

# chown openio:openio /etc/oio/sds/OPENIO/watch/rawx-2.yml

Then, to make your new rawx service available, you must reload the configuration and start the service:

# gridinit_cmd reload
# gridinit_cmd start OPENIO-rawx-2

And restart the conscience agent:

# gridinit_cmd restart @conscienceagent

Rdir assignation

In order to secure the new rawx, you must assign your new rawx to a rdir service.

You have two options:
  • Assign an existing Rdir to the new Rawx.
  • Install a new Rdir service (rdir-2) and assign it to the new Rawx.

Existing Rdir assignation

In the example below, you can see that the new rawx (10.0.0.36:6211) has no rdir assignation:

# openio rdir assignments rawx --aggregated
+----------------+-----------------+----------------+
| Rdir           | Number of bases | Bases          |
+----------------+-----------------+----------------+
| 10.0.0.36:6301 |               1 | 10.0.0.37:6201 |
| 10.0.0.37:6301 |               1 | 10.0.0.38:6201 |
| 10.0.0.38:6301 |               1 | 10.0.0.36:6201 |
| n/a            |               1 | 10.0.0.36:6211 |
+----------------+-----------------+----------------+

You must launch the following command to create the assignation:

# openio rdir bootstrap rawx
+----------------+----------------+-----------------+-----------------+
| Rdir           | Rawx           | Rdir location   | Rawx location   |
+----------------+----------------+-----------------+-----------------+
| 10.0.0.36:6301 | 10.0.0.37:6201 | node-1          | node-2          |
| 10.0.0.37:6301 | 10.0.0.38:6201 | node-2          | node-3          |
| 10.0.0.37:6301 | 10.0.0.36:6211 | node-2          | node-1          |
| 10.0.0.38:6301 | 10.0.0.36:6201 | node-3          | node-1          |
+----------------+----------------+-----------------+-----------------+

New Rdir assignation

First, you have to install a new rdir service on another server.

Create a new directory rdir-2 in /var/lib/oio/sds/OPENIO/.

# mkdir /var/lib/oio/sds/OPENIO/rdir-2/

Give the rights for this directory to the openio user:

# chown openio:openio /var/lib/oio/sds/OPENIO/rdir-2/

Create a new directory rdir-2 in /etc/oio/sds/OPENIO/

# mkdir /etc/oio/sds/OPENIO/rdir-2/

Give the rights for this directory to the openio user:

chown openio:openio /etc/oio/sds/OPENIO/rdir-2/

Create a new configuration file (rdir-2.conf) in your newly created directory /etc/oio/sds/OPENIO/rdir-2:

/etc/oio/sds/OPENIO/rdir-2/rdir-2.conf
# OpenIO managed
[rdir-server]
bind_addr = 10.0.0.38
bind_port = 6302
namespace = OPENIO
# Currently, only 1 worker is allowed to avoid concurrent access to leveldb database
workers = 1
worker_class = sync
threads = 1
db_path= /var/lib/oio/sds/OPENIO/rdir-2
log_facility = LOG_LOCAL0
log_level = info
log_address = /dev/log
syslog_prefix = OIO,OPENIO,rdir,2

Give the rights for this file to the openio user:

chown openio:openio /etc/oio/sds/OPENIO/rdir-2/rdir-2.conf

Create a new configuration file (OPENIO-rdir-2.conf) in the /etc/gridinit.d/ directory:

/etc/gridinit.d/OPENIO-rdir-2
# OpenIO managed
[Service.OPENIO-rdir-2]
command=/usr/bin/oio-rdir-server /etc/oio/sds/OPENIO/rdir-2/rdir-2.conf
enabled=true
start_at_boot=true
on_die=respawn
group=OPENIO,rdir,2
uid=openio
gid=openio
env.PATH=/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin

Give the rights for this file to the openio user:

chown openio:openio /etc/gridinit.d/OPENIO-rdir-2.conf

Create a new configuration file (rdir-2.yml) in the /etc/oio/sds/OPENIO/watch/ directory:

/etc/oio/sds/OPENIO/watch/rdir-2.yml
# OpenIO managed
host: 10.0.0.38
port: 6302
type: rdir
location: node-3.2
checks:
  - {type: tcp}
stats:
  - {type: volume, path: /var/lib/oio/sds/OPENIO/rdir-2}
  - {type: http, path: /status, parser: json}
  - {type: system}

Give the rights for this file to the openio user:

chown openio:openio /etc/oio/sds/OPENIO/watch/rdir-2.yml

Create a new directory rdir-2 in /var/log/oio/sds/OPENIO/.

# mkdir /var/log/oio/sds/OPENIO/rdir-2

Give the rights for this directory to the openio user:

# chown openio:openio /var/log/oio/sds/OPENIO/rdir-2/
# chmod 770 /var/log/oio/sds/OPENIO/rdir-2/

Then, to make your new rdir service available, you must reload the configuration and start the service:

# gridinit_cmd reload
# gridinit_cmd start OPENIO-rdir-2

And restart the conscience agent:

# gridinit_cmd restart @conscienceagent

Then, you must unlock your new service:

# openio cluster unlock rdir 10.0.0.38:6302

Now, you must assign your new rawx to a rdir service.

In the example below, you can see that the new rawx (10.0.0.36:6211) has no rdir assignation, and your new rdir (10.0.0.38:6302) does not manage any rawx:

# openio rdir assignments rawx --aggregated
+----------------+-----------------+----------------+
| Rdir           | Number of bases | Bases          |
+----------------+-----------------+----------------+
| 10.0.0.36:6301 |               1 | 10.0.0.37:6201 |
| 10.0.0.37:6301 |               1 | 10.0.0.38:6201 |
| 10.0.0.38:6302 |               0 |                |
| 10.0.0.38:6301 |               1 | 10.0.0.36:6201 |
| n/a            |               1 | 10.0.0.36:6211 |
+----------------+-----------------+----------------+

You must launch the following command to create the assignation:

# openio rdir bootstrap rawx
+----------------+----------------+-----------------+-----------------+
| Rdir           | Rawx           | Rdir location   | Rawx location   |
+----------------+----------------+-----------------+-----------------+
| 10.0.0.36:6301 | 10.0.0.37:6201 | node-1          | node-2          |
| 10.0.0.37:6301 | 10.0.0.38:6201 | node-2          | node-3          |
| 10.0.0.37:6301 | 10.0.0.36:6211 | node-2          | node-1          |
| 10.0.0.38:6301 | 10.0.0.36:6201 | node-3          | node-1          |
+----------------+----------------+-----------------+-----------------+

Finalize the installation

Finally, you must unlock your new service:

# openio cluster unlock rawx 10.0.0.36:6211

You can check that your new service is available using the openio cluster list command:

 # openio cluster list rawx

 +---------+----------------+------------+---------------------------------+------------+-------+------+-------+
 | Type    | Addr           | Service Id | Volume                          | Location   | Slots | Up   | Score |
 +---------+----------------+------------+---------------------------------+------------+-------+------+-------+
 | rawx    | 10.0.0.36:6211 | n/a        | /var/lib/oio/sds/OPENIO/rawx-2  | node-1     | n/a   | True |    90 |
 | rawx    | 10.0.0.38:6201 | n/a        | /var/lib/oio/sds/OPENIO/rawx-1  | node-3     | n/a   | True |    92 |
 | rawx    | 10.0.0.36:6201 | n/a        | /var/lib/oio/sds/OPENIO/rawx-1  | node-1     | n/a   | True |    90 |
 | rawx    | 10.0.0.37:6201 | n/a        | /var/lib/oio/sds/OPENIO/rawx-1  | node-2     | n/a   | True |    91 |
 +---------+----------------+------------+---------------------------------+------------+-------+------+-------+