Add a Rawx Service
Description
This documentation explains how to add a new Rawx service to your cluster.
It will rely on the following 3 nodes cluster:
+--------------------+ +--------------------+ +--------------------+
| node1 | | node2 | | node3 |
| | | | | |
| 10.0.0.36 | | 10.0.0.37 | | 10.0.0.38 |
+--------------------+ +--------------------+ +--------------------+
| | | | | |
| account | | account | | account |
| meta0 | | meta0 | | meta0 |
| meta1 | | meta1 | | meta1 |
| meta2 | | meta2 | | meta2 |
| rawx | | rawx | | rawx |
| rdir (rawx node2) | | rdir (rawx node3) | | rdir (rawx node1) |
| | | | | |
+--------------------+ +--------------------+ +--------------------+
Add a new rawx using ansible
This part of the documentation explains how to add a new Rawx service using ansible.
In this example, we will add a new mounted volume on the node1, which will automatically install a new rawx, a new rdir and a new oio-blob-indexer.
First, add a new volume in the host_vars/node1.yml configution file:
---
openio_data_mounts:
- { partition: '/dev/sdb', mountpoint: "/var/lib/oio/sds" }
- { partition: '/dev/sdb1', mountpoint: "/var/lib/oio/sds1" }
openio_metadata_mounts:
- { partition: '/dev/sdb', mountpoint: "/var/lib/oio/sds" }
Run the following command to configure your new mounted volume:
# ./deploy.sh
Then, on node1, restart the conscience agent:
# gridinit_cmd restart @conscienceagent
Check that the services have been correctly added:
# gridinit_cmd status
KEY STATUS PID GROUP
OPENIO-account-0 UP 17232 OPENIO,account,0
OPENIO-beanstalkd-0 UP 16654 OPENIO,beanstalkd,0
OPENIO-conscienceagent-0 UP 8420 OPENIO,conscienceagent,0
OPENIO-memcached-0 UP 21677 OPENIO,memcached,0
OPENIO-meta0-0 UP 21113 OPENIO,meta0,0
OPENIO-meta1-0 UP 21196 OPENIO,meta1,0
OPENIO-meta2-0 UP 18261 OPENIO,meta2,0
OPENIO-oio-blob-indexer-0 UP 19725 OPENIO,oio-blob-indexer,0
OPENIO-oio-blob-indexer-1 UP 6403 OPENIO,oio-blob-indexer,1
OPENIO-oio-blob-rebuilder-0 UP 20371 OPENIO,oio-blob-rebuilder,0
OPENIO-oio-event-agent-0 UP 20629 OPENIO,oio-event-agent,0
OPENIO-oioproxy-0 UP 16979 OPENIO,oioproxy,0
OPENIO-oioswift-0 UP 22457 OPENIO,oioswift,0
OPENIO-rawx-0 UP 19401 OPENIO,rawx,0
OPENIO-rawx-1 UP 5870 OPENIO,rawx,1
OPENIO-rdir-0 UP 20127 OPENIO,rdir,0
OPENIO-rdir-1 UP 6999 OPENIO,rdir,1
OPENIO-redis-0 UP 15877 OPENIO,redis,0
OPENIO-redissentinel-0 UP 3636 OPENIO,redissentinel,0
OPENIO-zookeeper-0 UP 2944 OPENIO,zookeeper,0
Finally, launch the following command to create the assignation of your new rawx:
# openio rdir bootstrap rawx
+----------------+----------------+---------------+---------------+
| Rdir | Rawx | Rdir location | Rawx location |
+----------------+----------------+---------------+---------------+
| 10.0.0.36:6300 | 10.0.0.38:6200 | node-1.0 | node-3.0 |
| 10.0.0.38:6300 | 10.0.0.36:6201 | node-3.0 | node-1.1 |
| 10.0.0.38:6300 | 10.0.0.37:6200 | node-3.0 | node-2.0 |
| 10.0.0.37:6300 | 10.0.0.36:6200 | node-2.0 | node-1.0 |
+----------------+----------------+---------------+---------------+
Add a new rawx manually
This part of the documentation explains how to manually add a new Rawx service to your cluster.
- Furthermore, to secure your new rawx service, you will have two options:
- Assign an existing Rdir to the new Rawx.
- Install a new Rdir service (rdir-2) and assign it to the new Rawx.
Both options will be explained below.
Prerequisites
You must know the IP addresses and ports that will be used for your new services. In this example, we will use the following:
- The new Rawx will listen on
10.0.0.36:6211 - The new Rdir will listen on
10.0.0.38:6302
Configure a new Rawx
Create a new directory rawx-2 in /var/lib/oio/sds/OPENIO/
# mkdir /var/lib/oio/sds/OPENIO/rawx-2
Give the rights for this directory to the openio user:
# chown openio:openio /var/lib/oio/sds/OPENIO/rawx-2/
Create a new directory rawx-2 in /etc/oio/sds/OPENIO/.
# mkdir /etc/oio/sds/OPENIO/rawx-2
Give the rights for this directory to the openio user:
# chown openio:openio /etc/oio/sds/OPENIO/rawx-2/
Create a new configuration file (rawx-2-httpd.conf) in your new directory /etc/oio/sds/OPENIO/rawx-2.
The following configuration must be adapted to your new service:
- Listen
- PidFile
- DocumentRoot
- SetEnv INFO_SERVICES
- SetEnv HOSTNAME
- ErrorLog
- CustomLog
- grid_docroot
- grid_dir_run
- VirtualHost
# OpenIO managed
LoadModule mpm_worker_module /usr/lib64/httpd/modules/mod_mpm_worker.so
LoadModule authz_core_module /usr/lib64/httpd/modules/mod_authz_core.so
LoadModule dav_module /usr/lib64/httpd/modules/mod_dav.so
LoadModule mime_module /usr/lib64/httpd/modules/mod_mime.so
LoadModule dav_rawx_module /usr/lib64/httpd/modules/mod_dav_rawx.so
LoadModule setenvif_module /usr/lib64/httpd/modules/mod_setenvif.so
LoadModule alias_module /usr/lib64/httpd/modules/mod_alias.so
LoadModule env_module /usr/lib64/httpd/modules/mod_env.so
LoadModule unixd_module /usr/lib64/httpd/modules/mod_unixd.so
LoadModule log_config_module /usr/lib64/httpd/modules/mod_log_config.so
LoadModule logio_module /usr/lib64/httpd/modules/mod_logio.so
Alias / /x/
Listen 10.0.0.36:6211
PidFile /run/rawx/OPENIO/rawx-2/OPENIO-rawx-2-httpd.pid
ServerRoot /var/lib/oio/sds/OPENIO/coredump
ServerName localhost
ServerSignature Off
ServerTokens Prod
DocumentRoot /var/lib/oio/sds/OPENIO/rawx-2
TypesConfig /etc/mime.types
User openio
Group openio
SetEnv INFO_SERVICES OIO,OPENIO,rawx,2
SetEnv LOG_TYPE access
SetEnv LEVEL INF
SetEnv HOSTNAME node-3.novalocal
SetEnvIf Remote_Addr "^" log-cid-out=1
SetEnvIf Remote_Addr "^" log-cid-in=0
SetEnvIf Request_Method "PUT" log-cid-in=1
SetEnvIf Request_Method "PUT" !log-cid-out
SetEnvIf log-cid-in 0 !log-cid-in
LogFormat "%{%b %d %T}t %{HOSTNAME}e %{INFO_SERVICES}e %{pid}P %{tid}P %{LOG_TYPE}e %{LEVEL}e %{Host}i %a:%{remote}p %m %>s %D %I %{x-oio-chunk-meta-container-id}i %{x-oio-req-id}i %U" log/cid-in
LogFormat "%{%b %d %T}t %{HOSTNAME}e %{INFO_SERVICES}e %{pid}P %{tid}P %{LOG_TYPE}e %{LEVEL}e %{Host}i %a:%{remote}p %m %>s %D %O %{x-oio-chunk-meta-container-id}o %{x-oio-req-id}i %U" log/cid-out
ErrorLog /var/log/oio/sds/OPENIO/rawx-2/rawx-2-httpd-errors.log
SetEnvIf Request_URI "/(stat|info)$" nolog=1
SetEnvIf nolog 1 !log-cid-out
SetEnvIf nolog 1 !log-cid-in
CustomLog /var/log/oio/sds/OPENIO/rawx-2/rawx-2-httpd-access.log log/cid-out env=log-cid-out
CustomLog /var/log/oio/sds/OPENIO/rawx-2/rawx-2-httpd-access.log log/cid-in env=log-cid-in
<IfModule worker.c>
MaxRequestsPerChild 0
MaxSpareThreads 256
MinSpareThreads 32
ServerLimit 16
StartServers 1
ThreadsPerChild 256
</IfModule>
DavDepthInfinity Off
grid_docroot /var/lib/oio/sds/OPENIO/rawx-2
# How many hexdigits must be used to name the indirection directories
grid_hash_width 3
# How many levels of directories are used to store chunks
grid_hash_depth 1
# At the end of an upload, perform a fsync() on the chunk file itself
grid_fsync enabled
# At the end of an upload, perform a fsync() on the directory holding the chunk
grid_fsync_dir enabled
# Preallocate space for the chunk file (enabled by default)
#grid_fallocate enabled
# Enable compression ('zlib' or 'lzo' or 'off')
grid_compression off
grid_namespace OPENIO
grid_dir_run /run/rawx/OPENIO/rawx-2
<Directory />
DAV rawx
AllowOverride None
Require all granted
Options -SymLinksIfOwnerMatch -FollowSymLinks -Includes -Indexes
</Directory>
<VirtualHost 10.0.0.36:6211>
# DO NOT REMOVE (even if empty) !
</VirtualHost>
Create a new directory rawx-2 in /var/log/oio/sds/OPENIO/.
# mkdir /var/log/oio/sds/OPENIO/rawx-2
Give the rights for this directory to the openio user:
# chown openio:openio /var/log/oio/sds/OPENIO/rawx-2/
# chmod 770 /var/log/oio/sds/OPENIO/rawx-2/
Create a new directory rawx-2 in /run/rawx/OPENIO/.
# mkdir /run/rawx/OPENIO/rawx-2
Give the rights for this directory to the openio user:
# chown openio:openio /run/rawx/OPENIO/rawx-2/
Create a new configuration file (OPENIO-rawx-2.conf) in the /etc/gridinit.d/ directory:
# OpenIO managed
[Service.OPENIO-rawx-2]
command=/usr/sbin/httpd -D FOREGROUND -f /etc/oio/sds/OPENIO/rawx-2/rawx-2-httpd.conf
enabled=true
start_at_boot=true
on_die=respawn
group=OPENIO,rawx,2
uid=openio
gid=openio
env.PATH=/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin
Give the rights for this file to the openio user:
# chown openio:openio /etc/gridinit.d/OPENIO-rawx-2.conf
Create a new configuration file (rawx-2.yml) in the /etc/oio/sds/OPENIO/watch/ directory:
The following configuration must be adapted to your new service:
- host
- port
- type: volume, path
# OpenIO managed
---
host: 10.0.0.36
port: 6211
type: rawx
location: node-3.0
checks:
- {type: http, uri: /info}
stats:
- {type: volume, path: /var/lib/oio/sds/OPENIO/rawx-2}
- {type: rawx, path: /stat}
- {type: system}
...
Give the rights for this file to the openio user:
# chown openio:openio /etc/oio/sds/OPENIO/watch/rawx-2.yml
Then, to make your new rawx service available, you must reload the configuration and start the service:
# gridinit_cmd reload
# gridinit_cmd start OPENIO-rawx-2
And restart the conscience agent:
# gridinit_cmd restart @conscienceagent
Rdir assignation
In order to secure the new rawx, you must assign your new rawx to a rdir service.
- You have two options:
- Assign an existing Rdir to the new Rawx.
- Install a new Rdir service (rdir-2) and assign it to the new Rawx.
Existing Rdir assignation
In the example below, you can see that the new rawx (10.0.0.36:6211) has no rdir assignation:
# openio rdir assignments rawx --aggregated
+----------------+-----------------+----------------+
| Rdir | Number of bases | Bases |
+----------------+-----------------+----------------+
| 10.0.0.36:6301 | 1 | 10.0.0.37:6201 |
| 10.0.0.37:6301 | 1 | 10.0.0.38:6201 |
| 10.0.0.38:6301 | 1 | 10.0.0.36:6201 |
| n/a | 1 | 10.0.0.36:6211 |
+----------------+-----------------+----------------+
You must launch the following command to create the assignation:
# openio rdir bootstrap rawx
+----------------+----------------+-----------------+-----------------+
| Rdir | Rawx | Rdir location | Rawx location |
+----------------+----------------+-----------------+-----------------+
| 10.0.0.36:6301 | 10.0.0.37:6201 | node-1 | node-2 |
| 10.0.0.37:6301 | 10.0.0.38:6201 | node-2 | node-3 |
| 10.0.0.37:6301 | 10.0.0.36:6211 | node-2 | node-1 |
| 10.0.0.38:6301 | 10.0.0.36:6201 | node-3 | node-1 |
+----------------+----------------+-----------------+-----------------+
New Rdir assignation
First, you have to install a new rdir service on another server.
Create a new directory rdir-2 in /var/lib/oio/sds/OPENIO/.
# mkdir /var/lib/oio/sds/OPENIO/rdir-2/
Give the rights for this directory to the openio user:
# chown openio:openio /var/lib/oio/sds/OPENIO/rdir-2/
Create a new directory rdir-2 in /etc/oio/sds/OPENIO/
# mkdir /etc/oio/sds/OPENIO/rdir-2/
Give the rights for this directory to the openio user:
chown openio:openio /etc/oio/sds/OPENIO/rdir-2/
Create a new configuration file (rdir-2.conf) in your newly created directory /etc/oio/sds/OPENIO/rdir-2:
# OpenIO managed
[rdir-server]
bind_addr = 10.0.0.38
bind_port = 6302
namespace = OPENIO
# Currently, only 1 worker is allowed to avoid concurrent access to leveldb database
workers = 1
worker_class = sync
threads = 1
db_path= /var/lib/oio/sds/OPENIO/rdir-2
log_facility = LOG_LOCAL0
log_level = info
log_address = /dev/log
syslog_prefix = OIO,OPENIO,rdir,2
Give the rights for this file to the openio user:
chown openio:openio /etc/oio/sds/OPENIO/rdir-2/rdir-2.conf
Create a new configuration file (OPENIO-rdir-2.conf) in the /etc/gridinit.d/ directory:
# OpenIO managed
[Service.OPENIO-rdir-2]
command=/usr/bin/oio-rdir-server /etc/oio/sds/OPENIO/rdir-2/rdir-2.conf
enabled=true
start_at_boot=true
on_die=respawn
group=OPENIO,rdir,2
uid=openio
gid=openio
env.PATH=/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin
Give the rights for this file to the openio user:
chown openio:openio /etc/gridinit.d/OPENIO-rdir-2.conf
Create a new configuration file (rdir-2.yml) in the /etc/oio/sds/OPENIO/watch/ directory:
# OpenIO managed
host: 10.0.0.38
port: 6302
type: rdir
location: node-3.2
checks:
- {type: tcp}
stats:
- {type: volume, path: /var/lib/oio/sds/OPENIO/rdir-2}
- {type: http, path: /status, parser: json}
- {type: system}
Give the rights for this file to the openio user:
chown openio:openio /etc/oio/sds/OPENIO/watch/rdir-2.yml
Create a new directory rdir-2 in /var/log/oio/sds/OPENIO/.
# mkdir /var/log/oio/sds/OPENIO/rdir-2
Give the rights for this directory to the openio user:
# chown openio:openio /var/log/oio/sds/OPENIO/rdir-2/
# chmod 770 /var/log/oio/sds/OPENIO/rdir-2/
Then, to make your new rdir service available, you must reload the configuration and start the service:
# gridinit_cmd reload
# gridinit_cmd start OPENIO-rdir-2
And restart the conscience agent:
# gridinit_cmd restart @conscienceagent
Then, you must unlock your new service:
# openio cluster unlock rdir 10.0.0.38:6302
Now, you must assign your new rawx to a rdir service.
In the example below, you can see that the new rawx (10.0.0.36:6211) has no rdir assignation, and your new rdir (10.0.0.38:6302) does not manage any rawx:
# openio rdir assignments rawx --aggregated
+----------------+-----------------+----------------+
| Rdir | Number of bases | Bases |
+----------------+-----------------+----------------+
| 10.0.0.36:6301 | 1 | 10.0.0.37:6201 |
| 10.0.0.37:6301 | 1 | 10.0.0.38:6201 |
| 10.0.0.38:6302 | 0 | |
| 10.0.0.38:6301 | 1 | 10.0.0.36:6201 |
| n/a | 1 | 10.0.0.36:6211 |
+----------------+-----------------+----------------+
You must launch the following command to create the assignation:
# openio rdir bootstrap rawx
+----------------+----------------+-----------------+-----------------+
| Rdir | Rawx | Rdir location | Rawx location |
+----------------+----------------+-----------------+-----------------+
| 10.0.0.36:6301 | 10.0.0.37:6201 | node-1 | node-2 |
| 10.0.0.37:6301 | 10.0.0.38:6201 | node-2 | node-3 |
| 10.0.0.37:6301 | 10.0.0.36:6211 | node-2 | node-1 |
| 10.0.0.38:6301 | 10.0.0.36:6201 | node-3 | node-1 |
+----------------+----------------+-----------------+-----------------+
Finalize the installation
Finally, you must unlock your new service:
# openio cluster unlock rawx 10.0.0.36:6211
You can check that your new service is available using the openio cluster list command:
# openio cluster list rawx
+---------+----------------+------------+---------------------------------+------------+-------+------+-------+
| Type | Addr | Service Id | Volume | Location | Slots | Up | Score |
+---------+----------------+------------+---------------------------------+------------+-------+------+-------+
| rawx | 10.0.0.36:6211 | n/a | /var/lib/oio/sds/OPENIO/rawx-2 | node-1 | n/a | True | 90 |
| rawx | 10.0.0.38:6201 | n/a | /var/lib/oio/sds/OPENIO/rawx-1 | node-3 | n/a | True | 92 |
| rawx | 10.0.0.36:6201 | n/a | /var/lib/oio/sds/OPENIO/rawx-1 | node-1 | n/a | True | 90 |
| rawx | 10.0.0.37:6201 | n/a | /var/lib/oio/sds/OPENIO/rawx-1 | node-2 | n/a | True | 91 |
+---------+----------------+------------+---------------------------------+------------+-------+------+-------+