OIOFS Node Installation

Tip

oio-fs is part of our paid plans.

Get in touch with the team or visit the page describing our plans.

Requirements

Licence

  • A login and a password provided by OpenIO Support

Hardware

  • Storage drive: A storage device for cache

Operating system

  • Centos 7
  • Ubuntu 18.04 (Server)

System

  • Root privileges are required (using sudo).
  • SELinux or AppArmor are disabled (managed at deployment).
SELinux
sudo sed -i -e 's@^SELINUX=enforcing$@SELINUX=disabled@g' /etc/selinux/config
sudo setenforce 0
sudo systemctl disable selinux.service
AppArmor
sudo service apparmor stop
sudo apparmor teardown
sudo update-rc.d -f apparmor remove
  • The system must be up to date.
RedHat
sudo yum update -y
sudo reboot
Ubuntu
sudo apt update -y
sudo apt upgrade -y
sudo reboot

Network

  • This node connected to the same lan of the OpenIO SDS.

Setup

You only need to perform this setup on one of the nodes in the cluster (or your laptop).

  • Install git.
RedHat
$> sudo yum install git -y
Ubuntu
$> sudo apt install git -y
  • Clone the OpenIO ansible playbook deployment repository
$> git clone https://github.com/open-io/ansible-playbook-openio-deployment.git --branch 19.10 oiosds
$> cd oiosds/products/sds
  • Install Ansible for the current user.
Ansible
$> python3 -m venv openio_venv
$> source openio_venv/bin/activate
$> pip install -r ansible.pip

Architecture

This playbook will deploy an oiofs mount connected to a SDS cluster as shown below:

+-------------------------+
|                         |
|                         |----+                                       +-------------------------+              .+--+.______
|                         |    |                                       |                         |              +CIFS       +
|                         |    |----+                                  |                         |              |    ___________
|                         |    |    |                                  |                         |              |   /          /
|                         |    |    |                                  |                         |              |  /          /
|                         |    |    |                                  |                         |        X X XX| /          /
|                         |    |    |                                  |                         |      X      +/__________/
|                         |    |    |   Provide file-oriented access   |                         |   XX
|           3             |    |    |                                  |           1             |XXX
|       OpenIO SDS        |    |    |  <---------------------------->  |       OpenIO FS         |   XX         .+--+.______
|         Servers         |    |    |                                  |         Server          |     X        +NFS        +
|                         |    |    |   on an object storage backend   |                         |       X X  XX|    ___________
|                         |    |    |                                  |                         |              |   /          /
|                         |    |    |                                  |                         |              |  /          /
|                         |    |    |                                  |                         |              | /          /
+-------------------------+    |    |                                  |                         |              +/__________/
    |                          |    |                                  |                         |
    +--------------------------+    |                                  +-------------------------+
          |                         |
          +-------------------------+

Configuration

Inventory

Replace the inventory.yml by the inventory named inventory_with_oiofs.yml

replace inventory.yml
  $> cp inventory_with_oiofs.yml inventory.yml

Fill the inventory according to your environment:

  • Edit the inventory.yml file and adapt the IP addresses and SSH user

    ip address
    ---
    all:
      hosts:
        # node for OIOFS
        node4:
          ansible_host: IP_ADDRESS_OF_NODE4 # Change it with the IP of the first server
          ansible_user: root
          openio_data_mounts: []
          openio_metadata_mounts: []
    
  • Set your credentials in the inventory file.

    oiofs repository credentials
    ---
    all:
      children:
        openio:
          vars:
            
            openio_repositories_credentials:
              oiofs:
                user: OIOFS_REPO_USER
                password: OIOFS_REPO_PASSWORD
    
  • You can check that everything is configured correctly using this command:

    RedHat
    $> ansible all -i inventory.yml -bv -m ping
    
    Ubuntu
    $> ansible all -i inventory.yml -bv -m ping -e 'ansible_python_interpreter=/usr/bin/python
    

Mandatory SDS services

An ecd and an oioproxy are required on the oiofs node.

Run this command to deploy them:

prepare oiofs node
$> ansible-playbook -i inventory.yml \
                 -t check,base,ecd,oioproxy,namespace \
                 -e "openio_maintenance_mode=false" \
                 main.yml

Here is the expected result:

result mandatory services
[root@node4 /]# cat /etc/oio/sds.conf.d/OPENIO
# OpenIO managed
[OPENIO]
# endpoints
conscience=172.28.0.4:6000
zookeeper=172.28.0.2:6005,172.28.0.4:6005,172.28.0.3:6005
proxy=172.28.0.5:6006
event-agent=beanstalk://172.28.0.5:6014
ecd=172.28.0.5:6017

udp_allowed=yes

ns.meta1_digits=1
ns.storage_policy=THREECOPIES
ns.chunk_size=104857600
ns.service_update_policy=meta2=KEEP|3|1|;rdir=KEEP|1|1|;
sqliterepo.repo.soft_max=1000
sqliterepo.repo.hard_max=1000

[root@node4 /]# gridinit_cmd status
KEY               STATUS      PID GROUP
OPENIO-ecd-0      UP         2190 OPENIO,ecd,0
OPENIO-oioproxy-0 UP         1799 OPENIO,oioproxy,0

OpenIO FS definition

The following parameters must be set:
  • oiofs_global_mount_directory: defines the directory where the oiofs mounts will be placed.
  • oiofs_cache_device: defines the dedicated block device used for cache.
  • oiofs_cache_folder: defines the cache folder ( it should already be mounted).
  • oiofs_cache_low_watermark and oiofs_cache_high_watermark: defines an acceptable percentage of cache fill.
mountpoints and cache
### OIOFS
    oiofs:
      hosts:
        node4: {}
      vars:
        oiofs_global_mount_directory: "/mnt"
        oiofs_global_redis_sentinel_servers: "{{ groups[oiofs_global_redis_inventory_groupname] \
          | map('extract', hostvars, ['openio_bind_address']) \
          | map('regex_replace', '$', ':6012')
          | list }}"

        ## CACHE
        oiofs_cache_device: /dev/vdb1
        oiofs_cache_folder: "{{ oiofs_global_mount_directory }}/cache"
        oiofs_cache_high_watermark: 80
        oiofs_cache_low_watermark: 50

    oiofs_redis:
      hosts: {}
  • oiofs_mountpoints: defines the OIOFS mounts that will be accessible through Samba and NFS.
    • active_mode: defines the management mode of this mount. If you are not on a high availability deployment, this paramater must be set to true.
    • namespace, account and container: defines the SDS context used by OpenIO FS.
    • state: present/absent ensures the mount is absent or defined and up.
    • http_server: defines a socket to request statistics about the mount. Must be unique by mountpoint.
mountpoints and cache
### OIOFS
    oiofs:
      vars:
        
        oiofs_mountpoints:
          - active_mode: true
            namespace: "{{ namespace }}"
            # account/container
            account: MY_ACCOUNT1
            container: MY_CONTAINER1
            state: present
            http_server: 127.0.0.1:6989

            # SDS
            openio_sds_conscience_url: "{{ openio_namespace_conscience_url }}"
            oioproxy_url: "{{ openio_bind_address }}:6006"
            ecd_url: "{{ openio_bind_address }}:6017"
            redis_sentinel_servers: "{{ oiofs_global_redis_sentinel_servers }}"
            redis_sentinel_name: "{{ namespace }}-master-1"

Exports

If you need to export the OpenIO FS mount, you must define some additionnal parameters in oiofs_mountpoints

  • user, group: defines the owners of the mount

  • ignore_flush: allows to ignore fsync and fflush calls, so only oio-fs choose when to flush and not the client.

  • auto_retry: if set to true, retries calls (write, read …) on full cache instead of returning EAGAIN.

  • exports: defines the unique type of export for the mount.

    • fsid: For NFS, this parameter must be unique.
    exports
    ### OIOFS
        oiofs:
          vars:
            
            oiofs_mountpoints:
              - active_mode: true
                
    
                # EXPORTS
                user: root
                group: root
                ignore_flush: true
                auto_retry: false
                export: nfs
                nfs_exports:
                  client: "*"
                  options:
                    - "rw"
                    - "async"
                    - "no_root_squash"
                    - "fsid=1"
                  uid: 0
                  gid: 0
    

Installation

Run this command to deploy the OpenIO FS:

deploy oiofs
$> ansible-playbook -i inventory.yml main.yml -t fact
$> ansible-playbook -i inventory.yml playbooks/oiofs.yml -e "openio_maintenance_mode=false"

Post-install Checks

The node is configured and the filesystem is mounted.

Run these commands on the node: gridinit_cmd status and df -h.

Sample outputs:

check mounts
[root@node4 /]# gridinit_cmd status
KEY                                                         STATUS      PID GROUP
OPENIO-ecd-0                                                UP         2128 OPENIO,ecd,0
OPENIO-oiofs-OPENIO-MY_ACCOUNT1-MY_CONTAINER_EXPORTED_NFS   UP         2615 OPENIO,oiofs,OPENIO-MY_ACCOUNT1-MY_CONTAINER_EXPORTED_NFS
OPENIO-oiofs-OPENIO-MY_ACCOUNT1-MY_CONTAINER_EXPORTED_SAMBA UP         2653 OPENIO,oiofs,OPENIO-MY_ACCOUNT1-MY_CONTAINER_EXPORTED_SAMBA
OPENIO-oioproxy-0                                           UP         1729 OPENIO,oioproxy,0

[root@node4 /]# df -h
Filesystem      Size  Used Avail Use% Mounted on
overlay          60G  8.6G   52G  15% /
tmpfs            64M     0   64M   0% /dev
/dev/vda1        60G  8.6G   52G  15% /etc/hosts
shm              64M     0   64M   0% /dev/shm
tmpfs           7.8G     0  7.8G   0% /sys/fs/cgroup
tmpfs           7.8G  8.1M  7.8G   1% /run
oiofs-fuse       16E     0   16E   0% /mnt/oiofs-OPENIO-MY_ACCOUNT1-MY_CONTAINER_EXPORTED_NFS
oiofs-fuse       16E     0   16E   0% /mnt/oiofs-OPENIO-MY_ACCOUNT1-MY_CONTAINER_EXPORTED_SAMBA
check SDS
[root@node4 /]# export OIO_NS=OPENIO
[root@node4 /]# cp -r /etc/  /mnt/oiofs-OPENIO-MY_ACCOUNT1-MY_CONTAINER_EXPORTED_NFS/
[root@node4 /]# openio --oio-account MY_ACCOUNT1 container list
+-------------------------------+---------+-------+------------------+------------------------------------------------------------------+
| Name                          |   Bytes | Count |            Mtime | CID                                                              |
+-------------------------------+---------+-------+------------------+------------------------------------------------------------------+
| MY_CONTAINER_EXPORTED_NFS_0   | 9439358 |   284 |  1575144085.8659 | A0C3BD03C85BD2B1810C5168016C2C31A7A81EC0BA2209264F6D9FFF296D5972 |
| MY_CONTAINER_EXPORTED_SAMBA_0 |       0 |     0 | 1575143929.50248 | 159DEBA12D8F4CBB03678DC8EB139B06548D04678941A225E1C14BF98180FB8B |
+-------------------------------+---------+-------+------------------+------------------------------------------------------------------+

[root@node4 /]# openio --oio-account MY_ACCOUNT1 container show MY_CONTAINER_EXPORTED_NFS_0
+-----------------+--------------------------------------------------------------------+
| Field           | Value                                                              |
+-----------------+--------------------------------------------------------------------+
| account         | MY_ACCOUNT1                                                        |
| base_name       | A0C3BD03C85BD2B1810C5168016C2C31A7A81EC0BA2209264F6D9FFF296D5972.1 |
| bytes_usage     | 9.439MB                                                            |
| container       | MY_CONTAINER_EXPORTED_NFS_0                                        |
| ctime           | 1575143927                                                         |
| damaged_objects | 0                                                                  |
| max_versions    | Namespace default                                                  |
| missing_chunks  | 0                                                                  |
| objects         | 284                                                                |
| quota           | Namespace default                                                  |
| status          | Enabled                                                            |
| storage_policy  | Namespace default                                                  |
+-----------------+--------------------------------------------------------------------+

[root@node4 /]# openio --oio-account MY_ACCOUNT1 object list MY_CONTAINER_EXPORTED_NFS_0
+------+---------+----------------------------------+------------------+
| Name |    Size | Hash                             |          Version |
+------+---------+----------------------------------+------------------+
| 100  |   65536 | A5AE49867124AC75F029A9A33AF31BAD | 1575144084612691 |
| 101  |   16384 | 1C5566B67C186B2EE1F58346B104785E | 1575144084621875 |
| 102  |      45 | 84C612BDA6CEB71C9ECC0A06B363F049 | 1575144084628161 || 96   |    1746 | 643B68A0994AA69649E5B3F13DCF5635 | 1575144084595684 |
| 97   |    1735 | B5EAAEE8A77829325A77CBB613380F90 | 1575144084608830 |
+------+---------+----------------------------------+------------------+
check exports
[root@node4 /]# showmount -e IP_ADDRESS_OF_NODE4
Export list for IP_ADDRESS_OF_NODE4:
/mnt/oiofs-OPENIO-MY_ACCOUNT1-MY_CONTAINER_EXPORTED_NFS *

[root@node4 /]# smbclient -L IP_ADDRESS_OF_NODE4 -U smbguest
Enter WORKGROUP\smbguest's password:

    Sharename                      Type      Comment
    ---------                      ----      -------
    MY_CONTAINER_EXPORTED_SAMBA    Disk      Samba oiofs
    IPC$                           IPC       IPC Service (Samba 4.8.3)
Reconnecting with SMB1 for workgroup listing.

    Server               Comment
    ---------            -------

    Workgroup            Master
    ---------            -------