Fix Invalid, Corrupted or Missing Databases

The problem

When you query a specific container, one of the services hosting it returns an error saying that the database file is missing, or is corrupt, or has missing entries in its admin table. In such a case, the replication mechanism should fetch a fresh copy of the database from another peer, but for some reason, it does not.

# openio container show FVE0 --debug
+----------------+--------------------------------------------------------------------+
| Field          | Value                                                              |
+----------------+--------------------------------------------------------------------+
| account        | myaccount                                                          |
| base_name      | 697ECB056A5F9339B36C3B0A020B5AB17B4B0160FBEBEA203315EA1FC1B61605.1 |
| bytes_usage    | 0B                                                                 |
| container      | FVE0                                                               |
| ctime          | 1507817632                                                         |
| max_versions   | Namespace default                                                  |
| meta.a         | 1                                                                  |
| objects        | 0                                                                  |
| quota          | Namespace default                                                  |
| storage_policy | Namespace default                                                  |
+----------------+--------------------------------------------------------------------+
# openio container show FVE0 --debug
'sys.m2.ctime'
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 387, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/lib/python2.7/site-packages/cliff/display.py", line 100, in run
    column_names, data = self.take_action(parsed_args)
  File "/home/fvennetier/src/public_git/oio-sds/oio/cli/container/container.py", line 231, in take_action
    ctime = float(sys['sys.m2.ctime']) / 1000000.
KeyError: 'sys.m2.ctime'
Exception raised: 'sys.m2.ctime'
# openio container show FVE0 --debug
+----------------+--------------------------------------------------------------------+
| Field          | Value                                                              |
+----------------+--------------------------------------------------------------------+
| account        | myaccount                                                          |
| base_name      | 697ECB056A5F9339B36C3B0A020B5AB17B4B0160FBEBEA203315EA1FC1B61605.1 |
| bytes_usage    | 0B                                                                 |
| container      | FVE0                                                               |
| ctime          | 1507817632                                                         |
| max_versions   | Namespace default                                                  |
| meta.a         | 1                                                                  |
| objects        | 0                                                                  |
| quota          | Namespace default                                                  |
| storage_policy | Namespace default                                                  |
+----------------+--------------------------------------------------------------------+

The solution

Since openio-sds 4.2, it is possible force the synchronization of the base on each peer by using openio-admin election sync with the type of service and the name of the database as parameters.

# openio-admin election sync meta2 FVE0
+----------------+--------+-----------+------+
| Id             | Status | Message   | Body |
+----------------+--------+-----------+------+
| 127.0.0.1:6015 |    200 | OK        |      |
| 127.0.0.1:6016 |    301 | not SLAVE |      |
| 127.0.0.1:6017 |    200 | OK        |      |
+----------------+--------+-----------+------+

Services 127.0.0.1:6015 and 127.0.0.1:6017 have fetched a fresh copy of the entire database from 127.0.0.1:6016.