oio.directory package


oio.directory.admin module

class oio.directory.admin.AdminClient(conf, **kwargs)[source]

Bases: oio.common.client.ProxyClient

Low level database administration client.


Instanciate a client object for ‘/cache/*’ proxy routes.

copy_base_from(service_type=None, account=None, reference=None, cid=None, service_id=None, **kwargs)[source]

Copy a base to another service, using DB_PIPEFROM.

  • svc_from – id of the source service.
  • svc_to – id of the destination service.
copy_base_to(service_type=None, account=None, reference=None, cid=None, service_id=None, **kwargs)[source]

Copy a base to another service, using DB_PIPETO. Source service is looked after in service directory.

Parameters:svc_to – id of the destination service.
election_debug(service_type=None, account=None, reference=None, cid=None, service_id=None, **kwargs)[source]

Get debugging information about an election.

election_leave(service_type=None, account=None, reference=None, cid=None, service_id=None, **kwargs)[source]

Force all peers to leave the election.

election_ping(service_type=None, account=None, reference=None, cid=None, service_id=None, **kwargs)[source]

Trigger or refresh an election.

election_status(service_type=None, account=None, reference=None, cid=None, service_id=None, **kwargs)[source]

Get the status of an election (trigger it if necessary).

Returns:a dict with ‘master’ (str), ‘slaves’ (list), ‘peers’ (dict) and ‘type’ (str)
‘peers’: {
‘’: {
{‘status’: 303,
‘message’: ‘’},

‘body’: u’’},

‘’: {
{‘status’: 200,
‘message’: ‘OK’},

‘body’: u’’},

‘’: {
{‘status’: 303,
‘message’: ‘’},

‘body’: u’’}

}, ‘master’: ‘’, ‘slaves’: [‘’, ‘’], ‘type’: ‘meta1’


election_sync(service_type=None, account=None, reference=None, cid=None, service_id=None, **kwargs)[source]

Try to synchronize a dubious election.


Instanciate a client object for ‘/forward/*’ proxy routes.

get_properties(service_type=None, account=None, reference=None, cid=None, service_id=None, **kwargs)[source]

Get user and system properties from the admin table of an sqliterepo base.

has_base(service_type=None, account=None, reference=None, cid=None, service_id=None, **kwargs)[source]

Ask each peer if base exists.

proxy_flush_cache(high=True, low=True, proxy_netloc=None, **kwargs)[source]

Flush “high” and “low” proxy caches. By default, flush the cache of the local proxy. If proxy_netloc is provided, flush the cache of this proxy.

proxy_get_cache_status(proxy_netloc=None, **kwargs)[source]

Get the status of the high (conscience and meta0) and low (meta1) cache, including the current number of entries.

proxy_get_live_config(proxy_netloc=None, **kwargs)[source]

Get all configuration parameters from the specified proxy service.

Returns:a dictionary with all configuration keys the service recognizes, and their current value.
Return type:dict
proxy_set_live_config(proxy_netloc=None, config=None, **kwargs)[source]

Set configuration parameters on the specified proxy service.

remove_base(service_type=None, account=None, reference=None, cid=None, service_id=None, **kwargs)[source]

Remove specific base.

service_balance_elections(svc_id, max_ops=0, inactivity=0, **kwargs)[source]

Balance elections to get an acceptable slave/master ratio.

  • svc_to – id of the service that should balance its elections.
  • max_ops – maximum number of balancing operations.
  • inactivity – avoid expiring election whose last activity is younger than the specified value.
service_flush_cache(svc_id, **kwargs)[source]

Flush the resolver cache of an sqliterepo-based service.

service_get_info(svc_id, **kwargs)[source]

Get all information from the specified service. Works on all services using ASN.1 protocol except conscience.

Returns:a dictionary with all information keys the service recognizes, and their current value.
Return type:dict
service_get_live_config(svc_id, **kwargs)[source]

Get all configuration parameters from the specified service. Works on all services using ASN.1 protocol.

Returns:a dictionary with all configuration keys the service recognizes, and their current value.
Return type:dict
service_reload_lb(svc_id, **kwargs)[source]

Force the service to reload its internal load balancer.

service_set_live_config(svc_id, config, **kwargs)[source]

Set some configuration parameters on the specified service. Works on all services using ASN.1 protocol. Notice that some parameters may not be taken into account, and no parameter will survice a service restart.

set_peers(service_type=None, account=None, reference=None, cid=None, service_id=None, **kwargs)[source]

Force the new peer set in the replicas of the old peer set.

set_properties(service_type=None, account=None, reference=None, cid=None, service_id=None, **kwargs)[source]

Set user or system properties in the admin table of an sqliterepo base.

vacuum_base(service_type=None, account=None, reference=None, cid=None, service_id=None, **kwargs)[source]

Vacuum (defragment) the database on the master service, then resynchronize it on the slaves.


Wrap database localization parameters in request parameters


Convert a list of service IDs to a comma separated string.

oio.directory.client module

class oio.directory.client.DirectoryClient(conf, **kwargs)[source]

Bases: oio.common.client.ProxyClient

Mid-level client for OpenIO SDS service directory (meta0, meta1).

create(account=None, reference=None, properties=None, **kwargs)[source]

Create a reference (a service container).

  • account – name of the account where the reference must be created
  • reference – name of the reference to create
  • properties (dict) – dictionary of properties to set on the reference

True if the reference has been created, False if it already existed

del_properties(account=None, reference=None, properties=None, cid=None, **kwargs)[source]

Delete properties for a reference.

delete(account=None, reference=None, cid=None, **kwargs)[source]

Delete a reference.

destroy(*args, **kwargs)[source]
Deprecated:use delete
force(account=None, reference=None, service_type=None, services=None, cid=None, autocreate=False, replace=False, **kwargs)[source]

Associate the specified services to the reference.

Parameters:replace (bool) – do not require the list of services of the specified type to be empty, overwrite it.
get_properties(account=None, reference=None, properties=None, cid=None, **kwargs)[source]

Get properties for a reference.

has(account=None, reference=None, cid=None, **kwargs)[source]

Poll and associate a new service to the reference.

list(account=None, reference=None, cid=None, service_type=None, **kwargs)[source]

List the services linked to the reference.

renew(account=None, reference=None, service_type=None, cid=None, **kwargs)[source]

Re-poll and re-associate a set of services to the reference. Will increment the sequence number.

set_properties(account=None, reference=None, properties=None, cid=None, **kwargs)[source]

Set properties for a reference.

show(*args, **kwargs)[source]
Deprecated:use list

Remove an associated service from the reference

oio.directory.indexer module

class oio.directory.indexer.Meta2Indexer(conf)[source]

Bases: oio.common.daemon.Daemon

A daemon that spawns a greenlet running a Meta2IndexingWorker for each volume.

run(*args, **kwargs)[source]
class oio.directory.indexer.Meta2IndexingWorker(volume_path, conf, pool_manager=None)[source]

Bases: object

Indexing worker responsible for a single volume.


Crawl the volume assigned to this worker, and index every database.


Add a meta2 database to the rdir index. Fails if the database isn’t handled by the current volume.

Parameters:db_id – The ContentID representing the reference to the database.

Log the status of indexer

Parameters:tag – One of three: starting, running, ended.

Main worker loop


Could be needed for eventually gracefully stopping.

warn(msg, container_id)[source]

oio.directory.meta module

class oio.directory.meta.MetaMapping(conf, service_types, admin_client=None, conscience_client=None, logger=None, rdir_client=None, **kwargs)[source]

Bases: object

Represents the content of the meta_n0 database

apply(moved=None, **kwargs)[source]

Upload the current mapping to the meta_n0 services, and set peers accordingly in meta_n1 databases.

Parameters:moved – list of bases that have moved.
static get_cid_and_seq(base)[source]

Reset the base allocations and reload the services from Conscience.

oio.directory.meta0 module

Meta0 client and meta1 balancing operations

class oio.directory.meta0.Meta0Client(conf, **kwargs)[source]

Bases: oio.common.client.ProxyClient

Meta0 administration client

force(mapping, **kwargs)[source]

Force the meta0 prefix mapping. The mapping may be partial to force only a subset of the prefixes.


Get the meta0 prefix mapping

class oio.directory.meta0.Meta0PrefixMapping(meta0_client, replicas=3, digits=None, min_dist=1, **kwargs)[source]

Bases: oio.directory.meta.MetaMapping

Represents the content of the meta0 database

assign_services(base, services, fail_if_already_set=False)[source]

Assign services to manage base.

Parameters:fail_if_already_set – raise ValueError if base is already managed by some services
bootstrap(level='volume', degradation=0)[source]

Spread the prefixe to balance accross sites. By default the token describing the site is <0>. Put one short_prefix into each slice (corresponding to a location level) and balance well into each level. At the end, expand each short prefixes into their complete prefixes. :param level: the kind of location token describing the level of

fair balancing.
Parameters:degradation – how many location slices of the given level we should tolerate.

Check that all bases have the right number of replicas


Build a dict with service addresses as keys and the number of managed bases as values.

decommission(svc, bases_to_remove=None, strategy=None, try_=False, compute_upper_limit=True)[source]

Unassign all bases of bases_to_remove from svc, and assign them to other services using strategy. :param svc: service to decommission :type svc: str or dict :param bases_to_remove: bases to remove of the service :type bases_to_remove: list :param strategy: strategy to move the bases :type strategy: function :param try_: keep the bases on this service

if the strategy doesn’t find better, otherwise obligatorily remove the bases of this service
Parameters:compute_upper_limit (bool) – compute the right number bases each service should host
static dist_between(loc1, loc2)[source]
find_services_less_bases(known=None, min_score=1, **_kwargs)[source]

Find replicas services, including the ones of known

find_services_more_availability(known=None, min_score=1, **_kwargs)[source]

Find replicas services, including the ones of known

get_loc(svc, default=None)[source]

Get the location of a service. If location is not defined, return: - service IP address if default is None or “addr” - default for any other value.


Get the list of bases managed by the service


Get the score of a service, or 0 if it is unknown

load_json(json_mapping, **kwargs)[source]

Load the mapping from a JSON string

load_meta0(json_mapping=None, **kwargs)[source]

Load the mapping from dictionary out of the cluster,


Get total the number of bases according to self.digits


Get the list of prefixes that share the same base as pfx.


Get the name of the base the prefix will be saved in. When self.digits is 4, the name of the base is pfx.


Reassign bases from the services which manage the most


Reset the base allocations and reload the services from Conscience.


Serialize the mapping to a JSON string suitable as input for ‘meta0_force’ request.


Returns the number of real prefixes in meta0/meta1. Raises an exception if the prefix number is not acceptable.


oio.directory.meta2 module

class oio.directory.meta2.Meta2Database(conf, logger=None, admin_client=None, conscience_client=None, rdir_client=None, directory_client=None, service_type='meta2')[source]

Bases: object

Execute maintenance operations on meta2 databases (or compatible services).

static get_cid_and_seq(base)[source]

Move a database from src to dst. If dst is None, find one automatically.


Rebuild a database.


Load the list of all services of type self.service_type.


Reset the base allocations and reload the services from Conscience.

oio.directory.meta2_rebuilder module

class oio.directory.meta2_rebuilder.ContentRepairerWorker(tool, queue_workers, queue_reply)[source]

Bases: oio.common.tool.ToolWorker

class oio.directory.meta2_rebuilder.Meta2Rebuilder(conf, input_file=None, service_id=None, **kwargs)[source]

Bases: oio.common.tool.Tool

Rebuild meta2 databases.

create_worker(queue_workers, queue_reply)[source]

Create worker to process the items.

static string_from_item(item)[source]

Module contents