oio.directory package

Submodules

oio.directory.admin module

class oio.directory.admin.AdminClient(conf, **kwargs)[source]

Bases: oio.common.client.ProxyClient

Low level database administration client.

cache_client

Instanciate a client object for ‘/cache/*’ proxy routes.

copy_base_from(service_type=None, account=None, reference=None, cid=None, service_id=None, **kwargs)[source]

Copy a base to another service, using DB_PIPEFROM.

Parameters:
  • svc_from – id of the source service.
  • svc_to – id of the destination service.
copy_base_to(service_type=None, account=None, reference=None, cid=None, service_id=None, **kwargs)[source]

Copy a base to another service, using DB_PIPETO. Source service is looked after in service directory.

Parameters:svc_to – id of the destination service.
election_debug(service_type=None, account=None, reference=None, cid=None, service_id=None, **kwargs)[source]

Get debugging information about an election.

election_leave(service_type=None, account=None, reference=None, cid=None, service_id=None, **kwargs)[source]

Force all peers to leave the election.

election_ping(service_type=None, account=None, reference=None, cid=None, service_id=None, **kwargs)[source]

Trigger or refresh an election.

election_status(service_type=None, account=None, reference=None, cid=None, service_id=None, **kwargs)[source]

Get the status of an election (trigger it if necessary).

Returns:a dict with ‘master’ (str), ‘slaves’ (list), ‘peers’ (dict) and ‘type’ (str)
example
{
‘peers’: {
‘127.0.0.3:6014’: {
‘status’:
{‘status’: 303,
‘message’: ‘127.0.0.1:6015’},

‘body’: u’’},

‘127.0.0.1:6015’: {
‘status’:
{‘status’: 200,
‘message’: ‘OK’},

‘body’: u’’},

‘127.0.0.2:6016’: {
‘status’:
{‘status’: 303,
‘message’: ‘127.0.0.1:6015’},

‘body’: u’’}

}, ‘master’: ‘127.0.0.1:6015’, ‘slaves’: [‘127.0.0.3:6014’, ‘127.0.0.2:6016’], ‘type’: ‘meta1’

}

election_sync(service_type=None, account=None, reference=None, cid=None, service_id=None, **kwargs)[source]

Try to synchronize a dubious election.

forwarder

Instanciate a client object for ‘/forward/*’ proxy routes.

get_properties(service_type=None, account=None, reference=None, cid=None, service_id=None, **kwargs)[source]

Get user and system properties from the admin table of an sqliterepo base.

has_base(service_type=None, account=None, reference=None, cid=None, service_id=None, **kwargs)[source]

Ask each peer if base exists.

proxy_flush_cache(high=True, low=True, proxy_netloc=None, **kwargs)[source]

Flush “high” and “low” proxy caches. By default, flush the cache of the local proxy. If proxy_netloc is provided, flush the cache of this proxy.

proxy_get_cache_status(proxy_netloc=None, **kwargs)[source]

Get the status of the high (conscience and meta0) and low (meta1) cache, including the current number of entries.

proxy_get_live_config(proxy_netloc=None, **kwargs)[source]

Get all configuration parameters from the specified proxy service.

Returns:a dictionary with all configuration keys the service recognizes, and their current value.
Return type:dict
proxy_set_live_config(proxy_netloc=None, config=None, **kwargs)[source]

Set configuration parameters on the specified proxy service.

remove_base(service_type=None, account=None, reference=None, cid=None, service_id=None, **kwargs)[source]

Remove specific base.

service_balance_elections(svc_id, max_ops=0, inactivity=0, **kwargs)[source]

Balance elections to get an acceptable slave/master ratio.

Parameters:
  • svc_to – id of the service that should balance its elections.
  • max_ops – maximum number of balancing operations.
  • inactivity – avoid expiring election whose last activity is younger than the specified value.
service_flush_cache(svc_id, **kwargs)[source]

Flush the resolver cache of an sqliterepo-based service.

service_get_info(svc_id, **kwargs)[source]

Get all information from the specified service. Works on all services using ASN.1 protocol except conscience.

Returns:a dictionary with all information keys the service recognizes, and their current value.
Return type:dict
service_get_live_config(svc_id, **kwargs)[source]

Get all configuration parameters from the specified service. Works on all services using ASN.1 protocol.

Returns:a dictionary with all configuration keys the service recognizes, and their current value.
Return type:dict
service_reload_lb(svc_id, **kwargs)[source]

Force the service to reload its internal load balancer.

service_set_live_config(svc_id, config, **kwargs)[source]

Set some configuration parameters on the specified service. Works on all services using ASN.1 protocol. Notice that some parameters may not be taken into account, and no parameter will survice a service restart.

set_peers(service_type=None, account=None, reference=None, cid=None, service_id=None, **kwargs)[source]

Force the new peer set in the replicas of the old peer set.

set_properties(service_type=None, account=None, reference=None, cid=None, service_id=None, **kwargs)[source]

Set user or system properties in the admin table of an sqliterepo base.

vacuum_base(service_type=None, account=None, reference=None, cid=None, service_id=None, **kwargs)[source]

Vacuum (defragment) the database on the master service, then resynchronize it on the slaves.

oio.directory.admin.loc_params(func)[source]

Wrap database localization parameters in request parameters

oio.directory.admin.service_id_to_string(service_id)[source]

Convert a list of service IDs to a comma separated string.

oio.directory.client module

class oio.directory.client.DirectoryClient(conf, **kwargs)[source]

Bases: oio.common.client.ProxyClient

Mid-level client for OpenIO SDS service directory (meta0, meta1).

create(account=None, reference=None, properties=None, **kwargs)[source]

Create a reference (a service container).

Parameters:
  • account – name of the account where the reference must be created
  • reference – name of the reference to create
  • properties (dict) – dictionary of properties to set on the reference
Returns:

True if the reference has been created, False if it already existed

del_properties(account=None, reference=None, properties=None, cid=None, **kwargs)[source]

Delete properties for a reference.

delete(account=None, reference=None, cid=None, **kwargs)[source]

Delete a reference.

destroy(*args, **kwargs)[source]
Deprecated:use delete
force(account=None, reference=None, service_type=None, services=None, cid=None, autocreate=False, replace=False, **kwargs)[source]

Associate the specified services to the reference.

Parameters:replace (bool) – do not require the list of services of the specified type to be empty, overwrite it.
get_properties(account=None, reference=None, properties=None, cid=None, **kwargs)[source]

Get properties for a reference.

has(account=None, reference=None, cid=None, **kwargs)[source]

Poll and associate a new service to the reference.

list(account=None, reference=None, cid=None, service_type=None, **kwargs)[source]

List the services linked to the reference.

renew(account=None, reference=None, service_type=None, cid=None, **kwargs)[source]

Re-poll and re-associate a set of services to the reference. Will increment the sequence number.

set_properties(account=None, reference=None, properties=None, cid=None, **kwargs)[source]

Set properties for a reference.

show(*args, **kwargs)[source]
Deprecated:use list

Remove an associated service from the reference

oio.directory.indexer module

class oio.directory.indexer.Meta2Indexer(conf)[source]

Bases: oio.common.daemon.Daemon

A daemon that spawns a greenlet running a Meta2IndexingWorker for each volume.

run(*args, **kwargs)[source]
stop()[source]
class oio.directory.indexer.Meta2IndexingWorker(volume_path, conf, pool_manager=None)[source]

Bases: object

Indexing worker responsible for a single volume.

crawl_volume()[source]

Crawl the volume assigned to this worker, and index every database.

index_meta2_database(db_id)[source]

Add a meta2 database to the rdir index. Fails if the database isn’t handled by the current volume.

Parameters:db_id – The ContentID representing the reference to the database.
report(tag)[source]

Log the status of indexer

Parameters:tag – One of three: starting, running, ended.
run()[source]

Main worker loop

stop()[source]

Could be needed for eventually gracefully stopping.

warn(msg, container_id)[source]

oio.directory.meta module

class oio.directory.meta.MetaMapping(conf, service_types, admin_client=None, conscience_client=None, logger=None, rdir_client=None, **kwargs)[source]

Bases: object

Represents the content of the meta_n0 database

admin
apply(moved=None, **kwargs)[source]

Upload the current mapping to the meta_n0 services, and set peers accordingly in meta_n1 databases.

Parameters:moved – list of bases that have moved.
conscience
static get_cid_and_seq(base)[source]
rdir
reset()[source]

Reset the base allocations and reload the services from Conscience.

oio.directory.meta0 module

Meta0 client and meta1 balancing operations

class oio.directory.meta0.Meta0Client(conf, **kwargs)[source]

Bases: oio.common.client.ProxyClient

Meta0 administration client

force(mapping, **kwargs)[source]

Force the meta0 prefix mapping. The mapping may be partial to force only a subset of the prefixes.

list(**kwargs)[source]

Get the meta0 prefix mapping

class oio.directory.meta0.Meta0PrefixMapping(meta0_client, replicas=3, digits=None, min_dist=1, **kwargs)[source]

Bases: oio.directory.meta.MetaMapping

Represents the content of the meta0 database

assign_services(base, services, fail_if_already_set=False)[source]

Assign services to manage base.

Parameters:fail_if_already_set – raise ValueError if base is already managed by some services
bootstrap(level='volume', degradation=0)[source]

Spread the prefixe to balance accross sites. By default the token describing the site is <0>. Put one short_prefix into each slice (corresponding to a location level) and balance well into each level. At the end, expand each short prefixes into their complete prefixes. :param level: the kind of location token describing the level of

fair balancing.
Parameters:degradation – how many location slices of the given level we should tolerate.
Returns:None
check_replicas()[source]

Check that all bases have the right number of replicas

count_pfx_by_svc()[source]

Build a dict with service addresses as keys and the number of managed bases as values.

decommission(svc, bases_to_remove=None, strategy=None)[source]

Unassign all bases of bases_to_remove from svc, and assign them to other services using strategy.

static dist_between(loc1, loc2)[source]
find_services_less_bases(known=None, min_score=1, **_kwargs)[source]

Find replicas services, including the ones of known

get_loc(svc, default=None)[source]

Get the location of a service. If location is not defined, return: - service IP address if default is None or “addr” - default for any other value.

get_managed_bases(svc)[source]

Get the list of bases managed by the service

get_score(svc)[source]

Get the score of a service, or 0 if it is unknown

load_json(json_mapping, **kwargs)[source]

Load the mapping from a JSON string

load_meta0(json_mapping=None, **kwargs)[source]

Load the mapping from dictionary out of the cluster,

num_bases()[source]

Get total the number of bases according to self.digits

prefix_siblings(pfx)[source]

Get the list of prefixes that share the same base as pfx.

prefix_to_base(pfx)[source]

Get the name of the base the prefix will be saved in. When self.digits is 4, the name of the base is pfx.

rebalance(max_loops=65536)[source]

Reassign bases from the services which manage the most

services
to_json(bases=None)[source]

Serialize the mapping to a JSON string suitable as input for ‘meta0_force’ request.

oio.directory.meta0.count_prefixes(digits)[source]

Returns the number of real prefixes in meta0/meta1. Raises an exception if the prefix number is not acceptable.

oio.directory.meta0.generate_prefixes(digits)[source]
oio.directory.meta0.generate_short_prefixes(digits)[source]

oio.directory.meta2 module

class oio.directory.meta2.Meta2Database(conf, logger=None, admin_client=None, conscience_client=None, rdir_client=None, directory_client=None, service_type='meta2')[source]

Bases: object

Execute maintenance operations on meta2 databases (or compatible services).

admin
conscience
directory
static get_cid_and_seq(base)[source]
move(**kwargs)[source]

Move a database from src to dst. If dst is None, find one automatically.

rdir
rebuild(**kwargs)[source]

Rebuild a database.

reload_all_services()[source]

Load the list of all services of type self.service_type.

reset_peers()[source]

Reset the base allocations and reload the services from Conscience.

oio.directory.meta2_rebuilder module

class oio.directory.meta2_rebuilder.ContentRepairerWorker(tool, queue_workers, queue_reply)[source]

Bases: oio.common.tool.ToolWorker

class oio.directory.meta2_rebuilder.Meta2Rebuilder(conf, input_file=None, service_id=None, **kwargs)[source]

Bases: oio.common.tool.Tool

Rebuild meta2 databases.

DEFAULT_RDIR_FETCH_LIMIT = 100
create_worker(queue_workers, queue_reply)[source]

Create worker to process the items.

static string_from_item(item)[source]

Module contents