oio.directory package

Submodules

oio.directory.admin module

class oio.directory.admin.AdminClient(conf, **kwargs)

Bases: oio.common.client.ProxyClient

Low level database administration client.

cache_client

Instanciate a client object for ‘/cache/*’ proxy routes.

copy_base_from(service_type=None, account=None, reference=None, cid=None, service_id=None, **kwargs)

Copy a base to another service, using DB_PIPEFROM.

Parameters:

• svc_from – id of the source service.
• svc_to – id of the destination service.

copy_base_to(service_type=None, account=None, reference=None, cid=None, service_id=None, **kwargs)

Copy a base to another service, using DB_PIPETO. Source service is looked after in service directory.

Parameters:svc_to – id of the destination service.

election_debug(service_type=None, account=None, reference=None, cid=None, service_id=None, **kwargs)

Get debugging information about an election.

election_leave(service_type=None, account=None, reference=None, cid=None, service_id=None, **kwargs)

Force all peers to leave the election.

election_ping(service_type=None, account=None, reference=None, cid=None, service_id=None, **kwargs)

Trigger or refresh an election.

election_status(service_type=None, account=None, reference=None, cid=None, service_id=None, **kwargs)

Get the status of an election (trigger it if necessary).

Returns:a dict with ‘master’ (str), ‘slaves’ (list), ‘peers’ (dict) and ‘type’ (str)

example

{

‘peers’: {

‘127.0.0.3:6014’: {

‘status’:

{‘status’: 303,

‘message’: ‘127.0.0.1:6015’},

‘body’: u’’},

‘127.0.0.1:6015’: {

‘status’:

{‘status’: 200,

‘message’: ‘OK’},

‘body’: u’’},

‘127.0.0.2:6016’: {

‘status’:

{‘status’: 303,

‘message’: ‘127.0.0.1:6015’},

‘body’: u’’}

}, ‘master’: ‘127.0.0.1:6015’, ‘slaves’: [‘127.0.0.3:6014’, ‘127.0.0.2:6016’], ‘type’: ‘meta1’

}

election_sync(service_type=None, account=None, reference=None, cid=None, service_id=None, **kwargs)

Try to synchronize a dubious election.

forwarder

Instanciate a client object for ‘/forward/*’ proxy routes.

get_properties(service_type=None, account=None, reference=None, cid=None, service_id=None, **kwargs)

Get user and system properties from the admin table of an sqliterepo base.

has_base(service_type=None, account=None, reference=None, cid=None, service_id=None, **kwargs)

Ask each peer if base exists.

proxy_flush_cache(high=True, low=True, proxy_netloc=None, **kwargs)

Flush “high” and “low” proxy caches. By default, flush the cache of the local proxy. If proxy_netloc is provided, flush the cache of this proxy.

proxy_get_cache_status(proxy_netloc=None, **kwargs)

Get the status of the high (conscience and meta0) and low (meta1) cache, including the current number of entries.

proxy_get_live_config(proxy_netloc=None, **kwargs)

Get all configuration parameters from the specified proxy service.

Returns:a dictionary with all configuration keys the service recognizes, and their current value. Return type:dict

proxy_set_live_config(proxy_netloc=None, config=None, **kwargs)

Set configuration parameters on the specified proxy service.

remove_base(service_type=None, account=None, reference=None, cid=None, service_id=None, **kwargs)

Remove specific base.

service_balance_elections(svc_id, max_ops=0, inactivity=0, **kwargs)

Balance elections to get an acceptable slave/master ratio.

Parameters:

• svc_to – id of the service that should balance its elections.
• max_ops – maximum number of balancing operations.
• inactivity – avoid expiring election whose last activity is younger than the specified value.

service_flush_cache(svc_id, **kwargs)

Flush the resolver cache of an sqliterepo-based service.

service_get_info(svc_id, **kwargs)

Get all information from the specified service. Works on all services using ASN.1 protocol except conscience.

Returns:a dictionary with all information keys the service recognizes, and their current value. Return type:dict

service_get_live_config(svc_id, **kwargs)

Get all configuration parameters from the specified service. Works on all services using ASN.1 protocol.

Returns:a dictionary with all configuration keys the service recognizes, and their current value. Return type:dict

service_reload_lb(svc_id, **kwargs)

Force the service to reload its internal load balancer.

service_set_live_config(svc_id, config, **kwargs)

Set some configuration parameters on the specified service. Works on all services using ASN.1 protocol. Notice that some parameters may not be taken into account, and no parameter will survice a service restart.

set_peers(service_type=None, account=None, reference=None, cid=None, service_id=None, **kwargs)

Force the new peer set in the replicas of the old peer set.

set_properties(service_type=None, account=None, reference=None, cid=None, service_id=None, **kwargs)

Set user or system properties in the admin table of an sqliterepo base.

vacuum_base(service_type=None, account=None, reference=None, cid=None, service_id=None, **kwargs)

Vacuum (defragment) the database on the master service, then resynchronize it on the slaves.

oio.directory.admin.loc_params(func)

Wrap database localization parameters in request parameters

oio.directory.admin.service_id_to_string(service_id)

Convert a list of service IDs to a comma separated string.

oio.directory.client module

class oio.directory.client.DirectoryClient(conf, **kwargs)

Bases: oio.common.client.ProxyClient

Mid-level client for OpenIO SDS service directory (meta0, meta1).

create(account=None, reference=None, properties=None, **kwargs)

Create a reference (a service container).

Parameters:

• account – name of the account where the reference must be created
• reference – name of the reference to create
• properties (dict) – dictionary of properties to set on the reference

Returns:

True if the reference has been created, False if it already existed

del_properties(account=None, reference=None, properties=None, cid=None, **kwargs)

Delete properties for a reference.

delete(account=None, reference=None, cid=None, **kwargs)

Delete a reference.

destroy(*args, **kwargs)

Deprecated:use delete

force(account=None, reference=None, service_type=None, services=None, cid=None, autocreate=False, replace=False, **kwargs)

Associate the specified services to the reference.

Parameters:replace (bool) – do not require the list of services of the specified type to be empty, overwrite it.

get_properties(account=None, reference=None, properties=None, cid=None, **kwargs)

Get properties for a reference.

has(account=None, reference=None, cid=None, **kwargs)

link(account=None, reference=None, service_type=None, cid=None, autocreate=False, **kwargs)

Poll and associate a new service to the reference.

list(account=None, reference=None, cid=None, service_type=None, **kwargs)

List the services linked to the reference.

renew(account=None, reference=None, service_type=None, cid=None, **kwargs)

Re-poll and re-associate a set of services to the reference. Will increment the sequence number.

set_properties(account=None, reference=None, properties=None, cid=None, **kwargs)

Set properties for a reference.

show(*args, **kwargs)

Deprecated:use list

unlink(account=None, reference=None, service_type=None, cid=None, **kwargs)

Remove an associated service from the reference

oio.directory.indexer module

class oio.directory.indexer.Meta2Indexer(conf)

Bases: oio.common.daemon.Daemon

A daemon that spawns a greenlet running a Meta2IndexingWorker for each volume.

run(*args, **kwargs)

stop()

class oio.directory.indexer.Meta2IndexingWorker(volume_path, conf, pool_manager=None)

Bases: object

Indexing worker responsible for a single volume.

crawl_volume()

Crawl the volume assigned to this worker, and index every database.

index_meta2_database(db_id)

Add a meta2 database to the rdir index. Fails if the database isn’t handled by the current volume.

Parameters:db_id – The ContentID representing the reference to the database.

report(tag)

Log the status of indexer

Parameters:tag – One of three: starting, running, ended.

run()

Main worker loop

stop()

Could be needed for eventually gracefully stopping.

warn(msg, container_id)

oio.directory.meta module

class oio.directory.meta.MetaMapping(conf, service_types, admin_client=None, conscience_client=None, logger=None, rdir_client=None, **kwargs)

Bases: object

Represents the content of the meta_n0 database

admin

apply(moved=None, **kwargs)

Upload the current mapping to the meta_n0 services, and set peers accordingly in meta_n1 databases.

Parameters:moved – list of bases that have moved.

conscience

static get_cid_and_seq(base)

rdir

reset()

Reset the base allocations and reload the services from Conscience.

oio.directory.meta0 module

Meta0 client and meta1 balancing operations

class oio.directory.meta0.Meta0Client(conf, **kwargs)

Bases: oio.common.client.ProxyClient

Meta0 administration client

force(mapping, **kwargs)

Force the meta0 prefix mapping. The mapping may be partial to force only a subset of the prefixes.

list(**kwargs)

Get the meta0 prefix mapping

class oio.directory.meta0.Meta0PrefixMapping(meta0_client, replicas=3, digits=None, min_dist=1, **kwargs)

Bases: oio.directory.meta.MetaMapping

Represents the content of the meta0 database

assign_services(base, services, fail_if_already_set=False)

Assign services to manage base.

Parameters:fail_if_already_set – raise ValueError if base is already managed by some services

bootstrap(level='volume', degradation=0)

Spread the prefixe to balance accross sites. By default the token describing the site is <0>. Put one short_prefix into each slice (corresponding to a location level) and balance well into each level. At the end, expand each short prefixes into their complete prefixes. :param level: the kind of location token describing the level of

fair balancing.

Parameters:degradation – how many location slices of the given level we should tolerate. Returns:None

check_replicas()

Check that all bases have the right number of replicas

count_pfx_by_svc()

Build a dict with service addresses as keys and the number of managed bases as values.

decommission(svc, bases_to_remove=None, strategy=None, try_=False, compute_upper_limit=True)

Unassign all bases of bases_to_remove from svc, and assign them to other services using strategy. :param svc: service to decommission :type svc: str or dict :param bases_to_remove: bases to remove of the service :type bases_to_remove: list :param strategy: strategy to move the bases :type strategy: function :param try_: keep the bases on this service

if the strategy doesn’t find better, otherwise obligatorily remove the bases of this service

Parameters:compute_upper_limit (bool) – compute the right number bases each service should host

static dist_between(loc1, loc2)

find_services_less_bases(known=None, min_score=1, **_kwargs)

Find replicas services, including the ones of known

find_services_more_availability(known=None, min_score=1, **_kwargs)

Find replicas services, including the ones of known

get_loc(svc, default=None)

Get the location of a service. If location is not defined, return: - service IP address if default is None or “addr” - default for any other value.

get_managed_bases(svc)

Get the list of bases managed by the service

get_score(svc)

Get the score of a service, or 0 if it is unknown

load_json(json_mapping, **kwargs)

Load the mapping from a JSON string

load_meta0(json_mapping=None, **kwargs)

Load the mapping from dictionary out of the cluster,

num_bases()

Get total the number of bases according to self.digits

prefix_siblings(pfx)

Get the list of prefixes that share the same base as pfx.

prefix_to_base(pfx)

Get the name of the base the prefix will be saved in. When self.digits is 4, the name of the base is pfx.

rebalance(max_loops=4096)

Reassign bases from the services which manage the most

reset()

Reset the base allocations and reload the services from Conscience.

services

to_json(bases=None)

Serialize the mapping to a JSON string suitable as input for ‘meta0_force’ request.

oio.directory.meta0.count_prefixes(digits)

Returns the number of real prefixes in meta0/meta1. Raises an exception if the prefix number is not acceptable.

oio.directory.meta0.generate_prefixes(digits)

oio.directory.meta0.generate_short_prefixes(digits)

oio.directory.meta2 module

class oio.directory.meta2.Meta2Database(conf, logger=None, admin_client=None, conscience_client=None, rdir_client=None, directory_client=None, service_type='meta2')

Bases: object

Execute maintenance operations on meta2 databases (or compatible services).

admin

conscience

directory

static get_cid_and_seq(base)

move(**kwargs)

Move a database from src to dst. If dst is None, find one automatically.

rdir

rebuild(**kwargs)

Rebuild a database.

reload_all_services()

Load the list of all services of type self.service_type.

reset_peers()

Reset the base allocations and reload the services from Conscience.

oio.directory.meta2_rebuilder module

class oio.directory.meta2_rebuilder.ContentRepairerWorker(tool, queue_workers, queue_reply)

Bases: oio.common.tool.ToolWorker

class oio.directory.meta2_rebuilder.Meta2Rebuilder(conf, input_file=None, service_id=None, **kwargs)

Bases: oio.common.tool.Tool

Rebuild meta2 databases.

DEFAULT_RDIR_FETCH_LIMIT = 100

create_worker(queue_workers, queue_reply)

Create worker to process the items.

static string_from_item(item)

Module contents