oio.common package

Submodules

oio.common.autocontainer module

class oio.common.autocontainer.AutocontainerBuilder(offset=0, size=None, mask=18446744073692774655L, base=16, con_format='%016X', **_kwargs)

Bases: oio.common.autocontainer.ContainerBuilder

Build a container name from the integer conversion of a user provided path and a clever mask.

path is expected to be something like “video/ABC/DEF/xxxxxxxxxFEDCBAxxxxxxxxxx_nomdufichier”

verify(name)

Verify that name is an autocontainer

class oio.common.autocontainer.ContainerBuilder(**_kwargs)

Bases: object

Base class for container name builders.

alternatives(path)

Generate all alternatives for the provided content path.

verify(name)

Verify that name is an autocontainer

class oio.common.autocontainer.HashedContainerBuilder(offset=0, size=None, bits=15, **_kwargs)

Bases: oio.common.autocontainer.ContainerBuilder

Build a container name from a SHA256 of the content path. Only the first (most significant) bits will be considered to generate the final prefix.

verify(name)

Verify that name is an autocontainer

exception oio.common.autocontainer.NoMatchFound

Bases: exceptions.ValueError

Exception raised when none of the configured patterns match the input object name.

class oio.common.autocontainer.RegexContainerBuilder(patterns, builder=<class 'oio.common.autocontainer.ContainerBuilder'>, **kwargs)

Bases: object

Build a container name from a regular expression applied on a user provided path. Use a concatenation of all matching groups as the container name if no custom builder provided.

Parameters:patterns (str or iterable of str) – regular expressions with at least one capture group

alternatives(path)

Generate all alternatives for the provided path, in case it matches several patterns.

verify(name)

oio.common.autocontainer.strtoll(val, base=10)

Mimics libc’s strtoll function

oio.common.cache module

oio.common.cache.del_cached_container_metadata(account=None, reference=None, cid=None, cache=None, **kwargs)

Delete the object metadata and location from the cache (if there is one)

oio.common.cache.del_cached_object_metadata(account=None, reference=None, path=None, cid=None, version=None, cache=None, **kwargs)

Delete the object metadata and location from the cache (if there is one)

oio.common.cache.get_cached_container_metadata(account=None, reference=None, cid=None, cache=None, **kwargs)

Get the container metadata from the cache (if there is one)

oio.common.cache.get_cached_object_metadata(account=None, reference=None, path=None, cid=None, version=None, properties=False, cache=None, **kwargs)

Get the object metadata and location from the cache (if there is one)

oio.common.cache.set_cached_container_metadata(container_meta, account=None, reference=None, cid=None, cache=None, **kwargs)

Set the object metadata and location in the cache (if there is one)

oio.common.cache.set_cached_object_metadata(content_meta, content_chunks, account=None, reference=None, path=None, cid=None, version=None, properties=False, cache=None, **kwargs)

Set the object metadata and location in the cache (if there is one)

oio.common.client module

class oio.common.client.ProxyClient(conf, request_prefix='', no_ns_in_url=False, endpoint=None, request_attempts=1, logger=None, **kwargs)

Bases: oio.api.base.HttpApi

Client directed towards oio-proxy, with logging facility

oio.common.configuration module

exception oio.common.configuration.InvalidServiceConfigError

Bases: exceptions.ValueError

class oio.common.configuration.NamespaceConfiguration(files, *args, **kwargs)

Bases: dict

Dictionary subclass that holds namespace configuration. Displays a nice message if a key is missing.

oio.common.configuration.load_namespace_conf(namespace, failsafe=False, fresh=False)

Load configuration for the namespace from the local configuration files.

Parameters:

• namespace – name of the namespace.
• failsafe – in case of error, return a dummy configuration.
• fresh – if True, reload configuration from files, do not use the cache.

Returns:

a dictionary with local namespace configuration.

oio.common.configuration.parse_config(conf_path)

oio.common.configuration.parse_options(parser=None)

oio.common.configuration.read_conf(conf_path, section_name=None, defaults=None, use_yaml=False)

oio.common.configuration.set_namespace_options(namespace, options, remove=None)

Set options in the local namespace configuration file. Can have nasty effects, be careful, only use in test code.

Parameters:

• namespace – the namespace to work with
• options – a dictionary with options to set
• remove – an iterable of options to remove

Returns:

a dictionary with all options of the namespace

oio.common.configuration.validate_service_conf(conf)

oio.common.constants module

oio.common.daemon module

class oio.common.daemon.Daemon(conf)

Bases: object

run(*args, **kwargs)

start(**kwargs)

stop()

oio.common.daemon.run_daemon(klass, conf_file, section_name=None, **kwargs)

oio.common.decorators module

oio.common.decorators.ensure_headers(func)

oio.common.decorators.ensure_request_id(func)

oio.common.decorators.ensure_request_id2(prefix='')

Ensure the subsequent RPCs will have a request ID.

oio.common.decorators.handle_account_not_found(fnc)

oio.common.decorators.handle_container_not_found(fnc)

oio.common.decorators.handle_object_not_found(fnc)

Catch oio.common.exceptions.NotFound exceptions and raise either oio.common.exceptions.NoSuchContainer or oio.common.exceptions.NoSuchObject respectively if the container is missing or the object is missing.

oio.common.decorators.patch_kwargs(fnc)

Patch keyword arguments with the ones passed to the class’ constructor. Compute a deadline if a timeout is provided and there is no deadline already. Requires the class to have a _global_kwargs member (dict).

oio.common.easy_value module

oio.common.easy_value.boolean_value(value, default=False)

oio.common.easy_value.convert_size(size, unit='')

oio.common.easy_value.debinarize(something)

Convert binary data to string.

For bytes arrays, return a string. For lists, call debinarize all items. For dicts, debinarize keys and values. For other types, return the item as is.

oio.common.easy_value.float_value(value, default)

oio.common.easy_value.int_value(value, default)

oio.common.easy_value.is_hexa(hexa, size=None)

oio.common.easy_value.true_value(value)

oio.common.exceptions module

exception oio.common.exceptions.BadRequest(http_status=400, status=None, message=None)

Bases: oio.common.exceptions.ClientException

Request is not correct.

exception oio.common.exceptions.ChunkException

Bases: oio.common.exceptions.OioException

exception oio.common.exceptions.ClientException(http_status, status=None, message=None)

Bases: oio.common.exceptions.StatusMessageException

exception oio.common.exceptions.ClientPreconditionFailed(http_status=412, status=None, message=None)

Bases: oio.common.exceptions.ClientException

exception oio.common.exceptions.CommandError

Bases: exceptions.Exception

exception oio.common.exceptions.ConfigurationException

Bases: oio.common.exceptions.OioException

exception oio.common.exceptions.Conflict(http_status=409, status=None, message=None)

Bases: oio.common.exceptions.ClientException

exception oio.common.exceptions.ContainerNotEmpty

Bases: oio.common.exceptions.OioException

exception oio.common.exceptions.ContentException

Bases: oio.common.exceptions.OioException

exception oio.common.exceptions.ContentNotFound

Bases: oio.common.exceptions.ContentException

exception oio.common.exceptions.CorruptedChunk

Bases: oio.common.exceptions.ChunkException

exception oio.common.exceptions.DeadlineReached

Bases: oio.common.exceptions.OioException

Special exception to be raised when a deadline is reached. This differs from the OioTimeout in that we are sure the operation won’t succeed silently in the background.

exception oio.common.exceptions.ECError

Bases: exceptions.Exception

exception oio.common.exceptions.EmptyByteRange

Bases: exceptions.Exception

exception oio.common.exceptions.EtagMismatch

Bases: oio.common.exceptions.OioException

exception oio.common.exceptions.ExplicitBury

Bases: oio.common.exceptions.OioException

exception oio.common.exceptions.FaultyChunk

Bases: oio.common.exceptions.ChunkException

Raised when a chunk misses some extended attributes, or they have invalid values.

exception oio.common.exceptions.FileNotFound

Bases: oio.common.exceptions.OioException

exception oio.common.exceptions.Forbidden(http_status=403, status=None, message=None)

Bases: oio.common.exceptions.ClientException

Operation is forbidden.

exception oio.common.exceptions.InvalidStorageMethod

Bases: oio.common.exceptions.OioException

exception oio.common.exceptions.LifecycleNotFound

Bases: oio.common.exceptions.OioException

exception oio.common.exceptions.Meta2Exception

Bases: oio.common.exceptions.OioException

exception oio.common.exceptions.MethodNotAllowed(http_status=405, status=None, message=None)

Bases: oio.common.exceptions.ClientException

Request method is not allowed. May be raised when the namespace is in WORM mode and user tries to delete.

exception oio.common.exceptions.MissingAttribute(attribute)

Bases: oio.common.exceptions.OioException

exception oio.common.exceptions.MissingContentLength

Bases: oio.common.exceptions.OioException

exception oio.common.exceptions.MissingData

Bases: oio.common.exceptions.OioException

exception oio.common.exceptions.MissingName

Bases: oio.common.exceptions.OioException

exception oio.common.exceptions.NoSuchAccount

Bases: oio.common.exceptions.OioException

exception oio.common.exceptions.NoSuchContainer

Bases: oio.common.exceptions.OioException

exception oio.common.exceptions.NoSuchObject

Bases: oio.common.exceptions.OioException

exception oio.common.exceptions.NoSuchReference

Bases: oio.common.exceptions.OioException

exception oio.common.exceptions.NotFound(http_status=404, status=None, message=None)

Bases: oio.common.exceptions.ClientException

Resource was not found.

exception oio.common.exceptions.OioException

Bases: exceptions.Exception

exception oio.common.exceptions.OioNetworkException

Bases: oio.common.exceptions.OioException

Network related exception (connection, timeout…).

exception oio.common.exceptions.OioTimeout

Bases: oio.common.exceptions.OioNetworkException

exception oio.common.exceptions.OrphanChunk

Bases: oio.common.exceptions.ChunkException

exception oio.common.exceptions.PreconditionFailed

Bases: oio.common.exceptions.OioException

exception oio.common.exceptions.RetryLater

Bases: oio.common.exceptions.OioException

Exception raised by workers that want a task to be rescheduled later.

exception oio.common.exceptions.ServerException

Bases: oio.common.exceptions.OioException

exception oio.common.exceptions.ServiceBusy(http_status=503, status=None, message=None)

Bases: oio.common.exceptions.ClientException

This kind of exceptions tell that the system was “busy” and could not handle the request at the moment. The user is invited to retry after a few seconds.

exception oio.common.exceptions.ServiceUnavailable

Bases: oio.common.exceptions.OioException

Exception raised when some services are temporarily not available. This does not mean data is lost.

exception oio.common.exceptions.SourceReadError

Bases: oio.common.exceptions.OioException

exception oio.common.exceptions.SourceReadTimeout

Bases: oio.common.exceptions.OioTimeout

Specialization of OioTimeout for the case when a timeout occurs while reading data from a client application.

exception oio.common.exceptions.SpareChunkException

Bases: oio.common.exceptions.Meta2Exception

Exception raised when no spare chunk has been found, or some may have been found but they don’t match all criteria.

exception oio.common.exceptions.StatusMessageException(http_status, status=None, message=None)

Bases: oio.common.exceptions.OioException

Error carrying an HTTP status, an OIO status and a message.

exception oio.common.exceptions.TooLarge(http_status=413, status=None, message=None)

Bases: oio.common.exceptions.ClientException

exception oio.common.exceptions.UnfinishedUploadException(exception, chunks_already_uploaded)

Bases: oio.common.exceptions.OioException

Exception raised when a number of chunks are not uploaded.

reraise()

Re-raise the wrapped exception. This is intended to be called after some sort of cleanup has been done.

exception oio.common.exceptions.UnrecoverableContent

Bases: oio.common.exceptions.ContentException

exception oio.common.exceptions.UnsatisfiableRange(http_status=416, status=None, message=None)

Bases: oio.common.exceptions.ClientException

exception oio.common.exceptions.VolumeException

Bases: oio.common.exceptions.OioException

Exception raised when someone is trying to contact a rdir service, but there is none assigned to the specified rawx.

oio.common.exceptions.from_response(resp, body=None)

oio.common.exceptions.from_status(status, reason='n/a')

oio.common.exceptions.reraise(exc_type, exc_value, extra_message=None)

Raise an exception of type exc_type with arguments of exc_value plus maybe extra_message at the beginning.

oio.common.fullpath module

oio.common.fullpath.decode_fullpath(fullpath)

Decode a “fullpath” string, extract its 5 parts.

Raises:ValueError if the string has invalid format. Returns:account, container, path, version and content ID.

oio.common.fullpath.decode_old_fullpath(fullpath)

oio.common.fullpath.encode_fullpath(account, container, path, version, content_id)

oio.common.fullpath.encode_old_fullpath(account, container, path, version)

oio.common.green module

exception oio.common.green.ChunkReadTimeout(seconds=None, exception=None)

Bases: oio.common.green.OioTimeout

msg_prefix = 'Chunk read '

exception oio.common.green.ChunkWriteTimeout(seconds=None, exception=None)

Bases: oio.common.green.OioTimeout

msg_prefix = 'Chunk write '

exception oio.common.green.ConnectionTimeout(seconds=None, exception=None)

Bases: oio.common.green.OioTimeout

msg_prefix = 'Connection '

class oio.common.green.ContextPool(size=1000)

Bases: eventlet.greenpool.GreenPool

exception oio.common.green.OioTimeout(seconds=None, exception=None)

Bases: eventlet.timeout.Timeout

Wrapper over eventlet.Timeout with better __str__.

msg_prefix = ''

exception oio.common.green.SourceReadTimeout(seconds=None, exception=None)

Bases: oio.common.green.OioTimeout

msg_prefix = 'Source read '

oio.common.green.eventlet_yield()

Swith to another eventlet coroutine.

oio.common.green.get_hub()

oio.common.green.ratelimit(run_time, max_rate, increment=1, rate_buffer=5, time_time=None)

oio.common.green.ratelimit_function_build(policy)

Given a policy, return a customized wrapper around ratelimit for a time aware rate limiter. :param policy: An array representing a rate limiting policy as described

by ratelimit_validate_policy.

Returns:A callable function similar in signature to ratelimit but that ignores all parameters other than the first one.

oio.common.green.ratelimit_function_curr_rate(curr_date, policy)

Given a validated policy and a datetime, return the applicable max_rate

Parameters:curr_date – The current date

:type curr_date datetime :param policy: An array representing a validated policy :return: The applicable max_rate (elements per second)

oio.common.green.ratelimit_function_next_rate(curr_date, policy)

Given a current date and a policy, calculate the date at which the next rate change is scheduled.

(Could be useful if the rate limited operation is fast, and as such we would want to cache the next rate date so that instead of selecting the rate each op, we’d just compare to a timestamp and return a cached value, which in the current implementation would make it go from a for loop with several comparisons to about a single comparison)

Parameters:

• curr_date (datetime) – The current datetime
• policy – A list representing a validated policy.

Returns:

the next scheduled rate and the datetime object for the next scheduled rate change.

oio.common.green.ratelimit_policy_from_string(policy_str)

Return type:list of 2-tuples with a datetime.timedelta and an integer.

oio.common.green.ratelimit_validate_policy(policy)

Validate a policy. The following rules are checked: - Each partition has a positive max_rate. - The start date of each partition is 0 or positive. - The start date of each partition is lower than 24h.

An example of a simple policy would be: [

(datetime.timedelta(0), 3),

]

Which would be a policy to have a constant max_rate of 3. A more complex policy would be: [

(datetime.timedelta(0, 1800), 10), # 0h30 to 6h45 (datetime.timedelta(0, 24300), 2), # 6h45 to 9h45 (datetime.timedelta(0, 35100), 5), # 9h45 to 15h30 (datetime.timedelta(0, 55800), 3), # 15h30 to 20h00 (datetime.timedelta(0, 72000), 8), # 20h00 to 0h30

]

Parameters:policy (list) – A list containing the policy that follows the aforementioned description. Raises:ValueError if one of the rules is not respected.

oio.common.http module

class oio.common.http.CustomHttpConnection(*args, **kwargs)

Bases: oio.common.http_eventlet.CustomHttpConnection

class oio.common.http.HeadersDict(headers, **kwargs)

Bases: dict

get(k[, d]) → D[k] if k in D, else d. d defaults to None.

pop(k[, d]) → v, remove specified key and return the corresponding value.

If key is not found, d is returned if given, otherwise KeyError is raised

update([E, ]**F) → None. Update D from dict/iterable E and F.

If E present and has a .keys() method, does: for k in E: D[k] = E[k] If E present and lacks .keys() method, does: for (k, v) in E: D[k] = v In either case, this is followed by: for k in F: D[k] = F[k]

oio.common.http.get_addr(host, port)

Generate the address for host (IPv4 or IPv6) and port

oio.common.http.headers_from_object_metadata(metadata)

Generate chunk PUT request headers from object metadata.

oio.common.http.http_header_from_ranges(ranges)

oio.common.http.parse_content_range(raw_content_range)

oio.common.http.parse_content_type(raw_content_type)

oio.common.http.ranges_from_http_header(val)

oio.common.http_eventlet module

class oio.common.http_eventlet.CustomHTTPResponse(sock, debuglevel=0, strict=0, method=None, url=None)

Bases: httplib.HTTPResponse

close()

force_close()

read(amount=None)

class oio.common.http_eventlet.CustomHttpConnection(host, port=None, strict=None, timeout=<object object>, source_address=None)

Bases: httplib.HTTPConnection

connect()

getresponse()

putrequest(method, url, skip_host=0, skip_accept_encoding=0)

response_class

alias of CustomHTTPResponse

set_cork(enabled=True)

Enable or disable TCP_CORK on the underlying socket.

set_nodelay(enabled=True)

Enable or disable TCP_NODELAY on the underlying socket.

class oio.common.http_eventlet.CustomHttpsConnection(host, port=None, key_file=None, cert_file=None, strict=None, timeout=<object object>, source_address=None, context=None, check_hostname=None)

Bases: httplib.HTTPSConnection

connect()

getresponse()

putrequest(method, url, skip_host=0, skip_accept_encoding=0)

response_class

alias of CustomHTTPResponse

set_cork(enabled=True)

Enable or disable TCP_CORK on the underlying socket.

oio.common.http_eventlet.http_connect(host, method, path, headers=None, query_string=None, scheme='http')

oio.common.http_urllib3 module

class oio.common.http_urllib3.SafePoolManager(num_pools=10, headers=None, **connection_pool_kw)

Bases: urllib3.poolmanager.PoolManager

urllib3.PoolManager wrapper that filters out keyword arguments not recognized by urllib3.

request(*args, **kwargs)

Filter out arguments that are not recognized by urllib3, then call urllib3.PoolManager.request.

oio.common.http_urllib3.get_pool_manager(pool_connections=32, pool_maxsize=32, max_retries=0, backoff_factor=0, **kwargs)

Get urllib3.PoolManager to manage pools of connections

Parameters:

• pool_connections (int) – number of connection pools (see “num_pools”).
• pool_maxsize (int) – number of connections per connection pool
• max_retries (int) – number of retries per request
• backoff_factor (float) – backoff factor to apply between attempts after second try

oio.common.http_urllib3.oio_exception_from_httperror(exc, reqid=None, url=None)

Convert an HTTPError from urllib3 to an OioException, and re-raise it.

oio.common.json module

oio.common.logger module

class oio.common.logger.StreamToLogger(logger, log_type='STDOUT')

Bases: object

close()

flush()

write(value)

writelines(values)

oio.common.logger.get_logger(conf, name=None, verbose=False, fmt='%(process)d %(thread)X %(name)s %(levelname)s %(message)s')

oio.common.logger.redirect_stdio(logger)

Close stdio, redirect stdout and stderr.

Parameters:logger –

oio.common.redis_conn module

class oio.common.redis_conn.RedisConnection(host=None, sentinel_hosts=None, sentinel_name=None, **kwargs)

Bases: object

URL_QUERY_ARGUMENT_PARSERS = {'health_check_interval': <type 'int'>, 'max_connections': <type 'int'>, 'retry_on_timeout': <function true_value at 0x7fdf932391b8>, 'socket_connect_timeout': <type 'float'>, 'socket_keepalive': <function true_value at 0x7fdf932391b8>, 'socket_timeout': <type 'float'>}

acquire_lock_with_timeout(lockname, acquire_timeout=10, lock_timeout=10)

Acquire a lock :lockname:

conn

Retrieve Redis connection (normal or sentinel)

conn_slave

Retrieve Redis connection (normal or sentinel)

register_script(script)

Register a LUA script and return Script object.

release_lock(lockname, identifier)

Release a previously acquired Lock

oio.common.redis_conn.catch_service_errors(func)

Catch errors attributable to the Redis service and raise ServiceBusy instead.

Raises:ServiceBusy – in case of a Redis service error

oio.common.storage_functions module

oio.common.storage_functions.fetch_stream(*args, **kwargs)

oio.common.storage_functions.fetch_stream_ec(*args, **kwargs)

oio.common.storage_functions.get_meta_ranges(ranges, chunks)

Convert object ranges to metachunks ranges.

Returns:a list of dictionaries indexed by metachunk positions

oio.common.storage_functions.obj_range_to_meta_chunk_range(obj_start, obj_end, meta_sizes)

Convert a requested object range into a list of meta_chunk ranges.

Parameters:meta_sizes (iterable, sorted in ascendant metachunk order.) – size of all object metachunks. Must be sorted! Returns:a dict of tuples (meta_chunk_start, meta_chunk_end) with metachunk positions as keys.

• meta_chunk_start is the first byte of the meta chunk, or None if this is a suffix byte range
• meta_chunk_end is the last byte of the meta_chunk, or None if this is a prefix byte range

oio.common.storage_functions.wrand_choice_index(scores)

Choose an element from the scores sequence and return its index

oio.common.storage_method module

class oio.common.storage_method.BackblazeStorageMethod(name, account_id, bucket_name)

Bases: oio.common.storage_method.StorageMethod

account_id

bucket_name

classmethod build(params)

class oio.common.storage_method.ECStorageMethod(name, ec_segment_size, ec_type, ec_nb_data, ec_nb_parity)

Bases: oio.common.storage_method.StorageMethod

classmethod build(params)

ec_fragment_size

ec_nb_data

ec_nb_parity

ec_segment_size

ec_type

expected_chunks

min_chunks_to_read

quorum

class oio.common.storage_method.ReplicatedStorageMethod(name, nb_copy)

Bases: oio.common.storage_method.StorageMethod

classmethod build(params)

expected_chunks

min_chunks_to_read

nb_copy

quorum

class oio.common.storage_method.StorageMethod(name, ec=False, backblaze=False)

Bases: object

backblaze

ec

expected_chunks

min_chunks_to_read

name

quorum

class oio.common.storage_method.StorageMethodLoad

Bases: object

class oio.common.storage_method.StorageMethods(methods)

Bases: object

load(chunk_method, **kwargs)

oio.common.storage_method.guess_storage_method(url)

oio.common.storage_method.load_methods()

oio.common.storage_method.parse_chunk_method(chunk_method)

oio.common.timestamp module

class oio.common.timestamp.Timestamp(timestamp=None)

Bases: object

isoformat

normal

oio.common.timestamp.cmp(x, y)

oio.common.tool module

class oio.common.tool.Tool(conf, beanstalkd_addr=None, logger=None)

Bases: object

Process all found items.

For the task_res variable, the following format must be respected: (item, info, error).

DEFAULT_BEANSTALKD_WORKER_TUBE = 'oio-process'

DEFAULT_CONCURRENCY = 1

DEFAULT_DISTRIBUTED_BEANSTALKD_WORKER_TUBE = 'oio-process'

DEFAULT_ITEM_PER_SECOND = 30

DEFAULT_REPORT_INTERVAL = 3600

DEFAULT_RETRY_DELAY = 3600

create_worker(queue_workers, queue_reply)

Create worker to process the items.

exit_gracefully(signum, frame)

fetch_items_with_beanstalkd_reply()

Fetch items with beanstalkd reply (useful if the task is distributed).

is_success()

Check if there are any errors.

static items_from_task_event(task_event)

Convert the task event into a list (generator) of items.

log_report(status, force=False)

Log a report with a fixed interval.

prepare_distributed_dispatcher()

The tool will dispatch the tasks on the platform.

prepare_local_dispatcher()

The tool will dispatch the tasks locally.

static res_event_from_task_res(task_res)

Convert the task result into a result event.

run()

Start processing all found items.

static string_from_item(item)

Convert the item into a string.

static task_event_from_item(item)

Convert the item into a task event.

static tasks_res_from_res_event(res_event)

Convert the result event into a list (generator) of tasks result.

update_counters(task_res)

Update all counters of the tool.

class oio.common.tool.ToolWorker(tool, queue_workers, queue_reply)

Bases: object

Process all items given by the tool.

run()

Starting processing all items given by the tool.

oio.common.tool.locate_tube(services, tube)

Get a list of beanstalkd services hosting the specified tube.

Parameters:

• services (iterable of dictionaries) – known beanstalkd services.
• tube – the tube to locate.

Returns:

a list of beanstalkd services hosting the the specified tube.

Return type:

list of dict

oio.common.utils module

class oio.common.utils.CacheDict(size=262144)

Bases: collections.OrderedDict

OrderedDict subclass which holds a limited number of items.

class oio.common.utils.GeneratorIO(gen, sub_generator=True, iter_size=8192)

Bases: io.RawIOBase

Make a file-like object from a generator. gen is the generator to read. sub_generator is a boolean telling that the generator yields sequences of bytes instead of bytes.

read(size=None)

readable()

Return whether object was opened for reading.

If False, read() will raise IOError.

readinto(b)

class oio.common.utils.RingBuffer(size=1)

Bases: list

append(value)

L.append(object) – append object to end

size

Get the size of the ring buffer

oio.common.utils.cid_from_name(account, ref)

Compute a container ID from an account and a reference name.

oio.common.utils.compute_perfdata_stats(perfdata, prefix='upload.')

Compute extra statistics from a dictionary of performance data.

oio.common.utils.deadline_to_timeout(deadline, check=False)

Convert a deadline (float seconds) to a timeout (float seconds)

oio.common.utils.depaginate(func, item_key=None, listing_key=None, marker_key=None, truncated_key=None, attempts=1, *args, **kwargs)

Yield items from the lists returned by the repetitive calls to func(*args, **kwargs). For each call (except the first), the marker is taken from the last element returned by the previous call (unless marker_key is provided). The listing stops after an empty listing is returned (unless truncated_key is provided).

Parameters:

• item_key – an accessor to the actual item that should be yielded, applied on each element of the listing
• listing_key – an accessor to the actual listing, applied on the result of func(*args, **kwargs)
• marker_key – an accessor to the next marker from the previous listing, applied on the result of func(*args, **kwargs)
• truncated_key – an accessor telling if the listing is truncated, applied on the result of func(*args, **kwargs)

oio.common.utils.drop_privileges(user)

oio.common.utils.encode(input, codec='utf-8')

Recursively encode a list of dictionnaries

oio.common.utils.fix_ranges(ranges, length)

oio.common.utils.getrandbits(k) → x. Generates a long int with k random bits.

oio.common.utils.group_chunk_errors(chunk_err_iter)

Group errors in a dictionary of lists. The keys are errors, the values are lists of chunk IDs.

oio.common.utils.lower_dict_keys(mydict)

Convert all dict keys to lower case.

oio.common.utils.monotonic_time()

Get the monotonic time as float seconds

oio.common.utils.paths_gen(volume_path)

Yield paths of all regular files under volume_path.

oio.common.utils.quote(value, safe='/')

oio.common.utils.request_id(prefix='')

Build a 128-bit request id string.

Parameters:prefix – optional prefix to the request id.

oio.common.utils.set_deadline_from_read_timeout(kwargs, force=False)

Compute a deadline from a read timeout, and set it in a keyword argument dictionary if there is none (or force is set).

oio.common.utils.set_fd_close_on_exec(fd)

oio.common.utils.set_fd_non_blocking(fd)

oio.common.utils.statfs(volume)

Parameters:volume – path to the mount point to get stats from. Returns:the free space ratio. Return type:float

oio.common.utils.timeout_to_deadline(timeout, now=None)

Convert a timeout (float seconds) to a deadline (float seconds).

oio.common.wsgi module

class oio.common.wsgi.Application(app, conf, logger_class=None)

Bases: gunicorn.app.base.BaseApplication

access_log_fmt = '%(bind0)s %(h)s:%({remote}p)s %(m)s %(s)s %(D)s %(B)s %(l)s %({x-oio-req-id}i)s %(U)s?%(q)s'

load()

load_config()

This method is used to load the configuration from one or several input(s). Custom Command line, configuration file. You have to override this method in your class.

class oio.common.wsgi.ServiceLogger(cfg)

Bases: gunicorn.glogging.Logger

access(resp, req, environ, request_time)

See http://httpd.apache.org/docs/2.0/logs.html#combined for format details

atoms(resp, req, environ, request_time)

Gets atoms for log formating.

class oio.common.wsgi.WerkzeugApp(url_map=None, logger=None)

Bases: object

dispatch_request(req)

wsgi_app(environ, start_response)

oio.common.wsgi.init_request_processor(conf_file, app_name, app_factory, *args, **kwargs)

oio.common.xattr module

oio.common.xattr.read_user_xattr(fd)

oio.common.xattr.set_fullpath_xattr(fd, new_fullpaths, remove_old_xattr=False, xattr_to_remove=None)

Insert new fullpath extended attributes, remove deprecated ones.

Parameters:

• new_fullpaths – dictionary of “fullpath” extended attributes that should be set on file. The key is the chunk ID (required to generate the attribute key), the value is the “fullpath”.
• remove_old_xattr (bool) – remove legacy attributes from file
• xattr_to_remove – list of extra extended attributes that should be removed from file

Module contents