oio.common package

Submodules

oio.common.autocontainer module

class oio.common.autocontainer.AutocontainerBuilder(offset=0, size=None, mask=18446744073692774655L, base=16, con_format='%016X', **_kwargs)

Bases: oio.common.autocontainer.ContainerBuilder

Build a container name from the integer conversion of a user provided path and a clever mask.

path is expected to be something like “video/ABC/DEF/xxxxxxxxxFEDCBAxxxxxxxxxx_nomdufichier”

verify(name)

Verify that name is an autocontainer

class oio.common.autocontainer.ContainerBuilder(**_kwargs)

Bases: object

Base class for container name builders.

alternatives(path)

Generate all alternatives for the provided content path.

verify(name)

Verify that name is an autocontainer

class oio.common.autocontainer.HashedContainerBuilder(offset=0, size=None, bits=15, **_kwargs)

Bases: oio.common.autocontainer.ContainerBuilder

Build a container name from a SHA256 of the content path. Only the first (most significant) bits will be considered to generate the final prefix.

verify(name)

Verify that name is an autocontainer

exception oio.common.autocontainer.NoMatchFound

Bases: exceptions.ValueError

Exception raised when none of the configured patterns match the input object name.

class oio.common.autocontainer.RegexContainerBuilder(patterns, builder=<class 'oio.common.autocontainer.ContainerBuilder'>, **kwargs)

Bases: object

Build a container name from a regular expression applied on a user provided path. Use a concatenation of all matching groups as the container name if no custom builder provided.

Parameters:patterns (str or iterable of str) – regular expressions with at least one capture group

alternatives(path)

Generate all alternatives for the provided path, in case it matches several patterns.

verify(name)

oio.common.autocontainer.strtoll(val, base=10)

Mimics libc’s strtoll function

oio.common.cache module

oio.common.cache.del_cached_container_metadata(account=None, reference=None, cid=None, cache=None, **kwargs)

Delete the object metadata and location from the cache (if there is one)

oio.common.cache.del_cached_object_metadata(account=None, reference=None, path=None, cid=None, version=None, cache=None, **kwargs)

Delete the object metadata and location from the cache (if there is one)

oio.common.cache.get_cached_container_metadata(account=None, reference=None, cid=None, cache=None, **kwargs)

Get the container metadata from the cache (if there is one)

oio.common.cache.get_cached_object_metadata(account=None, reference=None, path=None, cid=None, version=None, properties=False, cache=None, **kwargs)

Get the object metadata and location from the cache (if there is one)

oio.common.cache.set_cached_container_metadata(container_meta, account=None, reference=None, cid=None, cache=None, **kwargs)

Set the object metadata and location in the cache (if there is one)

oio.common.cache.set_cached_object_metadata(content_meta, content_chunks, account=None, reference=None, path=None, cid=None, version=None, properties=False, cache=None, **kwargs)

Set the object metadata and location in the cache (if there is one)

oio.common.client module

class oio.common.client.ProxyClient(conf, request_prefix='', no_ns_in_url=False, endpoint=None, request_attempts=1, logger=None, **kwargs)

Bases: oio.api.base.HttpApi

Client directed towards oio-proxy, with logging facility

oio.common.configuration module

exception oio.common.configuration.InvalidServiceConfigError

Bases: exceptions.ValueError

oio.common.configuration.config_paths()

Yield paths to potential namespace configuration files.

oio.common.configuration.load_namespace_conf(namespace)

oio.common.configuration.parse_config(conf_path)

oio.common.configuration.parse_options(parser=None)

oio.common.configuration.read_conf(conf_path, section_name=None, defaults=None, use_yaml=False)

oio.common.configuration.set_namespace_options(namespace, options, remove=None)

Set options in the local namespace configuration file. Can have nasty effects, be careful, only use in test code.

Parameters:

• namespace – the namespace to work with
• options – a dictionary with options to set
• remove – an iterable of options to remove

Returns:

a dictionary with all options of the namespace

oio.common.configuration.validate_service_conf(conf)

oio.common.constants module

oio.common.daemon module

class oio.common.daemon.Daemon(conf)

Bases: object

run(*args, **kwargs)

start(**kwargs)

stop()

oio.common.daemon.run_daemon(klass, conf_file, section_name=None, **kwargs)

oio.common.decorators module

oio.common.decorators.ensure_headers(func)

oio.common.decorators.ensure_request_id(func)

oio.common.decorators.handle_account_not_found(fnc)

oio.common.decorators.handle_container_not_found(fnc)

oio.common.decorators.handle_object_not_found(fnc)

Catch oio.common.exceptions.NotFound exceptions and raise either oio.common.exceptions.NoSuchContainer or oio.common.exceptions.NoSuchObject respectively if the container is missing or the object is missing.

oio.common.decorators.patch_kwargs(fnc)

Patch keyword arguments with the ones passed to the class’ constructor. Compute a deadline if a timeout is provided and there is no deadline already. Requires the class to have a _global_kwargs member (dict).

oio.common.easy_value module

oio.common.easy_value.boolean_value(value)

oio.common.easy_value.convert_size(size, unit='')

oio.common.easy_value.float_value(value, default)

oio.common.easy_value.int_value(value, default)

oio.common.easy_value.is_hexa(hexa, size=None)

oio.common.easy_value.true_value(value)

oio.common.exceptions module

exception oio.common.exceptions.ChunkException

Bases: oio.common.exceptions.OioException

exception oio.common.exceptions.ClientException(http_status, status=None, message=None)

Bases: oio.common.exceptions.StatusMessageException

exception oio.common.exceptions.CommandError

Bases: exceptions.Exception

exception oio.common.exceptions.ConfigurationException

Bases: oio.common.exceptions.OioException

exception oio.common.exceptions.Conflict(http_status=409, status=None, message=None)

Bases: oio.common.exceptions.ClientException

exception oio.common.exceptions.ContainerNotEmpty

Bases: oio.common.exceptions.OioException

exception oio.common.exceptions.ContentException

Bases: oio.common.exceptions.OioException

exception oio.common.exceptions.ContentNotFound

Bases: oio.common.exceptions.ContentException

exception oio.common.exceptions.CorruptedChunk

Bases: oio.common.exceptions.ChunkException

exception oio.common.exceptions.DeadlineReached

Bases: oio.common.exceptions.OioException

Special exception to be raised when a deadline is reached. This differs from the OioTimeout in that we are sure the operation won’t succeed silently in the background.

exception oio.common.exceptions.ECError

Bases: exceptions.Exception

exception oio.common.exceptions.EmptyByteRange

Bases: exceptions.Exception

exception oio.common.exceptions.EtagMismatch

Bases: oio.common.exceptions.OioException

exception oio.common.exceptions.ExplicitBury

Bases: oio.common.exceptions.OioException

exception oio.common.exceptions.FaultyChunk

Bases: oio.common.exceptions.ChunkException

Raised when a chunk misses some extended attributes, or they have invalid values.

exception oio.common.exceptions.FileNotFound

Bases: oio.common.exceptions.OioException

exception oio.common.exceptions.Forbidden(http_status=403, status=None, message=None)

Bases: oio.common.exceptions.ClientException

Operation is forbidden.

exception oio.common.exceptions.InvalidStorageMethod

Bases: oio.common.exceptions.OioException

exception oio.common.exceptions.LifecycleNotFound

Bases: oio.common.exceptions.OioException

exception oio.common.exceptions.Meta2Exception

Bases: oio.common.exceptions.OioException

exception oio.common.exceptions.MethodNotAllowed(http_status=405, status=None, message=None)

Bases: oio.common.exceptions.ClientException

Request method is not allowed. May be raised when the namespace is in WORM mode and user tries to delete.

exception oio.common.exceptions.MissingAttribute(attribute)

Bases: oio.common.exceptions.OioException

exception oio.common.exceptions.MissingContentLength

Bases: oio.common.exceptions.OioException

exception oio.common.exceptions.MissingData

Bases: oio.common.exceptions.OioException

exception oio.common.exceptions.MissingName

Bases: oio.common.exceptions.OioException

exception oio.common.exceptions.NoSuchAccount

Bases: oio.common.exceptions.OioException

exception oio.common.exceptions.NoSuchContainer

Bases: oio.common.exceptions.OioException

exception oio.common.exceptions.NoSuchObject

Bases: oio.common.exceptions.OioException

exception oio.common.exceptions.NoSuchReference

Bases: oio.common.exceptions.OioException

exception oio.common.exceptions.NotFound(http_status=404, status=None, message=None)

Bases: oio.common.exceptions.ClientException

Resource was not found.

exception oio.common.exceptions.OioException

Bases: exceptions.Exception

exception oio.common.exceptions.OioNetworkException

Bases: oio.common.exceptions.OioException

Network related exception (connection, timeout…).

exception oio.common.exceptions.OioTimeout

Bases: oio.common.exceptions.OioNetworkException

exception oio.common.exceptions.OrphanChunk

Bases: oio.common.exceptions.ChunkException

exception oio.common.exceptions.PreconditionFailed

Bases: oio.common.exceptions.OioException

exception oio.common.exceptions.RetryLater

Bases: oio.common.exceptions.OioException

Exception raised by workers that want a task to be rescheduled later.

exception oio.common.exceptions.ServerException

Bases: oio.common.exceptions.OioException

exception oio.common.exceptions.ServiceBusy(http_status=503, status=None, message=None)

Bases: oio.common.exceptions.ClientException

This kind of exceptions tell that the system was “busy” and could not handle the request at the moment. The user is invited to retry after a few seconds.

exception oio.common.exceptions.ServiceUnavailable

Bases: oio.common.exceptions.OioException

Exception raised when some services are temporarily not available. This does not mean data is lost.

exception oio.common.exceptions.SourceReadError

Bases: oio.common.exceptions.OioException

exception oio.common.exceptions.SourceReadTimeout

Bases: oio.common.exceptions.OioTimeout

Specialization of OioTimeout for the case when a timeout occurs while reading data from a client application.

exception oio.common.exceptions.SpareChunkException

Bases: oio.common.exceptions.Meta2Exception

Exception raised when no spare chunk has been found, or some may have been found but they don’t match all criteria.

exception oio.common.exceptions.StatusMessageException(http_status, status=None, message=None)

Bases: oio.common.exceptions.OioException

Error carrying an HTTP status, an OIO status and a message.

exception oio.common.exceptions.TooLarge(http_status=413, status=None, message=None)

Bases: oio.common.exceptions.ClientException

exception oio.common.exceptions.UnfinishedUploadException(exception, chunks_already_uploaded)

Bases: oio.common.exceptions.OioException

Exception raised when a number of chunks are not uploaded.

reraise()

exception oio.common.exceptions.UnrecoverableContent

Bases: oio.common.exceptions.ContentException

exception oio.common.exceptions.UnsatisfiableRange(http_status=416, status=None, message=None)

Bases: oio.common.exceptions.ClientException

exception oio.common.exceptions.VolumeException

Bases: oio.common.exceptions.OioException

Exception raised when someone is trying to contact a rdir service, but there is none assigned to the specified rawx.

oio.common.exceptions.from_response(resp, body=None)

oio.common.exceptions.from_status(status, reason='n/a')

oio.common.exceptions.reraise(exc_type, exc_value, extra_message=None)

Raise an exception of type exc_type with arguments of exc_value plus maybe extra_message at the beginning.

oio.common.fullpath module

oio.common.fullpath.decode_fullpath(fullpath)

oio.common.fullpath.decode_old_fullpath(fullpath)

oio.common.fullpath.encode_fullpath(account, container, path, version, content_id)

oio.common.fullpath.encode_old_fullpath(account, container, path, version)

oio.common.green module

exception oio.common.green.ChunkReadTimeout(seconds=None, exception=None)

Bases: oio.common.green.OioTimeout

msg_prefix = 'Chunk read '

exception oio.common.green.ChunkWriteTimeout(seconds=None, exception=None)

Bases: oio.common.green.OioTimeout

msg_prefix = 'Chunk write '

exception oio.common.green.ConnectionTimeout(seconds=None, exception=None)

Bases: oio.common.green.OioTimeout

msg_prefix = 'Connection '

class oio.common.green.ContextPool(size=1000)

Bases: eventlet.greenpool.GreenPool

exception oio.common.green.OioTimeout(seconds=None, exception=None)

Bases: eventlet.timeout.Timeout

Wrapper over eventlet.Timeout with better __str__.

msg_prefix = ''

exception oio.common.green.SourceReadTimeout(seconds=None, exception=None)

Bases: oio.common.green.OioTimeout

msg_prefix = 'Source read '

oio.common.green.eventlet_yield()

Swith to another eventlet coroutine.

oio.common.green.get_hub()

oio.common.green.ratelimit(run_time, max_rate, increment=1, rate_buffer=5)

oio.common.http module

class oio.common.http.CustomHttpConnection(*args, **kwargs)

Bases: oio.common.http_eventlet.CustomHttpConnection

class oio.common.http.HeadersDict(headers, **kwargs)

Bases: dict

get(k[, d]) → D[k] if k in D, else d. d defaults to None.

pop(k[, d]) → v, remove specified key and return the corresponding value.

If key is not found, d is returned if given, otherwise KeyError is raised

update([E, ]**F) → None. Update D from dict/iterable E and F.

If E present and has a .keys() method, does: for k in E: D[k] = E[k] If E present and lacks .keys() method, does: for (k, v) in E: D[k] = v In either case, this is followed by: for k in F: D[k] = F[k]

oio.common.http.headers_from_object_metadata(metadata)

Generate chunk PUT request headers from object metadata.

oio.common.http.http_header_from_ranges(ranges)

oio.common.http.parse_content_range(raw_content_range)

oio.common.http.parse_content_type(raw_content_type)

oio.common.http.ranges_from_http_header(val)

oio.common.http_eventlet module

class oio.common.http_eventlet.CustomHTTPResponse(sock, debuglevel=0, strict=0, method=None)

Bases: httplib.HTTPResponse

close()

force_close()

read(amount=None)

class oio.common.http_eventlet.CustomHttpConnection(host, port=None, strict=None, timeout=<object object>, source_address=None)

Bases: httplib.HTTPConnection

connect()

getresponse()

putrequest(method, url, skip_host=0, skip_accept_encoding=0)

response_class

alias of CustomHTTPResponse

set_cork(enabled=True)

Enable or disable TCP_CORK on the underlying socket.

oio.common.http_eventlet.http_connect(host, method, path, headers=None, query_string=None)

oio.common.http_urllib3 module

class oio.common.http_urllib3.SafePoolManager(num_pools=10, headers=None, **connection_pool_kw)

Bases: urllib3.poolmanager.PoolManager

urllib3.PoolManager wrapper that filters out keyword arguments not recognized by urllib3.

request(*args, **kwargs)

Filter out arguments that are not recognized by urllib3, then call urllib3.PoolManager.request.

oio.common.http_urllib3.get_pool_manager(pool_connections=32, pool_maxsize=32, max_retries=0, backoff_factor=0, **kwargs)

Get urllib3.PoolManager to manage pools of connections

Parameters:

• pool_connections (int) – number of connection pools
• pool_maxsize (int) – number of connections per connection pool
• max_retries (int) – number of retries per request
• backoff_factor (float) – backoff factor to apply between attempts after second try

oio.common.http_urllib3.oio_exception_from_httperror(exc, reqid=None, url=None)

Convert an HTTPError from urllib3 to an OioException, and re-raise it.

oio.common.json module

oio.common.logger module

class oio.common.logger.StreamToLogger(logger, log_type='STDOUT')

Bases: object

close()

flush()

write(value)

writelines(values)

oio.common.logger.get_logger(conf, name=None, verbose=False, fmt='%(process)d %(thread)X %(name)s %(levelname)s %(message)s')

oio.common.logger.redirect_stdio(logger)

Close stdio, redirect stdout and stderr.

Parameters:logger –

oio.common.redis_conn module

class oio.common.redis_conn.RedisConnection(host=None, sentinel_hosts=None, sentinel_name=None, **kwargs)

Bases: object

URL_QUERY_ARGUMENT_PARSERS = {'health_check_interval': <type 'int'>, 'max_connections': <type 'int'>, 'retry_on_timeout': <function true_value at 0x7f71f3f16cf8>, 'socket_connect_timeout': <type 'float'>, 'socket_keepalive': <function true_value at 0x7f71f3f16cf8>, 'socket_timeout': <type 'float'>}

acquire_lock_with_timeout(lockname, acquire_timeout=10, lock_timeout=10)

Acquire a lock :lockname:

conn

Retrieve Redis connection (normal or sentinel)

conn_slave

Retrieve Redis connection (normal or sentinel)

register_script(script)

Register a LUA script and return Script object.

release_lock(lockname, identifier)

Release a previously acquired Lock

oio.common.redis_conn.catch_service_errors(func)

Catch errors attributable to the Redis service and raise ServiceBusy instead.

Raises:ServiceBusy – in case of a Redis service error

oio.common.storage_functions module

oio.common.storage_functions.fetch_stream(*args, **kwargs)

oio.common.storage_functions.fetch_stream_ec(*args, **kwargs)

oio.common.storage_functions.get_meta_ranges(ranges, chunks)

Convert object ranges to metachunks ranges.

Returns:a list of dictionaries indexed by metachunk positions

oio.common.storage_functions.obj_range_to_meta_chunk_range(obj_start, obj_end, meta_sizes)

Convert a requested object range into a list of meta_chunk ranges.

Parameters:meta_sizes (iterable, sorted in ascendant metachunk order.) – size of all object metachunks. Must be sorted! Returns:a dict of tuples (meta_chunk_start, meta_chunk_end) with metachunk positions as keys.

• meta_chunk_start is the first byte of the meta chunk, or None if this is a suffix byte range
• meta_chunk_end is the last byte of the meta_chunk, or None if this is a prefix byte range

oio.common.storage_functions.wrand_choice_index(scores)

Choose an element from the scores sequence and return its index

oio.common.storage_method module

class oio.common.storage_method.BackblazeStorageMethod(name, account_id, bucket_name)

Bases: oio.common.storage_method.StorageMethod

account_id

bucket_name

classmethod build(params)

class oio.common.storage_method.ECStorageMethod(name, ec_segment_size, ec_type, ec_nb_data, ec_nb_parity)

Bases: oio.common.storage_method.StorageMethod

classmethod build(params)

ec_fragment_size

ec_nb_data

ec_nb_parity

ec_segment_size

ec_type

expected_chunks

min_chunks_to_read

quorum

class oio.common.storage_method.ReplicatedStorageMethod(name, nb_copy)

Bases: oio.common.storage_method.StorageMethod

classmethod build(params)

expected_chunks

min_chunks_to_read

nb_copy

quorum

class oio.common.storage_method.StorageMethod(name, ec=False, backblaze=False)

Bases: object

backblaze

ec

expected_chunks

min_chunks_to_read

name

quorum

class oio.common.storage_method.StorageMethodLoad

Bases: object

class oio.common.storage_method.StorageMethods(methods)

Bases: object

load(chunk_method, **kwargs)

oio.common.storage_method.guess_storage_method(url)

oio.common.storage_method.load_methods()

oio.common.storage_method.parse_chunk_method(chunk_method)

oio.common.timestamp module

class oio.common.timestamp.Timestamp(timestamp)

Bases: object

isoformat

normal

oio.common.utils module

class oio.common.utils.GeneratorIO(gen, sub_generator=True)

Bases: io.RawIOBase

Make a file-like object from a generator. gen is the generator to read. sub_generator is a boolean telling that the generator yields sequences of bytes instead of bytes.

read(size=None)

readable()

Return whether object was opened for reading.

If False, read() will raise IOError.

readinto(b)

class oio.common.utils.RingBuffer(size=1)

Bases: list

append(value)

L.append(object) – append object to end

size

Get the size of the ring buffer

oio.common.utils.cid_from_name(account, ref)

Compute a container ID from an account and a reference name.

oio.common.utils.deadline_to_timeout(deadline, check=False)

Convert a deadline (float seconds) to a timeout (float seconds)

oio.common.utils.depaginate(func, item_key=None, listing_key=None, marker_key=None, truncated_key=None, attempts=1, *args, **kwargs)

Yield items from the lists returned by the repetitive calls to func(*args, **kwargs). For each call (except the first), the marker is taken from the last element returned by the previous call (unless marker_key is provided). The listing stops after an empty listing is returned (unless truncated_key is provided).

Parameters:

• item_key – an accessor to the actual item that should be yielded, applied on each element of the listing
• listing_key – an accessor to the actual listing, applied on the result of func(*args, **kwargs)
• marker_key – an accessor to the next marker from the previous listing, applied on the result of func(*args, **kwargs)
• truncated_key – an accessor telling if the listing is truncated, applied on the result of func(*args, **kwargs)

oio.common.utils.drop_privileges(user)

oio.common.utils.fix_ranges(ranges, length)

oio.common.utils.getrandbits(k) → x. Generates a long int with k random bits.

oio.common.utils.group_chunk_errors(chunk_err_iter)

Group errors in a dictionary of lists. The keys are errors, the values are lists of chunk IDs.

oio.common.utils.monotonic_time()

Get the monotonic time as float seconds

oio.common.utils.paths_gen(volume_path)

Yield paths of all regular files under volume_path.

oio.common.utils.quote(value, safe='/')

oio.common.utils.request_id(prefix='')

Build a 128-bit request id string.

Parameters:prefix – optional prefix to the request id.

oio.common.utils.set_deadline_from_read_timeout(kwargs, force=False)

Compute a deadline from a read timeout, and set it in a keyword argument dictionary if there is none (or force is set).

oio.common.utils.set_fd_close_on_exec(fd)

oio.common.utils.set_fd_non_blocking(fd)

oio.common.utils.statfs(volume)

Parameters:volume – path to the mount point to get stats from. Returns:the free space ratio. Return type:float

oio.common.utils.timeout_to_deadline(timeout, now=None)

Convert a timeout (float seconds) to a deadline (float seconds).

oio.common.wsgi module

class oio.common.wsgi.Application(app, conf, logger_class=None)

Bases: gunicorn.app.base.BaseApplication

access_log_fmt = '%(bind0)s %(h)s:%({remote}p)s %(m)s %(s)s %(D)s %(B)s %(l)s %({x-oio-req-id}i)s %(U)s?%(q)s'

load()

load_config()

This method is used to load the configuration from one or several input(s). Custom Command line, configuration file. You have to override this method in your class.

class oio.common.wsgi.ServiceLogger(cfg)

Bases: gunicorn.glogging.Logger

access(resp, req, environ, request_time)

See http://httpd.apache.org/docs/2.0/logs.html#combined for format details

atoms(resp, req, environ, request_time)

Gets atoms for log formating.

class oio.common.wsgi.WerkzeugApp(url_map=None, logger=None)

Bases: object

dispatch_request(req)

wsgi_app(environ, start_response)

oio.common.wsgi.init_request_processor(conf_file, app_name, app_factory, *args, **kwargs)

oio.common.xattr module

oio.common.xattr.read_user_xattr(fd)

oio.common.xattr.set_fullpath_xattr(fd, new_fullpaths, remove_old_xattr=False, xattr_to_remove=None)

Insert new fullpath extended attributes, remove deprecated ones.

Parameters:

• new_fullpaths – dictionary of “fullpath” extended attributes that should be set on file. The key is the chunk ID (required to generate the attribute key), the value is the “fullpath”.
• remove_old_xattr (bool) – remove legacy attributes from file
• xattr_to_remove – list of extra extended attributes that should be removed from file

Module contents