oio.common package

Submodules

oio.common.autocontainer module

class oio.common.autocontainer.AutocontainerBuilder(offset=0, size=None, mask=18446744073692774655L, base=16, con_format='%016X', **_kwargs)[source]

Bases: oio.common.autocontainer.ContainerBuilder

Build a container name from the integer conversion of a user provided path and a clever mask.

path is expected to be something like “video/ABC/DEF/xxxxxxxxxFEDCBAxxxxxxxxxx_nomdufichier”

verify(name)[source]

Verify that name is an autocontainer

class oio.common.autocontainer.ContainerBuilder(**_kwargs)[source]

Bases: object

Base class for container name builders.

alternatives(path)[source]

Generate all alternatives for the provided content path.

verify(name)[source]

Verify that name is an autocontainer

class oio.common.autocontainer.HashedContainerBuilder(offset=0, size=None, bits=15, **_kwargs)[source]

Bases: oio.common.autocontainer.ContainerBuilder

Build a container name from a SHA256 of the content path. Only the first (most significant) bits will be considered to generate the final prefix.

verify(name)[source]

Verify that name is an autocontainer

exception oio.common.autocontainer.NoMatchFound[source]

Bases: exceptions.ValueError

Exception raised when none of the configured patterns match the input object name.

class oio.common.autocontainer.RegexContainerBuilder(patterns, builder=<class 'oio.common.autocontainer.ContainerBuilder'>, **kwargs)[source]

Bases: object

Build a container name from a regular expression applied on a user provided path. Use a concatenation of all matching groups as the container name if no custom builder provided.

Parameters:patterns (str or iterable of str) – regular expressions with at least one capture group
alternatives(path)[source]

Generate all alternatives for the provided path, in case it matches several patterns.

verify(name)[source]
oio.common.autocontainer.strtoll(val, base=10)[source]

Mimics libc’s strtoll function

oio.common.client module

class oio.common.client.ProxyClient(conf, pool_manager=None, request_prefix='', no_ns_in_url=False, endpoint=None, request_attempts=1, logger=None, **kwargs)[source]

Bases: oio.api.base.HttpApi

Client directed towards oio-proxy, with logging facility

oio.common.configuration module

exception oio.common.configuration.InvalidServiceConfigError[source]

Bases: exceptions.ValueError

oio.common.configuration.load_namespace_conf(namespace)[source]
oio.common.configuration.parse_config(conf_path)[source]
oio.common.configuration.parse_options(parser=None)[source]
oio.common.configuration.read_conf(conf_path, section_name=None, defaults=None, use_yaml=False)[source]
oio.common.configuration.validate_service_conf(conf)[source]

oio.common.constants module

oio.common.daemon module

class oio.common.daemon.Daemon(conf)[source]

Bases: object

run(*args, **kwargs)[source]
start(**kwargs)[source]
stop()[source]
oio.common.daemon.run_daemon(klass, conf_file, section_name=None, **kwargs)[source]

oio.common.decorators module

oio.common.decorators.ensure_headers(func)[source]
oio.common.decorators.ensure_request_id(func)[source]
oio.common.decorators.handle_account_not_found(fnc)[source]
oio.common.decorators.handle_container_not_found(fnc)[source]
oio.common.decorators.handle_object_not_found(fnc)[source]

Catch oio.common.exceptions.NotFound exceptions and raise either oio.common.exceptions.NoSuchContainer or oio.common.exceptions.NoSuchObject respectively if the container is missing or the object is missing.

oio.common.easy_value module

oio.common.easy_value.boolean_value(value)[source]
oio.common.easy_value.convert_size(size, unit='')[source]
oio.common.easy_value.float_value(value, default)[source]
oio.common.easy_value.int_value(value, default)[source]
oio.common.easy_value.is_hexa(hexa, size=None)[source]
oio.common.easy_value.true_value(value)[source]

oio.common.exceptions module

exception oio.common.exceptions.ChunkException[source]

Bases: oio.common.exceptions.OioException

exception oio.common.exceptions.ClientException(http_status, status=None, message=None)[source]

Bases: oio.common.exceptions.OioException

exception oio.common.exceptions.CommandError[source]

Bases: exceptions.Exception

exception oio.common.exceptions.ConfigurationException[source]

Bases: oio.common.exceptions.OioException

exception oio.common.exceptions.Conflict(http_status=409, status=None, message=None)[source]

Bases: oio.common.exceptions.ClientException

exception oio.common.exceptions.ContainerNotEmpty[source]

Bases: oio.common.exceptions.OioException

exception oio.common.exceptions.ContentException[source]

Bases: oio.common.exceptions.OioException

exception oio.common.exceptions.ContentNotFound[source]

Bases: oio.common.exceptions.ContentException

exception oio.common.exceptions.CorruptedChunk[source]

Bases: oio.common.exceptions.ChunkException

exception oio.common.exceptions.DeadlineReached[source]

Bases: oio.common.exceptions.OioException

Special exception to be raised when a deadline is reached. This differs from the OioTimeout in that we are sure the operation won’t succeed silently in the background.

exception oio.common.exceptions.ECError[source]

Bases: exceptions.Exception

exception oio.common.exceptions.EmptyByteRange[source]

Bases: exceptions.Exception

exception oio.common.exceptions.EtagMismatch[source]

Bases: oio.common.exceptions.OioException

exception oio.common.exceptions.ExplicitBury[source]

Bases: oio.common.exceptions.OioException

exception oio.common.exceptions.FaultyChunk[source]

Bases: oio.common.exceptions.ChunkException

exception oio.common.exceptions.FileNotFound[source]

Bases: oio.common.exceptions.OioException

exception oio.common.exceptions.Forbidden(http_status=403, status=None, message=None)[source]

Bases: oio.common.exceptions.ClientException

Operation is forbidden.

exception oio.common.exceptions.InconsistentContent[source]

Bases: oio.common.exceptions.ContentException

exception oio.common.exceptions.InvalidStorageMethod[source]

Bases: oio.common.exceptions.OioException

exception oio.common.exceptions.Meta2Exception[source]

Bases: oio.common.exceptions.OioException

exception oio.common.exceptions.MethodNotAllowed(http_status=405, status=None, message=None)[source]

Bases: oio.common.exceptions.ClientException

Request method is not allowed. May be raised when the namespace is in WORM mode and user tries to delete.

exception oio.common.exceptions.MissingAttribute(attribute)[source]

Bases: oio.common.exceptions.OioException

exception oio.common.exceptions.MissingContentLength[source]

Bases: oio.common.exceptions.OioException

exception oio.common.exceptions.MissingData[source]

Bases: oio.common.exceptions.OioException

exception oio.common.exceptions.MissingName[source]

Bases: oio.common.exceptions.OioException

exception oio.common.exceptions.NoSuchAccount[source]

Bases: oio.common.exceptions.OioException

exception oio.common.exceptions.NoSuchContainer[source]

Bases: oio.common.exceptions.OioException

exception oio.common.exceptions.NoSuchObject[source]

Bases: oio.common.exceptions.OioException

exception oio.common.exceptions.NoSuchReference[source]

Bases: oio.common.exceptions.OioException

exception oio.common.exceptions.NotFound(http_status=404, status=None, message=None)[source]

Bases: oio.common.exceptions.ClientException

Resource was not found.

exception oio.common.exceptions.OioException[source]

Bases: exceptions.Exception

exception oio.common.exceptions.OioNetworkException[source]

Bases: oio.common.exceptions.OioException

Network related exception (connection, timeout…).

exception oio.common.exceptions.OioTimeout[source]

Bases: oio.common.exceptions.OioNetworkException

exception oio.common.exceptions.OrphanChunk[source]

Bases: oio.common.exceptions.ChunkException

exception oio.common.exceptions.PreconditionFailed[source]

Bases: oio.common.exceptions.OioException

exception oio.common.exceptions.ServerException[source]

Bases: oio.common.exceptions.OioException

exception oio.common.exceptions.ServiceBusy(http_status=503, status=None, message=None)[source]

Bases: oio.common.exceptions.ClientException

exception oio.common.exceptions.ServiceUnavailable[source]

Bases: oio.common.exceptions.OioException

exception oio.common.exceptions.SourceReadError[source]

Bases: oio.common.exceptions.OioException

exception oio.common.exceptions.SourceReadTimeout[source]

Bases: oio.common.exceptions.OioTimeout

Specialization of OioTimeout for the case when a timeout occurs while reading data from a client application.

exception oio.common.exceptions.SpareChunkException[source]

Bases: oio.common.exceptions.Meta2Exception

exception oio.common.exceptions.TooLarge(http_status=413, status=None, message=None)[source]

Bases: oio.common.exceptions.ClientException

exception oio.common.exceptions.UnrecoverableContent[source]

Bases: oio.common.exceptions.ContentException

exception oio.common.exceptions.UnsatisfiableRange(http_status=416, status=None, message=None)[source]

Bases: oio.common.exceptions.ClientException

exception oio.common.exceptions.VolumeException[source]

Bases: oio.common.exceptions.OioException

Exception raised when someone is trying to contact a rdir service, but there is none assigned to the specified rawx.

oio.common.exceptions.from_response(resp, body=None)[source]
oio.common.exceptions.from_status(status, reason='n/a')[source]
oio.common.exceptions.reraise(exc_type, exc_value, extra_message=None)[source]

Raise an exception of type exc_type with arguments of exc_value plus maybe extra_message at the beginning.

oio.common.fullpath module

oio.common.fullpath.decode_fullpath(fullpath)[source]
oio.common.fullpath.decode_old_fullpath(fullpath)[source]
oio.common.fullpath.encode_fullpath(account, container, path, version, content_id)[source]
oio.common.fullpath.encode_old_fullpath(account, container, path, version)[source]

oio.common.green module

exception oio.common.green.ChunkReadTimeout(seconds=None, exception=None)[source]

Bases: oio.common.green.OioTimeout

msg_prefix = 'Chunk read '
exception oio.common.green.ChunkWriteTimeout(seconds=None, exception=None)[source]

Bases: oio.common.green.OioTimeout

msg_prefix = 'Chunk write '
exception oio.common.green.ConnectionTimeout(seconds=None, exception=None)[source]

Bases: oio.common.green.OioTimeout

msg_prefix = 'Connection '
class oio.common.green.ContextPool(size=1000)[source]

Bases: eventlet.greenpool.GreenPool

exception oio.common.green.OioTimeout(seconds=None, exception=None)[source]

Bases: eventlet.timeout.Timeout

Wrapper over eventlet.Timeout with better __str__.

msg_prefix = ''
exception oio.common.green.SourceReadTimeout(seconds=None, exception=None)[source]

Bases: oio.common.green.OioTimeout

msg_prefix = 'Source read '
oio.common.green.get_hub()[source]
oio.common.green.ratelimit(run_time, max_rate, increment=1, rate_buffer=5)[source]

oio.common.http module

class oio.common.http.CustomHttpConnection(*args, **kwargs)[source]

Bases: oio.common.http_eventlet.CustomHttpConnection

class oio.common.http.HeadersDict(headers, **kwargs)[source]

Bases: dict

get(k[, d]) → D[k] if k in D, else d. d defaults to None.[source]
pop(k[, d]) → v, remove specified key and return the corresponding value.[source]

If key is not found, d is returned if given, otherwise KeyError is raised

update([E, ]**F) → None. Update D from dict/iterable E and F.[source]

If E present and has a .keys() method, does: for k in E: D[k] = E[k] If E present and lacks .keys() method, does: for (k, v) in E: D[k] = v In either case, this is followed by: for k in F: D[k] = F[k]

oio.common.http.headers_from_object_metadata(metadata)[source]

Generate chunk PUT request headers from object metadata.

oio.common.http.http_header_from_ranges(ranges)[source]
oio.common.http.parse_content_range(raw_content_range)[source]
oio.common.http.parse_content_type(raw_content_type)[source]
oio.common.http.ranges_from_http_header(val)[source]

oio.common.http_eventlet module

class oio.common.http_eventlet.CustomHTTPResponse(sock, debuglevel=0, strict=0, method=None)[source]

Bases: httplib.HTTPResponse

close()[source]
force_close()[source]
read(amount=None)[source]
class oio.common.http_eventlet.CustomHttpConnection(host, port=None, strict=None, timeout=<object object>, source_address=None)[source]

Bases: httplib.HTTPConnection

connect()[source]
getresponse()[source]
putrequest(method, url, skip_host=0, skip_accept_encoding=0)[source]
response_class

alias of CustomHTTPResponse

oio.common.http_eventlet.http_connect(host, method, path, headers=None, query_string=None)[source]

oio.common.http_urllib3 module

oio.common.http_urllib3.get_pool_manager(pool_connections=32, pool_maxsize=32, max_retries=0, backoff_factor=0)[source]

Get urllib3.PoolManager to manage pools of connections

Parameters:
  • pool_connections (int) – number of connection pools
  • pool_maxsize (int) – number of connections per connection pool
  • max_retries (int) – number of retries per request
  • backoff_factor (float) – backoff factor to apply between attemps after second try
oio.common.http_urllib3.oio_exception_from_httperror(exc, reqid=None, url=None)[source]

Convert an HTTPError from urllib3 to an OioException, and re-raise it.

oio.common.json module

oio.common.logger module

class oio.common.logger.StreamToLogger(logger, log_type='STDOUT')[source]

Bases: object

close()[source]
flush()[source]
write(value)[source]
writelines(values)[source]
oio.common.logger.get_logger(conf, name=None, verbose=False, fmt='%(process)d %(thread)X %(name)s %(levelname)s %(message)s')[source]
oio.common.logger.redirect_stdio(logger)[source]

Close stdio, redirect stdout and stderr.

Parameters:logger

oio.common.redis_conn module

class oio.common.redis_conn.RedisConn(conf, connection=None, **kwargs)[source]

Bases: object

acquire_lock_with_timeout(lockname, acquire_timeout=10, lock_timeout=10)[source]

Acquire a lock :lockname:

conn

Retrieve Redis connection (normal or sentinel)

register_script(script)[source]

Register a LUA script and return Script object.

release_lock(lockname, identifier)[source]

Release a previously acquired Lock

oio.common.storage_functions module

oio.common.storage_functions.fetch_stream(*args, **kwargs)[source]
oio.common.storage_functions.fetch_stream_ec(*args, **kwargs)[source]
oio.common.storage_functions.get_meta_ranges(ranges, chunks)[source]

Convert object ranges to metachunks ranges.

Returns:a list of dictionaries indexed by metachunk positions
oio.common.storage_functions.obj_range_to_meta_chunk_range(obj_start, obj_end, meta_sizes)[source]

Convert a requested object range into a list of meta_chunk ranges.

Parameters:meta_sizes (iterable, sorted in ascendant metachunk order.) – size of all object metachunks. Must be sorted!
Returns:a dict of tuples (meta_chunk_start, meta_chunk_end) with metachunk positions as keys.
  • meta_chunk_start is the first byte of the meta chunk, or None if this is a suffix byte range
  • meta_chunk_end is the last byte of the meta_chunk, or None if this is a prefix byte range
oio.common.storage_functions.wrand_choice_index(scores)[source]

Choose an element from the scores sequence and return its index

oio.common.storage_method module

class oio.common.storage_method.BackblazeStorageMethod(name, account_id, bucket_name)[source]

Bases: oio.common.storage_method.StorageMethod

account_id
bucket_name
classmethod build(params)[source]
class oio.common.storage_method.ECStorageMethod(name, ec_segment_size, ec_type, ec_nb_data, ec_nb_parity)[source]

Bases: oio.common.storage_method.StorageMethod

classmethod build(params)[source]
ec_fragment_size
ec_nb_data
ec_nb_parity
ec_segment_size
ec_type
quorum
class oio.common.storage_method.ReplicatedStorageMethod(name, nb_copy)[source]

Bases: oio.common.storage_method.StorageMethod

classmethod build(params)[source]
nb_copy
quorum
class oio.common.storage_method.StorageMethod(name, ec=False, backblaze=False)[source]

Bases: object

backblaze
ec
name
class oio.common.storage_method.StorageMethodLoad[source]

Bases: object

class oio.common.storage_method.StorageMethods(methods)[source]

Bases: object

load(chunk_method, **kwargs)[source]
oio.common.storage_method.guess_storage_method(url)[source]
oio.common.storage_method.load_methods()[source]
oio.common.storage_method.parse_chunk_method(chunk_method)[source]

oio.common.timestamp module

class oio.common.timestamp.Timestamp(timestamp)[source]

Bases: object

isoformat
normal

oio.common.utils module

class oio.common.utils.GeneratorIO(gen, sub_generator=True)[source]

Bases: io.RawIOBase

Make a file-like object from a generator. gen is the generator to read. sub_generator is a boolean telling that the generator yields sequences of bytes instead of bytes.

read(size=None)[source]
readable()[source]

Return whether object was opened for reading.

If False, read() will raise IOError.

readinto(b)[source]
class oio.common.utils.RingBuffer(size=1)[source]

Bases: list

append(value)[source]

L.append(object) – append object to end

size

Get the size of the ring buffer

oio.common.utils.cid_from_name(account, ref)[source]
oio.common.utils.deadline_to_timeout(deadline, check=False)[source]

Convert a deadline (float seconds) to a timeout (float seconds)

oio.common.utils.depaginate(func, item_key=None, listing_key=None, marker_key=None, truncated_key=None, *args, **kwargs)[source]

Yield items from the lists returned by the repetitive calls to func(*args, **kwargs). For each call (except the first), the marker is taken from the last element returned by the previous call (unless marker_key is provided). The listing stops after an empty listing is returned (unless truncated_key is provided).

Parameters:
  • item_key – an accessor to the actual item that should be yielded, applied on each element of the listing
  • listing_key – an accessor to the actual listing, applied on the result of func(*args, **kwargs)
  • marker_key – an accessor to the next marker from the previous listing, applied on the result of func(*args, **kwargs)
  • truncated_key – an accessor telling if the listing is truncated, applied on the result of func(*args, **kwargs)
oio.common.utils.drop_privileges(user)[source]
oio.common.utils.fix_ranges(ranges, length)[source]
oio.common.utils.getrandbits(k) → x. Generates a long int with k random bits.
oio.common.utils.group_chunk_errors(chunk_err_iter)[source]

Group errors in a dictionary of lists. The keys are errors, the values are lists of chunk IDs.

oio.common.utils.monotonic_time()[source]

Get the monotonic time as float seconds

oio.common.utils.paths_gen(volume_path)[source]
oio.common.utils.quote(value, safe='/')[source]
oio.common.utils.request_id(prefix=None)[source]

Build a 128-bit request id string.

Parameters:prefix – optional prefix to the request id.
oio.common.utils.set_fd_close_on_exec(fd)[source]
oio.common.utils.set_fd_non_blocking(fd)[source]
oio.common.utils.statfs(volume)[source]
Parameters:volume – path to the mount point to get stats from.
Returns:the free space ratio.
Return type:float

oio.common.wsgi module

class oio.common.wsgi.Application(app, conf, logger_class=None)[source]

Bases: gunicorn.app.base.BaseApplication

access_log_fmt = '%(bind0)s %(h)s:%({remote}p)s %(m)s %(s)s %(D)s %(B)s %(l)s %({X-oio-req-id}i)s %(U)s?%(q)s'
load()[source]
load_config()[source]

This method is used to load the configuration from one or several input(s). Custom Command line, configuration file. You have to override this method in your class.

class oio.common.wsgi.ServiceLogger(cfg)[source]

Bases: gunicorn.glogging.Logger

access(resp, req, environ, request_time)[source]

See http://httpd.apache.org/docs/2.0/logs.html#combined for format details

atoms(resp, req, environ, request_time)[source]

Gets atoms for log formating.

class oio.common.wsgi.WerkzeugApp(url_map=None, logger=None)[source]

Bases: object

dispatch_request(req)[source]
wsgi_app(environ, start_response)[source]
oio.common.wsgi.init_request_processor(conf_file, app_name, app_factory, *args, **kwargs)[source]

oio.common.xattr module

oio.common.xattr.modify_xattr(fd, new_fullpaths, remove_old_xattr, xattr_to_remove)[source]
oio.common.xattr.read_user_xattr(fd)[source]

Module contents