oio.container package

Submodules

oio.container.app module

oio.container.app.create_app(conf=None)

oio.container.backup module

class oio.container.backup.ContainerBackup(conf)

Bases: oio.common.redis_conn.RedisConnection, oio.common.wsgi.WerkzeugApp

WSGI Application to dump or restore a container.

BLOCK_ALIGNMENT = 2048

REDIS_TIMEOUT = 86400

STREAMING = 52428800

generate_manifest()

on_dump(req)

Entry point for dump rule

on_restore(req)

Entry point for restore rule

redis

Redis connection object

class oio.container.backup.ContainerRestore(redis, proxy, logger)

Bases: object

MODE_FULL = 1

MODE_RANGE = 2

extract_tar_entry()

parse_reg_type(hdrs, account, container)

parse_xhd_type(hdrs)

enrich hdrs with new headers

prepare(account, container)

read(size)

restore(request, account, container)

Manage PUT method for restoring a container

class oio.container.backup.ContainerTarFile(storage_api, account, container, range_, oio_map, logger)

Bases: object

Expose a File Object API to be used with wrap_file

close()

create_tar_oio_properties(entry, range_, name)

Extract data from fake object for :name:

CONTAINER_PROPERTIES: contains properties of container CONTAINER_MANIFEST: map of object in Tar

create_tar_oio_stream(entry, range_)

Extract data from entry from object

next()

read(size=-1)

Stream TAR content each call will send object by object, or by chunk of size if object is too large

class oio.container.backup.LimitedStream(stream, size, entry=None, offset=0)

Bases: object

Wrap a stream to read no more than size bytes from input stream. Also verify checksums.

read(size=-1)

class oio.container.backup.OioTarEntry(conn, account, container, name, data=None)

Bases: object

buf

checksums

compute(conn, data=None)

data_blocks

Number of tar blocks required to store the entry data

filesize

header_blocks

Number of tar blocks required to store the entry header

slo

oio.container.backup.redis_cnx(fct)

oio.container.client module

class oio.container.client.ContainerClient(conf, **kwargs)

Bases: oio.common.client.ProxyClient

Intermediate level class to manage containers.

container_create(account, reference, properties=None, system=None, **kwargs)

Create a container.

Parameters:

• account (str) – account in which to create the container
• reference (str) – name of the container
• properties (dict) – properties to set on the container
• system (dict) – system properties to set on the container
• headers (dict) – extra headers to send to the proxy

Returns:

True if the container has been created, False if it already exists

container_create_many(account, containers, properties=None, **kwargs)

Create several containers.

Parameters:

• account (str) – account in which to create the containers
• containers (iterable of str) – names of the containers
• properties (dict) – properties to set on the containers
• headers (dict) – extra headers to send to the proxy

Returns:

a list of tuples with the name of the container and a boolean telling if the container has been created

Return type:

list of tuple

container_dedup(account=None, reference=None, cid=None, **kwargs)

container_del_properties(account=None, reference=None, properties=[], cid=None, **kwargs)

container_delete(account=None, reference=None, cid=None, **kwargs)

Delete a container.

Parameters:

• account (str) – account from which to delete the container
• reference (str) – name of the container
• cid (str) – container id that can be used instead of account and reference
• headers (dict) – extra headers to send to the proxy

container_enable(account=None, reference=None, cid=None, **kwargs)

Change the status of a container database to enable

Parameters:

• account (str) – account in which the container is
• reference (str) – name of the container
• cid – container id that can be used instead of account and reference

container_flush(account=None, reference=None, cid=None, **kwargs)

container_freeze(account=None, reference=None, cid=None, **kwargs)

Freeze the database of a container

Parameters:

• account (str) – account in which the container is
• reference (name of the container) – name of the container
• cid – container id that can be used instead of account and reference

container_get_properties(account=None, reference=None, path=None, cid=None, **kwargs)

Get information about a container (user and system properties).

Parameters:

• account (str) – account in which the container is
• reference (str) – name of the container
• cid (str) – container id that can be used instead of account and reference
• headers (dict) – extra headers to send to the proxy

Returns:

a dict with “properties” and “system” entries, containing respectively a dict of user properties and a dict of system properties.

container_purge(account=None, reference=None, cid=None, maxvers=None, **kwargs)

container_raw_delete(account=None, reference=None, data=None, cid=None, **kwargs)

Delete raw ‘beans’ from a container.

Parameters:data (list of dict items) – dictionaries representing the beans to delete. They must have a key for each column of the meta2 database, plus a ‘type’ telling which type of bean it is.

container_raw_insert(bean, account=None, reference=None, cid=None, **kwargs)

container_raw_update(old, new, account=None, reference=None, cid=None, **kwargs)

container_set_properties(account=None, reference=None, properties=None, clear=False, cid=None, system=None, **kwargs)

container_show(account=None, reference=None, cid=None, **kwargs)

Get information about a container (like user properties).

Parameters:

• account (str) – account in which the container is
• reference (str) – name of the container
• cid (str) – container id that can be used instead of account and reference
• headers (dict) – extra headers to send to the proxy

Returns:

a dict with “properties” containing a dict of user properties.

Deprecated:

use container_get_properties instead

container_snapshot(account=None, reference=None, dst_account=None, dst_reference=None, cid=None, **kwargs)

Create a snapshot of a the container.

This function duplicates only the database. It doesn’t duplicate the chunks of the contents.

Parameters:

• account (str) – account in which the container is
• reference (str) – name of the container
• cid (str) – container id that can be used instead of account and reference
• dst_account (str) – account in which the snapshot will be created
• dst_reference (str) – name of the snapshot

container_touch(account=None, reference=None, cid=None, recompute=False, damaged_objects=None, missing_chunks=None, **kwargs)

content_create(**kwargs)

Create a new object. This method does not upload any data, it just registers object metadata in the database.

Parameters:

• size (int) – size of the object
• checksum (hexadecimal str) – checksum of the object (may be None when appending)
• data – metadata of the object (list of chunks and

dict of properties) :type data: dict :param cid: container id that can be used in place of account

and reference

Parameters:content_id – the ID to set on the object, or the ID of the

existing object when appending :param stgpol: name of the storage policy for the object :param version: version of the object :type version: int :param mime_type: MIME type to set on the object :param chunk_method: :param headers: extra headers to send to the proxy :param append: append to an existing object instead of creating it :type append: bool :param change_policy: change policy of an existing object :type change_policy: bool

content_del_properties(account=None, reference=None, path=None, properties=[], cid=None, version=None, **kwargs)

Delete some properties from an object.

Parameters:properties (list) – list of property keys to delete Returns:True is the property has been deleted

content_delete(account=None, reference=None, path=None, cid=None, version=None, **kwargs)

Delete one object.

Returns:True if the object has been deleted

content_delete_many(account=None, reference=None, paths=None, cid=None, **kwargs)

Delete several objects.

Parameters:paths – an iterable of object paths (should not be a generator) Returns:a list of tuples with the path of the content and a boolean telling if the content has been deleted Return type:list of tuple

content_drain(account=None, reference=None, path=None, cid=None, version=None, **kwargs)

content_get_properties(account=None, reference=None, path=None, cid=None, **kwargs)

Get a description of the content along with its user properties.

content_list(account=None, reference=None, path=None, cid=None, **kwargs)

Get the list of contents of a container.

Returns:a tuple with container metadata dict as first element and a dict with “object” and “prefixes” as second element

content_locate(account=None, reference=None, path=None, cid=None, **kwargs)

Get a description of the content along with the list of its chunks.

Parameters:

• cid (hexadecimal str) – container id that can be used in place of account and reference
• content (hexadecimal str) – content id that can be used in place of path
• properties (bool) – should the request return object properties along with content description

Returns:

a tuple with content metadata dict as first element and chunk list as second element

content_prepare(account=None, reference=None, path=None, cid=None, **kwargs)

Prepare an upload: get URLs of chunks on available rawx.

Parameters:autocreate – create container if it doesn’t exist

content_purge(account=None, reference=None, path=None, cid=None, maxvers=None, **kwargs)

content_set_properties(account=None, reference=None, path=None, properties={}, cid=None, version=None, clear=False, **kwargs)

Set properties on an object.

Parameters:properties – dictionary of properties

content_spare(account=None, reference=None, path=None, cid=None, **kwargs)

content_touch(account=None, reference=None, path=None, cid=None, version=None, **kwargs)

content_truncate(account=None, reference=None, path=None, cid=None, version=None, size=0, **kwargs)

oio.container.client.extract_chunk_qualities(properties, raw=False)

Extract chunk quality information from a dictionary (or a list) of properties.

Parameters:

• properties – properties object.
• raw – False if properties is a dictionary, True if properties is a list of “raw” properties.

oio.container.client.extract_content_headers_meta(headers)

oio.container.client.extract_reference_params(func)

oio.container.lifecycle module

class oio.container.lifecycle.ContainerLifecycle(api, account, container, logger=None, recursive=False)

Bases: object

apply(obj_meta, **kwargs)

Match then apply the set of rules of this lifecycle configuration on the specified object.

Returns:tuples of (object metadata, rule name, action, status) Return type:generator of 4-tuples Notice:you must consume the results or the rules won’t be applied.

execute(use_precessed_versions=True, **kwargs)

Match then apply the set of rules of the lifecycle configuration on all objects of the container (and its relatives, if recursive mode is enabled).

Returns:tuples of (object metadata, rule name, action, status) Return type:generator of 4-tuples Notice:the results must be consumed or the rules won’t be applied.

get_configuration()

Get lifecycle configuration from container property.

is_current_version(obj_meta, **kwargs)

Check if the object is the current version

load()

Load lifecycle rules from container property.

Returns:True if a lifecycle configuration has been loaded

load_xml(xml_str)

Load lifecycle rules from LifecycleConfiguration XML document.

process_container(container, **kwargs)

Match then apply the set of rules of the lifecycle configuration on all objects of the container.

Returns:tuples of (object metadata, rule name, action, status) Return type:generator of 4-tuples Notice:the results must be consumed or the rules won’t be applied.

save(xml_str=None)

Save the lifecycle configuration in container property.

Parameters:xml_str – the configuration to save, or None to save the

configuration that has been loaded previously :type xml_str: str

class oio.container.lifecycle.DateActionFilter(date, **kwargs)

Bases: oio.container.lifecycle.LifecycleActionFilter

Specify the date when the specific rule action takes effect.

classmethod from_element(date_elt, **kwargs)

match(obj_meta, now=None, **kwargs)

Check if an object matches the conditions.

class oio.container.lifecycle.DaysActionFilter(days, **kwargs)

Bases: oio.container.lifecycle.LifecycleActionFilter

Specify the number of days after object creation when the specific rule action takes effect.

classmethod from_element(days_elt, **kwargs)

match(obj_meta, now=None, **kwargs)

Check if an object matches the conditions.

class oio.container.lifecycle.Expiration(filter_, **kwargs)

Bases: oio.container.lifecycle.LifecycleAction

Delete objects.

apply(obj_meta, version=None, **kwargs)

Match then apply the treatment on the object.

classmethod from_element(expiration_elt, **kwargs)

Load the expiration from an XML element

class oio.container.lifecycle.LifecycleAction(filter_, **kwargs)

Bases: oio.container.lifecycle.LifecycleActionFilter

Interface for Lifecycle actions.

Apply the action on the latest version.

apply(obj_meta, **kwargs)

Match then apply the treatment on the object.

match(obj_meta, **kwargs)

Check if an object matches the conditions.

class oio.container.lifecycle.LifecycleActionFilter(lifecycle=None, **kwargs)

Bases: object

Specify conditions when the specific rule action takes effect.

match(obj_meta, now=None, **kwargs)

Check if an object matches the conditions.

class oio.container.lifecycle.LifecycleRule(id_, filter_, enabled, actions)

Bases: object

Combination of a filter and a set of lifecycle actions.

apply(obj_meta, **kwargs)

Apply the set of actions of this rule.

Returns:the list of actions that have been applied Return type:list of tuple of a class and a bool or a class and an exception instance

classmethod from_element(rule_elt, **kwargs)

Load the rule from an XML element.

match(obj_meta, **kwargs)

Check if the specified object passes the filter of this rule.

class oio.container.lifecycle.LifecycleRuleFilter(prefix, tags)

Bases: object

Filter to determine on which objects to apply a lifecycle rule.

classmethod from_element(filter_elt, **kwargs)

Load the filter from an XML element.

match(obj_meta, **kwargs)

Check if an object matches the conditions defined by this filter.

class oio.container.lifecycle.NoncurrentCountActionFilter(count, **kwargs)

Bases: oio.container.lifecycle.LifecycleActionFilter

classmethod from_element(count_elt, **kwargs)

match(obj_meta, **kwargs)

Check if an object matches the conditions.

class oio.container.lifecycle.NoncurrentDaysActionFilter(days, **kwargs)

Bases: oio.container.lifecycle.DaysActionFilter

class oio.container.lifecycle.NoncurrentVersionExpiration(filter_, **kwargs)

Bases: oio.container.lifecycle.Expiration

Delete objects old versions.

apply(obj_meta, **kwargs)

Match then apply the treatment on the object.

classmethod from_element(expiration_elt, **kwargs)

Load the expiration from an XML element

match(obj_meta, **kwargs)

Check if an object matches the conditions.

class oio.container.lifecycle.NoncurrentVersionTransition(filter_, policy, **kwargs)

Bases: oio.container.lifecycle.Transition

Change object storage policy for old versions of the object only.

classmethod from_element(transition_elt, **kwargs)

Load the transition from an XML element

match(obj_meta, **kwargs)

Check if an object matches the conditions.

class oio.container.lifecycle.ProcessedVersions(**kwargs)

Bases: object

Save the processed versions of the last object

is_already_processed(obj_meta, **kwargs)

Check if the version of this object is already processed.

is_current(obj_meta, **kwargs)

Check if the object is the current version.

nb_processed(obj_meta, **kwargs)

Get the number of processed versions.

save_object(obj_meta, **kwargs)

Save object as processed.

class oio.container.lifecycle.Transition(filter_, policy, **kwargs)

Bases: oio.container.lifecycle.LifecycleAction

Change object storage policy.

STORAGE_POLICY_XML_TAG = 'StorageClass'

apply(obj_meta, **kwargs)

Match then apply the treatment on the object.

classmethod from_element(transition_elt, **kwargs)

Load the transition from an XML element

oio.container.lifecycle.int_to_iso8601(when)

oio.container.lifecycle.iso8601_to_int(when)

oio.container.md5py module

A sample implementation of MD5 in pure Python.

This is an implementation of the MD5 hash function, as specified by RFC 1321, in pure Python. It was implemented using Bruce Schneier’s excellent book “Applied Cryptography”, 2nd ed., 1996.

Surely this is not meant to compete with the existing implementation of the Python standard library (written in C). Rather, it should be seen as a Python complement that is more readable than C and can be used more conveniently for learning and experimenting purposes in the field of cryptography.

This module tries very hard to follow the API of the existing Python standard library’s “md5” module, but although it seems to work fine, it has not been extensively tested! (But note that there is a test module, test_md5py.py, that compares this Python implementation with the C one of the Python standard library.

BEWARE: this comes with no guarantee whatsoever about fitness and/or other properties! Specifically, do not use this in any production code! License is Python License!

Special thanks to Aurelian Coman who fixed some nasty bugs!

Copyright Dinu C. Gherman Copyright OpenIO SAS 2019

class oio.container.md5py.MD5

An implementation of the MD5 hash function in pure Python.

copy()

Return a clone object.

Return a copy (‘clone’) of the md5 object. This can be used to efficiently compute the digests of strings that share a common initial substring.

digest()

Terminate the message-digest computation and return digest.

Return the digest of the strings passed to the update() method so far. This is a 16-byte string which may contain non-ASCII characters, including null bytes.

hexdigest()

Terminate and return digest in HEX form.

Like digest() except the digest is returned as a string of length 32, containing only hexadecimal digits. This may be used to exchange the value safely in email or other non- binary environments.

init()

Initialize the message-digest and set all fields to zero.

update(inBuf)

Add to the current message.

Update the md5 object with the string arg. Repeated calls are equivalent to a single call with the concatenation of all the arguments, i.e. m.update(a); m.update(b) is equivalent to m.update(a+b).

oio.container.md5py.XX(func, a, b, c, d, x, s, ac)

Wrapper for call distribution to functions F, G, H and I.

This replaces functions FF, GG, HH and II from “Appl. Crypto. Rotation is separate from addition to prevent recomputation (now summed-up in one function).

oio.container.md5py.fF(x, y, z)

oio.container.md5py.fG(x, y, z)

oio.container.md5py.fH(x, y, z)

oio.container.md5py.fI(x, y, z)

oio.container.md5py.md5(arg=None)

Same as new().

For backward compatibility reasons, this is an alternative name for the new() function.

oio.container.md5py.new(arg=None)

Return a new md5 object.

If arg is present, the method call update(arg) is made.

oio.container.repairer module

class oio.container.repairer.ContainerRepairer(conf, containers=None, **kwargs)

Bases: oio.common.tool.Tool

Repair containers.

DEFAULT_REBUILD_BASES = True

DEFAULT_SYNC_BASES = True

DEFAULT_UPDATE_ACCOUNT = True

create_worker(queue_workers, queue_reply)

Create worker to process the items.

static string_from_item(item)

class oio.container.repairer.ContainerRepairerWorker(tool, queue_workers, queue_reply)

Bases: oio.common.tool.ToolWorker

Module contents