oio package

Subpackages

Module contents

OpenIO SDS Python API.

Basic object storage example:

>>> from oio import ObjectStorageApi
>>> api = ObjectStorageApi(namespace="OPENIO")
>>> api.object_create("myaccount", "mycontainer", "/etc/magic")
([{u'url': u'http://127.0.0.1:6008/DEADBEEFCAFEBABE1EE7',
   u'score': 65,
   u'hash': '8de4989188593b0419d387099c9e9872',
   u'pos': '0',
   u'size': 113}],
 113,
 '8de4989188593b0419d387099c9e9872')

class oio.ObjectStorageApi(namespace, logger=None, perfdata=None, cache=None, **kwargs)[source]

Bases: object

The Object Storage API.

High level API that wraps AccountClient, ContainerClient and DirectoryClient classes.

Every method that takes a kwargs argument accepts the at least the following keywords:

headers: dict of extra headers to pass to the proxy

connection_timeout: float

read_timeout: float

write_timeout: float

EXTRA_KEYWORDS = ('chunk_checksum_algo', 'autocreate', 'chunk_buffer_min', 'chunk_buffer_max')

account_create(*args, **kwargs)[source]

Create an account.

Parameters:	account (str) – name of the account to create
Returns:	True if the account has been created

account_del_properties(account=None, *args, **kwargs)[source]

account_delete(account=None, *args, **kwargs)[source]

Delete an account.

Parameters:	account (str) – name of the account to delete

account_flush(account=None, *args, **kwargs)[source]

Flush all containers of an account

Parameters:	account (str) – name of the account to flush

account_get_properties(account=None, *args, **kwargs)[source]: Get information about an account, including account properties.

account_list(*args, **kwargs)[source]

List known accounts.

Notice that account creation is asynchronous, and an autocreated account may appear in the listing only after several seconds.

account_refresh(account=None, *args, **kwargs)[source]

Refresh counters of an account.

Parameters:	account (str) – name of the account to refresh, or None to refresh all accounts (slow)

account_set_properties(account=None, *args, **kwargs)[source]

account_show(account=None, *args, **kwargs)[source]: Get information about an account.

all_accounts_refresh(**kwargs)[source]

Deprecated:	call account_refresh(None) instead

blob_client

A low-level client to rawx services.

Return type:	oio.blob.client.BlobClient

container_create(*args, **kwargs)[source]

Create a container.

Parameters:	account (str) – account in which to create the container container (str) – name of the container properties (dict) – properties to set on the container
Returns:	True if the container has been created, False if it already exists

container_create_many(*args, **kwargs)[source]

Create Many containers

Parameters:	account (str) – account in which to create the containers containers (list) – names of the containers properties (dict) – properties to set on the containers

container_del_properties(account, container, *args, **kwargs)[source]

Delete properties of a container.

Parameters:	account (str) – name of the account container (str) – name of the container to deal with properties (list) – a list of property keys

container_delete(account, container, *args, **kwargs)[source]

Delete a container.

Parameters:	account (str) – account from which to delete the container container (str) – name of the container

container_flush(account, container, *args, **kwargs)[source]

Flush a container

Parameters:	account (str) – account from which to delete the container container (str) – name of the container fast (bool) – flush container quickly, may put high pressure on the event system

container_get_properties(account, container, *args, **kwargs)[source]

Get information about a container (user and system properties).

Parameters:	account (str) – account in which the container is container (str) – name of the container properties – ignored
Returns:	a dict with “properties” and “system” entries, containing respectively a dict of user properties and a dict of system properties.

container_list(account=None, *args, **kwargs)[source]

Get the list of containers of an account.

Parameters:	account (str) – account from which to get the container list limit (int) – maximum number of results to return marker (bool) – name of the container from where to start the listing end_marker – prefix – delimiter – s3_buckets_only (bool) – list only S3 buckets.
Returns:	the list of containers of an account
Return type:	list of items (list) with 5 fields: name, number of objects, number of bytes, 1 if the item is a prefix or 0 if the item is actually a container, and modification time.

container_purge(account, container, *args, **kwargs)[source]

container_refresh(account, container, *args, **kwargs)[source]: Reset statistics of the specified container, and trigger an event that will update them. If the container does not exist, remove it from account.

container_set_properties(account, container, *args, **kwargs)[source]

Set properties on a container.

Parameters:	account (str) – name of the account container (str) – name of the container where to set properties properties (dict) – a dictionary of properties clear (bool) – system – dictionary of system properties to set

container_show(account, container, *args, **kwargs)[source]

Get information about a container (user properties).

Parameters:	account (str) – account in which the container is container (str) – name of the container
Returns:	a dict with “properties” containing a dict of user properties.

container_snapshot(account, container, *args, **kwargs)[source]

Take a snapshot of a container.

Create a separate database containing all information about the contents from the original database, but with copies of the chunks at the time of the snapshot. This new database is not replicated, and is frozen (you cannot write into it).

Pay attention to the fact that the source container is frozen during the snapshot capture. The capture may take some time, depending on the number of objects hosted by the container.

Parameters:	account (str) – account in which the source container is. container (str) – name of the source container. dst_account (str) – account in which the snapshot will be created. dst_container (str) – name of the new container (i.e. the snapshot).

container_touch(account, container, *args, **kwargs)[source]

Trigger a notification about the container state.

Parameters:	account (str) – account from which to delete the container container (str) – name of the container

container_update(account, container, metadata, clear=False, **kwargs)[source]

Deprecated:	use container_set_properties

object_analyze(*args, **kwargs)[source]

Deprecated:	use object_locate

object_change_policy(account, container, obj, *args, **kwargs)[source]

Change the storage policy of an object

Parameters:	account (str) – name of the account where to change the policy of the object container (str) – name of the container where to change the policy of the object obj (str) – name of the object to change the policy policy (str) – name of the new storage policy
Returns:	list of chunks, size, hash and metadata of object

object_create(account, container, *args, **kwargs)[source]

See documentation of object_create_ext for parameters

Returns:	list of chunks, size and hash of what has been uploaded

object_create_ext(account, container, *args, **kwargs)[source]

Create an object or append data to object in container of account with data taken from either data (str or generator) or file_or_path (path to a file or file-like object). The object will be named after obj_name if specified, or after the base name of file_or_path.

Parameters:

account (str) – name of the account where to create the object
container (str) – name of the container where to create the object
file_or_path (str or file-like object) – file-like object or path to a file from which to read object data
data (str or generator) – object data (if file_or_path is not set)
etag (str) – entity tag of the object
obj_name – name of the object to create. If not set, will use the base name of file_or_path.
mime_type (str) – MIME type of the object
properties (dict) – a dictionary of properties
policy (str) – name of the storage policy
key_file –
append (bool) – if set, data will be append to existing object (or object will be created if unset)
autocreate – if set to false, autocreation of container will be

disabled :type autocreate: bool

Parameters:	perfdata – optional dict that will be filled with metrics of time spent to resolve the meta2 address, to do the meta2 requests, and to upload chunks to rawx services. deadline (float seconds) – deadline for the request, in monotonic time (oio.common.utils.monotonic_time). This supersedes timeout or read_timeout keyword arguments.
Returns:	list of chunks, size, hash and metadata of what has been uploaded

object_del_properties(account, container, obj, *args, **kwargs)[source]

Delete some properties from an object.

Parameters:	properties (list) – list of property keys to delete
Returns:	True if the property has been deleted (or was missing)

object_delete(account, container, obj, *args, **kwargs)[source]

Delete an object from a container. If versioning is enabled and no version is specified, the object will be marked as deleted but not actually deleted.

Parameters:	account (str) – name of the account the object belongs to container (str) – name of the container the object belongs to obj – name of the object to delete version – version of the object to delete
Returns:	True on success

object_delete_many(*args, **kwargs)[source]

Delete several objects.

Parameters:	objs – an iterable of object names (should not be a generator)
Returns:	a list of tuples with the name of the object and a boolean telling if the object has been successfully deleted
Return type:	list of tuple

object_drain(*args, **kwargs)[source]

Remove all the chunks of a content, but keep all the metadata.

Parameters:	account (str) – name of the account where the object is present container (str) – name of the container where the object is present obj – name of the object to drain

object_fastcopy(*args, **kwargs)[source]

Deprecated:	use object_link.

object_fetch(*args, **kwargs)[source]

Download an object.

Parameters:	account – name of the account in which the object is stored container – name of the container in which the object is stored obj – name of the object to fetch version (str) – version of the object to fetch ranges (list of tuple) – a list of object ranges to download key_file – path to the file containing credentials properties (bool) – should the request return object properties along with content description (True by default) perfdata – optional dict that will be filled with metrics of time spent to resolve the meta2 address, to do the meta2 request, and the time-to-first-byte, as seen by this API.
Returns:	a dictionary of object metadata and a stream of object data
Return type:	tuple

object_get_properties(account, container, obj, *args, **kwargs)[source]

Get the description of an object along with its user properties.

Parameters:	account – name of the account in which the object is stored container – name of the container in which the object is stored obj – name of the object to query
Returns:	a dict describing the object

example

{‘hash’: ‘6BF60C17CC15EEA108024903B481738F’,

‘ctime’: ‘1481031763’, ‘deleted’: ‘False’, ‘properties’: {

u’projet’: u’OpenIO-SDS’},

‘length’: ‘43518’, ‘hash_method’: ‘md5’, ‘chunk_method’: ‘ec/algo=liberasurecode_rs_vand,k=6,m=3’, ‘version’: ‘1481031762951972’, ‘policy’: ‘EC’, ‘id’: ‘20BF2194FD420500CD4729AE0B5CBC07’, ‘mime_type’: ‘application/octet-stream’, ‘name’: ‘Makefile’}

object_head(account, container, obj, trust_level=0, **kwargs)[source]

Check for the presence of an object in a container.

Parameters:	trust_level (int) – 0: do not check chunks; 1: check if there are enough chunks to read the object; 2: check if all chunks are present.

object_link(*args, **kwargs)[source]: Make a shallow copy of an object. Works across accounts and across containers.

object_list(account, container, *args, **kwargs)[source]

Lists objects inside a container.

Parameters:

properties – if True, list object properties along with objects
versions – if True, list all versions of objects
deleted – if True, list also the deleted objects

Returns:

a dict which contains * ‘objects’: the list of object descriptions * ‘prefixes’: common prefixes (only if delimiter and prefix are set) * ‘properties’: a dict of container properties * ‘system’: a dict of system metadata * ‘truncated’: a bool telling if the listing was truncated * ‘next_marker’: a str to be used as marker to get the next

page of results (in case the listing was truncated)

object_locate(account, container, obj, *args, **kwargs)[source]

Get a description of the object along with the list of its chunks.

Parameters:

account – name of the account in which the object is stored
container – name of the container in which the object is stored
obj – name of the object to query
version – version of the object to query
chunk_info – if True, fetch additional information about chunks from rawx services (slow). The second element of the returned tuple becomes a generator (instead of a list).
properties (bool) – should the request return object properties along with content description

Returns:

a tuple with object metadata dict as first element and chunk list as second element

object_set_properties(account, container, obj, *args, **kwargs)[source]

Set properties on an object.

Parameters:	account – name of the account in which the object is stored container – name of the container in which the object is stored obj – name of the object to query properties – dictionary of properties

object_show(account, container, obj, *args, **kwargs)[source]

Get the description of an object along with the dictionary of user-set properties.

Deprecated:	prefer using object_get_properties, for consistency with container_get_properties.

object_touch(*args, **kwargs)[source]

Trigger a notification about an object (as if it just had been created).

Parameters:	account (str) – name of the account where to touch the object container (str) – name of the container where to touch the object obj (str) – name of the object to touch

object_truncate(account, container, obj, *args, **kwargs)[source]

Truncate object at specified size. Only shrink is supported. A download may occur if size is not on chunk boundaries.

Parameters:	account – name of the account in which the object is stored container – name of the container in which the object is stored obj – name of the object to query version – version of the object to query size – new size of object

object_update(account, container, obj, metadata, version=None, clear=False, **kwargs)[source]

Deprecated:	use object_set_properties

proxy_client