oio package

Subpackages

Module contents

OpenIO SDS Python API.

Basic object storage example:

>>> from oio import ObjectStorageApi
>>> api = ObjectStorageApi(namespace="OPENIO")
>>> api.object_create("myaccount", "mycontainer", "/etc/magic")
([{u'url': u'http://127.0.0.1:6008/DEADBEEFCAFEBABE1EE7',
   u'score': 65,
   u'hash': '8de4989188593b0419d387099c9e9872',
   u'pos': '0',
   u'size': 113}],
 113,
 '8de4989188593b0419d387099c9e9872')
class oio.ObjectStorageApi(namespace, logger=None, **kwargs)[source]

Bases: object

The Object Storage API.

High level API that wraps AccountClient, ContainerClient and DirectoryClient classes.

Every method that takes a kwargs argument accepts the at least the following keywords:

  • headers: dict of extra headers to pass to the proxy
  • connection_timeout: float
  • read_timeout: float
  • write_timeout: float
EXTRA_KEYWORDS = ('chunk_checksum_algo', 'autocreate')
TIMEOUT_KEYS = ('connection_timeout', 'read_timeout', 'write_timeout')
account_create(**kwargs)[source]

Create an account.

Parameters:account (str) – name of the account to create
Returns:True if the account has been created
account_del_properties(account=None, *args, **kwargs)[source]
account_delete(account=None, *args, **kwargs)[source]

Delete an account.

Parameters:account (str) – name of the account to delete
account_flush(account=None, *args, **kwargs)[source]

Flush all containers of an account

Parameters:account (str) – name of the account to flush
account_list(**kwargs)[source]

List known accounts.

Notice that account creation is asynchronous, and an autocreated account may appear in the listing only after several seconds.

account_refresh(account=None, *args, **kwargs)[source]

Refresh counters of an account.

Parameters:account (str) – name of the account to refresh, or None to refresh all accounts (slow)
account_set_properties(account=None, *args, **kwargs)[source]
account_show(account=None, *args, **kwargs)[source]

Get information about an account.

account_update(account=None, *args, **kwargs)[source]
all_accounts_refresh(**kwargs)[source]
Deprecated:call account_refresh(None) instead
blob_client

A low-level client to rawx services.

Return type:oio.blob.client.BlobClient
container_create(**kwargs)[source]

Create a container.

Parameters:
  • account (str) – account in which to create the container
  • container (str) – name of the container
  • properties (dict) – properties to set on the container
Returns:

True if the container has been created, False if it already exists

container_create_many(**kwargs)[source]

Create Many containers

Parameters:
  • account (str) – account in which to create the containers
  • containers (list) – names of the containers
  • properties (dict) – properties to set on the containers
container_del_properties(account, container, *args, **kwargs)[source]

Delete properties of a container.

Parameters:
  • account (str) – name of the account
  • container (str) – name of the container to deal with
  • properties (list) – a list of property keys
container_delete(account, container, *args, **kwargs)[source]

Delete a container.

Parameters:
  • account (str) – account from which to delete the container
  • container (str) – name of the container
container_flush(account, container, *args, **kwargs)[source]

Flush a container

Parameters:
  • account (str) – account from which to delete the container
  • container (str) – name of the container
  • fast (bool) – flush container quickly, may put high pressure on the event system
container_get_properties(account, container, *args, **kwargs)[source]

Get information about a container (user and system properties).

Parameters:
  • account (str) – account in which the container is
  • container (str) – name of the container
  • propertiesignored
Returns:

a dict with “properties” and “system” entries, containing respectively a dict of user properties and a dict of system properties.

container_list(account=None, *args, **kwargs)[source]

Get the list of containers of an account.

Parameters:
  • account (str) – account from which to get the container list
  • limit (int) – maximum number of results to return
  • marker (str) – name of the container from where to start the listing
  • end_marker
  • prefix
  • delimiter
Returns:

the list of containers of an account

Return type:

list of items (list) with 5 fields: name, number of objects, number of bytes, 1 if the item is a prefix or 0 if the item is actually a container, and modification time.

container_purge(account, container, *args, **kwargs)[source]
container_refresh(account, container, *args, **kwargs)[source]
container_set_properties(account, container, *args, **kwargs)[source]

Set properties on a container.

Parameters:
  • account (str) – name of the account
  • container (str) – name of the container where to set properties
  • properties (dict) – a dictionary of properties
  • clear (bool) –
  • system – dictionary of system properties to set
container_show(account, container, *args, **kwargs)[source]

Get information about a container (user properties).

Parameters:
  • account (str) – account in which the container is
  • container (str) – name of the container
Returns:

a dict with “properties” containing a dict of user properties.

container_snapshot(account, container, *args, **kwargs)[source]

Take a snapshot of a container.

Create a separate database containing all information about the contents from the original database, but with copies of the chunks at the time of the snapshot. This new database is not replicated, and is frozen (you cannot write into it).

Pay attention to the fact that the source container is frozen during the snapshot capture. The capture may take some time, depending on the number of objects hosted by the container.

Parameters:
  • account (str) – account in which the source container is.
  • container (str) – name of the source container.
  • dst_account (str) – account in which the snapshot will be created.
  • dst_container (str) – name of the new container (i.e. the snapshot).
container_touch(account, container, *args, **kwargs)[source]

Trigger a notification about the container state.

Parameters:
  • account (str) – account from which to delete the container
  • container (str) – name of the container
container_update(account, container, metadata, clear=False, **kwargs)[source]
Deprecated:use container_set_properties
object_analyze(*args, **kwargs)[source]
Deprecated:use object_locate
object_create(account, container, *args, **kwargs)[source]

Create an object or append data to object in container of account with data taken from either data (str or generator) or file_or_path (path to a file or file-like object). The object will be named after obj_name if specified, or after the base name of file_or_path.

Parameters:
  • account (str) – name of the account where to create the object
  • container (str) – name of the container where to create the object
  • file_or_path (str or file-like object) – file-like object or path to a file from which to read object data
  • data (str or generator) – object data (if file_or_path is not set)
  • etag (str) – entity tag of the object
  • obj_name – name of the object to create. If not set, will use the base name of file_or_path.
  • mime_type (str) – MIME type of the object
  • properties (dict) – a dictionary of properties
  • policy (str) – name of the storage policy
  • key_file
  • append (bool) – if set, data will be append to existing object (or object will be created if unset)
  • autocreate – if set to false, autocreation of container will be

disabled :type autocreate: bool

Parameters:
  • perfdata – optional dict that will be filled with metrics of time spent to resolve the meta2 address, to do the meta2 requests, and to upload chunks to rawx services.
  • deadline (float seconds) – deadline for the request, in monotonic time (oio.common.utils.monotonic_time). This supersedes timeout or read_timeout keyword arguments.
Returns:

list of chunks, size and hash of what has been uploaded

object_del_properties(account, container, obj, *args, **kwargs)[source]

Delete some properties from an object.

Parameters:properties (list) – list of property keys to delete
Returns:True if the property has been deleted (or was missing)
object_delete(account, container, obj, *args, **kwargs)[source]

Delete an object from a container. If versioning is enabled and no version is specified, the object will be marked as deleted but not actually deleted.

Parameters:
  • account (str) – name of the account the object belongs to
  • container (str) – name of the container the object belongs to
  • obj – name of the object to delete
  • version – version of the object to delete
Returns:

True on success

object_delete_many(**kwargs)[source]

Delete several objects.

Parameters:objs – an iterable of object names (should not be a generator)
Returns:a list of tuples with the name of the object and a boolean telling if the object has been successfully deleted
Return type:list of tuple
object_drain(**kwargs)[source]

Remove all the chunks of a content, but keep all the metadata.

Parameters:
  • account (str) – name of the account where the object is present
  • container (str) – name of the container where the object is present
  • obj – name of the object to drain
object_fastcopy(*args, **kwargs)[source]
Deprecated:use object_link.
object_fetch(*args, **kwargs)[source]

Download an object.

Parameters:
  • account – name of the account in which the object is stored
  • container – name of the container in which the object is stored
  • obj – name of the object to fetch
  • version (str) – version of the object to fetch
  • ranges (list of tuple) – a list of object ranges to download
  • key_file – path to the file containing credentials
  • properties (bool) – should the request return object properties along with content description (True by default)
  • perfdata – optional dict that will be filled with metrics of time spent to resolve the meta2 address, to do the meta2 request, and the time-to-first-byte, as seen by this API.
Returns:

a dictionary of object metadata and a stream of object data

Return type:

tuple

object_get_properties(account, container, obj, *args, **kwargs)[source]

Get the description of an object along with its user properties.

Parameters:
  • account – name of the account in which the object is stored
  • container – name of the container in which the object is stored
  • obj – name of the object to query
Returns:

a dict describing the object

example
{‘hash’: ‘6BF60C17CC15EEA108024903B481738F’,

‘ctime’: ‘1481031763’, ‘deleted’: ‘False’, ‘properties’: {

u’projet’: u’OpenIO-SDS’},

‘length’: ‘43518’, ‘hash_method’: ‘md5’, ‘chunk_method’: ‘ec/algo=liberasurecode_rs_vand,k=6,m=3’, ‘version’: ‘1481031762951972’, ‘policy’: ‘EC’, ‘id’: ‘20BF2194FD420500CD4729AE0B5CBC07’, ‘mime_type’: ‘application/octet-stream’, ‘name’: ‘Makefile’}

object_head(account, container, obj, trust_level=0, **kwargs)[source]

Check for the presence of an object in a container.

Parameters:trust_level (int) – 0: do not check chunks; 1: check if there are enough chunks to read the object; 2: check if all chunks are present.

Make a shallow copy of an object. Works across accounts and across containers.

object_list(account, container, *args, **kwargs)[source]

Lists objects inside a container.

Parameters:
  • properties – if True, list object properties along with objects
  • versions – if True, list all versions of objects
  • deleted – if True, list also the deleted objects
Returns:

a dict which contains * ‘objects’: the list of object descriptions * ‘prefixes’: common prefixes (only if delimiter and prefix are set) * ‘properties’: a dict of container properties * ‘system’: a dict of system metadata * ‘truncated’: a bool telling if the listing was truncated * ‘next_marker’: a str to be used as marker to get the next

page of results (in case the listing was truncated)

object_locate(account, container, obj, *args, **kwargs)[source]

Get a description of the object along with the list of its chunks.

Parameters:
  • account – name of the account in which the object is stored
  • container – name of the container in which the object is stored
  • obj – name of the object to query
  • version – version of the object to query
  • chunk_info – if True, fetch additional information about chunks from rawx services (slow). The second element of the returned tuple becomes a generator (instead of a list).
  • properties (bool) – should the request return object properties along with content description
Returns:

a tuple with object metadata dict as first element and chunk list as second element

object_set_properties(account, container, obj, *args, **kwargs)[source]

Set properties on an object.

Parameters:
  • account – name of the account in which the object is stored
  • container – name of the container in which the object is stored
  • obj – name of the object to query
  • properties – dictionary of properties
object_show(account, container, obj, *args, **kwargs)[source]

Get the description of an object along with the dictionary of user-set properties.

Deprecated:prefer using object_get_properties, for consistency with container_get_properties.
object_touch(**kwargs)[source]

Trigger a notification about an object (as if it just had been created).

Parameters:
  • account (str) – name of the account where to create the object
  • container (str) – name of the container where to create the object
  • obj – name of the object to touch
object_truncate(account, container, obj, *args, **kwargs)[source]

Truncate object at specified size. Only shrink is supported. A download may occur if size is not on chunk boundaries.

Parameters:
  • account – name of the account in which the object is stored
  • container – name of the container in which the object is stored
  • obj – name of the object to query
  • version – version of the object to query
  • size – new size of object
object_update(account, container, obj, metadata, version=None, clear=False, **kwargs)[source]
Deprecated:use object_set_properties
proxy_client