oio.api package

Submodules

oio.api.backblaze module

class oio.api.backblaze.BackblazeChunkDownloadHandler(meta, chunks, offset, size, headers=None, backblaze_info=None)

Bases: object

get_stream()

class oio.api.backblaze.BackblazeChunkWriteHandler(sysmeta, meta_chunk, checksum, storage_method, backblaze_info)

Bases: object

stream(source)

class oio.api.backblaze.BackblazeDeleteHandler(meta, chunks, backblaze_info)

Bases: object

delete()

class oio.api.backblaze.BackblazeDownloadHandler(sysmeta, meta_chunks, backblaze_info, headers, range_start=None, range_end=None)

Bases: object

get_iter()

class oio.api.backblaze.BackblazeWriteHandler(source, sysmeta, chunk_prep, storage_method, backblaze_info, **kwargs)

Bases: oio.api.io.WriteHandler

stream()

Only works with files, for the moment, because we need a file size to known when to stop.

oio.api.backblaze_http module

class oio.api.backblaze_http.Backblaze(account_id, application_key, authorization_required=None, upload_required=None, upload_part=False)

Bases: object

BACKBLAZE_BASE_API_URL = 'https://api.backblazeb2.com'

BACKBLAZE_MAX_CHUNK_SIZE = 209715200

authorization_token

delete(bucket_name, metadata)

delete_by_path_name(file_id, file_name)

download(bucket_name, metadata, headers=None)

download_by_path_name(bucket_name, link, headers=None)

get_backblaze_infos(bucket_name)

get_file_number(bucket_name)

get_list_buckets()

get_list_file_names(bucket_name)

get_list_file_names_by_bucket_id(bucket_id)

get_size(bucket_name)

upload(bucket_name, meta, data, sha1=None)

upload_part(file_id, data, part_number, sha1=None)

upload_part_begin(bucket_name, meta)

upload_part_end(file_id, sha1_array)

exception oio.api.backblaze_http.BackblazeException(status_code, message, response, headers_send)

Bases: exceptions.Exception

headers_received

headers_send

message

status_code

class oio.api.backblaze_http.BackblazeUtils

Bases: object

b2_authorization_list = {}

static get_credentials(storage_method, application_key_path=None, renew=False)

class oio.api.backblaze_http.Requests(error_handler=None)

Bases: object

get_response_from_request(content_type, url, headers=None, file_descriptor=None, json=False)

oio.api.base module

class oio.api.base.HttpApi(endpoint=None, pool_manager=None, connection='keep-alive', service_type='unknown', **kwargs)

Bases: object

Provides facilities to make HTTP requests towards the same endpoint, with a pool of connections.

oio.api.ec module

class oio.api.ec.ECChunkDownloadHandler(storage_method, chunks, meta_start, meta_end, headers, connection_timeout=None, read_timeout=None, reqid=None, perfdata=None, **_kwargs)

Bases: object

Handles the download of an EC meta chunk

get_stream()

class oio.api.ec.ECRebuildHandler(meta_chunk, missing, storage_method, connection_timeout=None, read_timeout=None, **_kwargs)

Bases: object

rebuild()

class oio.api.ec.ECStream(storage_method, readers, range_infos, meta_length, fragment_length, reqid=None, perfdata=None, logger=None)

Bases: object

Reads an EC meta chunk.

Handles the different readers.

close()

get_iter()

start()

class oio.api.ec.ECWriteHandler(source, sysmeta, chunk_preparer, storage_method, headers=None, **kwargs)

Bases: oio.api.io.WriteHandler

Handles writes to an EC content. For initialization parameters, see oio.api.io.WriteHandler.

stream()

Uploads a stream of data. :returns: a tuple of 3 which contains:

• the list of chunks to be saved in the container
• the number of bytes transfered
• the actual checksum of the data that went through the stream.

class oio.api.ec.EcChunkWriter(chunk, conn, write_timeout=None, chunk_checksum_algo='md5', perfdata=None, **kwargs)

Bases: object

Writes an EC chunk

chunk

conn

classmethod connect(chunk, sysmeta, reqid=None, connection_timeout=None, write_timeout=None, **kwargs)

finish(metachunk_size, metachunk_hash)

Send metachunk_size and metachunk_hash as trailers.

Returns:the chunk object if the upload has failed, else None

getresponse()

Read the HTTP response from the connection

send(data)

start(pool)

Spawn the send coroutine

wait()

Wait until all data in the queue has been processed by the send coroutine

class oio.api.ec.EcMetachunkWriter(sysmeta, meta_chunk, global_checksum, storage_method, connection_timeout=None, write_timeout=None, read_timeout=None, **kwargs)

Bases: oio.api.io.MetachunkWriter

encode_and_send(ec_stream, data, writers)

Encode a buffer of data through ec_stream, and dispatch the encoded data to the chunk writers.

Returns:the list of writers that are still writing

stream(source, size)

oio.api.ec.ec_encode(storage_method, n)

Encode EC segments

oio.api.ec.meta_chunk_range_to_segment_range(meta_start, meta_end, segment_size)

Converts a meta chunk range to a segment range.

Examples:

meta_chunk_range_to_segment_range(100, 600, 256) = (0, 767) meta_chunk_range_to_segment_range(100, 600, 512) = (0, 1023) meta_chunk_range_to_segment_range(300, None, 256) = (256, None)

Returns:a tuple (segment_start, segment_end)

• segment_start is the first byte of the first segment, or None if suffix byte range
• segment_end is the last byte of the last segment, or None if prefix byte range

oio.api.ec.segment_range_to_fragment_range(segment_start, segment_end, segment_size, fragment_size)

Converts a segment range into a fragment range.

Returns:a tuple (fragment_start, fragment_end)

• fragment_start is the first byte of the first fragment, or None if this is a suffix byte range
• fragment_end is the last byte of the last fragment, or None if this is a prefix byte range

oio.api.io module

class oio.api.io.ChunkReader(chunk_iter, buf_size, headers, connection_timeout=None, read_timeout=None, align=False, perfdata=None, resp_by_chunk=None, **_kwargs)

Bases: object

Reads a chunk.

fill_ranges(start, end, length)

Fill the request ranges.

get_iter()

static get_next_part(parts_iter)

Gets next part of the body

NOTE: for the moment only return one part

(single range only)

headers

iter_from_resp(source, parts_iter, part, chunk)

recover(nb_bytes)

Recover the request.

Parameters:

nb_bytes – number of bytes already consumed that we need to discard if we perform a recovery from another source.

Raises:

• ValueError – if range header is not valid
• oio.common.exceptions.UnsatisfiableRange –
• oio.common.exceptions.EmptyByteRange –

reqid

Returns:the request ID or None

stream()

Get a generator over chunk data. After calling this method, the headers field will be available (even if no data is read from the generator).

class oio.api.io.Closeable(*iterables)

Bases: object

close()

class oio.api.io.IOBaseWrapper(wrapped)

Bases: io.RawIOBase

Wrap any object that has a read method into an io.IOBase.

read(n=-1)

readable()

Return whether object was opened for reading.

If False, read() will raise IOError.

readinto(b)

class oio.api.io.LinkHandler(fullpath, chunk_preparer, storage_method, blob_client, headers=None, **kwargs)

Bases: oio.api.io._WriteHandler

link()

class oio.api.io.MetachunkLinker(meta_chunk_target, fullpath, blob_client, storage_method=None, quorum=None, reqid=None, perfdata=None, connection_timeout=None, write_timeout=None, **kwargs)

Bases: oio.api.io._MetachunkWriter

Base class for metachunk linkers

classmethod filter_kwargs(kwargs)

link()

class oio.api.io.MetachunkPreparer(container_client, account, container, obj_name, policy=None, **kwargs)

Bases: object

Get metadata for a new object and continuously yield new metachunks.

all_chunks_so_far()

Get the list of all chunks yielded so far.

class oio.api.io.MetachunkWriter(storage_method=None, quorum=None, chunk_checksum_algo='md5', reqid=None, chunk_buffer_min=32768, chunk_buffer_max=262144, perfdata=None, **_kwargs)

Bases: oio.api.io._MetachunkWriter

Base class for metachunk writers

buffer_size()

Return a progressive buffer size.

Start small to minimize initial dead time and parallelize early, then grow to avoid too much context switches.

classmethod filter_kwargs(kwargs)

class oio.api.io.WriteHandler(source, sysmeta, chunk_preparer, storage_method, headers=None, **kwargs)

Bases: oio.api.io._WriteHandler

read_timeout

stream()

Uploads a stream of data. :returns: a tuple of 3 which contains:

• the list of chunks to be saved in the container
• the number of bytes transfered
• the actual checksum of the data that went through the stream.

oio.api.io.chain(iterable)

oio.api.io.close_source(source, logger=None)

Safely close the connection behind source.

oio.api.io.consume(it)

oio.api.io.discard_bytes(buf_size, start)

Discard the right amount of bytes so the reader yields only full records.

oio.api.io.exp_ramp_gen(start, maximum)

Yield exponentially increasing numbers.

Multiply the yielded number by 2 in each iteration after the second one, until maximum is reached.

Parameters:

• start – the first number to be yielded.
• maximum – the maximum number to yield.

oio.api.io.iters_to_raw_body(parts_iter)

oio.api.io.make_iter_from_resp(resp)

Makes a part iterator from a HTTP response

iterator return tuples:

(start, end, length, headers, body_file)

oio.api.object_storage module

class oio.api.object_storage.ObjectStorageAPI(namespace, endpoint=None, **kwargs)

Bases: oio.api.object_storage.ObjectStorageApi

Deprecated:transitional wrapper for ObjectStorageApi

class oio.api.object_storage.ObjectStorageApi(namespace, logger=None, perfdata=None, **kwargs)

Bases: object

The Object Storage API.

High level API that wraps AccountClient, ContainerClient and DirectoryClient classes.

Every method that takes a kwargs argument accepts the at least the following keywords:

• headers: dict of extra headers to pass to the proxy
• connection_timeout: float
• read_timeout: float
• write_timeout: float

EXTRA_KEYWORDS = ('chunk_checksum_algo', 'autocreate', 'chunk_buffer_min', 'chunk_buffer_max', 'cache', 'tls')

account_create(*args, **kwargs)

Create an account.

Parameters:account (str) – name of the account to create Returns:True if the account has been created

account_del_properties(account=None, *args, **kwargs)

Delete some properties from the specified account.

account_delete(account=None, *args, **kwargs)

Delete an account.

Parameters:account (str) – name of the account to delete

account_flush(account=None, *args, **kwargs)

Flush all containers of an account

Parameters:account (str) – name of the account to flush

account_get_properties(account=None, *args, **kwargs)

Get information about an account, including account properties.

account_list(*args, **kwargs)

List known accounts.

Notice that account creation is asynchronous, and an autocreated account may appear in the listing only after several seconds.

account_refresh(account=None, *args, **kwargs)

Refresh counters of an account.

Parameters:account (str) – name of the account to refresh, or None to refresh all accounts (slow)

account_set_properties(account=None, *args, **kwargs)

account_show(account=None, *args, **kwargs)

Get information about an account.

all_accounts_refresh(**kwargs)

Deprecated:call account_refresh(None) instead

blob_client

A low-level client to rawx services.

Return type:oio.blob.client.BlobClient

container_create(*args, **kwargs)

Create a container.

Parameters:

• account (str) – account in which to create the container
• container (str) – name of the container
• properties (dict) – properties to set on the container

Returns:

True if the container has been created, False if it already exists

container_create_many(*args, **kwargs)

Create Many containers

Parameters:

• account (str) – account in which to create the containers
• containers (list) – names of the containers
• properties (dict) – properties to set on the containers

container_del_properties(account, container, *args, **kwargs)

Delete properties of a container.

Parameters:

• account (str) – name of the account
• container (str) – name of the container to deal with
• properties (list) – a list of property keys

container_delete(account, container, *args, **kwargs)

Delete a container.

Parameters:

• account (str) – account from which to delete the container
• container (str) – name of the container

container_flush(account, container, *args, **kwargs)

Flush a container

Parameters:

• account (str) – account from which to delete the container
• container (str) – name of the container
• fast (bool) – flush container quickly, may put high pressure on the event system

container_get_properties(account, container, *args, **kwargs)

Get information about a container (user and system properties).

Parameters:

• account (str) – account in which the container is
• container (str) – name of the container
• properties – ignored

Returns:

a dict with “properties” and “system” entries, containing respectively a dict of user properties and a dict of system properties.

container_list(account=None, *args, **kwargs)

Get the list of containers of an account.

Parameters:

• account (str) – account from which to get the container list
• limit (int) – maximum number of results to return
• marker (bool) – name of the container from where to start the listing
• end_marker –
• prefix –
• delimiter –
• s3_buckets_only (bool) – list only S3 buckets.

Returns:

the list of containers of an account

Return type:

list of items (list) with 5 fields: name, number of objects, number of bytes, 1 if the item is a prefix or 0 if the item is actually a container, and modification time.

container_purge(account, container, *args, **kwargs)

container_refresh(account, container, *args, **kwargs)

Reset statistics of the specified container, and trigger an event that will update them. If the container does not exist, remove it from account.

container_set_properties(account, container, *args, **kwargs)

Set properties on a container.

Parameters:

• account (str) – name of the account
• container (str) – name of the container where to set properties
• properties (dict) – a dictionary of properties
• clear (bool) –
• system – dictionary of system properties to set

container_show(account, container, *args, **kwargs)

Get information about a container (user properties).

Parameters:

• account (str) – account in which the container is
• container (str) – name of the container

Returns:

a dict with “properties” containing a dict of user properties.

container_snapshot(account, container, *args, **kwargs)

Take a snapshot of a container.

Create a separate database containing all information about the contents from the original database, but with copies of the chunks at the time of the snapshot. This new database is frozen (you cannot write into it).

Pay attention to the fact that the source container is frozen during the snapshot capture. The capture may take some time, depending on the number of objects hosted by the container. You should consider setting a long read_timeout on the request.

Parameters:

• account (str) – account in which the source container is.
• container (str) – name of the source container.
• dst_account (str) – account in which the snapshot will be created.
• dst_container (str) – name of the new container (i.e. the snapshot).
• batch_size – number of chunks to copy at a time.

container_touch(account, container, *args, **kwargs)

Trigger a notification about the container state.

Parameters:

• account (str) – account from which to delete the container
• container (str) – name of the container

container_update(account, container, metadata, clear=False, **kwargs)

Deprecated:use container_set_properties

object_analyze(*args, **kwargs)

Deprecated:use object_locate

object_change_policy(account, container, obj, *args, **kwargs)

Change the storage policy of an object

Parameters:

• account (str) – name of the account where to change the policy of the object
• container (str) – name of the container where to change the policy of the object
• obj (str) – name of the object to change the policy
• policy (str) – name of the new storage policy

Returns:

list of chunks, size, hash and metadata of object

object_create(account, container, *args, **kwargs)

See documentation of object_create_ext for parameters

Returns:list of chunks, size and hash of what has been uploaded

object_create_ext(account, container, *args, **kwargs)

Create an object or append data to object in container of account with data taken from either data (str or generator) or file_or_path (path to a file or file-like object). The object will be named after obj_name if specified, or after the base name of file_or_path.

Parameters:

• account (str) – name of the account where to create the object
• container (str) – name of the container where to create the object
• file_or_path (str or file-like object) – file-like object or path to a file from which to read object data
• data (str or generator) – object data (if file_or_path is not set)
• etag (str) – entity tag of the object
• obj_name – name of the object to create. If not set, will use the base name of file_or_path.
• mime_type (str) – MIME type of the object
• properties (dict) – a dictionary of properties
• policy (str) – name of the storage policy
• key_file –
• append (bool) – if set, data will be append to existing object (or object will be created if unset)
• autocreate – if set to false, autocreation of container will be

disabled :type autocreate: bool

Parameters:

• perfdata – optional dict that will be filled with metrics of time spent to resolve the meta2 address, to do the meta2 requests, and to upload chunks to rawx services.
• deadline (float seconds) – deadline for the request, in monotonic time (oio.common.utils.monotonic_time). This supersedes timeout or read_timeout keyword arguments.
• container_properties (dict) – when containers are automatically created, this keyword allows to set user and system properties.
• tls (bool) – if set, it will try to use TLS port exposed by rawx

Returns:

list of chunks, size, hash and metadata of what has been uploaded

object_del_properties(account, container, obj, *args, **kwargs)

Delete some properties from an object.

Parameters:properties (list) – list of property keys to delete Returns:True if the property has been deleted (or was missing)

object_delete(account, container, obj, *args, **kwargs)

Delete an object from a container. If versioning is enabled and no version is specified, the object will be marked as deleted but not actually deleted.

Parameters:

• account (str) – name of the account the object belongs to
• container (str) – name of the container the object belongs to
• obj – name of the object to delete
• version – version of the object to delete

Returns:

True on success

object_delete_many(*args, **kwargs)

Delete several objects.

Parameters:objs – an iterable of object names (should not be a generator) Returns:a list of tuples with the name of the object and a boolean telling if the object has been successfully deleted Return type:list of tuple

object_drain(*args, **kwargs)

Remove all the chunks of a content, but keep all the metadata.

Parameters:

• account (str) – name of the account where the object is present
• container (str) – name of the container where the object is present
• obj – name of the object to drain

object_fastcopy(*args, **kwargs)

Deprecated:use object_link.

object_fetch(account, container, obj, *args, **kwargs)

Download an object.

Parameters:

• account – name of the account in which the object is stored
• container – name of the container in which the object is stored
• obj – name of the object to fetch
• version (str) – version of the object to fetch
• ranges (list of tuple) – a list of object ranges to download
• key_file – path to the file containing credentials
• properties (bool) – should the request return object properties along with content description (True by default)
• perfdata – optional dict that will be filled with metrics of time spent to resolve the meta2 address, to do the meta2 request, and the time-to-first-byte, as seen by this API.

Returns:

a dictionary of object metadata and a stream of object data

Return type:

tuple

object_get_properties(account, container, obj, *args, **kwargs)

Get the description of an object along with its user properties.

Parameters:

• account – name of the account in which the object is stored
• container – name of the container in which the object is stored
• obj – name of the object to query

Returns:

a dict describing the object

example

{‘hash’: ‘6BF60C17CC15EEA108024903B481738F’,

‘ctime’: ‘1481031763’, ‘deleted’: ‘False’, ‘properties’: {

u’projet’: u’OpenIO-SDS’},

‘length’: ‘43518’, ‘hash_method’: ‘md5’, ‘chunk_method’: ‘ec/algo=liberasurecode_rs_vand,k=6,m=3’, ‘version’: ‘1481031762951972’, ‘policy’: ‘EC’, ‘id’: ‘20BF2194FD420500CD4729AE0B5CBC07’, ‘mime_type’: ‘application/octet-stream’, ‘name’: ‘Makefile’}

object_head(account, container, obj, trust_level=0, **kwargs)

Check for the presence of an object in a container.

Parameters:trust_level (int) – 0: do not check chunks; 1: check if there are enough chunks to read the object; 2: check if all chunks are present.

object_link(*args, **kwargs)

Make a shallow copy of an object. Works across accounts and across containers.

object_list(account, container, *args, **kwargs)

Lists objects inside a container.

Parameters:

• properties – if True, list object properties along with objects
• versions – if True, list all versions of objects
• deleted – if True, list also the deleted objects

Returns:

a dict which contains * ‘objects’: the list of object descriptions * ‘prefixes’: common prefixes (only if delimiter and prefix are set) * ‘properties’: a dict of container properties * ‘system’: a dict of system metadata * ‘truncated’: a bool telling if the listing was truncated * ‘next_marker’: a str to be used as marker to get the next

page of results (in case the listing was truncated)

object_locate(account, container, obj, *args, **kwargs)

Get a description of the object along with the list of its chunks.

Parameters:

• account – name of the account in which the object is stored
• container – name of the container in which the object is stored
• obj – name of the object to query
• version – version of the object to query
• chunk_info – if True, fetch additional information about chunks from rawx services (slow). The second element of the returned tuple becomes a generator (instead of a list).
• properties (bool) – should the request return object properties along with content description

Returns:

a tuple with object metadata dict as first element and chunk list as second element

object_set_properties(account, container, obj, *args, **kwargs)

Set properties on an object.

Parameters:

• account – name of the account in which the object is stored
• container – name of the container in which the object is stored
• obj – name of the object to query
• properties – dictionary of properties

object_show(account, container, obj, *args, **kwargs)

Get the description of an object along with the dictionary of user-set properties.

Deprecated:prefer using object_get_properties, for consistency with container_get_properties.

object_touch(*args, **kwargs)

Trigger a notification about an object (as if it just had been created).

Parameters:

• account (str) – name of the account where to touch the object
• container (str) – name of the container where to touch the object
• obj (str) – name of the object to touch

object_truncate(account, container, obj, *args, **kwargs)

Truncate object at specified size. Only shrink is supported. A download may occur if size is not on chunk boundaries.

Parameters:

• account – name of the account in which the object is stored
• container – name of the container in which the object is stored
• obj – name of the object to query
• version – version of the object to query
• size – new size of object

object_update(account, container, obj, metadata, version=None, clear=False, **kwargs)

Deprecated:use object_set_properties

proxy_client

resolve_cid(*args, **kwargs)

Resolve a CID into account and container names.

oio.api.replication module

class oio.api.replication.FakeChecksum(actual_checksum)

Bases: object

Acts as a checksum object but does not compute anything

hexdigest()

Returns the checksum passed as constructor parameter

update(*_args, **_kwargs)

class oio.api.replication.ReplicatedMetachunkWriter(sysmeta, meta_chunk, checksum, storage_method, quorum=None, connection_timeout=None, write_timeout=None, read_timeout=None, headers=None, **kwargs)

Bases: oio.api.io.MetachunkWriter

stream(source, size)

class oio.api.replication.ReplicatedWriteHandler(source, sysmeta, chunk_preparer, storage_method, headers=None, **kwargs)

Bases: oio.api.io.WriteHandler

Handles writes to a replicated content. For initialization parameters, see oio.api.io.WriteHandler.

stream()

Uploads a stream of data. :returns: a tuple of 3 which contains:

• the list of chunks to be saved in the container
• the number of bytes transfered
• the actual checksum of the data that went through the stream.

Module contents