oio.api package
Submodules
oio.api.backblaze module
-
class
oio.api.backblaze.
BackblazeChunkDownloadHandler
(meta, chunks, offset, size, headers=None, backblaze_info=None)[source] Bases:
object
-
get_stream
()[source]
-
-
class
oio.api.backblaze.
BackblazeChunkWriteHandler
(sysmeta, meta_chunk, checksum, storage_method, backblaze_info)[source] Bases:
object
-
stream
(source)[source]
-
-
class
oio.api.backblaze.
BackblazeDeleteHandler
(meta, chunks, backblaze_info)[source] Bases:
object
-
delete
()[source]
-
-
class
oio.api.backblaze.
BackblazeDownloadHandler
(sysmeta, meta_chunks, backblaze_info, headers, range_start=None, range_end=None)[source] Bases:
object
-
get_iter
()[source]
-
-
class
oio.api.backblaze.
BackblazeWriteHandler
(source, sysmeta, chunk_prep, storage_method, backblaze_info, **kwargs)[source] Bases:
oio.api.io.WriteHandler
-
stream
()[source] Only works with files, for the moment, because we need a file size to known when to stop.
-
oio.api.backblaze_http module
-
class
oio.api.backblaze_http.
Backblaze
(account_id, application_key, authorization_required=None, upload_required=None, upload_part=False)[source] Bases:
object
-
BACKBLAZE_BASE_API_URL
= 'https://api.backblazeb2.com'
-
BACKBLAZE_MAX_CHUNK_SIZE
= 209715200
-
delete
(bucket_name, metadata)[source]
-
delete_by_path_name
(file_id, file_name)[source]
-
download
(bucket_name, metadata, headers=None)[source]
-
download_by_path_name
(bucket_name, link, headers=None)[source]
-
get_backblaze_infos
(bucket_name)[source]
-
get_file_number
(bucket_name)[source]
-
get_list_buckets
()[source]
-
get_list_file_names
(bucket_name)[source]
-
get_list_file_names_by_bucket_id
(bucket_id)[source]
-
get_size
(bucket_name)[source]
-
upload
(bucket_name, meta, data, sha1=None)[source]
-
upload_part
(file_id, data, part_number, sha1=None)[source]
-
upload_part_begin
(bucket_name, meta)[source]
-
upload_part_end
(file_id, sha1_array)[source]
-
-
exception
oio.api.backblaze_http.
BackblazeException
(status_code, message, response, headers_send)[source] Bases:
exceptions.Exception
-
headers_received
-
headers_send
-
message
-
status_code
-
-
class
oio.api.backblaze_http.
BackblazeUtils
[source] Bases:
object
-
static
get_credentials
(storage_method, application_key_path=None, renew=False)[source]
-
static
-
exception
oio.api.backblaze_http.
BackblazeUtilsException
(string)[source] Bases:
exceptions.Exception
oio.api.base module
-
class
oio.api.base.
HttpApi
(endpoint=None, pool_manager=None, **kwargs)[source] Bases:
object
Provides facilities to make HTTP requests towards the same endpoint, with a pool of connections.
oio.api.ec module
-
class
oio.api.ec.
ECChunkDownloadHandler
(storage_method, chunks, meta_start, meta_end, headers, connection_timeout=None, read_timeout=None, **_kwargs)[source] Bases:
object
Handles the download of an EC meta chunk
-
get_stream
()[source]
-
-
class
oio.api.ec.
ECRebuildHandler
(meta_chunk, missing, storage_method, connection_timeout=None, read_timeout=None, **_kwargs)[source] Bases:
object
-
rebuild
()[source]
-
-
class
oio.api.ec.
ECStream
(storage_method, readers, range_infos, meta_length, fragment_length)[source] Bases:
object
Reads an EC meta chunk.
Handles the different readers.
-
close
()[source]
-
get_iter
()[source]
-
start
()[source]
-
-
class
oio.api.ec.
ECWriteHandler
(source, sysmeta, chunk_preparer, storage_method, headers=None, connection_timeout=None, write_timeout=None, read_timeout=None, deadline=None, chunk_checksum_algo='md5', **_kwargs)[source] Bases:
oio.api.io.WriteHandler
Handles writes to an EC content. For initialization parameters, see oio.api.io.WriteHandler.
-
stream
()[source] Uploads a stream of data. :returns: a tuple of 3 which contains:
- the list of chunks to be saved in the container
- the number of bytes transfered
- the actual checksum of the data that went through the stream.
-
-
class
oio.api.ec.
EcChunkWriter
(chunk, conn, write_timeout=None, chunk_checksum_algo='md5', **_kwargs)[source] Bases:
object
Writes an EC chunk
-
chunk
-
conn
-
classmethod
connect
(chunk, sysmeta, reqid=None, connection_timeout=None, write_timeout=None, **kwargs)[source]
-
finish
(metachunk_size, metachunk_hash)[source] Send metachunk_size and metachunk_hash as trailers
-
getresponse
()[source] Read the HTTP response from the connection
-
send
(data)[source]
-
start
(pool)[source] Spawn the send coroutine
-
wait
()[source] Wait until all data in the queue has been processed by the send coroutine
-
-
class
oio.api.ec.
EcMetachunkWriter
(sysmeta, meta_chunk, global_checksum, storage_method, reqid=None, connection_timeout=None, write_timeout=None, read_timeout=None, **kwargs)[source] Bases:
oio.api.io.MetachunkWriter
-
stream
(source, size)[source]
-
-
oio.api.ec.
ec_encode
(storage_method, n)[source] Encode EC segments
-
oio.api.ec.
meta_chunk_range_to_segment_range
(meta_start, meta_end, segment_size)[source] Converts a meta chunk range to a segment range.
- Examples:
- meta_chunk_range_to_segment_range(100, 600, 256) = (0, 767) meta_chunk_range_to_segment_range(100, 600, 512) = (0, 1023) meta_chunk_range_to_segment_range(300, None, 256) = (256, None)
Returns: a tuple (segment_start, segment_end) - segment_start is the first byte of the first segment, or None if suffix byte range
- segment_end is the last byte of the last segment, or None if prefix byte range
-
oio.api.ec.
segment_range_to_fragment_range
(segment_start, segment_end, segment_size, fragment_size)[source] Converts a segment range into a fragment range.
Returns: a tuple (fragment_start, fragment_end) - fragment_start is the first byte of the first fragment, or None if this is a suffix byte range
- fragment_end is the last byte of the last fragment, or None if this is a prefix byte range
oio.api.io module
-
class
oio.api.io.
ChunkReader
(chunk_iter, buf_size, headers, connection_timeout=None, read_timeout=None, align=False, **_kwargs)[source] Bases:
object
Reads a chunk.
-
fill_ranges
(start, end, length)[source] Fill the request ranges.
-
get_iter
()[source]
-
static
get_next_part
(parts_iter)[source] Gets next part of the body
- NOTE: for the moment only return one part
- (single range only)
-
headers
-
iter_from_resp
(source, parts_iter, part, chunk)[source]
-
recover
(nb_bytes)[source] Recover the request.
Parameters: nb_bytes – number of bytes already consumed that we need to discard if we perform a recovery from another source.
Raises: - ValueError – if range header is not valid
- oio.common.exceptions.UnsatisfiableRange –
- oio.common.exceptions.EmptyByteRange –
-
reqid
Returns: the request ID or None
-
stream
()[source] Get a generator over chunk data. After calling this method, the headers field will be available (even if no data is read from the generator).
-
-
class
oio.api.io.
IOBaseWrapper
(wrapped)[source] Bases:
io.RawIOBase
Wrap any object that has a read method into an io.IOBase.
-
read
(n=-1)[source]
-
readable
()[source] Return whether object was opened for reading.
If False, read() will raise IOError.
-
readinto
(b)[source]
-
-
class
oio.api.io.
MetachunkPreparer
(container_client, account, container, obj_name, policy=None, **kwargs)[source] Bases:
object
Get metadata for a new object and continuously yield new metachunks.
-
all_chunks_so_far
()[source] Get the list of all chunks yielded so far.
-
-
class
oio.api.io.
MetachunkWriter
(storage_method=None, quorum=None, chunk_checksum_algo='md5', **_kwargs)[source] Bases:
object
Base class for metachunk writers
-
quorum
Minimum number of chunks required to validate an upload
-
quorum_or_fail
(successes, failures)[source] Compare the number of uploads against the quorum.
Parameters: - successes (list or tuple) – a list of chunk objects whose upload succeded
- failures (list or tuple) – a list of chunk objects whose upload failed
Raises: - exc.SourceReadError – if there is an error while reading data from the client
- exc.SourceReadTimeout – if there is a timeout while reading data from the client
- exc.OioTimeout – if there is a timeout among the errors
- exc.OioException – if quorum has not been reached for any other reason
-
-
class
oio.api.io.
WriteHandler
(source, sysmeta, chunk_preparer, storage_method, headers=None, connection_timeout=None, write_timeout=None, read_timeout=None, deadline=None, chunk_checksum_algo='md5', **_kwargs)[source] Bases:
object
-
read_timeout
-
stream
()[source] Uploads a stream of data. :returns: a tuple of 3 which contains:
- the list of chunks to be saved in the container
- the number of bytes transfered
- the actual checksum of the data that went through the stream.
-
write_timeout
-
-
oio.api.io.
chain
(iterable)[source]
-
oio.api.io.
close_source
(source)[source] Safely close the connection behind source.
-
oio.api.io.
consume
(it)[source]
-
oio.api.io.
discard_bytes
(buf_size, start)[source] Discard the right amount of bytes so the reader yields only full records.
-
oio.api.io.
iters_to_raw_body
(parts_iter)[source]
-
oio.api.io.
make_iter_from_resp
(resp)[source] Makes a part iterator from a HTTP response
iterator return tuples:
(start, end, length, headers, body_file)
oio.api.object_storage module
-
class
oio.api.object_storage.
ObjectStorageAPI
(namespace, endpoint=None, **kwargs)[source] Bases:
oio.api.object_storage.ObjectStorageApi
Deprecated: transitional wrapper for ObjectStorageApi
-
class
oio.api.object_storage.
ObjectStorageApi
(namespace, logger=None, **kwargs)[source] Bases:
object
The Object Storage API.
High level API that wraps AccountClient, ContainerClient and DirectoryClient classes.
Every method that takes a kwargs argument accepts the at least the following keywords:
- headers: dict of extra headers to pass to the proxy
- connection_timeout: float
- read_timeout: float
- write_timeout: float
-
EXTRA_KEYWORDS
= ('chunk_checksum_algo', 'autocreate')
-
TIMEOUT_KEYS
= ('connection_timeout', 'read_timeout', 'write_timeout')
-
account_create
(**kwargs)[source] Create an account.
Parameters: account (str) – name of the account to create Returns: True if the account has been created
-
account_del_properties
(account=None, *args, **kwargs)[source]
-
account_delete
(account=None, *args, **kwargs)[source] Delete an account.
Parameters: account (str) – name of the account to delete
-
account_flush
(account=None, *args, **kwargs)[source] Flush all containers of an account
Parameters: account (str) – name of the account to flush
-
account_list
(**kwargs)[source] List known accounts.
Notice that account creation is asynchronous, and an autocreated account may appear in the listing only after several seconds.
-
account_refresh
(account=None, *args, **kwargs)[source] Refresh counters of an account.
Parameters: account (str) – name of the account to refresh, or None to refresh all accounts (slow)
-
account_set_properties
(account=None, *args, **kwargs)[source]
-
account_show
(account=None, *args, **kwargs)[source] Get information about an account.
-
account_update
(account=None, *args, **kwargs)[source]
-
all_accounts_refresh
(**kwargs)[source] Deprecated: call account_refresh(None) instead
-
blob_client
A low-level client to rawx services.
Return type: oio.blob.client.BlobClient
-
container_create
(**kwargs)[source] Create a container.
Parameters: - account (str) – account in which to create the container
- container (str) – name of the container
- properties (dict) – properties to set on the container
Returns: True if the container has been created, False if it already exists
-
container_create_many
(**kwargs)[source] Create Many containers
Parameters: - account (str) – account in which to create the containers
- containers (list) – names of the containers
- properties (dict) – properties to set on the containers
-
container_del_properties
(account, container, *args, **kwargs)[source] Delete properties of a container.
Parameters: - account (str) – name of the account
- container (str) – name of the container to deal with
- properties (list) – a list of property keys
-
container_delete
(account, container, *args, **kwargs)[source] Delete a container.
Parameters: - account (str) – account from which to delete the container
- container (str) – name of the container
-
container_flush
(account, container, *args, **kwargs)[source] Flush a container
Parameters: - account (str) – account from which to delete the container
- container (str) – name of the container
- fast (bool) – flush container quickly, may put high pressure on the event system
-
container_get_properties
(account, container, *args, **kwargs)[source] Get information about a container (user and system properties).
Parameters: - account (str) – account in which the container is
- container (str) – name of the container
- properties – ignored
Returns: a dict with “properties” and “system” entries, containing respectively a dict of user properties and a dict of system properties.
-
container_list
(account=None, *args, **kwargs)[source] Get the list of containers of an account.
Parameters: - account (str) – account from which to get the container list
- limit (int) – maximum number of results to return
- marker (str) – name of the container from where to start the listing
- end_marker –
- prefix –
- delimiter –
Returns: the list of containers of an account
Return type: list of items (list) with 5 fields: name, number of objects, number of bytes, 1 if the item is a prefix or 0 if the item is actually a container, and modification time.
-
container_purge
(account, container, *args, **kwargs)[source]
-
container_refresh
(account, container, *args, **kwargs)[source]
-
container_set_properties
(account, container, *args, **kwargs)[source] Set properties on a container.
Parameters: - account (str) – name of the account
- container (str) – name of the container where to set properties
- properties (dict) – a dictionary of properties
- clear (bool) –
- system – dictionary of system properties to set
-
container_show
(account, container, *args, **kwargs)[source] Get information about a container (user properties).
Parameters: - account (str) – account in which the container is
- container (str) – name of the container
Returns: a dict with “properties” containing a dict of user properties.
-
container_snapshot
(account, container, *args, **kwargs)[source] Create a copy of the container (only the content of the database)
Parameters: - account (str) – account in which the target is
- container (str) – name of the target
- dst_account (str) – account in which the snapshot will be.
- dst_container (str) – name of the snapshot
-
container_touch
(account, container, *args, **kwargs)[source] Trigger a notification about the container state.
Parameters: - account (str) – account from which to delete the container
- container (str) – name of the container
-
container_update
(account, container, metadata, clear=False, **kwargs)[source] Deprecated: use container_set_properties
-
object_analyze
(*args, **kwargs)[source] Deprecated: use object_locate
-
object_create
(account, container, *args, **kwargs)[source] Create an object or append data to object in container of account with data taken from either data (str or generator) or file_or_path (path to a file or file-like object). The object will be named after obj_name if specified, or after the base name of file_or_path.
Parameters: - account (str) – name of the account where to create the object
- container (str) – name of the container where to create the object
- file_or_path (str or file-like object) – file-like object or path to a file from which to read object data
- data (str or generator) – object data (if file_or_path is not set)
- etag (str) – entity tag of the object
- obj_name – name of the object to create. If not set, will use the base name of file_or_path.
- mime_type (str) – MIME type of the object
- properties (dict) – a dictionary of properties
- policy (str) – name of the storage policy
- key_file –
- append (bool) – if set, data will be append to existing object (or object will be created if unset)
- autocreate – if set to false, autocreation of container will be
disabled :type autocreate: bool
Parameters: - perfdata – optional dict that will be filled with metrics of time spent to resolve the meta2 address, to do the meta2 requests, and to upload chunks to rawx services.
- deadline (float seconds) – deadline for the request, in monotonic time (oio.common.utils.monotonic_time). This supersedes timeout or read_timeout keyword arguments.
Returns: list of chunks, size and hash of the what has been uploaded
-
object_del_properties
(account, container, obj, *args, **kwargs)[source] Delete some properties from an object.
Parameters: properties (list) – list of property keys to delete Returns: True if the property has been deleted (or was missing)
-
object_delete
(account, container, obj, *args, **kwargs)[source] Delete an object from a container. If versioning is enabled and no version is specified, the object will be marked as deleted but not actually deleted.
Parameters: - account (str) – name of the account the object belongs to
- container (str) – name of the container the object belongs to
- obj – name of the object to delete
- version – version of the object to delete
Returns: True on success
-
object_delete_many
(**kwargs)[source] Delete several objects.
Parameters: objs – an iterable of object names (should not be a generator) Returns: a list of tuples with the name of the object and a boolean telling if the object has been successfully deleted Return type: list of tuple
-
object_drain
(**kwargs)[source] Remove all the chunks of a content, but keep all the metadata.
Parameters: - account (str) – name of the account where the object is present
- container (str) – name of the container where the object is present
- obj – name of the object to drain
-
object_fastcopy
(*args, **kwargs)[source] Deprecated: use object_link.
-
object_fetch
(*args, **kwargs)[source] Download an object.
Parameters: - account – name of the account in which the object is stored
- container – name of the container in which the object is stored
- obj – name of the object to fetch
- version (str) – version of the object to fetch
- ranges (list of tuple) – a list of object ranges to download
- key_file – path to the file containing credentials
- properties (bool) – should the request return object properties along with content description (True by default)
- perfdata – optional dict that will be filled with metrics of time spent to resolve the meta2 address, to do the meta2 request, and the time-to-first-byte, as seen by this API.
Returns: a dictionary of object metadata and a stream of object data
Return type: tuple
-
object_get_properties
(account, container, obj, *args, **kwargs)[source] Get the description of an object along with its user properties.
Parameters: - account – name of the account in which the object is stored
- container – name of the container in which the object is stored
- obj – name of the object to query
Returns: a dict describing the object
-
example
- {‘hash’: ‘6BF60C17CC15EEA108024903B481738F’,
‘ctime’: ‘1481031763’, ‘deleted’: ‘False’, ‘properties’: {
u’projet’: u’OpenIO-SDS’},‘length’: ‘43518’, ‘hash_method’: ‘md5’, ‘chunk_method’: ‘ec/algo=liberasurecode_rs_vand,k=6,m=3’, ‘version’: ‘1481031762951972’, ‘policy’: ‘EC’, ‘id’: ‘20BF2194FD420500CD4729AE0B5CBC07’, ‘mime_type’: ‘application/octet-stream’, ‘name’: ‘Makefile’}
-
object_head
(account, container, obj, trust_level=0, **kwargs)[source] Check for the presence of an object in a container.
Parameters: trust_level (int) – 0: do not check chunks; 1: check if there are enough chunks to read the object; 2: check if all chunks are present.
-
object_link
(*args, **kwargs)[source] Make a shallow copy of an object. Works across accounts and across containers.
-
object_list
(account, container, *args, **kwargs)[source] Lists objects inside a container.
Parameters: - properties – if True, list object properties along with objects
- versions – if True, list all versions of objects
- deleted – if True, list also the deleted objects
Returns: a dict which contains * ‘objects’: the list of object descriptions * ‘prefixes’: common prefixes (only if delimiter and prefix are set) * ‘properties’: a dict of container properties * ‘system’: a dict of system metadata * ‘truncated’: a bool telling if the listing was truncated * ‘next_marker’: a str to be used as marker to get the next
page of results (in case the listing was truncated)
-
object_locate
(account, container, obj, *args, **kwargs)[source] Get a description of the object along with the list of its chunks.
Parameters: - account – name of the account in which the object is stored
- container – name of the container in which the object is stored
- obj – name of the object to query
- version – version of the object to query
- chunk_info – if True, fetch additional information about chunks from rawx services (slow). The second element of the returned tuple becomes a generator (instead of a list).
- properties (bool) – should the request return object properties along with content description
Returns: a tuple with object metadata dict as first element and chunk list as second element
-
object_set_properties
(account, container, obj, *args, **kwargs)[source] Set properties on an object.
Parameters: - account – name of the account in which the object is stored
- container – name of the container in which the object is stored
- obj – name of the object to query
- properties – dictionary of properties
-
object_show
(account, container, obj, *args, **kwargs)[source] Get the description of an object along with the dictionary of user-set properties.
Deprecated: prefer using object_get_properties, for consistency with container_get_properties.
-
object_touch
(**kwargs)[source] Trigger a notification about an object (as if it just had been created).
Parameters: - account (str) – name of the account where to create the object
- container (str) – name of the container where to create the object
- obj – name of the object to touch
-
object_truncate
(account, container, obj, *args, **kwargs)[source] Truncate object at specified size. Only shrink is supported. A download may occur if size is not on chunk boundaries.
Parameters: - account – name of the account in which the object is stored
- container – name of the container in which the object is stored
- obj – name of the object to query
- version – version of the object to query
- size – new size of object
-
object_update
(account, container, obj, metadata, version=None, clear=False, **kwargs)[source] Deprecated: use object_set_properties
-
proxy_client
-
oio.api.object_storage.
patch_kwargs
(fnc)[source] Patch keyword arguments with the ones passed to the class’ constructor.
oio.api.replication module
-
class
oio.api.replication.
FakeChecksum
(actual_checksum)[source] Bases:
object
Acts as a checksum object but does not compute anything
-
hexdigest
()[source] Returns the checksum passed as constructor parameter
-
update
(*_args, **_kwargs)[source]
-
-
class
oio.api.replication.
ReplicatedMetachunkWriter
(sysmeta, meta_chunk, checksum, storage_method, quorum=None, connection_timeout=None, write_timeout=None, read_timeout=None, headers=None, **kwargs)[source] Bases:
oio.api.io.MetachunkWriter
-
stream
(source, size=None)[source]
-
-
class
oio.api.replication.
ReplicatedWriteHandler
(source, sysmeta, chunk_preparer, storage_method, headers=None, connection_timeout=None, write_timeout=None, read_timeout=None, deadline=None, chunk_checksum_algo='md5', **_kwargs)[source] Bases:
oio.api.io.WriteHandler
Handles writes to a replicated content. For initialization parameters, see oio.api.io.WriteHandler.
-
stream
()[source] Uploads a stream of data. :returns: a tuple of 3 which contains:
- the list of chunks to be saved in the container
- the number of bytes transfered
- the actual checksum of the data that went through the stream.
-