oio.blob package

Submodules

oio.blob.auditor module

class oio.blob.auditor.BlobAuditor(conf, **kwargs)[source]

Bases: oio.common.daemon.Daemon

Walk through the chunks of a volume, and check for incoherencies: missing extended attributes, invalid hash, position or size, or orphaned chunk.

run(*args, **kwargs)[source]
class oio.blob.auditor.BlobAuditorWorker(conf, logger, volume)[source]

Bases: object

audit_pass()[source]
chunk_audit(path, chunk_id)[source]
chunk_file_audit(chunk_file, chunk_id)[source]
safe_chunk_audit(path)[source]
class oio.blob.auditor.ChunkReader(fp, size, md5_checksum)[source]

Bases: object

close()[source]

oio.blob.client module

class oio.blob.client.BlobClient(conf=None, perfdata=None, logger=None, connection_pool=None, **kwargs)[source]

Bases: object

A low-level client to rawx services.

chunk_copy(*args, **kwargs)[source]
chunk_delete(*args, **kwargs)[source]
chunk_delete_many(**kwargs)[source]
Return type:list of either urllib3.response.HTTPResponse or urllib3.exceptions.HTTPError, with an extra “chunk” attribute.
chunk_get(*args, **kwargs)[source]
Parameters:check_headers – when True (the default), raise FaultyChunk if a mandatory response header is missing.
Returns:a tuple with a dictionary of chunk metadata and a stream to the chunk’s data.
chunk_head(*args, **kwargs)[source]

Perform a HEAD request on a chunk.

Parameters:
  • url – URL of the chunk to request.
  • xattr – when False, ask the rawx not to read extended attributes of the chunk.
  • check_hash – when True, ask the rawx to validate checksum of the chunk.
Returns:

a dict with chunk metadata (empty when xattr is False).

chunk_put(*args, **kwargs)[source]
resolve_url(url)[source]
oio.blob.client.extract_headers_meta(headers, check=True)[source]

Extract chunk metadata from a dictionary of rawx response headers.

Parameters:
  • headers – a dictionary of headers, as returned by a HEAD or GET request to a rawx service.
  • check – if True (the default), raise FaultyChunk if one or several mandatory response headers are missing.
Returns:

a dictionary of chunk metadata.

oio.blob.client.update_rawx_perfdata(func)[source]

oio.blob.converter module

class oio.blob.converter.BlobConverter(conf, logger=None, **kwargs)[source]

Bases: object

cid_from_name(account, container)[source]
content_id_from_name(cid, path, version, search=False)[source]
convert_chunk(fd, chunk_id)[source]
converter_pass(input_file=None)[source]
decode_fullpath(fullpath)[source]
decode_old_fullpath(old_fullpath)[source]
encode_fullpath(chunk_inode, chunk_id, account, container, path, version, content_id)[source]
get_chunk_id_and_fullpath(chunk_inode, chunk_pos, container_id, path, version, chunk_id=None, account=None, container=None, content_id=None)[source]
is_fullpath_error(err)[source]
name_from_cid(cid)[source]
paths_gen(input_file=None)[source]
rdir

Get an instance of RdirClient.

recover_chunk_fullpath(path, chunk_id=None)[source]
safe_convert_chunk(path, chunk_id=None)[source]
save_xattr(chunk_id, xattr)[source]

oio.blob.indexer module

class oio.blob.indexer.BlobIndexer(conf, **kwargs)[source]

Bases: oio.common.daemon.Daemon

index_pass()[source]
report(tag, start_time)[source]
run(*args, **kwargs)[source]
safe_recover_fullpath(path)[source]
safe_update_index(path)[source]
update_index(path, chunk_id)[source]
oio.blob.indexer.random() → x in the interval [0, 1).

oio.blob.mover module

class oio.blob.mover.BlobMover(conf, **kwargs)[source]

Bases: oio.common.daemon.Daemon

run(*args, **kwargs)[source]
class oio.blob.mover.BlobMoverWorker(conf, logger, volume)[source]

Bases: object

chunk_move(path, chunk_id)[source]
load_chunk_metadata(path, chunk_id)[source]
mover_pass(**kwargs)[source]
safe_chunk_move(path)[source]

oio.blob.operator module

class oio.blob.operator.ChunkOperator(conf, logger=None)[source]

Bases: object

Execute maintenance operations on chunks.

rebuild(container_id, content_id, chunk_id_or_pos, rawx_id=None, try_chunk_delete=False, allow_frozen_container=True, allow_same_rawx=True)[source]

Try to find the chunk in the metadata of the specified object, then rebuild it.

oio.blob.rebuilder module

class oio.blob.rebuilder.BlobRebuilder(conf, input_file=None, service_id=None, **kwargs)[source]

Bases: oio.common.tool.Tool

Rebuild chunks.

DEFAULT_ALLOW_FROZEN_CT = False
DEFAULT_ALLOW_SAME_RAWX = True
DEFAULT_BEANSTALKD_WORKER_TUBE = 'oio-rebuild'
DEFAULT_DISTRIBUTED_BEANSTALKD_WORKER_TUBE = 'oio-rebuild'
DEFAULT_DRY_RUN = False
DEFAULT_RDIR_FETCH_LIMIT = 100
DEFAULT_RDIR_TIMEOUT = 60.0
DEFAULT_TRY_CHUNK_DELETE = False
create_worker(queue_workers, queue_reply)[source]

Create worker to process the items.

static items_from_task_event(task_event)[source]
static res_event_from_task_res(task_res)[source]
run()[source]

Start processing all found items.

static string_from_item(item)[source]
static task_event_from_item(item)[source]
static tasks_res_from_res_event(res_event)[source]
update_counters(task_res)[source]

Update all counters of the tool.

class oio.blob.rebuilder.BlobRebuilderWorker(tool, queue_workers, queue_reply)[source]

Bases: oio.common.tool.ToolWorker

oio.blob.registrator module

class oio.blob.registrator.BlobRegistrator(conf, logger, volume, container_ids)[source]

Bases: object

BEAN_TYPES = ('alias', 'header', 'chunk')
DEFAULT_CHUNKS_PER_SECOND = 30
DEFAULT_REPORT_INTERVAL = 3600
log_report(status, force=False)[source]
pass_bean(meta, bean)[source]
pass_chunk_file(path)[source]
pass_volume()[source]

oio.blob.utils module

oio.blob.utils.check_volume(volume_path)[source]

Check if volume_path points to a rawx directory.

Returns:the namespace name and the service ID
Raises:oio.common.exceptions.OioException – when the specified path does not belong to a rawx service or misses some attributes.
oio.blob.utils.check_volume_for_service_type(volume_path, required_type)[source]

Check if volume_path points to a directory for the specified service type.

Returns:the namespace name and the service ID
Raises:oio.common.exceptions.OioException – when the specified path does not belong to a service from the specified type or is missing some attributes.
oio.blob.utils.read_chunk_metadata(fd, chunk_id, check_chunk_id=True)[source]

Module contents