oio.blob package

Submodules

oio.blob.auditor module

class oio.blob.auditor.BlobAuditor(conf, **kwargs)

Bases: oio.common.daemon.Daemon

Walk through the chunks of a volume, and check for incoherencies: missing extended attributes, invalid hash, position or size, or orphaned chunk.

run(*args, **kwargs)

class oio.blob.auditor.BlobAuditorWorker(conf, logger, volume)

Bases: object

audit_pass()

chunk_audit(path, chunk_id)

chunk_file_audit(chunk_file, chunk_id)

safe_chunk_audit(path)

class oio.blob.auditor.ChunkReader(fp, size, md5_checksum, compression=None)

Bases: object

close()

Perform checks on what has been read before closing, if no error has occurred yet.

oio.blob.client module

class oio.blob.client.BlobClient(conf=None, perfdata=None, logger=None, connection_pool=None, **kwargs)

Bases: object

A low-level client to rawx services.

chunk_copy(*args, **kwargs)

chunk_delete(*args, **kwargs)

chunk_delete_many(**kwargs)

Return type:list of either urllib3.response.HTTPResponse or urllib3.exceptions.HTTPError, with an extra “chunk” attribute.

chunk_get(*args, **kwargs)

Parameters:check_headers – when True (the default), raise FaultyChunk if a mandatory response header is missing. Returns:a tuple with a dictionary of chunk metadata and a stream to the chunk’s data.

chunk_head(*args, **kwargs)

Perform a HEAD request on a chunk.

Parameters:

• url – URL of the chunk to request.
• xattr – when False, ask the rawx not to read extended attributes of the chunk.
• check_hash – when True, ask the rawx to validate checksum of the chunk.

Returns:

a dict with chunk metadata (empty when xattr is False).

chunk_link(*args, **kwargs)

chunk_put(*args, **kwargs)

resolve_url(url)

oio.blob.client.extract_headers_meta(headers, check=True)

Extract chunk metadata from a dictionary of rawx response headers.

Parameters:

• headers – a dictionary of headers, as returned by a HEAD or GET request to a rawx service.
• check – if True (the default), raise FaultyChunk if one or several mandatory response headers are missing.

Returns:

a dictionary of chunk metadata.

oio.blob.client.update_rawx_perfdata(func)

oio.blob.converter module

class oio.blob.converter.BlobConverter(conf, logger=None, **kwargs)

Bases: object

cid_from_name(account, container)

content_id_from_name(cid, path, version, search=False)

convert_chunk(fd, chunk_id)

converter_pass(input_file=None)

decode_fullpath(fullpath)

decode_old_fullpath(old_fullpath)

encode_fullpath(fd, chunk_id, account, container, path, version, content_id)

get_chunk_id_and_fullpath(fd, chunk_pos, container_id, path, version, chunk_id=None, account=None, container=None, content_id=None)

is_fullpath_error(err)

name_from_cid(cid)

paths_gen(input_file=None)

rdir

Get an instance of RdirClient.

recover_chunk_fullpath(path, chunk_id=None)

safe_convert_chunk(path, chunk_id=None)

save_xattr(fd, chunk_id, xattr)

oio.blob.indexer module

class oio.blob.indexer.BlobIndexer(conf, **kwargs)

Bases: oio.common.daemon.Daemon

index_pass()

report(tag, start_time)

run(*args, **kwargs)

safe_recover_fullpath(path)

safe_update_index(path)

update_index(path, chunk_id)

oio.blob.indexer.random() → x in the interval [0, 1).

oio.blob.mover module

class oio.blob.mover.BlobMover(conf, **kwargs)

Bases: oio.common.daemon.Daemon

run(*args, **kwargs)

class oio.blob.mover.BlobMoverWorker(conf, logger, volume)

Bases: object

chunk_move(path, chunk_id)

load_chunk_metadata(path, chunk_id)

mover_pass(**kwargs)

safe_chunk_move(path)

oio.blob.operator module

class oio.blob.operator.ChunkOperator(conf, logger=None)

Bases: object

Execute maintenance operations on chunks.

rebuild(container_id, content_id, chunk_id_or_pos, rawx_id=None, try_chunk_delete=False, allow_frozen_container=True, allow_same_rawx=True)

Try to find the chunk in the metadata of the specified object, then rebuild it.

oio.blob.rebuilder module

class oio.blob.rebuilder.BlobRebuilder(conf, input_file=None, service_id=None, **kwargs)

Bases: oio.common.tool.Tool

Rebuild chunks.

DEFAULT_ALLOW_FROZEN_CT = False

DEFAULT_ALLOW_SAME_RAWX = True

DEFAULT_BEANSTALKD_WORKER_TUBE = 'oio-rebuild'

DEFAULT_DISTRIBUTED_BEANSTALKD_WORKER_TUBE = 'oio-rebuild'

DEFAULT_DRY_RUN = False

DEFAULT_RDIR_FETCH_LIMIT = 100

DEFAULT_RDIR_TIMEOUT = 60.0

DEFAULT_TRY_CHUNK_DELETE = False

create_worker(queue_workers, queue_reply)

Create worker to process the items.

static items_from_task_event(task_event)

static res_event_from_task_res(task_res)

run()

Start processing all found items.

static string_from_item(item)

static task_event_from_item(item)

static tasks_res_from_res_event(res_event)

update_counters(task_res)

Update all counters of the tool.

class oio.blob.rebuilder.BlobRebuilderWorker(tool, queue_workers, queue_reply)

Bases: oio.common.tool.ToolWorker

oio.blob.registrator module

class oio.blob.registrator.BlobRegistrator(conf, logger, volume, container_ids)

Bases: object

BEAN_TYPES = ('alias', 'header', 'chunk')

DEFAULT_CHUNKS_PER_SECOND = 30

DEFAULT_REPORT_INTERVAL = 3600

log_report(status, force=False)

pass_bean(meta, bean)

pass_chunk_file(path)

pass_volume()

oio.blob.utils module

oio.blob.utils.check_volume(volume_path)

Check if volume_path points to a rawx directory.

Returns:the namespace name and the service ID Raises:oio.common.exceptions.OioException – when the specified path does not belong to a rawx service or misses some attributes.

oio.blob.utils.check_volume_for_service_type(volume_path, required_type)

Check if volume_path points to a directory for the specified service type.

Returns:the namespace name and the service ID Raises:oio.common.exceptions.OioException – when the specified path does not belong to a service from the specified type or is missing some attributes.

oio.blob.utils.read_chunk_metadata(fd, chunk_id, for_conversion=False)

Module contents