oio.crawler package

Submodules

oio.crawler.integrity module

Recursively check account, container, content and chunk integrity.

class oio.crawler.integrity.Checker(namespace, concurrency=50, error_file=None, rebuild_file=None, check_xattr=True, limit_listings=0, request_attempts=1, logger=None, verbose=False, check_hash=False, min_time_in_error=0.0, required_confirmations=0, beanstalkd_addr=None, beanstalkd_tube='oio-rebuild', cache_size=16777216, **_kwargs)

Bases: object

check(target, recurse=0)

check_account(target, recurse=0)

check_all_accounts(recurse=0)

check_chunk(target)

check_container(target, recurse=0)

check_obj(target, recurse=0)

Check one object version. If no version is specified, all versions of the object will be checked. :returns: the result of the check of the most recent version,

or the one that is explicitly targeted.

check_obj_versions(target, versions, recurse=0)

Run checks of all versions of the targeted object in parallel.

complete_target_from_chunk_metadata(target, xattr_meta)

Complete a Target object from metadata found in chunk’s extended attributes. In case the “fullpath” is not available, try to read legacy metadata, and maybe ask meta1 to resolve the CID into account and container names.

fetch_results(rate_limiter=None)

log_result(target)

Log a check result, if it shows errors. Dispatch the errors to the appropriate destinations (log files, queues, etc.).

merge_with_delayed_target(target)

Merge the specified target with a delayed one.

Returns:the delayed target, if there is one, with an error log including the errors of the new target. Return the new target otherwise.

recover_and_complete_object_meta(target, chunk)

report()

reset_stats()

run(rate_limiter=None)

Fetch results and write logs until all jobs have finished.

Returns:a generator yielding check results.

send_chunk_job(target, irreparable=False)

Send a “content broken” event, to trigger the reconstruction of the chunk.

send_result(target, errors=None, irreparable=False)

Put an item in the result queue.

stop()

write_chunk_error(target, chunk=None, irreparable=False)

write_error(target, irreparable=False)

write_rebuilder_input(target, irreparable=False)

class oio.crawler.integrity.ItemResult(errors=None, irreparable=False)

Bases: object

Hold the result of a check. Must be serializable to be used in the Checker’s return queue.

errors_to_str(separator='\n', err_format='%s')

Pretty print errors stored in this result.

class oio.crawler.integrity.Target(account, container=None, obj=None, chunk=None, content_id=None, version=None, cid=None)

Bases: object

Identify the target of a check, hold a log of errors.

append_result(result)

cid

copy()

copy_account()

copy_container()

copy_object()

has_errors

Tell if this target still presents errors. Will return False if it showed errors in the past but does not show them anymore.

irreparable

Tell if the target presents irreparable errors.

Check only the latest result. The “irreparable” situation may have been temporary, for example if a rawx went down then up again.

latest_error_result()

time_in_error()

Tell for how long this target has shown errors.

Return type:tuple Returns:the duration (in seconds) since we detected an error, and the number of consecutive error confirmations.

type

Tell which type of item this object targets.

oio.crawler.integrity.main()

Main function for legacy integrity crawler.

oio.crawler.integrity.run_indefinitely(checker, entries=None, rate_limiter=None, pause_between_passes=0.0)

oio.crawler.integrity.run_once(checker, entries=None, rate_limiter=None)

oio.crawler.storage_tierer module

class oio.crawler.storage_tierer.StorageTierer(conf, **kwargs)

Bases: oio.common.daemon.Daemon

run(*args, **kwargs)

class oio.crawler.storage_tierer.StorageTiererWorker(conf, logger)

Bases: object

change_policy(account, container, obj, version)

run()

safe_change_policy(account, container, obj, version)

Module contents