Storage Policies

Storage Policies are the way to describe different storage tiers in your storage platform.

Each storage policy must at least contain a Data Security policy, and can be used to target only certain pools.

Storage policies are defined for the Conscience service, and are thus available at the Namespace level.

Note

By default, the configuration file can be found at /etc/oio/sds/[NS]/conscience-X/conscience-X-policies.conf

Configuration

Suppose we have configured the following pool in /etc/oio/sds/[NS]/conscience-X/conscience-X-services.conf

[pool:rawx21]
targets=2,rawx-site1,rawx;1,rawx-site2,rawx

We can then define a storage policy that uses this pool to replicate chunks 3 times on 2 different sites following the 2+1 model.

[STORAGE_POLICIES]
DUPONETHREE=NONE:THREECOPIES
DUPONETHREE_MULTISITE=rawx21:THREECOPIES

As shown in the example above, a custom storage policy has been created for multi-site replication.

See also: Conscience.

Data Security

The Data Security describes the way an object is stored on the storage pool.

Each data security policy is derived from one of the supported security types. For the moment being, these are:

  • plain replication security (replicated data chunks)
  • ec erasure coding security (data chunks + parity chunks)

By default, you have 3 data security policies available:

[DATA_SECURITY]
THREECOPIES=plain/distance=1,nb_copy=3
TWOCOPIES=plain/distance=1,nb_copy=2
ERASURECODE=ec/k=6,m=3,algo=liberasurecode_rs_vand,distance=1

The policies defined above can be interpreted as the following:

  • 2 replication policies (THREECOPIES/TWOCOPIES for 2x/3x replication),
  • a 6+3 Erasure Coding policy (6 data chunks + 3 parity chunks using Reed Solomon with liberasurecode).

You can add more data security policies on top of the existing ones, or even alter the ones provided by default. Please note that it is not recommended to alter a data security entry after objects have already been created using the corresponding Storage Policy, as it may result in data loss.

Options
Option Description
nb_copy replication only: defines the number of copy to store
distance defines the minimum distance between chunks to ensure security
algo erasure coding only: defines the erasure coding algorithm to use
k erasure coding only: defines the number of data chunks
m erasure coding only: defines the number of parity chunks

See also: Erasure Coding section.

Usage

When an object is pushed, the storage policy is chosen in the following order:

  • Object-level: when the Storage Policy is explicitely specified when pushing
  • Container-level: when the container where the object is created specifies a Storage Policy to use
  • Namespace-level: default behavior, will use the policy defined in /etc/oio/sds/[NS]/conscience-X/conscience-X.conf

See also: OpenIO SDS Configuration.