C SDK

This sections describes how to install and use the OpenIO SDS C SDK.

Install

Last version directly from Github:

git clone git://github.com/open-io/oio-sds.git

Basic Concepts

An Object Storage API differs from a conventional filesystem: instead of directories and files, you manipulate containers where you store objects. A container can hold millions of objects.

Note that there is no hierarchy notion with containers: you cannot nest a container within an other, however you can emulate a nested folder structure with a naming convention for your objects. For example with an object name such as “documents/work/2015/finance/report.pdf” you can retrieve your files using the appropriate “path” prefix.

In this SDK, you manipulate Container and Object, all you need is to initialize an oio_sds_s structure. To initialize it, you need the namespace name. Endpoint URLs and all other configuration options will be loaded from /etc/oio/sds.conf.d/NAMESPACE (or ~/.oio/sds.conf if you have deployed from source).

struct oio_sds_s *client = NULL;
struct oio_error_s *err = NULL;
err = oio_sds_init(&client, NAMESPACE);
g_assert_no_error((GError*)err);

All of the sample code that follows assumes that you have correctly initialized an oio_sds_s structure.

Accounts

Accounts are a convenient way to manage the storage containers. Containers always belong to a specific Account.

You can list containers for a specified Account. Accounts are also a great way to track your storage usage (Total bytes used, Total number of objects, Total number of containers).

The API lets you set and retrieve your own metadata on accounts.

Creating a Container

Start by creating a container:

struct oio_url_s *url = oio_url_empty();
g_assert_nonnull(url);
oio_url_set(url, OIOURL_NS, NAMESPACE);
oio_url_set(url, OIOURL_ACCOUNT, ACCOUNT);
oio_url_set(url, OIOURL_USER, CONTAINER);

err = oio_sds_create(client, url);
oio_url_clean(url);
g_assert_no_error((GError*)err);

Note that if you try to create a container more than once with the same name, the request is ignored

Showing the description of a Container

To show the description of a container:

struct oio_url_s *url = oio_url_empty();
g_assert_nonnull(url);
oio_url_set(url, OIOURL_NS, NAMESPACE);
oio_url_set(url, OIOURL_ACCOUNT, ACCOUNT);
oio_url_set(url, OIOURL_USER, CONTAINER);

void _print_element (void *ctx, const char *key, const char *value) {
    (void) ctx;
    g_print("\"%s\": \"%s\", ", key, value);
}

err = oio_sds_get_container_properties(client, url, print_element, NULL);
oio_url_clean(url);
g_assert_no_error((GError*)err);

Note that if you try to get a non-existent container, a oio_error_s is returned.

Storing Objects

This example creates an object named object.txt with the data provided, in the container CONTAINER:

struct oio_url_s *url = oio_url_empty();
g_assert_nonnull(url);
oio_url_set(url, OIOURL_NS, NAMESPACE);
oio_url_set(url, OIOURL_ACCOUNT, ACCOUNT);
oio_url_set(url, OIOURL_USER, CONTAINER);
oio_url_set(url, OIOURL_PATH, "object.txt");

struct oio_sds_ul_dst_s ul_dst = OIO_SDS_UPLOAD_DST_INIT;
ul_dst.url = url;
gchar data[] = "Content example";

err = oio_sds_upload_from_buffer(client, &ul_dst, data, sizeof(data));
oio_url_clean(url);
g_assert_no_error((GError*)err);

Note that if you try to store an object in a non-existent container, a oio_error_s is returned.

Retrieving Object

Note that if you try to retrieve a non-existent object, a oio_error_s is returned.

This sample code stores an object and retrieves it using the different parameters.

struct oio_url_s *url = oio_url_empty();
g_assert_nonnull(url);
oio_url_set(url, OIOURL_NS, NAMESPACE);
oio_url_set(url, OIOURL_ACCOUNT, ACCOUNT);
oio_url_set(url, OIOURL_USER, CONTAINER);
oio_url_set(url, OIOURL_PATH, "object.txt");

guchar data[1024];
struct oio_sds_dl_src_s src = { .url = url, .ranges = NULL };
struct oio_sds_dl_dst_s dst = {
    .type = OIO_DL_DST_BUFFER,
    .data = {.buffer = {.ptr = data, .length=sizeof(data)}}
};

err = oio_sds_download(client, &src, &dst);
oio_url_clean(url);
g_assert_no_error((GError*)err);
g_print("%s", data);

Deleting Objects

Example:

struct oio_url_s *url = oio_url_empty();
g_assert_nonnull(url);
oio_url_set(url, OIOURL_NS, NAMESPACE);
oio_url_set(url, OIOURL_ACCOUNT, ACCOUNT);
oio_url_set(url, OIOURL_USER, CONTAINER);
oio_url_set(url, OIOURL_PATH, "object.txt");

err = oio_sds_delete(client, url);
oio_url_clean(url);
g_assert_no_error((GError*)err);

Note that if you try to delete a non-existent object, a oio_error_s is returned.

Containers and Objects Metadata

The Object Storage API lets you set and retrieve your own metadata on containers and objects.

struct oio_url_s *url = oio_url_empty();
g_assert_nonnull(url);
oio_url_set(url, OIOURL_NS, NAMESPACE);
oio_url_set(url, OIOURL_ACCOUNT, ACCOUNT);
oio_url_set(url, OIOURL_USER, CONTAINER);

void _print_element (void *ctx, const char *key, const char *value) {
    (void) ctx;
    g_print("\"%s\": \"%s\", ", key, value);
}

g_print("Metadata: {");
err = oio_sds_get_container_properties(client, url, _print_element, NULL);
oio_url_clean(url);
g_assert_no_error((GError*)err);
g_print("}\n");

It should output an empty dict, unless you added metadata to this container.

struct oio_url_s *url = oio_url_empty();
g_assert_nonnull(url);
oio_url_set(url, OIOURL_NS, NAMESPACE);
oio_url_set(url, OIOURL_ACCOUNT, ACCOUNT);
oio_url_set(url, OIOURL_USER, CONTAINER);

const gchar* const properties[5] = {"color", "blue", "flag", "true", NULL};

err = oio_sds_set_container_properties(client, url, properties);
g_assert_no_error((GError*)err);

void _print_element (void *ctx, const char *key, const char *value) {
    (void) ctx;
    g_print("\"%s\": \"%s\", ", key, value);
}

g_print("Metadata: {");
err = oio_sds_get_container_properties(client, url, _print_element, NULL);
oio_url_clean(url);
g_assert_no_error((GError*)err);
g_print("}\n");

It should now output:

Metadata: {"color": "blue", "flag": "true", }

This is very similar for objects. You can use the methods oio_sds_get_content_properties() and oio_sds_set_content_properties().

Listing Objects

struct oio_url_s *url = oio_url_empty();
g_assert_nonnull(url);
oio_url_set(url, OIOURL_NS, NAMESPACE);
oio_url_set(url, OIOURL_ACCOUNT, ACCOUNT);
oio_url_set(url, OIOURL_USER, CONTAINER);

struct oio_sds_list_param_s list_in = {
    .url = url,
    .prefix = NULL, .marker = NULL, .end = NULL, .delimiter = 0, .max_items = 0,
    .flag_allversions = 1, .flag_nodeleted = 1, .flag_properties = 1
};

int _print_item (void *ctx, const struct oio_sds_list_item_s *item) {
    (void) ctx;
    g_print("%s\n", item->name);
    return 0;
}
struct oio_sds_list_listener_s list_out = {
    .ctx = NULL,
    .on_item = _print_item, .on_prefix = NULL, .on_bound = NULL,
};

err = oio_sds_list(client, &list_in, &list_out);
oio_url_clean(url);
g_assert_no_error((GError*)err);

This returns a list of objects stored in the container.

Since containers can hold millions of objects, there are several methods to filter the results.

Filters:

  • marker - Indicates where to start the listing from.
  • end - Indicates where to stop the listing.
  • prefix - If set, the listing only includes objects whose name begin with its value.
  • delimiter - If set, excludes the objects whose name contains its value. delimiter only takes a single character.
  • max_items - Indicates the maximum number of objects to return in the listing.

To illustrate these features, we create some objects in a container:

struct oio_url_s *url = oio_url_empty();
g_assert_nonnull(url);
oio_url_set(url, OIOURL_NS, NAMESPACE);
oio_url_set(url, OIOURL_ACCOUNT, ACCOUNT);
oio_url_set(url, OIOURL_USER, CONTAINER);

err = oio_sds_create(client, url);
g_assert_no_error((GError*)err);

struct oio_sds_ul_dst_s dst = OIO_SDS_UPLOAD_DST_INIT;
dst.url = url;
gchar data[] = "sample";

gchar *name = NULL;
for (int i = 0; i < 5; i++) {
    name = g_strdup_printf("object%d", i);
    oio_url_set(url, OIOURL_PATH, name);
    err = oio_sds_upload_from_buffer(client, &dst, data, sizeof(data));
    g_free(name);
    g_assert_no_error((GError*)err);
}

for (gchar id = 'a'; id <= 'd'; id++) {
    name = g_strdup_printf("foo/%c", id);
    oio_url_set(url, OIOURL_PATH, name);
    err = oio_sds_upload_from_buffer(client, &dst, data, sizeof(data));
    g_free(name);
    g_assert_no_error((GError*)err);
}

oio_url_clean(url);

First list all the objects:

struct oio_url_s *url = oio_url_empty();
g_assert_nonnull(url);
oio_url_set(url, OIOURL_NS, NAMESPACE);
oio_url_set(url, OIOURL_ACCOUNT, ACCOUNT);
oio_url_set(url, OIOURL_USER, CONTAINER);

struct oio_sds_list_param_s list_in = {
    .url = url,
    .prefix = NULL, .marker = NULL, .end = NULL, .delimiter = 0, .max_items = 0,
    .flag_allversions = 1, .flag_nodeleted = 1, .flag_properties = 1
};

int _print_item (void *ctx, const struct oio_sds_list_item_s *item) {
    (void) ctx;
    g_print("%s\n", item->name);
    return 0;
}
struct oio_sds_list_listener_s list_out = {
    .ctx = NULL,
    .on_item = _print_item, .on_prefix = NULL, .on_bound = NULL,
};

err = oio_sds_list(client, &list_in, &list_out);
oio_url_clean(url);
g_assert_no_error((GError*)err);

It should output:

object4
object3
object2
object1
object0
foo/d
foo/c
foo/b
foo/a

Then let’s use the paginating features:

struct oio_url_s *url = oio_url_empty();
g_assert_nonnull(url);
oio_url_set(url, OIOURL_NS, NAMESPACE);
oio_url_set(url, OIOURL_ACCOUNT, ACCOUNT);
oio_url_set(url, OIOURL_USER, CONTAINER);

struct oio_sds_list_param_s list_in = {
    .url = url,
    .prefix = NULL, .marker = NULL, .end = NULL, .delimiter = 0, .max_items = 4,
    .flag_allversions = 1, .flag_nodeleted = 1, .flag_properties = 1
};

gboolean save_marker;
gchar marker[16];
int _print_item (void *ctx, const struct oio_sds_list_item_s *item) {
    (void) ctx;
    g_print("%s, ", item->name);
    if (save_marker) {
        g_strlcpy(marker, item->name, 16);
        save_marker = FALSE;
    }
    return 0;
}
struct oio_sds_list_listener_s list_out = {
    .ctx = NULL,
    .on_item = _print_item, .on_prefix = NULL, .on_bound = NULL,
};

marker[0] = '\0';
do {
    g_print("Objects: [");
    save_marker = TRUE;
    list_in.marker = marker;
    err = oio_sds_list(client, &list_in, &list_out);
    g_assert_no_error((GError*)err);
    g_print("]\n");
} while (list_out.out_count);

oio_url_clean(url);

Here is the result:

Objects: [foo/d, foo/c, foo/b, foo/a, ]
Objects: [object3, object2, object1, object0, ]
Objects: [object4, ]
Objects: []

How to use the prefix parameter:

struct oio_url_s *url = oio_url_empty();
g_assert_nonnull(url);
oio_url_set(url, OIOURL_NS, NAMESPACE);
oio_url_set(url, OIOURL_ACCOUNT, ACCOUNT);
oio_url_set(url, OIOURL_USER, CONTAINER);

struct oio_sds_list_param_s list_in = {
    .url = url,
    .prefix = "foo", .marker = NULL, .end = NULL, .delimiter = 0, .max_items = 0,
    .flag_allversions = 1, .flag_nodeleted = 1, .flag_properties = 1
};

int _print_item (void *ctx, const struct oio_sds_list_item_s *item) {
    (void) ctx;
    g_print("%s, ", item->name);
    return 0;
}
struct oio_sds_list_listener_s list_out = {
    .ctx = NULL,
    .on_item = _print_item, .on_prefix = NULL, .on_bound = NULL,
};

g_print("Objects: [");
err = oio_sds_list(client, &list_in, &list_out);
oio_url_clean(url);
g_assert_no_error((GError*)err);
g_print("]\n");

This only outputs the objects starting with “foo”:

Objects: [foo/d, foo/c, foo/b, foo/a, ]

How to use the delimiter parameter:

struct oio_url_s *url = oio_url_empty();
g_assert_nonnull(url);
oio_url_set(url, OIOURL_NS, NAMESPACE);
oio_url_set(url, OIOURL_ACCOUNT, ACCOUNT);
oio_url_set(url, OIOURL_USER, CONTAINER);

struct oio_sds_list_param_s list_in = {
    .url = url,
    .prefix = NULL, .marker = NULL, .end = NULL, .delimiter = '/', .max_items = 0,
    .flag_allversions = 1, .flag_nodeleted = 1, .flag_properties = 1
};

int _print_item (void *ctx, const struct oio_sds_list_item_s *item) {
      (void) ctx;
      g_print("%s, ", item->name);
      return 0;
  }
struct oio_sds_list_listener_s list_out = {
    .ctx = NULL,
    .on_item = _print_item, .on_prefix = NULL, .on_bound = NULL,
};

g_print("Objects: [");
err = oio_sds_list(client, &list_in, &list_out);
oio_url_clean(url);
g_assert_no_error((GError*)err);
g_print("]\n");

This excludes all the objects in the nested foo folder.

Objects: [object4, object3, object2, object1, object0, ]

Note that if you try to list a non-existent container, a oio_error_s is returned.

Deleting Containers

There is several options to delete containers. Example:

struct oio_url_s *url = oio_url_empty();
g_assert_nonnull(url);
oio_url_set(url, OIOURL_NS, NAMESPACE);
oio_url_set(url, OIOURL_ACCOUNT, ACCOUNT);
oio_url_set(url, OIOURL_USER, CONTAINER);

err = oio_sds_delete_container(client, url);
g_assert_no_error((GError*)err);
oio_url_clean(url);

You can not delete a container if it still holds objects, if you try to do so a oio_error_s is returned.

Note that if you try to delete a non-existent container, a oio_error_s is returned.