Trace Requests

OpenIO SDS provides a way to trace requests in the cluster.

openio cli
$ openio object create TEST_CONTAINER /etc/os-release --oio-account TEST_ACCOUNT
Jul 27 08:17:35 node1 OIO,OPENIO,meta1,1: 15404 8A9D access INF 172.17.0.123:6111 172.17.0.3:39100 M1_LINK 200 4402 154 DD83434968FE3D4D0815BEE837572D5B4669241A8156E4F59DB55A06788DD1CD 4E54A5BDA1EA1812B7DED90F964B1971 t=4362 OPENIO/TEST_ACCOUNT/TEST_CONTAINER/meta2/os-release|DD83434968FE3D4D0815BEE837572D5B4669241A8156E4F59DB55A06788DD1CD|meta2|0
Jul 27 08:17:35 node1 OIO,OPENIO,meta1,1: 15404 8A9D access INF 172.17.0.123:6111 172.17.0.4:40866 M1_LIST 200 301 154 DD83434968FE3D4D0815BEE837572D5B4669241A8156E4F59DB55A06788DD1CD 4E54A5BDA1EA1812B7DED90F964B1971 t=262 OPENIO|DD83434968FE3D4D0815BEE837572D5B4669241A8156E4F59DB55A06788DD1CD|meta2
Jul 27 08:17:35 node1 OIO,OPENIO,meta2,1: 14629 9E2C access INF 172.17.0.123:6121 172.17.0.4:6121 DB_USE 200 4089 67 DD83434968FE3D4D0815BEE837572D5B4669241A8156E4F59DB55A06788DD1CD 6B3821A4F1D175079D388D36CF0C3F7E t=4056 DD83434968FE3D4D0815BEE837572D5B4669241A8156E4F59DB55A06788DD1CD.1.meta2
Jul 27 08:17:35 node1 OIO,OPENIO,meta1,1: 15404 8A9D access INF 172.17.0.123:6111 172.17.0.2:52740 M1_LIST 200 283 154 DD83434968FE3D4D0815BEE837572D5B4669241A8156E4F59DB55A06788DD1CD 733843DB9638408A5861C0DD293FE1CB t=251 OPENIO|DD83434968FE3D4D0815BEE837572D5B4669241A8156E4F59DB55A06788DD1CD|meta2
Jul 27 08:17:35 node1 OIO,OPENIO,meta2,1: 14629 9E2C access INF 172.17.0.123:6121 172.17.0.2:6121 DB_USE 200 141 67 DD83434968FE3D4D0815BEE837572D5B4669241A8156E4F59DB55A06788DD1CD 733854551253CF0185909B0D0E0C0285 t=37 DD83434968FE3D4D0815BEE837572D5B4669241A8156E4F59DB55A06788DD1CD.1.meta2
Jul 27 08:17:35 node1 OIO,OPENIO,meta2,1: 14629 6B64 access INF 172.17.0.123:6121 172.17.0.4:37296 DB_VERS 200 5463 183 DD83434968FE3D4D0815BEE837572D5B4669241A8156E4F59DB55A06788DD1CD 6B38AC154B22F5DA982046D1B267D5D0 t=3022 DD83434968FE3D4D0815BEE837572D5B4669241A8156E4F59DB55A06788DD1CD.1.meta2
Jul 27 08:17:35 node1 OIO,OPENIO,meta2,1: 14629 6624 log WRN oio.m2v2 Missing sys.account or sys.user.name in database /var/lib/oio/sds/OPENIO/meta2-1/DD8/DD83434968FE3D4D0815BEE837572D5B4669241A8156E4F59DB55A06788DD1CD.1.meta2
Jul 27 08:17:35 node1 OIO,OPENIO,meta2,1: 14629 6624 access INF 172.17.0.123:6121 172.17.0.3:45442 DB_PIPEFROM 200 21921 67 DD83434968FE3D4D0815BEE837572D5B4669241A8156E4F59DB55A06788DD1CD 2539C7CAC8C2FCF1AD5A6D27406EC948 t=20662 DD83434968FE3D4D0815BEE837572D5B4669241A8156E4F59DB55A06788DD1CD.1.meta2 DD83434968FE3D4D0815BEE837572D5B4669241A8156E4F59DB55A06788DD1CD.1.meta2|172.17.0.4:6121
Jul 27 08:17:35 node1 OIO,OPENIO,meta2,1: 14629 6B64 access INF 172.17.0.123:6121 172.17.0.4:37308 DB_REPLI 200 711 67 DD83434968FE3D4D0815BEE837572D5B4669241A8156E4F59DB55A06788DD1CD 4E54A5BDA1EA1812B7DED90F964B1971 t=663 DD83434968FE3D4D0815BEE837572D5B4669241A8156E4F59DB55A06788DD1CD.1.meta2
Jul 27 08:17:35 node1 OIO,OPENIO,oioproxy,1: 14649 FAF8 access INF 172.17.0.123:6006 172.17.0.3:38886 POST 200 107665 844 DD83434968FE3D4D0815BEE837572D5B4669241A8156E4F59DB55A06788DD1CD 4E54A5BDA1EA1812B7DED90F964B1971 /v3.0/OPENIO/content/prepare?acct=TEST_ACCOUNT&ref=TEST_CONTAINER&path=os-release t=107565
Jul 27 08:17:35 node1 OIO,OPENIO,meta2,1: 14629 6B64 access INF 172.17.0.123:6121 172.17.0.4:37334 DB_REPLI 200 880 67 DD83434968FE3D4D0815BEE837572D5B4669241A8156E4F59DB55A06788DD1CD 4E54A5BDA1EA1812B7DED90F964B1971 t=668 DD83434968FE3D4D0815BEE837572D5B4669241A8156E4F59DB55A06788DD1CD.1.meta2
Jul 27 08:17:35 node1 OIO,OPENIO,oioproxy,1: 14649 FB58 access INF 172.17.0.123:6006 172.17.0.3:39078 POST 204 8450 0 DD83434968FE3D4D0815BEE837572D5B4669241A8156E4F59DB55A06788DD1CD 4E54A5BDA1EA1812B7DED90F964B1971 /v3.0/OPENIO/content/create?acct=TEST_ACCOUNT&ref=TEST_CONTAINER&path=os-release t=8403
Jul 27 08:17:35 node1 OIO,OPENIO,rawx,1 14685 140695335397120 access INF 172.17.0.123:6201 172.17.0.3:47000 PUT 201 16849 1194 DD83434968FE3D4D0815BEE837572D5B4669241A8156E4F59DB55A06788DD1CD 4E54A5BDA1EA1812B7DED90F964B1971 /4E54A5E9BBBC97D1168E36E3C5E73F8D9511D93F5D4CA9078CB8457597A1AEB7

Here DD83434968FE3D4D0815BEE837572D5B4669241A8156E4F59DB55A06788DD1CD is the container ID of the object creation.

aws cli
$ aws --endpoint-url http://172.17.0.123:6007 --no-verify-ssl s3 mb s3://bucket && aws --endpoint-url http://172.17.0.123:6007 --no-verify-ssl s3 cp /etc/os-release s3://bucket
Jul 27 08:35:39 node1 OIO,OPENIO,meta1,1: 20515 FAB4 access INF 172.17.0.123:6111 172.17.0.3:58960 M1_LINK 200 2160 156 0B61014329490527829B95D2F540A64D0935419CB9EB0750D32CB6F5E05D196C tx2e76d2532dfa4e12bb85c-005b5ad95b t=2132 OPENIO/AUTH_demo/bucket/meta2|0B61014329490527829B95D2F540A64D0935419CB9EB0750D32CB6F5E05D196C|meta2|0
Jul 27 08:35:39 node1 OIO,OPENIO,meta1,1: 20515 FAB4 access INF 172.17.0.123:6111 172.17.0.2:44304 M1_LIST 200 285 156 0B61014329490527829B95D2F540A64D0935419CB9EB0750D32CB6F5E05D196C tx2e76d2532dfa4e12bb85c-005b5ad95b t=247 OPENIO|0B61014329490527829B95D2F540A64D0935419CB9EB0750D32CB6F5E05D196C|meta2
Jul 27 08:35:39 node1 OIO,OPENIO,meta1,1: 20515 FAB4 access INF 172.17.0.123:6111 172.17.0.2:44310 M1_LIST 200 266 154 0B61014329490527829B95D2F540A64D0935419CB9EB0750D32CB6F5E05D196C 73388D503EDB670D7C10B50B8E3E272D t=238 OPENIO|0B61014329490527829B95D2F540A64D0935419CB9EB0750D32CB6F5E05D196C|meta2
Jul 27 08:35:39 node1 OIO,OPENIO,meta1,1: 20515 FAB4 access INF 172.17.0.123:6111 172.17.0.3:58992 M1_LIST 200 300 154 0B61014329490527829B95D2F540A64D0935419CB9EB0750D32CB6F5E05D196C 73382B764B2DCAFD4F725385E537781D t=269 OPENIO|0B61014329490527829B95D2F540A64D0935419CB9EB0750D32CB6F5E05D196C|meta2
Jul 27 08:35:39 node1 OIO,OPENIO,meta2,1: 20425 DFD2 access INF 172.17.0.123:6121 172.17.0.2:6121 DB_USE 200 5695 67 0B61014329490527829B95D2F540A64D0935419CB9EB0750D32CB6F5E05D196C 73382B764B2DCAFD4F725385E537781D t=5627 0B61014329490527829B95D2F540A64D0935419CB9EB0750D32CB6F5E05D196C.1.meta2
Jul 27 08:35:39 node1 OIO,OPENIO,meta1,1: 20515 FAB4 access INF 172.17.0.123:6111 172.17.0.4:60764 M1_LIST 200 289 154 0B61014329490527829B95D2F540A64D0935419CB9EB0750D32CB6F5E05D196C 6B38684B3667AD651236D474DDDC8E03 t=252 OPENIO|0B61014329490527829B95D2F540A64D0935419CB9EB0750D32CB6F5E05D196C|meta2
Jul 27 08:35:39 node1 OIO,OPENIO,meta2,1: 20425 DFD2 access INF 172.17.0.123:6121 172.17.0.4:6121 DB_USE 200 57 67 0B61014329490527829B95D2F540A64D0935419CB9EB0750D32CB6F5E05D196C 6B38E4FF5AA8379D5C04677DBF28286F t=30 0B61014329490527829B95D2F540A64D0935419CB9EB0750D32CB6F5E05D196C.1.meta2
Jul 27 08:35:39 node1 OIO,OPENIO,meta2,1: 20425 DF82 access INF 172.17.0.123:6121 172.17.0.2:44688 DB_VERS 200 3271 183 0B61014329490527829B95D2F540A64D0935419CB9EB0750D32CB6F5E05D196C 73383765478418379A400752DB09A814 t=2202 0B61014329490527829B95D2F540A64D0935419CB9EB0750D32CB6F5E05D196C.1.meta2
Jul 27 08:35:39 node1 OIO,OPENIO,meta2,1: 20425 DF82 access INF 172.17.0.123:6121 172.17.0.4:57140 DB_VERS 200 263 183 0B61014329490527829B95D2F540A64D0935419CB9EB0750D32CB6F5E05D196C 6B38D34367A8986718DFE25AC8EA09A1 t=96 0B61014329490527829B95D2F540A64D0935419CB9EB0750D32CB6F5E05D196C.1.meta2
Jul 27 08:35:39 node1 OIO,OPENIO,meta2,1: 20425 DF82 access INF 172.17.0.123:6121 172.17.0.4:57168 DB_REPLI 200 525 69 0B61014329490527829B95D2F540A64D0935419CB9EB0750D32CB6F5E05D196C tx2e76d2532dfa4e12bb85c-005b5ad95b t=485 0B61014329490527829B95D2F540A64D0935419CB9EB0750D32CB6F5E05D196C.1.meta2
Jul 27 08:35:39 node1 OIO,OPENIO,oioproxy,1: 20536 674C access INF 172.17.0.123:6006 172.17.0.3:58744 POST 201 99967 0 0B61014329490527829B95D2F540A64D0935419CB9EB0750D32CB6F5E05D196C tx2e76d2532dfa4e12bb85c-005b5ad95b /v3.0/OPENIO/container/create?acct=AUTH_demo&ref=bucket t=99932
Jul 27 08:35:39 node1 OIO,OPENIO,meta2,1: 20425 DF82 access INF 172.17.0.123:6121 172.17.0.4:57176 DB_REPLI 200 459 69 0B61014329490527829B95D2F540A64D0935419CB9EB0750D32CB6F5E05D196C tx2e76d2532dfa4e12bb85c-005b5ad95b t=422 0B61014329490527829B95D2F540A64D0935419CB9EB0750D32CB6F5E05D196C.1.meta2
Jul 27 08:35:39 node1 OIO,OPENIO,oioproxy,1: 20536 3B4D access INF 172.17.0.123:6006 172.17.0.3:58928 POST 200 2278 2 0B61014329490527829B95D2F540A64D0935419CB9EB0750D32CB6F5E05D196C tx2e76d2532dfa4e12bb85c-005b5ad95b /v3.0/OPENIO/container/set_properties?acct=AUTH_demo&ref=bucket t=2247
Jul 27 08:35:39 node1 OIO,OPENIO,meta2,1: 20425 DF82 access INF 172.17.0.123:6121 172.17.0.3:37146 DB_PGET 200 161 745 0B61014329490527829B95D2F540A64D0935419CB9EB0750D32CB6F5E05D196C tx92f6fddb4cd94baf9413b-005b5ad95b t=120 0B61014329490527829B95D2F540A64D0935419CB9EB0750D32CB6F5E05D196C.1.meta2
Jul 27 08:35:39 node1 OIO,OPENIO,oioproxy,1: 20536 674C access INF 172.17.0.123:6006 172.17.0.3:58990 POST 200 539 691 0B61014329490527829B95D2F540A64D0935419CB9EB0750D32CB6F5E05D196C tx92f6fddb4cd94baf9413b-005b5ad95b /v3.0/OPENIO/container/get_properties?acct=AUTH_demo&ref=bucket t=484
Jul 27 08:35:39 node1 OIO,OPENIO,oioproxy,1: 20536 3B4D access INF 172.17.0.123:6006 172.17.0.3:58994 POST 200 737 844 0B61014329490527829B95D2F540A64D0935419CB9EB0750D32CB6F5E05D196C tx92f6fddb4cd94baf9413b-005b5ad95b /v3.0/OPENIO/content/prepare?acct=AUTH_demo&ref=bucket&path=os-release t=698
Jul 27 08:35:40 node1 OIO,OPENIO,meta2,1: 20425 DF82 access INF 172.17.0.123:6121 172.17.0.4:57258 DB_REPLI 200 720 69 0B61014329490527829B95D2F540A64D0935419CB9EB0750D32CB6F5E05D196C tx92f6fddb4cd94baf9413b-005b5ad95b t=681 0B61014329490527829B95D2F540A64D0935419CB9EB0750D32CB6F5E05D196C.1.meta2
Jul 27 08:35:40 node1 OIO,OPENIO,oioproxy,1: 20536 3B4D access INF 172.17.0.123:6006 172.17.0.3:59008 POST 204 3748 0 0B61014329490527829B95D2F540A64D0935419CB9EB0750D32CB6F5E05D196C tx92f6fddb4cd94baf9413b-005b5ad95b /v3.0/OPENIO/content/create?acct=AUTH_demo&ref=bucket&path=os-release t=3700
Jul 27 08:35:39 node1 OIO,OPENIO,rawx,1 20272 140553240884992 access INF 172.17.0.123:6201 172.17.0.3:38700 PUT 201 11731 1185 0B61014329490527829B95D2F540A64D0935419CB9EB0750D32CB6F5E05D196C tx92f6fddb4cd94baf9413b-005b5ad95b /B3E58CCAEB6ECC61581C4FA6AB406B696F63E6115F81B8A5477185F1CF45CA2D

In addition to the ID OpenIO SDS, the ID AWD S3 is trackable too (here: tx92f6fddb4cd94baf9413b-005b5ad95b).