No Reponse body when using TFDS with AWS S3 buckets

[neel04]

Hi, I'm using TFDS to stream a dataset from TFrecords stored on a s3 path s3://..... I can confirm they exist, and are non-empty (few terabytes of data in total). This is my log:

2022-07-23 21:47:21.762290: I tensorflow/core/platform/s3/aws_logging.cc:84] Attempting to acquire curl connection.
2022-07-23 21:47:21.762296: I tensorflow/core/platform/s3/aws_logging.cc:71] Connection has been released. Continuing.
2022-07-23 21:47:21.762302: I tensorflow/core/platform/s3/aws_logging.cc:84] Returning connection handle 0x7f616800a000
2022-07-23 21:47:21.762308: I tensorflow/core/platform/s3/aws_logging.cc:84] Obtained connection handle 0x7f616800a000
2022-07-23 21:47:21.773612: I tensorflow/core/platform/s3/aws_logging.cc:84] HTTP/1.1 404 Not Found

2022-07-23 21:47:21.773625: I tensorflow/core/platform/s3/aws_logging.cc:84] x-amz-request-id: 65SDTM7WR91XR7HW

2022-07-23 21:47:21.773637: I tensorflow/core/platform/s3/aws_logging.cc:84] x-amz-id-2: r4e2bY2f07y0QWgZsggOUh48tBowli6SjLZI2PNCaezgkVmbkIoU1VkoevrlLjZEHdCHWCamiwo=

2022-07-23 21:47:21.773645: I tensorflow/core/platform/s3/aws_logging.cc:84] Content-Type: application/xml

2022-07-23 21:47:21.773653: I tensorflow/core/platform/s3/aws_logging.cc:84] Date: Sat, 23 Jul 2022 21:47:21 GMT

2022-07-23 21:47:21.773663: I tensorflow/core/platform/s3/aws_logging.cc:84] Server: AmazonS3

2022-07-23 21:47:21.773671: I tensorflow/core/platform/s3/aws_logging.cc:84]

2022-07-23 21:47:21.773684: I tensorflow/core/platform/s3/aws_logging.cc:84] Returned http response code 404
2022-07-23 21:47:21.773691: I tensorflow/core/platform/s3/aws_logging.cc:84] Returned content type application/xml
2022-07-23 21:47:21.773697: I tensorflow/core/platform/s3/aws_logging.cc:84] Releasing curl handle 0x7f616800a000
2022-07-23 21:47:21.773706: I tensorflow/core/platform/s3/aws_logging.cc:84] Releasing curl handle 0x7f616800a000
2022-07-23 21:47:21.773712: I tensorflow/core/platform/s3/aws_logging.cc:84] Notified waiting threads.
2022-07-23 21:47:21.773725: I tensorflow/core/platform/s3/aws_logging.cc:84] Request returned error. Attempting to generate appropriate error codes from response
2022-07-23 21:47:21.773735: E tensorflow/core/platform/s3/aws_logging.cc:77] HTTP response code: 404
Exception name:
Error message: No response body.
5 response headers:
content-type : application/xml
date : Sat, 23 Jul 2022 21:47:21 GMT
server : AmazonS3
x-amz-id-2 : r4e2bY2f07y0QWgZsggOUh48tBowli6SjLZI2PNCaezgkVmbkIoU1VkoevrlLjZEHdCHWCamiwo=
x-amz-request-id : 65SDTM7WR91XR7HW
2022-07-23 21:47:21.773750: W tensorflow/core/platform/s3/aws_logging.cc:74] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
2022-07-23 21:47:21.773760: I tensorflow/core/platform/s3/aws_logging.cc:84] Server time is Sat, 23 Jul 2022 21:47:21 GMT, while client time is Sat, 23 Jul 2022 21:47:21 GMT
2022-07-23 21:47:21.773775: E tensorflow/core/platform/s3/aws_logging.cc:77] Transfer handle [F66BACBE-4F50-4351-B448-BF8064BA7DA7] Failed to get download parts information for object in Bucket: [s-laion] with Key: [ssd-videos/new_tfrecs/bdd100k/1.0.0/bdd100k-train.tfrecord-00056-of-47044] HTTP response code: 404
Exception name:
Error message: No response body.
5 response headers:
content-type : application/xml
date : Sat, 23 Jul 2022 21:47:21 GMT
server : AmazonS3
x-amz-id-2 : r4e2bY2f07y0QWgZsggOUh48tBowli6SjLZI2PNCaezgkVmbkIoU1VkoevrlLjZEHdCHWCamiwo=
x-amz-request-id : 65SDTM7WR91XR7HW
2022-07-23 21:47:21.773786: I tensorflow/core/platform/s3/aws_logging.cc:71] Transfer handle ID [F66BACBE-4F50-4351-B448-BF8064BA7DA7] Updated handle status from [NOT_STARTED] to [FAILED].

If anyone has any idea how to resolve this, please help! 🙏

[Conchylicultor]

It would be helpful to get a reproductible minimal code snippet which correspond to the above stacktrace which uses only TF.

[Conchylicultor]

I also think this is a https://github.com/tensorflow/tensorflow issue. Not a TFDS issue.

[neel04]

I haven't fully triangulated the bug yet, but its definitely with TFDS. The problem is that somehow the internal modules which glob the filepaths on calling tfds.load() are NOT globbing the latest filepaths - I'm not familiar with the internals but I assume it might be caching it somewhere due to the dataset name provided as a parameter staying the same.

I'll do a deep dive in the code, but if you happen to have any idea about it (or a code snippet you think might be troublesome) that would be nice!

[neel04]

For a reproducible snippet, you'd need an S3 bucket with dummy files, say named:

random-test.tfrecord-00000-of-00001
random-train.tfrecord-00000-of-00001

Perform a:

import tensorflow_datasets as tfds
data = tfds.load(name='random', ... ) #S3 path goes here

Then simply delete those dummy TFRecords for something different:

random-test.tfrecord-00000-of-00002
random-test.tfrecord-00001-of-00002
random-train.tfrecord-00000-of-00002
random-train.tfrecord-00001-of-00002

Because the filepath changes (00002 instead of 00001) the paths TFDS got from my dir were the same, but they'd been updated with more files - so there were more TFRecords as a result x-of-y the value of y changed - and not by much, so its a subtle change..

[neel04]

I think the reason is that once you update the dataset, one has to NOT use the same version 1.0.0 like I did. If this indeed reproduces the issue, then it would be nice if a warning can be put!