Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DateTime fields need to be passed as strings or else we get an error #448

Open
ychaker opened this issue Jun 15, 2020 · 4 comments
Open

DateTime fields need to be passed as strings or else we get an error #448

ychaker opened this issue Jun 15, 2020 · 4 comments

Comments

@ychaker
Copy link

ychaker commented Jun 15, 2020

howdy!
First of all I'd like to thank everyone for their effort on this gem, I truly appreciate all of you work.

I'm new to the gem and DynamoDB overall, so excuse my noob question.

According to the README (and rspec files), I was under the impression that we could pass a DateTime object to a field that is configured to be of type :datetime, however when trying to set a field to a value like this:

Time.at(str&.to_i).utc.to_datetime

I get an error saying unsupported type, expected Hash, Array, Set, String, Numeric, IO, true, false, or nil, got DateTime

you can see the stack trace here:

{"name":"runtime","hostname":"docker-desktop","pid":14,"level":20,"time":"2020-06-12T22:27:48.211+00:00","v":0,"msg":"Data source: guidepoint"}
{"name":"runtime","hostname":"docker-desktop","pid":14,"level":20,"time":"2020-06-12T22:27:48.227+00:00","v":0,"msg":"list_tables | Request \"{}\" | Response \"{\\\"TableNames\\\":[]}\""}
{"name":"runtime","hostname":"docker-desktop","pid":14,"level":20,"time":"2020-06-12T22:27:48.227+00:00","v":0,"msg":"(14.98 ms) LIST TABLES"}
{"name":"runtime","hostname":"docker-desktop","pid":14,"level":20,"time":"2020-06-12T22:27:48.227+00:00","v":0,"msg":"(15.16 ms) CACHE TABLES"}
{"name":"runtime","hostname":"docker-desktop","pid":14,"level":30,"time":"2020-06-12T22:27:48.227+00:00","v":0,"msg":"Creating telematics_local_events table. This could take a while."}
{"name":"runtime","hostname":"docker-desktop","pid":14,"level":20,"time":"2020-06-12T22:27:48.243+00:00","v":0,"msg":"create_table | Request #<String \"{\\\"TableName\\\":\\\"telematics_local_events\\\",\\\"KeySchema\\\":[{\\\"AttributeName\\\":\\\"id\\\",\\\"KeyType\\\
":\\\"HASH\\\"},{\\\"AttributeName\\\":\\\"recorded_at\\\",\\\"KeyType\\\":\\\"RANGE\\\"}],\\\"AttributeDefinitions\\\":[{\\\"AttributeName\\\":\\\"id\\\",\\\"AttributeType\\\":\\\"S\\\"},{\\\"AttributeName\\\":\\\"recorded_at\\\",\\\"AttributeType\\\":\\\"N\\\"},{\\\"A
ttributeName\\\":\\\"device_id\\\",\\\"AttributeType\\\":\\\"S\\\"},{\\\"AttributeName\\\":\\\"event_type\\\",\\\"AttributeType\\\":\\\"S\\\"},{\\\"AttributeName\\\":\\\"provider\\\",\\\"AttributeType\\\":\\\"S\\\"},{\\\"AttributeName\\\":\\\"vin\\\",\\\"AttributeType\\
\":\\\"S\\\"}],\\\"BillingMode\\\":\\\"PROVISIONED\\\",\\\"ProvisionedThroughput\\\":{\\\"ReadCapacityUnits\\\":10,\\\"WriteCapacityUnits\\\":10},\\\"GlobalSecondaryIndexes\\\":[{\\\"IndexName\\\":\\\"telematics_local_events_index_device_id\\\",\\\"KeySchema\\\":[{\\\"A
ttributeName\\\":\\\"device_id\\\",\\\"KeyType\\\":\\\"HASH\\\"}],\\\"Projection\\\":{\\\"ProjectionType\\\":\\\"ALL\\\"},\\\"ProvisionedThroughput\\\":{\\\"ReadCapacityUnits\\\":100,\\\"WriteCapacityUnits\\\":20}},{\\\"IndexName\\\":\\\"telematics_local_events_index_ev
ent_type\\\",\\\"KeySchema\\\":[{\\\"AttributeName\\\":\\\"event_type\\\",\\\"KeyType\\\":\\\"HASH\\\"}],\\\"Projection\\\":{\\\"ProjectionType\\\":\\\"ALL\\\"},\\\"ProvisionedThroughput\\\":{\\\"ReadC\" ... (1722 bytes)> | Response #<String \"{\\\"TableDescription\\\":
{\\\"AttributeDefinitions\\\":[{\\\"AttributeName\\\":\\\"id\\\",\\\"AttributeType\\\":\\\"S\\\"},{\\\"AttributeName\\\":\\\"recorded_at\\\",\\\"AttributeType\\\":\\\"N\\\"},{\\\"AttributeName\\\":\\\"device_id\\\",\\\"AttributeType\\\":\\\"S\\\"},{\\\"AttributeName\\\"
:\\\"event_type\\\",\\\"AttributeType\\\":\\\"S\\\"},{\\\"AttributeName\\\":\\\"provider\\\",\\\"AttributeType\\\":\\\"S\\\"},{\\\"AttributeName\\\":\\\"vin\\\",\\\"AttributeType\\\":\\\"S\\\"}],\\\"TableName\\\":\\\"telematics_local_events\\\",\\\"KeySchema\\\":[{\\\"A
ttributeName\\\":\\\"id\\\",\\\"KeyType\\\":\\\"HASH\\\"},{\\\"AttributeName\\\":\\\"recorded_at\\\",\\\"KeyType\\\":\\\"RANGE\\\"}],\\\"TableStatus\\\":\\\"ACTIVE\\\",\\\"CreationDateTime\\\":1592000868.240,\\\"ProvisionedThroughput\\\":{\\\"LastIncreaseDateTime\\\":0.
000,\\\"LastDecreaseDateTime\\\":0.000,\\\"NumberOfDecreasesToday\\\":0,\\\"ReadCapacityUnits\\\":10,\\\"WriteCapacityUnits\\\":10},\\\"TableSizeBytes\\\":0,\\\"ItemCount\\\":0,\\\"TableArn\\\":\\\"arn:aws:dynamodb:ddblocal:000000000000:table/telematics_local_events\\\"
,\\\"GlobalSecondaryIndexes\\\":[{\\\"IndexName\\\":\\\"telematics_local_events_index_device_id\\\",\\\"KeySchema\\\":[{\\\"AttributeName\\\":\\\"device_id\\\",\\\"KeyType\\\":\\\"HASH\\\"}],\\\"Projection\\\":{\\\"ProjectionType\\\":\\\"ALL\\\"},\\\"IndexStatus\\\"\" .
.. (2888 bytes)>"}
{"name":"runtime","hostname":"docker-desktop","pid":14,"level":20,"time":"2020-06-12T22:27:48.243+00:00","v":0,"msg":"(15.86 ms) CREATE TABLE"}
/var/task/vendor/bundle/ruby/2.7.0/gems/dynamoid-3.5.0/lib/dynamoid/adapter.rb:158: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
/var/task/vendor/bundle/ruby/2.7.0/gems/dynamoid-3.5.0/lib/dynamoid/adapter_plugin/aws_sdk_v3.rb:281: warning: The called method `update_time_to_live' is defined here
{"name":"runtime","hostname":"docker-desktop","pid":14,"level":20,"time":"2020-06-12T22:27:48.260+00:00","v":0,"msg":"update_time_to_live | Request \"{\\\"TableName\\\":\\\"telematics_local_events\\\",\\\"TimeToLiveSpecification\\\":{\\\"AttributeName\\\":\\\"expires_at
\\\",\\\"Enabled\\\":true}}\" | Response \"{\\\"TimeToLiveSpecification\\\":{\\\"Enabled\\\":true,\\\"AttributeName\\\":\\\"expires_at\\\"}}\""}
{"name":"runtime","hostname":"docker-desktop","pid":14,"level":20,"time":"2020-06-12T22:27:48.261+00:00","v":0,"msg":"(17.15 ms) UPDATE TIME TO LIVE - [{:table_name=>\"telematics_local_events\", :attribute=>\"expires_at\"}]"}
Error raised from handler method
{
  "errorMessage": "unsupported type, expected Hash, Array, Set, String, Numeric, IO, true, false, or nil, got DateTime",
  "errorType": "Function<ArgumentError>",
  "stackTrace": [
    "/var/task/vendor/bundle/ruby/2.7.0/gems/aws-sdk-dynamodb-1.49.1/lib/aws-sdk-dynamodb/attribute_value.rb:53:in `format'",
    "/var/task/vendor/bundle/ruby/2.7.0/gems/aws-sdk-dynamodb-1.49.1/lib/aws-sdk-dynamodb/attribute_value.rb:31:in `block in format'",
    "/var/task/vendor/bundle/ruby/2.7.0/gems/aws-sdk-dynamodb-1.49.1/lib/aws-sdk-dynamodb/attribute_value.rb:30:in `each'",
    "/var/task/vendor/bundle/ruby/2.7.0/gems/aws-sdk-dynamodb-1.49.1/lib/aws-sdk-dynamodb/attribute_value.rb:30:in `with_object'",
    "/var/task/vendor/bundle/ruby/2.7.0/gems/aws-sdk-dynamodb-1.49.1/lib/aws-sdk-dynamodb/attribute_value.rb:30:in `format'",
    "/var/task/vendor/bundle/ruby/2.7.0/gems/aws-sdk-dynamodb-1.49.1/lib/aws-sdk-dynamodb/attribute_value.rb:16:in `marshal'",
    "/var/task/vendor/bundle/ruby/2.7.0/gems/aws-sdk-dynamodb-1.49.1/lib/aws-sdk-dynamodb/plugins/simple_attributes.rb:195:in `translate'",
    "/var/task/vendor/bundle/ruby/2.7.0/gems/aws-sdk-dynamodb-1.49.1/lib/aws-sdk-dynamodb/plugins/simple_attributes.rb:189:in `block in map'",
    "/var/task/vendor/bundle/ruby/2.7.0/gems/aws-sdk-dynamodb-1.49.1/lib/aws-sdk-dynamodb/plugins/simple_attributes.rb:188:in `each'",
    "/var/task/vendor/bundle/ruby/2.7.0/gems/aws-sdk-dynamodb-1.49.1/lib/aws-sdk-dynamodb/plugins/simple_attributes.rb:188:in `with_object'",
    "/var/task/vendor/bundle/ruby/2.7.0/gems/aws-sdk-dynamodb-1.49.1/lib/aws-sdk-dynamodb/plugins/simple_attributes.rb:188:in `map'",
    "/var/task/vendor/bundle/ruby/2.7.0/gems/aws-sdk-dynamodb-1.49.1/lib/aws-sdk-dynamodb/plugins/simple_attributes.rb:205:in `translate_complex'",
    "/var/task/vendor/bundle/ruby/2.7.0/gems/aws-sdk-dynamodb-1.49.1/lib/aws-sdk-dynamodb/plugins/simple_attributes.rb:197:in `translate'",
    "/var/task/vendor/bundle/ruby/2.7.0/gems/aws-sdk-dynamodb-1.49.1/lib/aws-sdk-dynamodb/plugins/simple_attributes.rb:170:in `block in structure'",
    "/var/task/vendor/bundle/ruby/2.7.0/gems/aws-sdk-dynamodb-1.49.1/lib/aws-sdk-dynamodb/plugins/simple_attributes.rb:169:in `each'",
    "/var/task/vendor/bundle/ruby/2.7.0/gems/aws-sdk-dynamodb-1.49.1/lib/aws-sdk-dynamodb/plugins/simple_attributes.rb:169:in `with_object'",
    "/var/task/vendor/bundle/ruby/2.7.0/gems/aws-sdk-dynamodb-1.49.1/lib/aws-sdk-dynamodb/plugins/simple_attributes.rb:169:in `structure'",
    "/var/task/vendor/bundle/ruby/2.7.0/gems/aws-sdk-dynamodb-1.49.1/lib/aws-sdk-dynamodb/plugins/simple_attributes.rb:157:in `apply'",
    "/var/task/vendor/bundle/ruby/2.7.0/gems/aws-sdk-dynamodb-1.49.1/lib/aws-sdk-dynamodb/plugins/simple_attributes.rb:126:in `translate_input'",
    "/var/task/vendor/bundle/ruby/2.7.0/gems/aws-sdk-dynamodb-1.49.1/lib/aws-sdk-dynamodb/plugins/simple_attributes.rb:116:in `call'",
    "/var/task/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.99.2/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:20:in `call'",
    "/var/task/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.99.2/lib/aws-sdk-core/plugins/idempotency_token.rb:17:in `call'",
    "/var/task/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.99.2/lib/aws-sdk-core/plugins/param_converter.rb:24:in `call'",
    "/var/task/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.99.2/lib/aws-sdk-core/plugins/response_paging.rb:10:in `call'",
    "/var/task/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.99.2/lib/seahorse/client/plugins/response_target.rb:23:in `call'",
    "/var/task/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.99.2/lib/seahorse/client/request.rb:70:in `send_request'",
    "/var/task/vendor/bundle/ruby/2.7.0/gems/aws-sdk-dynamodb-1.49.1/lib/aws-sdk-dynamodb/client.rb:3536:in `put_item'",
    "/var/task/vendor/bundle/ruby/2.7.0/gems/dynamoid-3.5.0/lib/dynamoid/adapter_plugin/aws_sdk_v3.rb:451:in `put_item'",
    "/var/task/vendor/bundle/ruby/2.7.0/gems/dynamoid-3.5.0/lib/dynamoid/adapter.rb:150:in `block (3 levels) in <class:Adapter>'",
    "/var/task/vendor/bundle/ruby/2.7.0/gems/dynamoid-3.5.0/lib/dynamoid/adapter.rb:55:in `benchmark'",
    "/var/task/vendor/bundle/ruby/2.7.0/gems/dynamoid-3.5.0/lib/dynamoid/adapter.rb:150:in `block (2 levels) in <class:Adapter>'",
    "/var/task/vendor/bundle/ruby/2.7.0/gems/dynamoid-3.5.0/lib/dynamoid/adapter.rb:70:in `write'",
    "/var/task/vendor/bundle/ruby/2.7.0/gems/dynamoid-3.5.0/lib/dynamoid/persistence/save.rb:23:in `call'",
    "/var/task/vendor/bundle/ruby/2.7.0/gems/dynamoid-3.5.0/lib/dynamoid/persistence/save.rb:7:in `call'",
    "/var/task/vendor/bundle/ruby/2.7.0/gems/dynamoid-3.5.0/lib/dynamoid/persistence.rb:300:in `block (2 levels) in save'",
    "/var/task/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:135:in `run_callbacks'",
    "/var/task/vendor/bundle/ruby/2.7.0/gems/dynamoid-3.5.0/lib/dynamoid/persistence.rb:299:in `block in save'",
    "/var/task/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:135:in `run_callbacks'",
    "/var/task/vendor/bundle/ruby/2.7.0/gems/dynamoid-3.5.0/lib/dynamoid/persistence.rb:298:in `save'",
    "/var/task/vendor/bundle/ruby/2.7.0/gems/dynamoid-3.5.0/lib/dynamoid/dirty.rb:48:in `save'",
    "/var/task/vendor/bundle/ruby/2.7.0/gems/dynamoid-3.5.0/lib/dynamoid/validations.rb:18:in `save'",
    "/var/task/vendor/bundle/ruby/2.7.0/gems/dynamoid-3.5.0/lib/dynamoid/identity_map.rb:64:in `save'",
    "/opt/lib/ingestion/handlers/kinesis.rb:10:in `block in call'",
    "/opt/lib/ingestion/handlers/kinesis.rb:8:in `map'",
    "/opt/lib/ingestion/handlers/kinesis.rb:8:in `call'",
    "/opt/lib/ingestion/handler.rb:15:in `call'",
    "/var/task/handler.rb:8:in `lambda_handler'"
  ]
}

However, if I change the value to be:

Time.at(str&.to_i).utc.to_datetime.to_formatted_s(:iso8601)

while keeping the type as :datetime everything works. I feel it's odd that I'd have to pass a String, which if I'm not mistaken, then gets converted to a DateTime by the type caster, and then to a number.

Am I doing something wrong?
@andrykonchin
Copy link
Member

andrykonchin commented Jun 15, 2020 •
edited

Could you please provide a code snippet to reproduce the issue?

The issue is DynamoDB doesn't support Date/Time/DateTime type internally so such columns are stored either as number of seconds or as a string. Looks like an attribute value somehow isn't type casted by Dynamoid.

@jmonsanto
Copy link

jmonsanto commented Jul 22, 2020 •
edited

Something like this:

record.update do |r|
  r.set data: data
  r.set synced_at: DateTime.current
end

When getting a record from DynamoDB there are no issues casting a DynamoDB string to ruby datetime, however when attempting to save a datetime object, it isn't casted to string to persist on DynamoDB.

But shouldn't datetime values be auto-casted if the declared field attribute is string?

@andrykonchin
Copy link
Member

I see. Yes, it's definitely a bug in the #update/#update! methods.

@ychaker
Copy link
Author

ychaker commented Jul 23, 2020

thank you @jmonsanto for providing the example, I dropped the ball on that when I saw the notification and then completely forgot about it, sorry!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@ychaker @andrykonchin @jmonsanto