Bug when posting file_data with non-000Z milliseconds

[ustayready]

What is the installation of Nemesis?

Debian 11, k3s

What is the issue?

Nemesis API endpoint for posting file_data to /api/data using curl or python3 requests requires the timestamp/expiration to have 000Z milliseconds in order to succeed. I've attempted dozens of alternative timestamp formats and none seem to work.

Example Curl Statements:

BAD: Invalid expiration value in metadata field

curl -X POST http://192.168.68.66/api/data -v --user 'nemesis:password' \
   -H "Content-Type: application/json" \
   -d '{
         "metadata": {
           "agent_id": "3249337381",
           "agent_type": "monitor",
           "automated": true,
           "data_type": "file_data",
           "expiration": "2024-06-14T21:43:07.900Z",
           "project": "7249",
           "timestamp": "2024-05-15T21:43:07.400Z",
           "source": "blob://monitor"
         },
         "data": [
           {
             "path": "./test/Certify.exe",
             "size": 583568,
             "object_id": "8c0f2413-12b2-47e5-82fc-0b46e0535c13"
           }
         ]
       }'

BAD: Invalid file_data message

curl -X POST http://192.168.68.66/api/data -v --user 'nemesis:password' \
   -H "Content-Type: application/json" \
   -d '{
         "metadata": {
           "agent_id": "3249337381",
           "agent_type": "monitor",
           "automated": true,
           "data_type": "file_data",
           "expiration": "2024-06-14T21:43:07",
           "project": "7249",
           "timestamp": "2024-05-15T21:43:07",
           "source": "blob://monitor"
         },
         "data": [
           {
             "path": "./test/Certify.exe",
             "size": 583568,
             "object_id": "8c0f2413-12b2-47e5-82fc-0b46e0535c13"
           }
         ]
       }'

GOOD

curl -X POST http://192.168.68.66/api/data -v --user 'nemesis:password' \
   -H "Content-Type: application/json" \
   -d '{
         "metadata": {
           "agent_id": "3249337381",
           "agent_type": "monitor",
           "automated": true,
           "data_type": "file_data",
           "expiration": "2024-06-14T21:43:07.000Z",
           "project": "7249",
           "timestamp": "2024-05-15T21:43:07.000Z",
           "source": "blob://monitor"
         },
         "data": [
           {
             "path": "./test/Certify.exe",
             "size": 583568,
             "object_id": "8c0f2413-12b2-47e5-82fc-0b46e0535c13"
           }
         ]
       }'

Problem Line in Code

It looks like the strptime() is requiring .000Z:

Nemesis/cmd/enrichment/enrichment/tasks/webapi/nemesis_api.py

Line 370 in 84d5986

expiration = datetime.strptime(expiration_string, "%Y-%m-%dT%H:%M:%S.000Z")

Proposed Solution

Consider using the format string "%Y-%m-%dT%H:%M:%S.%fZ" for flexible formatting.

If I get a moment, I will make a PR to fix this. I tested it locally and it works at parsing all the variants.