Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Can't used --JSON-input from JSON (export from Splunk) #1083

Closed
haicoiok opened this issue Jun 5, 2023 · 9 comments · Fixed by #1286 or #1346
Closed

Can't used --JSON-input from JSON (export from Splunk) #1083

haicoiok opened this issue Jun 5, 2023 · 9 comments · Fixed by #1286 or #1346
Assignees
@hitenkoku
Labels
enhancement New feature or request
Milestone
v2.16.0

Comments

@haicoiok
Copy link

haicoiok commented Jun 5, 2023

Can you please assist me in utilizing the DFIR Timeline feature from a JSON file exported from Splunk? I am currently facing difficulties in executing this task.

image
@haicoiok
Copy link
Author

haicoiok commented Jun 5, 2023 •
edited

If I use the command hayabusa.exe csv-timeline -f C:\Users\***\Downloads\data1.json -J it doesn't trigger the rule. However, another application shows many results.
image
Another App:
image

@YamatoSecurity
Copy link
Collaborator

@haicoiok Unfortunately we don't support all JSON formats at the moment and haven't tested with JSON files exported from splunk. If you can create a JSON file that we can test we can see if it is possible to support it.

@haicoiok
Copy link
Author

haicoiok commented Jun 5, 2023 •
edited

@YamatoSecurity This sample data export from Splunk. "Deleted Data"

@YamatoSecurity
Copy link
Collaborator

@haicoiok Thanks!
@hitenkoku Whenever you get free time, could you take a look at the splunk JSON file and see how hard it would be to support it?

@YamatoSecurity YamatoSecurity added the under-investigation under investigation to develop label Jun 5, 2023
@hitenkoku hitenkoku self-assigned this Jun 6, 2023
@hitenkoku
Copy link
Collaborator

@YamatoSecurity I found the cause.
I found that the cause was a difference between the originally expected json format and the exported JSON format from Splunk. The Splunk JSON file contains various information in the result field, but the expected JSON file does not have such a structure.

  • expected json format
{
  "EventTime": "2020-05-01 22:55:23",
  "port": 60737,
  "Message": "Process accessed:\r\nRuleName: -\r\nUtcTime: 2020-05-02 02:55:23.551\r\nSourceProcessGUID: {6bbf237a-cafb-5eac-1000-000000000400}\r\nSourceProcessId: 900\r\nSourceThreadId: 504\r\nSourceImage: C:\\windows\\system32\\svchost.exe\r\nTargetProcessGUID: {6bbf237a-cb97-5eac-6202-000000000400}\r\nTargetProcessId: 2092\r\nTargetImage: C:\\windows\\System32\\svchost.exe\r\nGrantedAccess: 0x1000\r\nCallTrace: C:\\windows\\SYSTEM32\\ntdll.dll+9c584|C:\\windows\\SYSTEM32\\psmserviceexthost.dll+222a3|C:\\windows\\SYSTEM32\\psmserviceexthost.dll+1a172|C:\\windows\\SYSTEM32\\psmserviceexthost.dll+19e3b|C:\\windows\\SYSTEM32\\psmserviceexthost.dll+19318|C:\\windows\\SYSTEM32\\ntdll.dll+3089d|C:\\windows\\SYSTEM32\\ntdll.dll+34634|C:\\windows\\System32\\KERNEL32.DLL+17bd4|C:\\windows\\SYSTEM32\\ntdll.dll+6ced1",
  "SourceThreadId": "504",
  "EventID": 10,
  "TargetProcessId": "2092",
  "SourceModuleName": "eventlog",
  "tags": [
    "mordorDataset"
  ],
  "@version": "1",
  "SourceImage": "C:\\windows\\system32\\svchost.exe",
  "SourceName": "Microsoft-Windows-Sysmon",
  "AccountType": "User",
  "TargetImage": "C:\\windows\\System32\\svchost.exe",
  "host": "wec.internal.cloudapp.net",
  "Task": 10,
  "ThreadID": 4396,
  "EventReceivedTime": "2020-05-01 22:55:26",
  "CallTrace": "C:\\windows\\SYSTEM32\\ntdll.dll+9c584|C:\\windows\\SYSTEM32\\psmserviceexthost.dll+222a3|C:\\windows\\SYSTEM32\\psmserviceexthost.dll+1a172|C:\\windows\\SYSTEM32\\psmserviceexthost.dll+19e3b|C:\\windows\\SYSTEM32\\psmserviceexthost.dll+19318|C:\\windows\\SYSTEM32\\ntdll.dll+3089d|C:\\windows\\SYSTEM32\\ntdll.dll+34634|C:\\windows\\System32\\KERNEL32.DLL+17bd4|C:\\windows\\SYSTEM32\\ntdll.dll+6ced1",
  "Domain": "NT AUTHORITY",
  "UtcTime": "2020-05-02 02:55:23.551",
  "Keywords": -9223372036854776000,
  "RecordNumber": 138294,
  "SourceProcessId": "900",
  "SourceModuleType": "im_msvistalog",
  "@timestamp": "2020-05-02T02:55:26.493Z",
  "SeverityValue": 2,
  "Version": 3,
  "OpcodeValue": 0,
  "Severity": "INFO",
  "Channel": "Microsoft-Windows-Sysmon/Operational",
  "AccountName": "SYSTEM",
  "GrantedAccess": "0x1000",
  "SourceProcessGUID": "{6bbf237a-cafb-5eac-1000-000000000400}",
  "EventType": "INFO",
  "UserID": "S-1-5-18",
  "TargetProcessGUID": "{6bbf237a-cb97-5eac-6202-000000000400}",
  "ProcessId": "900",
  "Hostname": "UTICA.dmevals.local",
  "RuleName": "-",
  "ExecutionProcessID": 3496,
  "ProviderGuid": "{5770385F-C22A-43E0-BF4C-06F5698FFBD9}"
}
  • splunk JSON file format
{
  "preview": false,
  "offset": 0,
  "result": {
    "Channel": "Microsoft-Windows-Sysmon/Operational",
    "CommandLine": "\"C:\\Program Files\\SplunkUniversalForwarder\\bin\\splunk-MonitorNoHandle.exe\"",
    ...
  }
}

@hitenkoku hitenkoku added enhancement New feature or request and removed under-investigation under investigation to develop labels Jun 6, 2023
@hitenkoku hitenkoku added this to the v2.7.0 milestone Jun 17, 2023
@hitenkoku hitenkoku modified the milestones: v2.7.0, v2.8.0 Jul 2, 2023
@YamatoSecurity YamatoSecurity modified the milestones: v2.8.0, v2.9.0 Aug 15, 2023
@YamatoSecurity YamatoSecurity modified the milestones: v2.9.0, v2.10.0 Sep 14, 2023
@YamatoSecurity YamatoSecurity modified the milestones: v2.10.0, v2.11.0 Oct 18, 2023
@YamatoSecurity YamatoSecurity modified the milestones: v2.11.0, v2.13.0 Dec 13, 2023
@YamatoSecurity YamatoSecurity modified the milestones: v2.13.0, v2.14.0 Jan 23, 2024
hitenkoku added a commit that referenced this issue Feb 26, 2024
@hitenkoku
feat(main, metrics): adjusted JSON input file exported from Splunk #1083
3d8872d
hitenkoku added a commit that referenced this issue Feb 26, 2024
@hitenkoku
docs(CHANGELOG): added #1083
a22d3af
@hitenkoku hitenkoku linked a pull request Feb 26, 2024 that will close this issue
Adjusted json input from json export from splunk #1286
Merged
hitenkoku added a commit that referenced this issue Feb 28, 2024
@hitenkoku
feat(detection): fixed missing Details field in JSON output when splu…
478db5e 
…nk JSON input is specified. #1083
@chacobsa
Copy link

chacobsa commented Apr 4, 2024

Hello! I hope you are doing well. I would like to bring to your attention a minor issue I have encountered with a feature. I have noticed that the JSON output downloaded from Splunk Web is different from the JSON output downloaded from Splunk REST API. This is why Hayabusa can't scan through JSON fetched via REST API. Would you address this issue in the next update? Thank you!

@YamatoSecurity
Copy link
Collaborator

@chacobsa Thank you for letting us know. Is it possible to provide a sample JSON file downloaded from the Splunk REST API to test?

@YamatoSecurity YamatoSecurity reopened this Apr 4, 2024
@chacobsa
Copy link

chacobsa commented Apr 9, 2024

@YamatoSecurity
splunk_rest_api_output.zip
A sample JSON to test in the attachment. I got it by sending a REST API request on Splunk Enterprise version 9.1.2

@YamatoSecurity
Copy link
Collaborator

@chacobsa Thank you! We will take a look at it.

@YamatoSecurity YamatoSecurity modified the milestones: v2.14.0, v2.16.0 Apr 20, 2024
hitenkoku added a commit that referenced this issue May 12, 2024
@hitenkoku
feat(main): adjusted splunk api json format #1083
f1b0a26
@hitenkoku hitenkoku linked a pull request May 12, 2024 that will close this issue
feat(main): adjusted splunk api json format #1083 #1346
Merged
hitenkoku added a commit that referenced this issue May 12, 2024
@hitenkoku
fix(main): fixed errors in variable assignments corrected. #1083
fce0100
@fukusuket fukusuket mentioned this issue May 12, 2024
Merged
hitenkoku added a commit that referenced this issue May 15, 2024
@hitenkoku
fix(main, Cargo): removed quickxml_to_serde and fixed all record info…
5a4290b 
…rmation output in Details fields #1083
hitenkoku added a commit that referenced this issue May 16, 2024
@hitenkoku
fix(main): fixed time fmt parse error to regular json input file #1083
cf4027d
hitenkoku added a commit that referenced this issue May 17, 2024
@hitenkoku
Merge pull request #1346 from Yamato-Security/1083-cant-used-json-inp…
2845ad1 
…ut-from-json-export-from-splunk-rest-api

feat(main): adjusted splunk api json format #1083
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@hitenkoku hitenkoku

Labels
enhancement New feature or request
Projects
None yet
Milestone
v2.16.0
4 participants
@hitenkoku @haicoiok @YamatoSecurity @chacobsa