Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add JSON type detection to handle both arrays and objects #750

Open
wants to merge 5 commits into
base: main
Choose a base branch
from

Conversation

rivkode
Copy link

@rivkode rivkode commented May 22, 2024

  • Handle both JSON arrays and Objects
  • Type checking for JSON using JsonNode
  • Utilize Stream API to process JSON arrays

I am considering the following for future improvements as below.

  • Looking at how other open source Json readers are using it (e.g. springboot JsonReader), and referencing it.
  • Modify the constructor to make the ObjectMapper a bit more flexible. Allow users to create any object they want.

issue link

- Type checking for JSON root node using JsonNode
- Utilize Stream API to process JSON arrays
- Modify parsing logic to handle both JSON arrays and objects
@rivkode rivkode changed the title fix: Add JSON root type detection to handle both arrays and objects Add JSON root type detection to handle both arrays and objects May 22, 2024
@rivkode rivkode changed the title Add JSON root type detection to handle both arrays and objects Add JSON type detection to handle both arrays and objects May 22, 2024
@@ -58,41 +62,37 @@ public JsonReader(Resource resource, JsonMetadataGenerator jsonMetadataGenerator

@Override
public List<Document> get() {
ObjectMapper objectMapper = new ObjectMapper();
List<Document> documents = new ArrayList<>();
try {
// TODO, not all json will be an array

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we also remove this TODO?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, I will remove it!

- max depth: 4
- array length: 2
- number of root elements: 6
Comment on lines 48 to +51
@Test
void loadJson() {
assertThat(resource).isNotNull();
JsonReader jsonReader = new JsonReader(resource, "description");
void loadJsonObject() {
assertThat(ObjectResource).isNotNull();
JsonReader jsonReader = new JsonReader(ObjectResource, "description");
Copy link
Author

@rivkode rivkode May 24, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Before
image

After
image

@markpollack
Copy link
Member

Thanks for the contribution, will review after we release M1, moving this to M2.

@markpollack markpollack added this to the 1.0.0-M2 milestone May 24, 2024
@tzolov tzolov added the ETL label Jun 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants