Issues with searching nested fields with arrays of objects

[lukew-cogapp]

The problem

Attempting to search nested fields with arrays of objects causes no search results to be returned. A similar situation also occurs when enabling searchable on a RefinementList and searching for text, but here an error occurs.

Search issue

The data looks like this:

"Author_100": [
  {
    "firstName": "value",
    "lastName": "value",
    "title": "value",
  },
  {
    "firstName": "value",
    "lastName": "value",
    "title": "value",
  }
]

With mapping of:

"Author_100": {
  "type": "nested",
  "properties": {
    "firstName": {
      "type": "text",
      "fields": {
        "keyword": {
          "type": "keyword",
          "ignore_above": 256
        }
      }
    },
    "lastName": {
      "type": "text",
      "fields": {
        "keyword": {
          "type": "keyword",
          "ignore_above": 256
        }
      }
    },
    "title": {
      "type": "text",
      "fields": {
        "keyword": {
          "type": "keyword",
          "ignore_above": 256
        }
      }
    }
  }
}

The combination of packages we're using is:

    "@searchkit/api": "^4.0.1",
    "@searchkit/client": "^3.0.0",
    "@searchkit/instantsearch-client": "^4.1.1",

With Node 16.

The simplified configuration for the search field is:
search_attributes: ["Author_100.firstName"]

After enabling debug mode and running the query directly, I can see it gets no results because the query doesn't include the nested property, as per the ES docs. I appreciate this is likely a feature request rather than a bug, to be able to define a search term as nested, so the necessary modifications to the query can be made.

Facet issue

You do have support for nested facets, where we define the following config for facet_attributes:

        attribute: "author",
        field: "firstName.keyword",
        type: "string",
        nestedPath: "Author_100",
      },

The facet works as expected for selection & showMore functionality, but when enabling searchable and attempting a search, an error occurs:

error - TypeError: Cannot read properties of undefined (reading 'map')
    at transformFacetValuesResponse (/my-proj-dir/node_modules/searchkit/dist/index.js:820:67)
    at /my-proj-dir/node_modules/searchkit/dist/index.js:1026:16
    at Array.map (<anonymous>)
    at Searchkit.handleInstantSearchRequests (/my-proj-dir/node_modules/searchkit/dist/index.js:1023:48)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async handler (webpack-internal:///(api)/./pages/api/search.js:121:21)
    at async Object.apiResolver (/my-proj-dir/node_modules/next/dist/server/api-utils/node.js:363:9)
    at async DevServer.runApi (/my-proj-dir/node_modules/next/dist/server/next-server.js:488:9)
    at async Object.fn (/my-proj-dir/node_modules/next/dist/server/next-server.js:750:37)
    at async Router.execute (/my-proj-dir/node_modules/next/dist/server/router.js:253:36)
    at async DevServer.run (/my-proj-dir/node_modules/next/dist/server/base-server.js:384:29)
    at async DevServer.run (/my-proj-dir/node_modules/next/dist/server/dev/next-dev-server.js:741:20)
    at async DevServer.handleRequest (/my-proj-dir/node_modules/next/dist/server/base-server.js:322:20) {
  page: '/api/search'
}

When running debug and inspecting the query manually, the query does work, but Searchkit can't seem to parse the result.

Here's a snippet of the result:

"aggregations": {
		"Author_100.": {
			"doc_count": 81080,
			"author": {
				"doc_count_error_upper_bound": 0,
				"sum_other_doc_count": 12,
				"buckets": [
					{
						"key": "Luke",
						"doc_count": 6
					},

My guess is it's struggling to parse the Author_100. including the fullstop at the end.

[joemcelroy]

Thanks so much for a detailed issue!

You're correct, nested search attributes are not supported. I "deprioritised" it as developers can override the ES query.query via getQuery function but its definitely a good feature to have :) I've opened an issue here #1209

The facet search is a bug and one thing I didn't think about when developing nested fields. I will fix that soon.

Also happy for PRs for nested field search_attributes too!

[lukew-cogapp]

Thanks for the speedy reply.

Could you give me an example of how to use getQuery to do a nested field search?

[joemcelroy]

Something like this. Searchkit ships with a default "QueryHandler" but you can override via getQuery fn. query argument is the searchterm. Here we are just searching the nested field user.first. codesandbox link of it working

const results = await apiClient.handleRequest(req.body, {
    getQuery: (query) => {
      return {
        bool: {
          must: [
            {
              nested: {
                path: "user",
                query: {
                  combined_fields: {
                    query: query,
                    fields: ["user.first"]
                  }
                }
              }
            }
          ]
        }
      };
    }
  });

[joemcelroy]

the facet query bug is fixed now in latest version :)