• currently the index for body is INDEX body_idx body TYPE tokenbf_v1(10240, 3, 0) GRANULARITY 4
• When you use something like body ilike 'data' the index doesn't work while body like 'data' does work.

One solutionI found to get the index working was

• change index to INDEX body_idx lower(body) TYPE tokenbf_v1(10240, 3, 0) GRANULARITY 4 i.e added lower
• Now when I search lower(body) like 'data' the index is used.

Though I have to verify if is some performance improvements, there should be definately some as previously the index was not even getting used.
also have to verify how it works with json query.

cc @ankitnayan

There is definately improvements when the index is used, a lot less granules is scanned.
ex: searching for 3ee1645f-98f8-4a70-a3c9-b4429d475ae8 in body

without the token index

│         Parts: 145/145                      │
│         Granules: 467345/472640

with the index

│         Parts: 125/144                        │
│         Granules: 162090/472638               │

A lot of granules are getting skipped when the index gets actually used.