FT.SEARCH Beta Feedback November 2023

[admtech]

The index for Dragonfly has been available since version 1.12. The following FT commands are currently available:
FT.CREATE, FT.SEARCH, FT.PROFILE, FT._LIST, FT.INFO, FT.DROPINDEX. This makes the index compatible with RediSearch.

Here are a few examples:

Create index:

FT.CREATE search ON JSON PREFIX 1 adm:content: SCHEMA $.title AS title TEXT $.tags.* AS tags TAG $.stamp_create as stamp_create TEXT SORTABLE $.stamp AS stamp NUMERIC SORTABLE

Search or display index samples:

FT.SEARCH search "Linux"
FT.SEARCH search * NOCONTENT LIMIT 0 10
FT.SEARCH search "@title: Windows @tags:{972739643}" RETURN 1 $.title as title SORTBY stamp_create DESC LIMIT 0 10

In principle, an index in Dragonfly enables something similar to what an SQL DB does with "SELECT * FROM search WHERE". The speed of creating the index is incredibly fast.

For example, if I create an index for our comments (1.5 million), it takes less than 11 seconds. Compared to Redis RediSearch with 5 minutes and 27 seconds for the same index (but with prefix/infix), this is really very good progress. A big thank you to @dranikpg and the dragonflyDB team.

Here is my feedback on version 1.12.1

1. Currently, the FT commands are not synchronized in a master/slave configuration. Is this intentional?

2. With the TAGs type, it seems to stop after 10 characters. This is how it works:

FT.SEARCH search "@tags:{123456789}" RETURN 2 $.title as title $.tags as tags SORTBY stamp_create DESC LIMIT 0 10
1) (integer) 0

while

FT.SEARCH search "@tags:{1234567890}" RETURN 2 $.title as title $.tags as tags SORTBY stamp_create DESC LIMIT 0 10
(error) ERR Query syntax error

reports an error ((error) ERR Query syntax error). The 1234... are only meant as an example, these are keys that consist of numbers but are saved as strings. Current workaround: If you save it in "TEXT" instead of the type "TAG", i.e. "$.tags.* AS tags TEXT" then it works.

3. The "RETURN" syntax is slightly different from the original RediSearch.

You can write the following at Dragonfly:

FT.SEARCH search * RETURN 1 $.title as title oder FT.SEARCH search * RETURN 1 $.title

Under "RediSearch" the count and the "$." work differently:

FT.SEARCH content * RETURN 1 $.title as title
(error) Unknown argument `as` at position 4 for <main>

FT.SEARCH content * RETURN 2 $.title AS title
(error) RETURN path AS name - must be accompanied with NAME

Here's how it works under RediSearch:

FT.SEARCH content * RETURN 1 $.title 
FT.SEARCH content * RETURN 3 $.title AS title
FT.SEARCH content * RETURN 1 title

You don't need a "$." with the "title" if it's a json array. Or you have to set the count to 3 for it to work with "AS". I didn't quite understand that either. However, I think Dragonfly's solution is better. For compatibility, however, the "title" should also work without "$.".

4. FT.INFO search-index provides some information about the index.

FT.INFO search
1) index_name
2) search
3) fields
4) 1) 1) identifier
      2) $.stamp
      3) attribute
      4) stamp
      5) type
      6) NUMERIC
   2) 1) identifier
      2) $.stamp_create
      3) attribute
      4) stamp_create
      5) type
      6) TEXT
   3) 1) identifier
      2) $.title
      3) attribute
      4) title
      5) type
      6) TEXT
   4) 1) identifier
      2) $.tags.*
      3) attribute
      4) tags
      5) type
      6) TAG
5) num_docs
6) (integer) 275557 

Here the "SORTABLE" information is not yet displayed for the respective fields. Under RedSearch these are not called "fields" but "attributes". The information "index_definition" and the information about the total size of the index in MB would also be nice. All other values can be neglected for the time being. Here is an FT.INFO from RediSearch:

FT.INFO search
 1) index_name
 2) content
 3) index_options
 4) (empty array)
 5) index_definition
 6) 1) key_type
    2) JSON
    3) prefixes
    4) 1) adm:content:
    5) default_score
    6) "1"
 7) attributes
 8)  1) 1) identifier
        2) $.title
        3) attribute
        4) title
        5) type
        6) TEXT
        7) WEIGHT
        8) "1"
     2) 1) identifier
        2) $.tags.*
        3) attribute
        4) tags
        5) type
        6) TEXT
        7) WEIGHT
        8) "1"
    3)  1) identifier
         2) $.stamp_create
         3) attribute
         4) stamp_create
         5) type
         6) TEXT
         7) WEIGHT
         8) "1"
         9) SORTABLE
        10) UNF
 9) num_docs
10) "275550"
11) max_doc_id
12) "275550"
13) num_terms
14) "1252264"
15) num_records
16) "33919082"
17) inverted_sz_mb
18) "215.0576171875"
19) vector_index_sz_mb
20) "0"
21) total_inverted_index_blocks
22) "4777164"
23) offset_vectors_sz_mb
24) "56.05750274658203"
25) doc_table_size_mb
26) "21.999134063720703"
27) sortable_values_size_mb
28) "12.876462936401367"
29) key_table_size_mb
30) "8.388565063476563"
31) geoshapes_sz_mb
32) "0"
33) records_per_doc_avg
34) "123.09591674804688"
35) bytes_per_record_avg
36) "6.648301124572754"
37) offsets_per_term_avg
38) "1.4423855543136597"
39) offset_bits_per_record_avg
40) "9.611656188964844"
41) hash_indexing_failures
42) "4"
43) total_indexing_time
44) "79933.401"
45) indexing
46) "0"
47) percent_indexed
48) "1"
49) number_of_uses
50) (integer) 9
51) cleaning
52) (integer) 0
53) gc_stats
54)  1) bytes_collected
     2) "0"
     3) total_ms_run
     4) "0"
     5) total_cycles
     6) "0"
     7) average_cycle_time_ms
     8) "nan"
     9) last_run_time_ms
    10) "0"
    11) gc_numeric_trees_missed
    12) "0"
    13) gc_blocks_denied
    14) "0"
55) cursor_stats
56) 1) global_idle
    2) (integer) 0
    3) global_total
    4) (integer) 0
    5) index_capacity
    6) (integer) 128
    7) index_total
    8) (integer) 0
57) dialect_stats
58) 1) dialect_1
    2) (integer) 1
    3) dialect_2
    4) (integer) 0
    5) dialect_3
    6) (integer) 0
    7) dialect_4
    8) (integer) 0

5. According to which sorting is the search result currently displayed? By any random order?

6. The log file often shows "doc_accessors.cc:177] Invalid Json path: $comment Syntax error" if the wrong "Json path" is used. Shouldn't this rather appear as the output of the corresponding wrong command? The CLI is then silent about the error.

7. Should index errors or features be entered individually or collectively in "Issue" in future or is it sufficient in the beta stage here in the forum?

Wishes:

• Search Scrore (WITHSCORES)
• Prefix (Word*)/ Infix (*Word)
• show WITHSORTKEYS
• Integrate GEO search with the GEO type in the index (now that GEOSEARCH is also available in Dragonfly ;-))
• Index Feature Matrix, what has already been implemented and what has not yet been implemented.

Otherwise the index works absolutely stable. Since DragonflyDB created the index in a few seconds, I first thought it was an error and it wouldn't work. I was wrong, Dragonfly is just damn fast and the index works. Well done, keep up the good work! 👍

Frank

[dranikpg]

Hi!

I truly appreciate your extensive feedback 🙂 It really helps us grow and spot mistakes for real world use cases

I'll quickly answer your questions:

1. Replication support will be finished soon
2. Oh yes, this is really a pitfall with the current query lexer 😅 Whenever you use strings that consist only of integers, please use quotes, single or double. Libraries like redis-om always put their strings in quotes for safety. So @tag:{'$var'} is the safest way that should be preferred
3. Yes, will fix this. I also noticed that with our RETURNING clause you specify the number of returned values and not arguments
4. I already fixed this partially (but it wasn't cherry picked into 1.12.1 😓) It now shows some generic info and flags, but not the detailed stats yet
5. If you don't use SORTBY, the output is "semi-random". This means that on average for a large result set older records will come first, however then can move a few positions back and forth. It might be not that useful on its own, but fits best for counting the number of results / checking if items exists at all without the overhead of sorting.
6. Yes, that's on my TODO. Currently the field should be just absent if the wrong path is used
7. I can currently keep track of everything, so you can write where you prefer. If it gets more traction over time once we announce it, you can open separate issues / search for them 🙂

[claridgicus]

@dranikpg
This was the missing feature set I would need to swap
Is the goal to implement a one to one implementation for redissearch features or is their scope for a bit of improvement on the shortcomings?

[romange]

if you have a feedback to share or you have requests for missing features or improvements (even those that do not appear in redis implementation) - please tell us.