Payload indexing

[eatsleepraverepeat]

Hi,
my question regarding payload fields index, it's seems like in certain scenarios it's does not improve ANN search time with given set of filters.

Here's a vector collection data, respective payload schema and NHSW config,

{
  "result": {
    "status": "yellow",
    "optimizer_status": "ok",
    "vectors_count": 1782689,
    "segments_count": 41,
    "disk_data_size": 0,
    "ram_data_size": 0,
    "config": {
      "params": {
        "vector_size": 128,
        "distance": "Cosine",
        "shard_number": 8
      },
      "hnsw_config": {
        "m": 16,
        "ef_construct": 100,
        "full_scan_threshold": 10000
      },
      "optimizer_config": {
        "deleted_threshold": 0.2,
        "vacuum_min_vector_number": 1000,
        "default_segment_number": 5,
        "max_segment_size": 300000,
        "memmap_threshold": 50000,
        "indexing_threshold": 20000,
        "payload_indexing_threshold": 2500,
        "flush_interval_sec": 1,
        "max_optimization_threads": 0
      },
      "wal_config": {
        "wal_capacity_mb": 32,
        "wal_segments_ahead": 0
      }
    },
    "payload_schema": {
      "date": {
        "data_type": "integer"
      },
      "availability": {
        "data_type": "integer"
      },
      "type": {
        "data_type": "keyword"
      },
      "disambiguation_key": {
        "data_type": "keyword"
      },
      "cluster": {
        "data_type": "integer"
      },
      "altered": {
        "data_type": "keyword"
      },
      "duration": {
        "data_type": "integer"
      },
      "artists_id": {
        "data_type": "integer"
      },
      "release_id": {
        "data_type": "integer"
      },
      "label_id": {
        "data_type": "integer"
      }
    }
  },
  "status": "ok",
  "time": 0.000272679
}

I've made a simple experiment to compare response time of collection with and without payload attributes index and have not seen any difference. Requests are sent through collections/{collection_name}/points/search method, about 20 additional filters were included, response is limited to 2048 objects without vectors and with some of payload fields.

Any feedback on this behaviour would be appreciated,
@generall @orsk-moscow

Best regards,
Alexey, компания Звук

[generall]

Hi @eatsleepraverepeat!

The first thing I notice in the info - "status": "yellow". It means that some segments are still in the process of index building. All vectors are available for search, but some segments might perform slower. It should not affect the production use, as all segments are updated only during the initial upload, and the streaming updates be distributed in time.

Other possible reasons for the observed speed are related to query planning and approaches to combining vector & scalar searches.

In the first step, the query planner tries to estimate what would be the cardinality of the filtering condition (how many points satisfy the given criteria). If the expected cardinality is too high (above the threshold), the query planner will decide to use the vector index. In this case, search speed should be comparable with regular HNSW retrieval and no filters.

If the expected cardinality is below the threshold - the query planner will attempt to retrieve points from ids first and vector-based reranking later. Retrieving ids from the index is, on its own, might go with several different scenarios. If a condition on the indexed field could be used to select most of the points, the query planner would prefer to use it and filter out the rest of the points using all other conditions. Unfortunately, it is not always possible to detect this primary condition in the query, so some changes in query structure might help if it is the case.

Additionally, in the next release of Qdrant there will be a significant improvement in a filtering speed: #461

Overall, there are several opportunities to improve the performance, but it would require some additional examination of the data and queries you have.

[eatsleepraverepeat]

I see,

seems like we facing scenario with low cardinality and later vector-based reranking, your clarification brings sense on why we see no difference, the number of objects under conditions is lower than threshold. Is this parameter configurable? If so, where to find it?

I'm taking time to do additional timings under various conditions

Thank your very much for your assistance!

[generall]

https://github.com/qdrant/qdrant/blob/master/config/config.yaml#L68

[orsk-moscow]

Follow your link, I see:

    # Max number of threads, which can be used for optimization. If 0 - `NUM_CPU - 1` will be used
    max_optimization_threads: 0

IMO you want to point to:

    # Number of neighbours to consider during the index building. Larger the value - more accurate the search, more time required to build index.
    ef_construct: 100

Is that right?

[generall]

heh, sorry we are in the release process, things are moving ;)

i mean this option:

    # If payload chunk is smaller than `full_scan_threshold_kb` additional indexing won't be used -
    # in this case full-scan search should be preferred by query planner and additional indexing is not required.
    # Note: 1Kb = 1 vector of size 256
    full_scan_threshold_kb: 10000