【ef_search】set hnsw.ef_search = 1001 failed

[frg01]

Why is the boundary value of this parameter only 1,000, and there are tens of thousands of data in my table, but only 1,000 entries can be returned through the index? I don't think it makes sense

1. 
CREATE TABLE items_update4 (id bigserial ,embedding vector(3));

2. 
CREATE OR REPLACE FUNCTION public.add_vector_data(
 tb_name varchar,
    row_cnt integer,
 beg_id integer,
    vector_dimen integer, 
 vector_v_mul integer 
 )
    RETURNS void
    LANGUAGE 'plpgsql'

    COST 100
    VOLATILE 
AS $BODY$
declare
v_row_idx integer := 0;
embedding varchar := '';
v_strSql varchar := '';
begin         
  while v_row_idx < row_cnt loop  
 embedding := '''[' || array_to_string(ARRAY(SELECT random() * vector_v_mul FROM generate_series(1, vector_dimen)), ',') || ']''';
 v_strSql := 'insert into '||tb_name||'(id,embedding)values('||beg_id+v_row_idx||','||embedding||') ;';
 EXECUTE v_strSql; 
 v_row_idx = v_row_idx+1;
 if v_row_idx%1000=0 then
     raise notice'row_cnt: %',v_row_idx;
 end if; 
 
  end loop; 
end
$BODY$;

3.
SELECT COUNT(*)
FROM (
    SELECT * 
    FROM items_update4 
    ORDER BY embedding <-> '[1,2,3]' 
    LIMIT 10000
) AS subquery;

4. 
create index items_update4_spann_1 on items_update4 using hnsw (embedding vector_l2_ops);

5.
SELECT COUNT(*)
FROM (
    SELECT * 
    FROM items_update4 
    ORDER BY embedding <-> '[1,2,3]' 
    LIMIT 10000
) AS subquery;

No index can return any quantity, and creating an index can only return the number specified by the ef_search.
If you know, please tell me why

[jkatz]

@frg01 What is your use case where you're trying to return more than 1000 entries in a single query?

[ankane]

Hi @frg01, hnsw.ef_search is limited to 1,000 to limit memory usage. IVFFlat is recommended if you need more than 500 results (docs).