feat: add config db-hoisted-tx-settings to allow only hoisted function settings

[taimoorzaeem]

Fixes #3242.

@steve-chavez @laurenceisla @wolfgangwalther Even though I have applied the filter for function settings using the new config, but the setting is still being applied in io tests. Am I testing it right?

[wolfgangwalther]

The settings will always be set inside the function, so this RPC will not be helpful to test whether the setting is hoisted or not.

To actually test the hoisting behavior... you could:

• either test the expected behavior, so for example add a statement_timeout here with 1s, then wait for 2s in the io test, then check that the statement was timed out. This should differ when you enable the hoisting and when you disable it. Maybe you can find a different setting, where this is easier to test, though.
• Or test the current_setting stuff like you currently do, but do it in a computed column. So, let the RPC return some view/table type, add a computed column to return the current_setting for this view/table, then assert that output. The computed column will be outside the scope of the RPC, and thus the settings will only apply when they are actually hoisted.

The computed columns approach is very similar to what you have already, so probably easiest.

[taimoorzaeem]

@wolfgangwalther How about this?

-- returns computed fields
create or replace function not_hoisted_should_not_apply()
returns table (
  work_mem text,
  statement_timeout text
)
as $$
begin
  return query
          select
            current_setting('work_mem',false) as work_mem,
            current_setting('statement_timeout',false) as statement_timeout;
end;
$$ language plpgsql
set work_mem = '5000'
set statement_timeout = '5s';

-- calls the above function
create or replace function test_not_hoisted_should_not_apply()
returns text
as $$
  select work_mem || ' ' || statement_timeout
  from not_hoisted_should_not_apply();
$$ language sql

Calling test_not_hoisted_should_not_apply yields 5000kb 5s which is strange because only work_mem is hoisted but statement_timeout is still being applied. Is this right way to write computed fields?

[wolfgangwalther]

Is this right way to write computed fields?

No, you are not even using a computed field, yet.

Here's how:

create function rpc_with_settings() returns items as $$
  select 1
$$ language sql
set work_mem = '5000'
set statement_timeout = '5s';

create function get_work_mem(items) returns text as $$
  select current_setting('work_mem', true)
$$ language sql;

create function get_statement_timeout(items) returns text as $$
  select current_setting('statement_timeout', true)
$$ language sql;

Then query it like this:

GET /rpc/rpc_with_settings?select=get_work_mem,get_statement_timeout

Depending on your hoisting settings, this will return different results.

[taimoorzaeem]

@wolfgangwalther Thanks a lot! It worked.

[taimoorzaeem]

It appears that the test case is failing, however, the test passes when run with postgrest-test-io -k test_only_hoisted_settings_are_applied.

[wolfgangwalther]

It appears that the test case is failing, however, the test passes when run with postgrest-test-io -k test_only_hoisted_settings_are_applied.

You could try to make the RPC plpgsql instead of SQL to make sure to avoid any inlining - although that should not happen in this case anyway.

Or maybe some other test sets the statement timeout to 5s? What if you change the timeout on the RPC to e.g. 6s? Does wrong expectation change as well? Then you'd be sure that this is actually the value we are talking about.

Not exactly sure what happens.

[wolfgangwalther]

It seems like you need to update the pytest snapshot for schema cache, too.

[wolfgangwalther]

This test is still broken. It was broken before - you can't receive the config values from inside the function. You need to use the computed fields.

[wolfgangwalther]

This test was broken before already, too. Same thing - we can't put the current_setting inside the RPC.

[taimoorzaeem]

Or maybe some other test sets the statement timeout to 5s? What if you change the timeout on the RPC to e.g. 6s? Does wrong expectation change as well? Then you'd be sure that this is actually the value we are talking about.

Thanks for this suggestion. I tried the changing the value to 6s and 7s and it still showed 5s at test time.

My guess is that statement_timeout has default values of 5s when all io-tests are run and 2s when an individual io-test is run with -k flag. I've changed the value to 5s to indicate default value in the test.

[wolfgangwalther]

My guess is that statement_timeout has default values of 5s when all io-tests are run and 2s when an individual io-test is run with -k flag. I've changed the value to 5s to indicate default value in the test.

The more likely explanation is that some other test is leaking it's settings.

Right, we have this in another test:

       # reload statement_timeout with NOTIFY
        response = postgrest.session.post(
            "/rpc/change_role_statement_timeout", data={"timeout": "5s"}
        )
        assert response.status_code == 204

This test needs to clean up behind and reset the timeout...

[wolfgangwalther]

This is the wrong place to do the filtering. It should be done in the SQL query in SchemaCache.hs. This already has a filter for default_transaction_isolation - which should be replaced with the stuff in configDbHoistedTxSettings:

  LEFT JOIN LATERAL (
    SELECT
      array_agg(row(
        substr(setting, 1, strpos(setting, '=') - 1),
        lower(substr(setting, strpos(setting, '=') + 1))
      )) as kvs
    FROM unnest(proconfig) setting
    WHERE setting not LIKE 'default_transaction_isolation%'
  ) func_settings ON TRUE

[taimoorzaeem]

@wolfgangwalther By the changing the filter, io-tests are failing as the config is not applying now.

LEFT JOIN LATERAL (
    SELECT
      array_agg(row(
        substr(setting, 1, strpos(setting, '=') - 1),
        lower(substr(setting, strpos(setting, '=') + 1))
      )) as kvs
    FROM unnest(proconfig) setting
    WHERE setting NOT LIKE ALL(ARRAY['|] <> encodeUtf8 (T.intercalate "," hoistedSets) <> [q|'])
  ) func_settings ON TRUE
  WHERE t.oid <> 'trigger'::regtype AND COALESCE(a.callable, true)
|] <> (if pgVer >= pgVersion110 then "AND prokind = 'f'" else "AND NOT (proisagg OR proiswindow)")

I get,

FAILED test/io/test_io.py::test_first_hoisted_setting_is_applied - assert '{"get_work_m...imeout":"7s"}' == '{"get_work_m...imeout":"2s"}'
FAILED test/io/test_io.py::test_second_hoisted_setting_is_applied - assert '{"get_work_m...imeout":"7s"}' == '{"get_work_m...imeout":"7s"}'
============= 2 failed, 157 passed, 1 skipped in 67.99s (0:01:07) ==============