Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[bug] when using a validator with a default for nested data it parses value via toml twice #979

Open
rochacbruno opened this issue Aug 16, 2023 · 0 comments
Assignees
Labels
Milestone

Comments

@rochacbruno
Copy link
Member

Describe the bug

from __future__ import annotations

from dynaconf import Dynaconf
from dynaconf import Validator

settings = Dynaconf()
settings.validators.register(
    Validator("group.something_new", default=5),
)
settings.validators.validate()

assert settings.group.test_list == ["1", "2"], settings.group

Execution

$ DYNACONF_GROUP__TEST_LIST="['1','2']" python app.py

expectation:

settings.group.test_list == ["1", "2"]

current behavior

$ DYNACONF_GROUP__TEST_LIST="['1','2']" python app.py
Traceback (most recent call last):
  File "/home/rochacbruno/Projects/dynaconf/tests_functional/issues/905_item_duplication_in_list/app.py", line 21, in <module>
    assert settings.group.test_list == ["1", "2"], settings.group
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: {'something_new': 5, 'TEST_LIST': [1, 2]}

What is happening?

At the end of validators.validate.validate the Validator("group.something_new", default=5), is executed and then default value is merged with current data, during that process dynaconf is calling parse_conf_data passing tomlfy=True which forces toml to evaluate the existing data again so.

In [1]: from dynaconf.utils.parse_conf import parse_conf_data

In [2]: data = {"TEST_LIST": ["1", "2"]}

In [3]: parse_conf_data(data, tomlfy=True)
Out[3]: {'TEST_LIST': [1, 2]}

In [4]: parse_conf_data(data, tomlfy=False)
Out[4]: {'TEST_LIST': ['1', '2']}

Possible solutions:

Change setdefault method to use tomlfy=False
Change parse_with_toml to avoid doing that transformation.

@rochacbruno rochacbruno self-assigned this Aug 16, 2023
rochacbruno added a commit that referenced this issue Aug 16, 2023
3 years ago I made the wrong decision to pass validators defaults to
TOML parser.

Validator object is created at Python level, or at YAML level where
there is no need to use toml as parser.

TOML parser must ideally be used only on environment variables
where there is no type system available.

That said, this PR ensures toml will keep parsing values coming from
envvars, but setdefault method will stop passing `tomlfy=True`
so the type informed on the validator will be the type set as default
value.

The only possible breaking change on this is if someone is setting
defaults relying on TOML to parse the data, in that case, user
will need to manually call `parse_with_toml` on default data.

I think that is not a common case, if someone reports it as a broken
change, we can provide a new argument `parse_default_with_toml` on
the Validator object later.
@rochacbruno rochacbruno linked a pull request Aug 16, 2023 that will close this issue
@rochacbruno rochacbruno added this to the 3.3.0 milestone Aug 21, 2023
@rochacbruno rochacbruno modified the milestones: 3.3.0, 3.2.3, 3.2.4 Sep 7, 2023
@pedro-psb pedro-psb modified the milestones: 3.2.4, 3.2.10 Sep 21, 2023
@rochacbruno rochacbruno modified the milestones: 3.2.10, 3.3.0 Oct 24, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants