Dynaconf 4.0 changes and ideas

[rochacbruno]

Dynaconf 4.0

What is Dynaconf

Dynaconf is a configuration management library written in Python, it is designed to provide the following features:

• Environment first config management
  • Environment variables rules
    • unless you explicitly say it doesn't (e.g: CLI arguments)
  • If not then config maps and settings servers (Vault, Redis, Etcd)
  • If not then fall back to settings.{anyformat} files on the filesystem
  • If not then go with default values
• Schema is optional but can be strict and validated
• Everything is Lazy and it allows interpolations, transformations, parsing
• Pure Python, Django, Flask, FastAPI are supported
• Dependencies are vendored for easy packaging and distribution
• Settings can be layered on multiple environments

Quick Start

project/config.py

from dynaconf import Dynaconf

settings = Dynaconf(
    settings_file="settings.yaml"
)

then

project/app.py

from project.config import settings
print(settings.NAME)

NAME will hold the value of one of those

If your CLI app receives arguments it has the highest precedence once you mark it as an important key.

$ myapp.py --key=value

Then environment variables will rule over anything else.

$ export DYNACONF_KEY=value
$ myapp.py

Then if there is no environment variable defined, values coming from services such as config maps, Redis, Vault, Etcd and others.

Assuming redis is an enabled loader for your Dynaconf instance

$ redis-cli set key value
$ myapp.py

Also can work for Vault, Etcd, k8s config map and any other custom loader.

Then if there is no external service:

Assuming there is a settings.yaml on the local file system.

key: value

$ myapp.py

Then if there is NO source of configuration, the default values might be used.

# app.py
settings = Dynaconf(key="value")
print(settings.get("key", "default value))

There are some ways to define default values, on Validators, as explicit passed in keyword arguments.

---

Everything explained on https://dynaconf.com works out of the box, no breaking changes (except big / design bug fixes)

---

To be NEW in 4.0

💡 Schema and class-based Validation (optional, opt-in)

from dynaconf import Dynaconf, schema

class ServerType(schema.StringType):
    host: str
    port: int

    def read(self, value):
        """Hipotetically take http://foo.bar:5000 """
        host, port = value.rpartition(":")
        self.host = host.strip()
        self.port = int(port.strip())

    def validate(self, settings, value):
        if not value.startswith("http"):
            raise schema.ValidationError("Invalid Server")

class MySettings(schema.BaseSettings):
    name: str
    server: ServerType   # Types are composable
    port: int

    class Meta:
        strict = True
        strict_fields = ['name']  # for when wide strict is False
        readonly = ["name"]

    def validate_<field_name>(...):

    def read_<field_name>(...):

    def write_<field_name>(...):

then

settings = Dynaconf(
    settings_file="settings.yaml",
    schema=MySettings
)

• Loaders will occur normally, but there will be a de-serialization step based on the schema
• Validation will be performed first using the schema, then fall back to legacy Validators
• Dynaconf Validators will be deprecated in favor of class-based schema
• When strict=False (the default) the Schema will be used for validation and parsing, but variables out of the schema will still be loaded
• When strict=True only variables defined on the schema will be loaded

Questions

• Pydantic or purely Python types?
• Refactor or write from scratch?
• Keep Box or replace with ChainMap? (how to do that without breaks)
• YAML must be the preferred format instead of TOML?
• Make TOML parser for envvars optional?

[ZhymabekRoman]

#TODO: feat in 4.0 - make able writing the configuration values directly to a file. For example - settings["param"] = "key"

[rochacbruno]

@ZhymabekRoman you mean mutating the setting file?

This is difficult, take this scenario:

# there is a settings.toml
"""
foo = "bar"
"""

# there is a settings.yaml
"""
foo: bar
other: value
"""

# there is the settings loading both files
settings = Dynaconf(settings_files=["settings.toml", "settings.yaml"])

# User also enabled the vault/redis source
export VAULT_CONFIG=....
export REDIS_CONFIG=...

So in the above we have 5 data sources

• toml
• yaml
• redis
• vault
• environment variables

When calling

settings["key"] = "value"

Other than the in-memory object, to what source it should write?

It is already possible to dump settings or specific keys individually.
https://www.dynaconf.com/advanced/#exporting

[ZhymabekRoman]

@rochacbruno, Maybe it makes sense then to implement something like the priorities of mutating and/or reading configuration files? Or may be to implement something in addition like:

settings.redis["key"] = "value"
settings.ini["key1"] = "value1"

For example, why do I need to change the configuration file myself? Suppose I gave my utility for solving some business problem to someone who is not very good at programming and doesn't want to get into it. And I say to him - open a terminal and type in there - python3 init_configurations.py and answer the script questions and that's it, you'll have a configured utility. Up to this point, I've implemented it like this:

import os
import sys

USER_DATA_PATH = "bot/user_data"

print("\n===Config Master===\n")

if os.path.isdir(f"{USER_DATA_PATH}/") is True:
    print("Users data and configuration folder is already exists! Exiting...")
    sys.exit(1)
else:
    os.makedirs(USER_DATA_PATH)

tlgrm_bot_api = input("Enter Telegram bot API token: ")
audfprint_mode = input("Select the audfprint working mode: \n    1 - Fast audio recognition speed, but worse accuracy\n    0 - High recognition accuracy, but will take longer time\nEnter 0 or 1: ")

with open(f"{USER_DATA_PATH}/config.py", "w") as file:
    file.write("# Declare Telegram Bot API token\n")
    file.write(f"API_TOKEN = '{base64_encode(tlgrm_bot_api)}'\n")
    file.write("# Declare users data path\n")
    file.write(f"USER_DATA_PATH = '{USER_DATA_PATH}'\n")
    file.write("# Declare audio recognizing mode\n")
    file.write(f"audfprint_mode = '{audfprint_mode}'\n")
    file.write("# Declare users FSM state storage\n")
    file.write(f"FSM_FILE_STORAGE = '{USER_DATA_PATH}/FSM_state_storage.json'\n")

print("Done!")

Scary script, isn't it? This script is very difficult to maintain and the most dangerous thing is that you can easily drop the utility which will read this file by writing one of the quotes to the value of parameter. I would like it to look something like this:
settings["bot_api_token"] = "987198719:TOP_SECRET_BY_FBI"

[phitoduck]

Hi! I'm super excited about this document. Do you have an ETA for something like an alpha release for Dynaconf v4?

I've just started seriously looking into Dynaconf (found a YouTube presentation from 2021). The pydantic schema feature looks amazing. We use FastAPI and Pydantic at work. We like the pydantic.BaseSettings class, but it still requires you to write a lot of code if you want features like: having different environments, reading from sources that aren't ENV vars, writing to files, etc. The main reason we like pydantic for our config is that it takes care of reading ENV vars, it gives us autocompletion, and we can define our own validators--however, the auto ENV var reading breaks down for pydantic if you do nested schemas. It looks like your plan doesn't, though!

Idk if Dynaconf has this feature, but something cool about Pydantic is community-maintained validators/types, for example EmailStr or PostgresDsn. These two essentially check if (a) the config value is a string, and (b) the config value matches a certain regular expression. Does Dynaconf already have something like this? If not, I think it'd be awesome to either hook into Pydantics validators or make a clear plugin system that enables the community to develop these for Dynaconf.

The moment the schema-driven Dynaconf lands, I'm going to present it and try to make it our standard at work. Thanks for the work your doing making config easy!

[hugoprudente]

Pydantic or purely Python types?

I have been using Pydantic over purely Python for over 2 years now on a internal SDK and I'm really enjoying the way that the validators and the power to generate OpenAPI 3.1 schemas works.

I know that Pydantic in the overall is a little bit slower than dataclass & attr but we have extra validations. I think the bast path on this should be evaluate the trade-off.

Keep Box or replace with ChainMap? (how to do that without breaks)

I'm all in favor of ChainMap :), but can be trick create a backwards compatibility transformation.

YAML must be the preferred format instead of TOML?

I do like TOML but i think YAML for multilevel configurations are much easier to read, but that is personal preference.

Make TOML parser for envvars optional?

If YAML is preferred method then the TOML parser for envvars should be optional .

[nische]

Hi together,

is there any timeline for the Version 4? Espacially the schema thing would be nice.

BR,
Nico

[jorpilo]

Keyring integration (loader and export)