Centralized config package | External hooks for platformdirs, __package__, package_dir, etc.

[chasealanbrown]

Problem

I am trying to set up a single configuration package that all of my other packages will "inherit" from (let's call it confpkg).
The idea is to be able to change the code in confpkg to add "standard" configuration values, which can then be overridden in whatever package is being developed (call it yourpkg), or some dependency that also uses confpkg (call it mid_pkg).

Desired Solution

It is desired to have this work with absolute minimal user interaction, and to have a very nice api exposed in the code of yourpkg.

For example, in yourpkg:

from confpkg import pkg_setup
CONF, CACHE, LOG = pkg_setup()

Those two lines are all that need to be included.

Now this package ('yourpkg') has these available:
- CONF['data_dir'] == "/home/user/.local/share/yourpkg/data" (Or w/e platformdirs outputs)
- CONF['cache_dir'] == "/home/user/.cache/yourpkg"
- etc - many different directories from package dirs
- CONF['pkg_name'] == __package__ (The package name will be 'yourpkg' here, but will change for different packages)
- CONF['pkg_dir'] == {some logic typically like: os.path.dirname(os.path.realpath(__file__))} (This should work to give the 'yourpkg' top-level package directory, automatically)
These values should of course be overridden by the 'yourpkg' directory's config.toml file.

This can allow easily setting things like a 'default' database filepath, etc.
(The "Another Example / Perspective" section covers how confpkg should handle other logic around this.)

---

Another example (from yourpkg still) using mid_pkg, to demonstrate this further:

from mid_pkg import BigDatabase  # NOTE:  Also uses this same `confpkg` line within the `mid_pkg` code
from confpkg import pkg_setup
CONF, LOG = pkg_setup()

db = BigDatabase()
# DB is created using it's packages code (where CONF from `confpkg` is called),
#   which can contain a default `data_dir` computed by `platformdirs` - specific to the `mid_pkg`
db.query("select * from Docs") 

However, specifically, it is desired to not have to have the same config_maker.py script in package A, B, C, etc - but rather just to have one config_maker.py or config.py in the confpkg package, which will load the directories and add the package name and dir - for the package that is calling the CONF, LOG = pkgsetup().

What is a good way to make this happen?

Solution Attempts

I have solved some of this with the inspect package, or having to load the string of the file from a package, and then execute it from within the calling package.

The solution used functions similar to this: https://gitlab.com/chase_brown/myappname/-/blob/12113f788b84a4d642e4f7f275fe4200b15f0685/myappname/util.py#L15-41
(*Yes, the package is basically useless and you shouldn't use it, but it illustrates the point)

However, I just noticed Dynaconf exists, and it seems to be very close to what I need (if not a perfect fit - I am still learning this package).

So I figured I should not re-invent the wheel, and rather use the standard tools that are extremely popular in the language.

Another Example / Perspective

A good example to illustrate the problem is the following:

• Pkg_A --(depends on)--> Pkg_B --(depends on)--> Pkg_C
• Pkg_C needs a data_dir which will hold, say 3TB of space (to construct a database for Pkg_A).
  This can be achieved with the system desired in this post - specifically a separate confpkg that can construct or append to a config.toml, (or just create entries in CONF without touching config.toml) for Pkg_A, Pkg_B, and Pkg_C, that has defaults from platform_dirs.
  However, importantly here - the confpkg should check if that default can allow for this space, and ask the user for input to alter the config.toml if and only if needed.

It's clear that copy/pasting this code (probably a few hundred lines) to every single package created by the user is not a viable solution. Especially if an institution wants to make an addition for all of the packages that confpkg uses (e.g. a standard 'welcome message' for a LOG or something.

---

So what is a good way to deal with this (extremely common) type of scenario?

[pedro-psb]

Hello, sorry for the delay. I have looked at this but didn't know how to respond immediately.

This looks like a useful use case, but we don't provide a specific out-of-the-box solution for the problem as you describe. As a library, Dynaconf provides some features for handling config, such as getting config from multiple sources, merging then, lazy evaluating, defining validators, etc. I believe you could leverage those features to implement this kind of "extensible framework" that you want. And if there is any specific gap/feature that you feel is missing to support this, it might be easier to request and implement that (as opposed to this more complex proposal).

Your use case looks somehow similar to the project I work on.
It uses a plugin system for django, and it use dynaconf to layer the settings like (sorted by loading order):

1. base_settings from core (everything in here)
2. base settings from loaded plugins
3. settings defined by users (via their settings file or envvars).

Let me know if that's helpful.