Explicitly Hidden (from the UI) Configuration Values #19033
Replies: 3 comments
-
Could someone please chime in on this one in regards to if will be considered for the roadmap. In my example I am grabbing secrets from hashicorp vault but it's the same concept (see code/ui below for context). To make testing easier and keep objects decoupled I would prefer to just set the api_key on the resource rather than inject the vault client. If I move to CyberArk etc later it would require less refactoring etc not to mention it makes unit tests cleaner/easier as your passing simple type (string) vs complex (object etc). The workaround above is likely the path I would go down at this stage. Thanks for sharing @Daniel-Vetter-Coverwhale. Suggestion Thanks in advance. Really liking Dagster I have looked at ALOT of other tools and the assets as code, lineage and materialization really make a huge difference. CODE import os
import hvac
from dagster import Definitions, load_assets_from_modules
from .assets import stocks
from .resources import FinancialModelingPrepApi
stock_assets = load_assets_from_modules([stocks], group_name="stocks")
vault_client = hvac.Client(url=os.getenv("VAULT_URL"), verify=True)
vault_client.auth.approle.login(role_id=os.getenv("VAULT_APP_ROLE_ID"), secret_id=os.getenv("VAULT_APP_ROLE_SECRET"))
fmp_api_key = vault_client.secrets.kv.v2.read_secret_version(path="financial-modeling-prep-api", mount_point="kv")['data']['data']['API_KEY']
defs = Definitions(
assets=[*stock_assets],
resources={
"fmp_api": FinancialModelingPrepApi(api_key=fmp_api_key),
},
) |
Beta Was this translation helpful? Give feedback.
-
Also would love to see this. |
Beta Was this translation helpful? Give feedback.
-
I imagine that subclassing the
|
Beta Was this translation helpful? Give feedback.
-
Is there any way to mark a configuration value as secret to avoid showing in the UI without it being an envvar?
My current use case is pulling database configuration including passwords from AWS secrets manager and building sling connections from that info, and if I open the launchpad for one of the sling assets using the sling resource it shows everything. I think it would be awesome if for things like SlingResources or other third party published configurable resources to accept things like passwords for fully specified instances, but mark them so the user doesn't have to worry about whether they show up in the UI.
I realize that I can set an environment variable and then reference it as part of loading the secret (which is what I'm doing for now), but it would feel better and more generally useful I think to just have some other way of specifying that a particular configuration value should never be shown in plaintext.
Some sample code showing how I'm building the sling connections from the secrets, currently using dagster.EnvVar which requires of course that we set the value from the secret into a visible (from the system) environment variable. As mentioned above I would prefer to not have to use the environment variable to hide the password from the UI. Also I wonder at the value of the having separate Source and Target connection classes, but that's a different issue.
Beta Was this translation helpful? Give feedback.
All reactions