Error Handling Surrounding Flattening

[matthew-skinner]

When setting flattening_enabled to true then running meltano select tap-stackexchange --list --all an error message returns without a needed change expressed.

Example of the error:

Cannot list the selected attributes: Catalog discovery failed: command ['/meltano/.meltano/extractors/tap-stackexchange/venv/bin/tap-stackexchange', '--config', '/meltano/.meltano/run/tap-stackexchange/tap.07702730-c1d8-433b-ab2c-92b6271df905.config.json', '--discover'] returned 1

I found a bread crumb by going back and running meltano invoke tap-stackexchange --about, which returned this error:

Traceback (most recent call last):
  File "/meltano/.meltano/extractors/tap-stackexchange/venv/bin/tap-stackexchange", line 8, in <module>
    sys.exit(TapStackExchange.cli())
  File "/meltano/.meltano/extractors/tap-stackexchange/venv/lib/python3.8/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/meltano/.meltano/extractors/tap-stackexchange/venv/lib/python3.8/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/meltano/.meltano/extractors/tap-stackexchange/venv/lib/python3.8/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/meltano/.meltano/extractors/tap-stackexchange/venv/lib/python3.8/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/meltano/.meltano/extractors/tap-stackexchange/venv/lib/python3.8/site-packages/singer_sdk/tap_base.py", line 467, in cli
    tap = cls(  # type: ignore  # Ignore 'type not callable'
  File "/meltano/.meltano/extractors/tap-stackexchange/venv/lib/python3.8/site-packages/singer_sdk/tap_base.py", line 87, in __init__
    self.mapper = PluginMapper(
  File "/meltano/.meltano/extractors/tap-stackexchange/venv/lib/python3.8/site-packages/singer_sdk/mapper.py", line 540, in __init__
    self.flattening_options = get_flattening_options(plugin_config)
  File "/meltano/.meltano/extractors/tap-stackexchange/venv/lib/python3.8/site-packages/singer_sdk/helpers/_flattening.py", line 34, in get_flattening_options
    return FlatteningOptions(max_level=int(plugin_config["flattening_max_depth"]))
KeyError: 'flattening_max_depth'

I took a guess at setting flattening_max_depth config and it fixed the issue, but there is nothing that I can find stating flattening_max_depth is required when setting flattening_enabled.

A possible solution instead of updating the error handling would be to remove flattening_enabled altogether as a config and have it's state set by the flattening_max_depth. For instance, if flattening_max_depth is set then flattening is enabled to that depth, otherwise flattening is off.