--extensions_require does not exit on fail, and keeps trying to load bad extension

[getvictor]

Bug report

When I try to load a bad extension, like:

./osqueryi --extensions_autoload=/Users/victor/work/fleet/extensions.load --extensions_require=example_extension,example_extension2

osquery does not exit. Instead, it keeps trying to reload the extension every 2 seconds, giving me warnings. Setting extensions_timeout and extensions_interval do not change the behavior.

E0123 15:20:32.208709 1805398016 virtual_table.cpp:1115] Error creating named virtual table: example_table (1)
W0123 15:20:32.208743 1805398016 interface.cpp:143] Could not add extension example_extension2: SQLITE_ERROR
2024/01/23 15:20:32 status 1 deregistering extension: No extension UUID found
W0123 15:20:32.218493 -638578688 extensions.cpp:781] Required extension not found or not loaded: example_extension2
E0123 15:20:32.218505 -638578688 init.cpp:714] An error occurred during extension manager startup: Required extension not found or not loaded: example_extension2
Using a virtual database. Need help, type '.help'
osquery> W0123 15:20:34.971463 1802530816 watcher.cpp:738] Extension respawning too quickly: /Users/victor/work/fleet/example_table2.ext
W0123 15:20:35.187198 1804824576 interface.cpp:143] Could not add extension example_extension2: Duplicate registry item: example_table
2024/01/23 15:20:35 status 1 deregistering extension: No extension UUID found
W0123 15:20:37.975450 1802530816 watcher.cpp:738] Extension respawning too quickly: /Users/victor/work/fleet/example_table2.ext
W0123 15:20:38.190068 1805398016 interface.cpp:143] Could not add extension example_extension2: Duplicate registry item: example_table
2024/01/23 15:20:38 status 1 deregistering extension: No extension UUID found
W0123 15:20:40.979302 1802530816 watcher.cpp:738] Extension respawning too quickly: /Users/victor/work/fleet/example_table2.ext
W0123 15:20:41.197801 1804824576 interface.cpp:143] Could not add extension example_extension2: Duplicate registry item: example_table
2024/01/23 15:20:41 status 1 deregistering extension: No extension UUID found
W0123 15:20:43.985483 1802530816 watcher.cpp:738] Extension respawning too quickly: /Users/victor/work/fleet/example_table2.ext
W0123 15:20:44.198812 1805398016 interface.cpp:143] Could not add extension example_extension2: Duplicate registry item: example_table
2024/01/23 15:20:44 status 1 deregistering extension: No extension UUID found
W0123 15:20:46.991468 1802530816 watcher.cpp:738] Extension respawning too quickly: /Users/victor/work/fleet/example_table2.ext
W0123 15:20:47.206677 1804824576 interface.cpp:143] Could not add extension example_extension2: Duplicate registry item: example_table
2024/01/23 15:20:47 status 1 deregistering extension: No extension UUID found
W0123 15:20:49.998178 1802530816 watcher.cpp:738] Extension respawning too quickly: /Users/victor/work/fleet/example_table2.ext
W0123 15:20:50.223445 1805398016 interface.cpp:143] Could not add extension example_extension2: Duplicate registry item: example_table
2024/01/23 15:20:50 status 1 deregistering extension: No extension UUID found
W0123 15:20:53.220268 1804824576 interface.cpp:143] Could not add extension example_extension2: Duplicate registry item: example_table
2024/01/23 15:20:53 status 1 deregistering extension: No extension UUID found

This issue is somewhat related to #7256 since they touch the same code.

What operating system and version are you using?

macOS

What version of osquery are you using?

5.11.0

What steps did you take to reproduce the issue?

What did you expect to see?

I expect osquery to exit if --extensions_require is not satisfied.

I expect osquery to not keep trying to reload a bad extension.

What did you see instead?

[getvictor]

If the extension fails to load, it should sleep instead of trying again. sleep can be added here:

osquery/osquery/extensions/impl_thrift.cpp

Line 268 in 612656b

_return.code = (int)extensions::ExtensionCode::EXT_FAILED;

[Smjert]

I wouldn't put the sleep so deep; that's more a generic interface toward thrift.

That being said, there are two pieces at play here, the watcher/watchdog which attempts to start the extensions passed via --extensions_autoload, and the worker that waits some time the extensions to be registered, if extensions_require is given.

The watcher starts the extensions and if they die, they get restarted immediately, or at least it attempts to, but limits itself and warns about the fact that the extension is being restarted too quickly.
The logic is here:

osquery/osquery/core/watcher.cpp

Line 390 in 612656b

createExtension(extension.first);

Then you have the worker which waits for the extensions to be registered before proceeding with its own initialization internally, because if the extensions are providing some fundamental feature (like a config or logging plugin), then that will cause failures for sure in the worker.
This is handled here:

osquery/osquery/core/init.cpp

Line 655 in 612656b

auto status = applyExtensionDelay(([type, name](bool& stop) {

But worker and watcher do not communicate in this.

Furthermore, I'm not sure that extensions_require was ever intended to have the osquery worker exit if it cannot register the extensions in time (or if the extensions themselves are not started in time).
You have have to consider that extension may be loaded manually and not be controlled by osquery.

If an extension is providing a fundamental feature, like a config or logger plugin which is enabled via flags, osquery will fail anyway to enable them, and will exit.
If the extensions provide tables, then simply osquery won't be able to query those and the rest of the functionality will work, which might be desirable, instead of not having access to osquery at all.
Some do still put extensions_require with extensions providing tables because they don't want to see a query returning with an error (once), due to the table not being found, if the extension doesn't really have an issue.

I would also consider this desire here: #8067

EDIT: Oh and the definition of "bad extension" I think needs to be clarified. When the watcher is launching a managed extension, it restarts it if it exits, but doesn't know why it exited.
It just treats it as the worker; it's an important process that the user asked to keep alive.

[getvictor]

Yes, I agree that --extensions_require doesn't seem useful for table extensions. I wouldn't want to specify it every time for every extension.

I opened a PR to change warnings to errors when extension could not be added: #8260

[directionless]

Yes, I agree that --extensions_require doesn't seem useful for table extensions. I wouldn't want to specify it every time for every extension.

If the tables are used for decorators, one might want this on table extensions