Stale Account Index Cache

[mikegrima]

One potential issue that can arise is that our singletons may contain stale cached data. AWS Lambda will periodically rotate out the container running the lambda function but until it does, there could be stale data affecting workloads.

This can particularly be an issue with the Account Index and the Starbase. The following scenario can happen:

1. The Starbase runs and loads the account index
2. A new AWS account is created and added to the organization
3. At this point, the account index is now stale - both on S3 and the cached singletons
4. Eventually, the AccountIndexGeneratorShip runs to update the account index in S3
5. The Starbase may still have a lingering Lambda function or 2 primed up with the stale index cached
6. A new task appears and workers need to be tasked
7. An older Starbase lambda is running the logic to task the workers - the workers are not tasked on the new account because the cached account index on the Starbase doesn't see it yet
8. After a while, AWS Lambda rotates out the old Starbase lambdas and new ones are invoked and run with the latest index
9. The task runs again (eventually via an EventBridge CRON if set) and the workers are tasked with the newly created account - all is well now

Eventually, everything will iron itself out. However, I can see value in having the default account index keep a timestamp variable of when it was last instantiated and have it refresh after maybe 30 minutes after that timestamp has passed. That would reduce the likelihood staleness in the Starbase.