Site generator: make history scale

[fahad19]

Background

We have a status site generator here: https://featurevisor.com/docs/site/

It generates history data from Git in out/history-full.json file.

Challenge

As Git repo grows over time, this file will grow too. And it's worth coming up with a strategy for scaling it, while staying a statically generated site.

Ideas

• Split the history-full.json file into paginated files like:
  • history-1.json
  • history-2.json
• Generate history on a per entity basis as well, like:
  • history-feature-foo.json
  • history-segment-netherlands.json
• If we have scaling concerns for history of individual entities, we can paginate them too:
  • history-feature-foo-1.json
  • history-feature-foo-2.json

[jagadeeshgade008]

@fahad19 It will be a good option if we use https://www.npmjs.com/package/better-sqlite3 with the filesystem as db.
I have implemented a sample example here https://github.com/jagadeeshgade008/fs-db

The good part is we can paginate easily once history is prepared

[fahad19]

@jagadeeshgade008: looks really cool!

but we have to understand that we have limitations to how we serve this site statically. there's no Node.js involved. so we cannot rely on DB-like solution that cannot run in the browser.

[jagadeeshgade008]

@fahad19 How about implementing similar to this

will create jsons based on year or if we want we can create based on day as well and when on click of day will fetch respective json instead of getting entire history json on load

[fahad19]

too many clicks I would say.

most often, people would be only interested to see the latest changes. so current list view does well there.

if user wants to load even older results, those can be fetched on demand.