I think this should be better integrated with Gatsby, at least downloading the files within something like onPreInit so that we can ensure the required data is at least attempted to be fetched no matter where gatsby is called from.

I have been wondering about a better place for data fetching and onPreInit seems to be a good idea instead of a node script. I also wanted this part to be integrated into the theme itself, but the implementation I was thinking of would require lots of async refactoring to already implemented sidebar logic.

We also need to keep in mind the possibility that we open this theme up for general use in the future.

The implementation is trying to be general to our use case where we have sidebar.json in GitHub repos. But, we can easily generalize by taking the source of the JSON file instead of the repo and main branch.

const tools = [
  {
    name: 'dvc',
    data: dvcSidebar
  },
  {
    name: 'cml',
    repo: 'cml.dev',
    mainBranch: 'master'
  }
]

onPreInit would be the right place for data fetching but for this particular case it doesn't seem to fit well.
We have the packages/gatsby-theme-iterative/src/utils/shared/sidebar.js file load before the onPreInit is triggered.

The way we have set up the sidebar plus(sync operation) and the data fetching being an async operation, it is difficult to fit in the process. I don't think converting this into a source node and consuming from there is also not a good idea. WDYT? cc: @rogermparent

The way we have set up the sidebar plus(sync operation) and the data fetching being an async operation, it is difficult to fit in the process. I don't think converting this into a source node and consuming from there is also not a good idea. WDYT?

I actually do think using Gatsby's GraphQL system somehow would be the best option, but it would take a lot of dev work. We can still make a useful iteration by adding a basic data-fetching pattern for external sidebars. The first thing that comes to my mind is a thunk-like implementation that exposes the sidebar values from a module, but delays downloading until onPreInit, or otherwise on first usage.

I think we're definitely going to have to turn a few things async no matter how we go about it, since fetching sidebars from external repos is inherently async. Hopefully we can isolate all of that to build-time, and the site itself can just consume static data.

I actually do think using Gatsby's GraphQL system somehow would be the best option, but it would take a lot of dev work.

I gave another round look into this implementation and it feels like converting the sidebar data into nodes will not only need a lot of dev work but also consuming those nodes also doesn't seem to be efficient.

If we were to use it for simple data rendering it would be a good idea to use the gatsby solution for it. But, we have more than such use case, we have to use the data beyond data rendering and for the command-line, api-linker, etc. I also find it difficult to consume it over gatsby-remark plugins. Also, I think the approach will going to use multiple graphql data-query for same data since it's going to be used on multiple places.

Most of the sidebar logic is here.

My take on this is that let's proceed with the simple straightforward solution for now and review/suggestions for improving the current solution would be a great way to proceed.

My take on this is that let's proceed with the simple straightforward solution for now and review/suggestions for improving the current solution would be a great way to proceed.

100%, nitpicks aside you have a very viable solution here that fixes the problem and can be built on later. Nice work!

Closing as it will require setup changes in all the websites and also its significance decreased as we are moving towards unifying if possible. We can come back anytime later if required.