Automate monthly reports

[huddlej]

Context

We produce monthly reports describing seasonal influenza circulation patterns. A substantial portion of time spent on these reports is tedious copying and pasting or running commands manually that could be automated.

Description

The following specific tasks will minimize the manual work associated with reports and allow us to focus on the science.

• Schedule nextflu-private builds to run and deploy after data upload (like the public builds). This should be easy to do now. We need to add rules that produce dated Auspice JSONs and deploy these JSONs to the desired group.
  • Add GitHub Action workflow to run nextflu-private builds
  • Add rules to the nextflu-private workflow to rename Auspice JSONs based on the build date and deploy the JSONs to the Nextstrain group #139
  • Trigger nextflu-private builds from upload action #157
• Plot counts per lineage by region with matplotlib instead of altair #162
• Use existing public 2y builds as “representative” trees in our reports #164
• Automatically generate Markdown tables of new sequences per clade and clade coverage by titer references. The workflow logic to do this was added in Make Markdown output for reports #129, but we still need to add explicit dependencies for tabulate to our Docker and Conda environments.
  • Summarize all new sequences per clade using Nextclade annotations from the full dataset instead of using subsampled data associated with representative or titers builds. Depends on the addition of an earlier step in the workflow that runs Nextclade on all data as mentioned in Filter low-quality sequences by Nextclade QC status #144.
  • Summarize haplotype coverage by titer references using frequencies per haplotype from all available data (i.e., flu_frequecies data) and include distinct references per haplotype across all titer collections (e.g., cell FRA, cell HI, egg FRA, egg HI, etc.)
• Use a narrative template file to stub out a date-specific narrative for a given set of dated builds #163
• Automatically add links for the latest builds to the nextflu-private group landing page after builds have been deployed.
• Run TreeKnit and/or pathogen-embed on HA/NA trees/alignments and report a list of reassorted clades. Richard has already implemented workflow logic for TreeKnit. We can’t run TreeKnit from Conda, but we could add it to the Docker image and run it on AWS Batch. We don’t have a Conda package for pathogen-embed yet, but that would be easy to add.