NotGitRepository at /course/applicant-quiz1/ - after delete of local cache

[KlavsKlavsen]

if I delete local cache (because I am running on k8s and pod restarts) - relate does not DL course again - it just fails with:
"NotGitRepository at /course/applicant-quiz1/"

is there a command to "re-download" cached courses - I could run at pod startup to fix this? instead of having to use persistent storage to keep cache forever (which would also mean it was necessary to have a backup of a cache (highly unusual and should really be unnecessary) - for otherwise things won't work, unless I delete all courses and recreate them - possibly loosing previous exam results?)

[inducer]

I agree that such a thing could be implemented.

That said, I have tended towards viewing the course repos in Relate as "authoritative" information, rather than "cached" information, and future changes that I have planned are headed further into that direction, such as with a built-in dev environment that allows you to make changes (i.e. commit to the repo) from within the web UI. So, operationally, I would recommend backing up those repos (and not viewing them as caches).

[KlavsKlavsen]

Well.. since you ask for GIT repos and access to them - I don't see why you distrust those same git repos so much, so you keep the "first copy you get" - and then suddenly only trust that.
That seems unnecessary to say you don't trust upstream (git repo with course data) after first pull - and yet when admin choose "update course" - you don't do anything to help show a diff - if you wanted to protect against anything nefarious..

So I don't see any benefit from treating relate's cache as nothing more than that with the current way its used - and frankly I don't see any huge benefit in it.

[KlavsKlavsen]

You COULD add GPG checking of signing if you wanted to create some real security. I gpg sign ALL my commits (setup in git config) - with my Yubikey - so I could add my public gpg key in the list of "trusted signining keys" in relate f.ex. and relate could verify entire gpg repo before using it. That would benefit security.

[inducer]

I think we're talking past each other a bit here. I'm not super worried about the authenticity of the stuff in the git repo.

What I'm trying to say is that I hesitate to call the git repo maintained by Relate just a passive copy of something for which the authoritative source of truth lives somewhere else (i.e., a cache). That's for the following reasons:

• Before too long, I want to be able to edit the course through the web interface. (cf. IDE-like interface #70, Configurable Forms for flows #679) That would produce commits that, at least for a time, exist only in Relate's copy of the repo. This is the point where that repo becomes a source of truth rather than a cache.
• Operationally, if you look at the list of courses on the instance I maintain (https://relate.cs.illinois.edu/), I can virtually guarantee that 80% of the configured git sources no longer work, particularly for older courses. (SSH keys removed, servers moved, SSH keys no longer usable because of SHA-1, what have you). So if I threw away the repos corresponding to them, I'd no longer be able to render any pages for those courses, potentially losing lots of info that was previously on the open web.

[KlavsKlavsen]

@inducer re. the operational part - it should detech such discrepancies (out of sync with master git repo or master git repo gone) and display alert to teachers, staff, admins IMHO. Or atleast that should be an option to do so, so one notices and is enticed to fix it before its too late.
It should also HEAVILY complain if it can't sync with an upstream git repo thats given as the source (or that its ssh key does not give the access to do so) - so one knows to fix that before it becomes a real problem.
Backups do fail - and having pushed data to its upstream git repo would indeed protect it against failure on relate side of things.

Its also an unnecessary operational problem, that if the local copy of these repos are gone - even though upstream git repos works just fine - and it could just pull data from there - it doesn't even try, or offer an opportunity to do so, forcing one to delete all course data (in pg too) and recreating course :(

[inducer]

it should detech such discrepancies (out of sync with master git repo or master git repo gone) and display alert to teachers, staff, admins IMHO.

I understand where you're coming from, and I agree that what you describe is a self-consistent view of one way to operate the system. It just doesn't have much to do with my operational reality.

In my world, I'm on the hook for archiving courses for instructors who may have long left the university and who can't be counted on to have an interest in keeping their old stuff working. In addition, my perception of git commits existing only on the server is that that's not a bad place to keep them (compared to a TA's laptop); it's well-secured with backups and (hopefully) competently operated. So I don't view that as a three-alarm-fire in the same way as you do.

I think the two world views are not so disconnected that couldn't both be supported by the same code, but I think it's important to recognize that they are distinct. If you'd like to write code to support your model, I'm happy to consider it. Just recognize that it's distinct from mine.

[KlavsKlavsen]

@inducer I totally understand that - and if I were you I would be happy to have relate tell the operator (you) that a specific course no longer is consistent "on the git end" - so you can do something about it - ensuring you have the code (and history of it) you need - which is also what documents copyright etc. - and if relate helps you ensure all courses HAVE a working "git repo" - AND you ensure pgsql db backup - you're well set. You really should prefer that as well.

In automatic scaling land (k8s :) - you would preferrably NOT split the same filesystem amongst all webservers - as that would be a common bottleneck - which means your current view is bad for scaling - and it would be much better if the system could just scale up - and then automaticly (or by way of run-script commands) ensure a new set of webservers work - when scaleup is suddenly needed (around exam time f.ex. :)

[inducer]

In automatic scaling land (k8s :) - you would preferrably NOT split the same filesystem amongst all webservers

That makes sense! Even so, I feel like even gitlab.com lived reasonably happily with git repos on NFS for a long while. If/when this bottleneck shows up, I think there's still a way out by implementing on-demand fetches for local copies of the repos from either upstream or a designated write replica. So this feels like a manageable risk to me, and the benefits of just living with this outweigh the (IMO relatively solveable) future scaling challenges.

[KlavsKlavsen]

Which settings currently tell relate what paths on the filesystem is expected to be there "always" and be backed up? and is it anything other than the downloaded courses (git copy) that needs to be there?

One can definetely add NFS setup (using Rook operator in k8s) and presumably that runs "releatively well and HA" - I have not tried, I tend to use Ceph also managed by Rook operator) it - but it is all eggs in one basket IO and HA wise - if NFS has issues - everything using it has issues :)

Without that need - I can scale postgres using postgres-operator from Zalando (which uses local NVMe disks for its masters+slaves), which also takes care of backups and failover.

I do feel it annoying to be unable to scale beyond 1 pod (1 instance) - due to a simple thing as course data, which in my case could simply be fetched from git, upon setup of new instance - and I would have a scaleable setup - where only the write load on the postgres could be an issue.

Once you begin to want features, where relate writes to the git repo - sync becomes an issue - but that could relatively easily be fixed, by simply putting a last write timestamp for each course in the DB - so the code woulc check its local timestamp against that - and updating it from git - if the timestamp has been updated.

[inducer]

RELATE_BULK_STORAGE:

relate/local_settings_example.py

Line 96 in 2186f96

RELATE_BULK_STORAGE = FileSystemStorage(path.join(_BASEDIR, "bulk-storage"))

This is larger data (like PDFs or larger codes, >256 bytes) submitted by students. Once it's written, it's static. This used to be in the database, but it's too big to be in there. This already uses the Django storage framework and could be pointed at, like, an S3 bucket.

GIT_ROOT:

relate/local_settings_example.py

Line 86 in 2186f96

GIT_ROOT = path.join(_BASEDIR, "git-roots")

(as discussed above)

That's it.

[inducer]

Once you begin to want features, where relate writes to the git repo - sync becomes an issue - but that could relatively easily be fixed, by simply putting a last write timestamp for each course in the DB - so the code woulc check its local timestamp against that - and updating it from git - if the timestamp has been updated.

I agree. It's just that the code to do that isn't there yet. It's even easier than timestamps with git, you can just use the git hash, and the expected one is already in the database.