Request-only masters without shared server directory

[Agalin]

Hi, I'm aware of the discussion in #539 about scaling Devpi Server through replicas and requests-only nodes as well as official limitations of the requests-only mode.

In short:

• You can run multiple masters using the same shared server directory as long as exactly one of them is not running in the requests-only mode.
• You can use replicas to replicate from those masters. That one master will handle event propagation.
• You should be able to run replicas in requests-only mode with the same limitations applied as for masters - only one node should be responsible for the synchronization thread.
• There are additional limitations if the default storage (sqlite) is used due to write concurrency limitations. In this case requests-only nodes cannot be used for anything package-related.
• You can expect strange issues if PyPI mirroring is used.

Currently we have a single master + a few replicas in different datacenters (scenario replicas were designed for) with the default (sqlite) storage used.

I'm testing a kubernetes setup for Devpi instead. In this case devpi-postgresql is used for storage. Each pod has its own server directory with only the secret file being shared. LDAP is used for authentication.

What I've observed till now:

• Single master works just fine.
• Single replica works just fine.
• Multiple full replicas using the same database cause race conditions and break synchronization. Sad but not really unexpected.
• Requests-only replicas fail when packages mirrored from PyPI are accessed with an error related to the synchronization thread not being there. That's expected I guess.
• Multiple masters in the requests-only mode using the same database appear to work just fine for reads.
• Multiple masters in the requests-only mode using the same database appear to work fine for PyPI-mirrored packages and uploaded packages too.
• The only full master attached to the database shared with requests-only masters works fine when used as a target for package uploads.
• It seems to crash without any errors (trying to reproduce with --debug right now) when files are uploaded (or synced from PyPI) by another instance. Currently running in debug and for some reason it didn't crash despite crashing each time a package was changed by another node when running in non-debug mode.

I've tried to check the implementation but I'm still not sure what exactly is the full master responsible for besides replication. What would I lose by running all masters in the requests-only mode? In this setup I don't need devpi-powered replication at all as all instances can access the same database and replication for disaster recovery can be handled on the database level. Would setting UUID of all requests-only masters to the same value help?

Yes, I'm aware of devpi not supporting read-only replicas for postgres (which is unfortunate but fixable to a degree - I see there are already connection requests explicitly stating write=False that is just ignored). It would still be much easier for me to have all data stored in a separate database.

[fschulze]

Nice analysis of the current state.

With current devpi-server and devpi-web you might loose some search index updates when using all servers in --requests-only mode. The event serial update loop won't run and thus not trigger any of the hooks which cause the search index to be updated. Some other plugins like devpi-jenkins also use the event hooks.

Have you tried to route all non GET requests to the primary (rename from master is planned) running in default mode? With current devpi-server and devpi-postgresql the setup with on regular devpi-server and several --requests-only which only serve GET using one shared DB should be quite performant and stable, but I haven't tested it myself.

The UUID only matters for actual replications from a primary.

The --requests-only option has no connection to regular replication. By default devpi-server runs in standalone mode and doesn't offer the replication routes, you explicitly have to set the master role.

Do you still have any of the tracebacks you got when you ran several devpi-server against the same PostgreSQL DB? There might still be some caches and other places which assume to run in exclusive mode and fail when there are multiple writers, even with devpi-postgresql.

[fschulze]

Hmm, from the top of my head I currently can't think of what could cause the memory spike. The two known things are the indexing and on replicas looping over all the files to check whether they exist or still have to be downloaded. Can you try to run the primary with --debug and create an issue with the last log lines before the crash, preferably without any other activity like requests? If it doesn't happen without any requests, then it might be the project names fetching from PyPI.

[Agalin]

I'll try but it my take me a few days to find time for that.

[Agalin]

Well. There aren't any useful logs printed with --debug either. 🤷

2023-10-13 14:33:18,015 INFO  NOCTX Loading node info from /devpi/.nodeinfo
2023-10-13 14:33:18,016 INFO  NOCTX wrote nodeinfo to: /devpi/.nodeinfo
2023-10-13 14:33:18,088 DEBUG NOCTX Using use_copy=True
2023-10-13 14:33:18,192 DEBUG NOCTX Using use_copy=True
2023-10-13 14:33:18,195 DEBUG [Rtx79170] closing transaction at 79170
2023-10-13 14:33:18,213 INFO  NOCTX running with role 'master'
2023-10-13 14:33:18,213 DEBUG NOCTX creating application in process 1
2023-10-13 14:33:18,225 INFO  NOCTX Using secret file '/devpi/secret/secret'.

Is it possible to get any additional logs through --logger-cfg?

[fschulze]

Does it crash after that last log line? The --debug setting is for all loggers. It might be possible to squeeze out some more from libraries, but I doubt those would be relevant to the crash.

[Agalin]

Yeah, that's the last line in logs before the crash.