Unable to create party or guild on local docker install

[bootymcjudy]

General Info

• Locally installed using Docker

Description

When trying to create a party or guild the site just spills out Unexpected error has occurred on the top right of the page. What is odd is I could have sworn this worked earlier this week, but now on a fresh install it stopped working

Console Errors

when looking at the logs of the server,

  method: 'POST',
  originalUrl: '/api/v4/groups/',
  headers: {
  ...
  },
  body: { type: 'party', name: "booty's Party" },
  query: {},
  httpCode: 500,
  isHandledError: false,
  fullError: MongoError: This MongoDB deployment does not support retryable writes. Please add retryWrites=false to your connection string.
      at getMMAPError (/usr/src/habitica/node_modules/mongoose/node_modules/mongodb/lib/core/topologies/shared.js:413:18)
      at handler (/usr/src/habitica/node_modules/mongoose/node_modules/mongodb/lib/core/sdam/topology.js:955:15)
      at /usr/src/habitica/node_modules/mongoose/node_modules/mongodb/lib/cmap/connection_pool.js:348:13
      at handleOperationResult (/usr/src/habitica/node_modules/mongoose/node_modules/mongodb/lib/core/sdam/server.js:565:5)
      at commandResponseHandler (/usr/src/habitica/node_modules/mongoose/node_modules/mongodb/lib/core/wireprotocol/command.js:120:25)
      at MessageStream.messageHandler (/usr/src/habitica/node_modules/mongoose/node_modules/mongodb/lib/cmap/connection.js:272:11)
      at MessageStream.emit (events.js:400:28)
      at MessageStream.emit (domain.js:475:12)
      at processIncomingData (/usr/src/habitica/node_modules/mongoose/node_modules/mongodb/lib/cmap/message_stream.js:144:12)
      at MessageStream._write (/usr/src/habitica/node_modules/mongoose/node_modules/mongodb/lib/cmap/message_stream.js:42:5)
      at writeOrBuffer (internal/streams/writable.js:358:12)
      at MessageStream.Writable.write (internal/streams/writable.js:303:10)
      at Socket.ondata (internal/streams/readable.js:731:22)
      at Socket.emit (events.js:400:28)
      at Socket.emit (domain.js:475:12)
      at addChunk (internal/streams/readable.js:293:12)
      at readableAddChunk (internal/streams/readable.js:267:9)
      at Socket.Readable.push (internal/streams/readable.js:206:10)
      at TCP.onStreamRead (internal/stream_base_commons.js:188:23)
      at TCP.callbackTrampoline (internal/async_hooks.js:130:17) {
    originalError: MongoError: Transaction numbers are only allowed on a replica set member or mongos
        at MessageStream.messageHandler (/usr/src/habitica/node_modules/mongoose/node_modules/mongodb/lib/cmap/connection.js:272:20)
        at MessageStream.emit (events.js:400:28)
        at MessageStream.emit (domain.js:475:12)
        at processIncomingData (/usr/src/habitica/node_modules/mongoose/node_modules/mongodb/lib/cmap/message_stream.js:144:12)
        at MessageStream._write (/usr/src/habitica/node_modules/mongoose/node_modules/mongodb/lib/cmap/message_stream.js:42:5)
        at writeOrBuffer (internal/streams/writable.js:358:12)
        at MessageStream.Writable.write (internal/streams/writable.js:303:10)
        at Socket.ondata (internal/streams/readable.js:731:22)
        at Socket.emit (events.js:400:28)
        at Socket.emit (domain.js:475:12)
        at addChunk (internal/streams/readable.js:293:12)
        at readableAddChunk (internal/streams/readable.js:267:9)
        at Socket.Readable.push (internal/streams/readable.js:206:10)
        at TCP.onStreamRead (internal/stream_base_commons.js:188:23)
        at TCP.callbackTrampoline (internal/async_hooks.js:130:17) {
      ok: 0,
      code: 20,
      codeName: 'IllegalOperation'
    }
  },
  level: 'error',
  message: 'MongoError: This MongoDB deployment does not support retryable writes. Please add retryWrites=false to your connection string.'

So it looks like the issue has something to do with,
MongoError: This MongoDB deployment does not support retryable writes. Please add retryWrites=false to your connection string.

I have tried appending ?retryWrites=false into docker-compose but when I do the build just fails

[mariahlaqua]

The real issue is "originalError: MongoError: Transaction numbers are only allowed on a replica set member or mongos." Locally it needs to be run as a replica set rather than a standalone server. Checkout:

https://www.mongodb.com/docs/manual/tutorial/convert-standalone-to-replica-set/

Let me know if that helps. :)

[alexiswatson]

This does appear to be the culprit:

[user@localhost habitica]# docker exec -it habitica-mongo-1 sh
# mongo habitica-dev
MongoDB shell version v3.6.23
connecting to: mongodb://127.0.0.1:27017/habitica-dev?gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("d2a7665c-ccf2-4b02-b1f6-8ee72fe09683") }
MongoDB server version: 3.6.23
Welcome to the MongoDB shell.
For interactive help, type "help".
For more comprehensive documentation, see
	http://docs.mongodb.org/
Questions? Try the support group
	http://groups.google.com/group/mongodb-user
Server has startup warnings: 
2022-12-24T07:21:32.189+0000 I CONTROL  [initandlisten] 
2022-12-24T07:21:32.189+0000 I CONTROL  [initandlisten] ** WARNING: Access control is not enabled for the database.
2022-12-24T07:21:32.189+0000 I CONTROL  [initandlisten] **          Read and write access to data and configuration is unrestricted.
2022-12-24T07:21:32.189+0000 I CONTROL  [initandlisten] 
2022-12-24T07:21:32.190+0000 I CONTROL  [initandlisten] 
2022-12-24T07:21:32.190+0000 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'.
2022-12-24T07:21:32.190+0000 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
2022-12-24T07:21:32.190+0000 I CONTROL  [initandlisten] 
> rs.status()
{
	"ok" : 0,
	"errmsg" : "not running with --replSet",
	"code" : 76,
	"codeName" : "NoReplicationEnabled"
}

As this doesn't appear to be by design (replica sets appear to be referenced elsewhere in the docs), I'll investigate further, though my guess is that the flag is missing from the Dockerfile.

[alexiswatson]

Not only is the flag missing, though I noticed the docker-compose.dev is still pulling Mongo 3.6. As these API routes use transactions (for v3 and v4, it seems), it seems we want to be using Mongo 4.2.8, as mongo:dev in package.json seems to dictate.

The answer, in broad strokes, would be to do something like this for new installs:

  mongo:
     image: mongo:4.2.8
     networks:
       - habitica
     ports:
       - "27017:27017"
     command: --replSet rs

Then there's the matter of running rs.initiate() when needed. I'll look for an elegant way to do this before offering a PR, but after doing so manually, I find that I can now create a Guild.

Caveat: Folks should note that this solution will not upgrade an existing database from 3.6 to 4.2.8. Those who need to keep their data are going to have to follow the Mongo upgrade path from 3.6 to 4.0 to 4.2.8 incrementally, as dictated by the Mongo documentation. Most devs should be fine simply dropping their containers and rebuilding, while most self-hosters are saavy enough for it to be acceptable to leave it an exercise to the reader.

Feedback on the approach is welcome in the interim.

[alexiswatson]

I've seen a number of folks try to use something like heartbeat to do something like this, but it's a bit of a kludge: it's a startup task, not a health check. We only need to run rs.initiate() on startup, and only if it hasn't run yet. Thankfully the Mongo image will execute JavaScript in /docker-entrypoint-initdb.d once loaded. A two-line script can be tucked there using volumes to wait for the database to start, and then conditionally run rs.initiate() if needed.

[WestonJ87]

I encountered this as well, wish I'd known before, now to either migrate Mongo myself or just re-install and add back all my data I added.

that said, has anyone fiddled with like.. externalizing guild and stuff? I think it would be nice to have my "party" stuff be specific to the local install, say for kids and significant other, but it might be nice for the actual guild functionalities to look at the actual habitica.com data so you can tap into communities and stuff.

that said I'm sure my user @ would not work w/ their apis and what not.

[Neko-Box-Coder]

I tried to use your changes @alexiswatson (after updating db to 4.2.8)
rs.status().myState doesn't exist if rs.initiate failed which happen to me.

And the default rs.initiate doesn't work for the rs.config().members[0].host since it is not using localhost and the container host changes everytime.

My current workaround is just manually go into the container and do mongo --eval 'rsconf = rs.conf();rsconf.members = [{_id: 0, host: "localhost:27017"}];rs.reconfig(rsconf, {force: true})'
since I couldn't get it to work in the mongo-init.js.

If anyone is trying to solve the party/guild problem and planning to use this, do back up the database before doing anything.

[CuriousMagpie]

@SabreCat and @phillipthelen Can y'all take a look at this?

[alexiswatson]

I've since retired my local Habitica, and with all the respect I have for the team and what they've built, I'm not in a position to iterate on this further--especially absent feedback on the direction after a year. I'd invite folks to use my PR as the foundation for future changes.

[diyoyo]

I tried to use your changes @alexiswatson (after updating db to 4.2.8) rs.status().myState doesn't exist if rs.initiate failed which happen to me.

And the default rs.initiate doesn't work for the rs.config().members[0].host since it is not using localhost and the container host changes everytime.

My current workaround is just manually go into the container and do mongo --eval 'rsconf = rs.conf();rsconf.members = [{_id: 0, host: "localhost:27017"}];rs.reconfig(rsconf, {force: true})' since I couldn't get it to work in the mongo-init.js.

If anyone is trying to solve the party/guild problem and planning to use this, do back up the database before doing anything.

@Neko-Box-Coder So are you using v4.2.8 or the default config, using this workaround?

I don't know what this workaround is doing and what you mean by "the host changes every time". Yet, if you're talking about the hostname of the mongo-db container, one way to avoid it to change is using "hostname: mongo-db-host" in the yml file. But again, I'm saying this just in case it could magically help, as I don't see a reason why it would.

Anyways, using @alexiswatson workaround, I'm facing the issue highlighted in the Wiki:

[diyoyo]

Not only is the flag missing, though I noticed the docker-compose.dev is still pulling Mongo 3.6. As these API routes use transactions (for v3 and v4, it seems), it seems we want to be using Mongo 4.2.8, as mongo:dev in package.json seems to dictate.

The answer, in broad strokes, would be to do something like this for new installs:

  mongo:
     image: mongo:4.2.8
     networks:
       - habitica
     ports:
       - "27017:27017"
     command: --replSet rs

Then there's the matter of running rs.initiate() when needed. I'll look for an elegant way to do this before offering a PR, but after doing so manually, I find that I can now create a Guild.

Caveat: Folks should note that this solution will not upgrade an existing database from 3.6 to 4.2.8. Those who need to keep their data are going to have to follow the Mongo upgrade path from 3.6 to 4.0 to 4.2.8 incrementally, as dictated by the Mongo documentation. Most devs should be fine simply dropping their containers and rebuilding, while most self-hosters are saavy enough for it to be acceptable to leave it an exercise to the reader.

Feedback on the approach is welcome in the interim.

Actually, I don't know whether it's been changed since then, but the package.json file states v4.1.1 rather than v4.2.8
...

[diyoyo]

2024-04-16 Update:
Got the PR of @alexiswatson to work with the following changes:

• git checkout tags/v5.19.1 ( I still can't get docker to build with npm20: No gulpfile found in postinstall, but that's another story)
• Using mongo:5.0 in docker-compose.dev.yml
• Keeping mongoose:5.13.20 in package.json for now.

This is very empirical, and works on MyComputer(TM). I chose to keep v5 everywhere because I read this line in package.json :
"mongo:dev": "run-rs -v 5.0.23 -l ubuntu1804 --keep --dbpath mongodb-data -...
The above line is still refering 5.0.23 even after upgrade to mongo 7, fyi.

I'll keep you posted on my attempts to increase mongo version and also node version.

[CuriousMagpie]

Thanks for picking this up, @diyoyo! We greatly appreciate the time you're spending on this. Keep us posted!

And thanks for getting this as far as you did, @alexiswatson--we also appreciate your time investment.

[diyoyo]

@CuriousMagpie So far I have a running dev version on Docker on RPi CM4 (arm) with the following:

• Habitica v5.19.1 (node 14.21.3, npm 6.4.18)
• Mongo 4.4.17

Now, I'm trying to build v5.20.0. I found an issue with the "npm run install" that was trying to run postinstall too soon, apparently. So I renamed the postinstall to mypostinstall in package.json , then changed npm run postinstall to npm run mypostinstall in Dockerfile-dev .

Now, it builds, but then I get: Invalid host header on http://localhost:8080
Which, imho, is probably related to the unavailable network, as read at the end of docker logs habitica-client :

  App running at:
  - Local:   http://localhost:8080/ 
  - Network: unavailable

Any idea why the network is not created ?

PS: if I run docker network ls, It still shows up:

NETWORK ID     NAME                DRIVER    SCOPE
45d527bdd707   bridge              bridge    local
34a1e2268a29   habitica_habitica   bridge    local

Any idea?

Thanks.

[diyoyo]

Ok, I still have this Network: unavailable thing, but now I no longer have the invalid Host Header
What helped was this: https://bobbyhadz.com/blog/invalid-host-header#specify-the-allowed-hosts-in-the-allowedhosts-array

So far I have my own install.sh file with plenty of sed calls and my personal bash functions.
It creates running docker images of habitica and mongo. I reached level 5.24.1 for Habitica, but stuck at Mongo 4.4.17 on my quest :D , because of an ARM limitation with the official mongo 5 image.
I guess the next step is to prepare a PR for people wanting to test the local install with docker.
Or I could just make sure that config files are kept in mounted volumes and publish the docker image on hub.docker.com.

Progress log:

• Party creation is working fine.
• There are still some websocket errors because of the network ip thing, but it doesn't seem to make the client less functional.
• I need to test the errors that were observer for 5.19.1 :
  • Quest enrollment notification never disappear after a party member accepts it.
  • Fake subscriptions not working
  • Public challenges created by superadmin are completely dysfunctional:
    • member's count not updated upwards (but can go below 0 if people leave the challenge)
    • tasks don't show up in member's tasklist
    • ...

[CuriousMagpie]

@SabreCat Can you take a look at this? Thanks!