Instructions unclear

[KMikeeU]

Hey, I am trying to get an sb-mirror with HTTP API running over postgres. However, after starting the containers with docker-compose up and waiting for files to download, the postgres database is empty and the http server returns 404's for any query.

Configs

docker-compose.yml

version: '3'
services:
  postgres:
    ports:
      - '127.0.0.1:5432:5432'
    environment:
      - POSTGRES_USER=mirror_db_user
      - POSTGRES_PASSWORD=mirror_db_pass
    volumes:
      - ./mirror:/mirror
    image: postgres:alpine
  sb-mirror:
    image: mchangrh/sb-mirror:latest
    build: ./build/sb-mirror
    # map port externally
    ports:
      - "127.0.0.1:873:873"
    environment:
      - MIRROR=TRUE # enable cronjob
      - MIRROR_URL=qc.mchang.xyz # override to set upstream mirror
      # - VALIDATE=TRUE # enable rsync checksum validation
      # - CSVLINT=TRUE # lint csv files (will just stop sqlite3 from complaining)
      # - SQLITE=TRUE # generate .db in /export
        # SQLITE will not always generate usable files since postgres does not export files correctly.
      - PADDING_VAR=false # here to make compose not complain
    volumes:
      - ./mirror:/mirror
      - ./export:/export
  sb-server:
    ports:
     - "127.0.0.1:6000:8080"
    volumes:
      - ./export/:/app/database/
      - ./mirror/:/mirror
      # - ./sqlite-config.json:/app/config.json
      - ./postgres-config.json:/app/config.json
    image: ghcr.io/ajayyy/sb-server:latest

postgres-config.json

{
  "port": 8080,
  "globalSalt": "mirrorsalt",
  "adminUserID": "c132d179bfa6a48f4014c163e4f530ecb401f505cf2e5fd02f5b1bb55ac97f5c",
  "behindProxy": true,
  "postgres": {
      "user": "mirror_db_user",
      "password": "mirror_db_pass",
      "host": "postgres",
      "port": 5432
  },
  "mode": "mirror",
  "dumpDatabase": {
    "postgresExportPath": "/mirror"
  }
}

Output

Creating sb-mirror_postgres_1  ... done
Creating sb-mirror_sb-server_1 ... done
Creating sb-mirror_sb-mirror_1 ... done
Attaching to sb-mirror_sb-mirror_1, sb-mirror_sb-server_1, sb-mirror_postgres_1
sb-mirror_1  | Uses SponsorBlock data from https://sponsor.ajay.app/
sb-mirror_1  | Downloading from mirror: qc.mchang.xyz
sb-server_1  | Entrypoint script
postgres_1   | The files belonging to this database system will be owned by user "postgres".
postgres_1   | This user must also own the server process.
postgres_1   |
postgres_1   | The database cluster will be initialized with locale "en_US.utf8".
postgres_1   | The default database encoding has accordingly been set to "UTF8".
postgres_1   | The default text search configuration will be set to "english".
postgres_1   |
postgres_1   | Data page checksums are disabled.
postgres_1   |
postgres_1   | fixing permissions on existing directory /var/lib/postgresql/data ... ok
postgres_1   | creating subdirectories ... ok
postgres_1   | selecting dynamic shared memory implementation ... posix
postgres_1   | selecting default max_connections ... 100
postgres_1   | selecting default shared_buffers ... 128MB
postgres_1   | selecting default time zone ... UTC
postgres_1   | creating configuration files ... ok
postgres_1   | running bootstrap script ... ok
sb-mirror_1  | receiving incremental file list
postgres_1   | performing post-bootstrap initialization ... sh: locale: not found
postgres_1   | 2022-11-09 18:48:54.437 UTC [30] WARNING:  no usable system locales were found
postgres_1   | ok
postgres_1   | syncing data to disk ... ok
postgres_1   |
postgres_1   |
postgres_1   | Success. You can now start the database server using:
postgres_1   |
postgres_1   |     pg_ctl -D /var/lib/postgresql/data -l logfile start
postgres_1   |
postgres_1   | initdb: warning: enabling "trust" authentication for local connections
postgres_1   | initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.
postgres_1   | waiting for server to start....2022-11-09 18:48:55.647 UTC [36] LOG:  starting PostgreSQL 15.0 on x86_64-pc-linux-musl, compiled by gcc (Alpine 11.2.1_git20220219) 11.2.1 20220219, 64-bit
postgres_1   | 2022-11-09 18:48:55.649 UTC [36] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
postgres_1   | 2022-11-09 18:48:55.656 UTC [39] LOG:  database system was shut down at 2022-11-09 18:48:55 UTC
postgres_1   | 2022-11-09 18:48:55.661 UTC [36] LOG:  database system is ready to accept connections
postgres_1   |  done
postgres_1   | server started
postgres_1   | CREATE DATABASE
postgres_1   |
postgres_1   |
postgres_1   | /usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
postgres_1   |
postgres_1   | waiting for server to shut down....2022-11-09 18:48:55.853 UTC [36] LOG:  received fast shutdown request
postgres_1   | 2022-11-09 18:48:55.884 UTC [36] LOG:  aborting any active transactions
postgres_1   | 2022-11-09 18:48:55.886 UTC [36] LOG:  background worker "logical replication launcher" (PID 42) exited with exit code 1
postgres_1   | 2022-11-09 18:48:55.886 UTC [37] LOG:  shutting down
postgres_1   | 2022-11-09 18:48:55.888 UTC [37] LOG:  checkpoint starting: shutdown immediate
postgres_1   | 2022-11-09 18:48:55.961 UTC [37] LOG:  checkpoint complete: wrote 918 buffers (5.6%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.024 s, sync=0.045 s, total=0.076 s; sync files=250, longest=0.037 s, average=0.001 s; distance=4221 kB, estimate=4221 kB
postgres_1   | 2022-11-09 18:48:55.972 UTC [36] LOG:  database system is shut down
postgres_1   |  done
postgres_1   | server stopped
postgres_1   |
postgres_1   | PostgreSQL init process complete; ready for start up.
postgres_1   |
postgres_1   | 2022-11-09 18:48:56.084 UTC [1] LOG:  starting PostgreSQL 15.0 on x86_64-pc-linux-musl, compiled by gcc (Alpine 11.2.1_git20220219) 11.2.1 20220219, 64-bit
postgres_1   | 2022-11-09 18:48:56.084 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
postgres_1   | 2022-11-09 18:48:56.084 UTC [1] LOG:  listening on IPv6 address "::", port 5432
postgres_1   | 2022-11-09 18:48:56.127 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
postgres_1   | 2022-11-09 18:48:56.133 UTC [51] LOG:  database system was shut down at 2022-11-09 18:48:55 UTC
postgres_1   | 2022-11-09 18:48:56.137 UTC [1] LOG:  database system is ready to accept connections
sb-mirror_1  | ./
sb-mirror_1  | categoryVotes.csv
      8,771,182 100%    6.83MB/s    0:00:01 (xfr#1, to-chk=8/10)
sb-mirror_1  | lockCategories.csv
      1,609,363 100%    4.25MB/s    0:00:00 (xfr#2, to-chk=7/10)
sb-mirror_1  | ratings.csv
        924,650 100%    2.06MB/s    0:00:00 (xfr#3, to-chk=6/10)
sb-mirror_1  | sponsorTimes.csv
  1,328,630,890 100%   11.10MB/s    0:01:54 (xfr#4, to-chk=5/10)
sb-mirror_1  | unlistedVideos.csv
      5,729,807 100%   11.43MB/s    0:00:00 (xfr#5, to-chk=4/10)
sb-mirror_1  | userNames.csv
      7,971,851 100%    6.55MB/s    0:00:01 (xfr#6, to-chk=3/10)
sb-mirror_1  | videoInfo.csv
    408,766,072 100%   13.63MB/s    0:00:28 (xfr#7, to-chk=2/10)
sb-mirror_1  | vipUsers.csv
          3,558 100%    5.70kB/s    0:00:00 (xfr#8, to-chk=1/10)
sb-mirror_1  | warnings.csv
        153,796 100%  240.69kB/s    0:00:00 (xfr#9, to-chk=0/10)
sb-mirror_1  |
sb-mirror_1  | sent 209 bytes  received 1,690,016,195 bytes  11,304,457.55 bytes/sec
sb-mirror_1  | total size is 1,762,561,169  speedup is 1.04
sb-mirror_sb-mirror_1 exited with code 0

What I also tried:

• Setting DBINIT=TRUE environment variable in sb-server seems to create a "sponsorTimes" tables in postgres, which contains many sane tables (which all stay empty however)
• Changing /app/config.json to /usr/src/app/config.json because that seems to be the correct location for sb-server having checked the source code, doing so prints this warning to the console: sb-server_1 | WARN 2022-11-09T19:18:46.410Z: [dumpDatabase] No tables configured
• Enabling all flags in sb-mirror which results in countless errors from sb-mirror, which all look like this

sb-mirror_1  | /mirror/sponsorTimes.csv:2802: INSERT failed: NOT NULL constraint failed: sponsorTimes.startTime
sb-mirror_1  | /mirror/sponsorTimes.csv:2803: expected 20 columns but found 1 - filling the rest with NULL

[mchangrh]

DBINIT=TRUE is documented as only generating the db and exiting

some of the paths are broken since I switched over to the upstream container, which uses a different home path, that's why the configs aren't binding. I'll try to patch it, haven't had much time to sit down and work on it lately

[modelD-svg]

Having similar issues. Appreciate your work on this