Visiting admin page shows 502 Bad Gateway nginx #3635

mitchellvanbijleveld · 2024-01-02T21:16:24Z

Steps To Reproduce

Upgrading the Bitwarden installation from the December beta release to the beta release of the 2nd of January, 2024.

Expected Result

The browser shows Bitwarden's admin page to put in an email address to get a login link.

Actual Result

The browser shows: 502 Bad Gateway nginx

Screenshots or Videos

No response

Additional Context

nginx shows errors, something like

connect() failed (111: Connection refused) while connecting to upstream, client:

Also, data/bitwarden/logs/admin.log shows:

2024-01-02 21:20:00.189 +00:00 [INF] Starting job DeleteSendsJob at "2024-01-02T21:20:00.1889573Z".
2024-01-02 21:20:01.980 +00:00 [INF] Migrating database.
2024-01-02 21:20:02.242 +00:00 [ERR] Failed executing DbCommand (3ms) [Parameters=[], CommandType='"Text"', CommandTimeout='30']
CALL POMELO_BEFORE_DROP_PRIMARY_KEY(NULL, 'Grant');
ALTER TABLE `Grant` DROP PRIMARY KEY;

Githash Version

919d759-dirty

Environment Details

Running Debian 12, AMD64. This is nginx's config file:

server {
    listen 443 ssl http2; # managed by Certbot
    listen [::]:443 ssl http2; # managed by Certbot

    server_name <url>;

    ssl_certificate /etc/letsencrypt/live/<url>/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/<url>/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot

    add_header Strict-Transport-Security "max-age=31536000" always; # managed by Certbot

    ssl_trusted_certificate /etc/letsencrypt/live/<url>/chain.pem; # managed by Certbot
    ssl_stapling on; # managed by Certbot
    ssl_stapling_verify on; # managed by Certbot

    location / {
        proxy_pass http://localhost:8080;
        proxy_set_header Host $host;
    }

}

server {
    if ($host = <url>) {
        return 301 https://$host$request_uri;
    } # managed by Certbot

    listen 80 ;
    listen [::]:80 ;

    server_name <url>;
}

Database Image

docker-compose.yml:

version: "3.8"

services:
  bitwarden:
    depends_on:
      - db
    env_file:
      - /XXX.env
    image: bitwarden/self-host:beta
    restart: always
    ports:
      - "8080:8080"
    volumes:
      - /XXX:/etc/bitwarden

  db:
    environment:
      MARIADB_USER: "<HIDDEN>"
      MARIADB_PASSWORD: "<HIDDEN>"
      MARIADB_DATABASE: "<HIDDEN>"
      MARIADB_RANDOM_ROOT_PASSWORD: "true"
    image: mariadb:10
    restart: always
    volumes:
      - /XXX:/var/lib/mysql

Issue-Link

#2480

Issue Tracking Info

I understand that work is tracked outside of Github. A PR will be linked to this issue should one be opened to address it, but Bitwarden doesn't use fields like "assigned", "milestone", or "project" to track progress.

The text was updated successfully, but these errors were encountered:

juanmanuelbc · 2024-01-03T08:02:46Z

More or less the same here.

Just updated my unified self-host container and this is what I get in the logs:

Any help will be apreciated.

Thank you...

karolszklarski · 2024-01-03T12:11:45Z

gitHash cf4d8a4 is working, ea162d5 is not

That commit modifies GrantRepository, and logs say something about 'grant' table

Duplicate: #3637

karolszklarski · 2024-01-03T12:18:59Z

Working older version (11 days ago):
docker image pull bitwarden/self-host@sha256:495424ccf31ca2ce0dca4b96774531efbc49ab595600d004694a78a31170bc2c

Krychaz · 2024-01-03T16:21:55Z

Hi there,

Thank you for your report!

I was able to reproduce this issue, and I have flagged this to our engineering team.

If you wish to add any further information/screenshots/recordings etc., please feel free to do so at any time - our engineering team will be happy to review these.

Thanks once again!

DuckThom · 2024-01-10T23:33:02Z

As I've just upgraded my Bitwarden install from 2023.12.0 to 2024.1.0, I too ran into this issue. However, I've done some digging around and added some findings to this issue: #3651 (comment)

To summarize, what I could see was that ALTER TABLE GrantADD CONSTRAINTPK_Grant PRIMARY KEY (Id); is actually the migration that's failing due to the existing data. The error Can't DROP INDEX PRIMARY; check that it exists is a side-effect because of the system attempting to re-run migrations that have already altered the database.

I do like to point out though that I am running MySQL 8.0.35 and the platform is ARM64, so this issue does not appear to be specific to MariaDB nor AMD64.

I did find a way to somewhat work around the issue which involved some manual queries but I managed to get the Bitwarden instance running again

(Warning for anyone attempting this manual "fix": these commands are destructive and can cause data loss if you're not careful. Make sure you know what you are doing!)

Create a backup of your database Very important
Stop bitwarden process
Log into the DB: mysql -u root -p bitwarden_vault
Check the state of the Grant table: show create table `Grant`;. There should not be a PRIMARY KEY and there should be an Id column.
Remove Id with alter table `Grant` drop column Id;
Re-add PRIMARY KEY: alter table `Grant` add constraint PK_Grant_Id PRIMARY KEY (`Key`); (Note: I made the assumption that Key was the original PK, not 100% sure but it's just to make the migration dropping the PK not fail)
Exit mysql
Create dump of table: mysqldump --no-create-info --complete-insert -u root -p bitwarden_vault Grant > grant.sql. Make sure to not lose this file, you'll need it later on!
Log into the DB: mysql -u root -p bitwarden_vault
Truncate table: TRUNCATE `Grant`; (DESTRUCTIVE)
Exit mysql
Start bitwarden
Check admin.log output, ie. docker compose exec bitwarden tail -f /var/log/bitwarden/admin.log
Ensure it logs: Application is shutting down... and Application started. Press Ctrl+C to shut down. shortly after.
Stop bitwarden process
Import dump made in step 7: mysql -u root -p bitwarden_vault < grant.sql
Exit mysql
Start bitwarden process

withinfocus · 2024-01-11T21:08:10Z

@mitchellvanbijleveld can you pull the latest from main and give this a try? We made a modification to the database migration.

mitchellvanbijleveld · 2024-01-11T21:38:48Z

@withinfocus I am sorry to have you ask, but pulling from main? I might be confused but the Docker self-host image is only available as dev/beta, right? I tried the dev which was updated like 15 minutes ago but that one isn't working.

The docker compose logs shows a lot of MariaDB related connection errors:

bitwarden-db-1  | 2024-01-11 21:33:58 126 [Warning] Aborted connection 126 to db: 'unconnected' user: 'BWUSER' host: '172.21.0.3' (Got an error reading communication packets)
bitwarden-db-1  | 2024-01-11 21:34:23 132 [Warning] Aborted connection 132 to db: 'BWUSER' user: 'BWUSER' host: '172.21.0.3' (Got an error reading communication packets)
bitwarden-db-1  | 2024-01-11 21:34:23 130 [Warning] Aborted connection 130 to db: 'unconnected' user: 'BWUSER' host: '172.21.0.3' (Got an error reading communication packets)
bitwarden-db-1  | 2024-01-11 21:34:47 136 [Warning] Aborted connection 136 to db: 'BWUSER' user: 'BWUSER' host: '172.21.0.3' (Got an error reading communication packets)
bitwarden-db-1  | 2024-01-11 21:34:47 134 [Warning] Aborted connection 134 to db: 'unconnected' user: 'BWUSER' host: '172.21.0.3' (Got an error reading communication packets)
bitwarden-db-1  | 2024-01-11 21:35:12 139 [Warning] Aborted connection 139 to db: 'BWUSER' user: 'BWUSER' host: '172.21.0.3' (Got an error reading communication packets)
bitwarden-db-1  | 2024-01-11 21:35:12 138 [Warning] Aborted connection 138 to db: 'unconnected' user: 'BWUSER' host: '172.21.0.3' (Got an error reading communication packets)

mitchellvanbijleveld · 2024-01-11T21:40:16Z

@withinfocus the admin logs show the following:

fail: Microsoft.EntityFrameworkCore.Database.Command[20102]
      Failed executing DbCommand (4ms) [Parameters=[], CommandType='Text', CommandTimeout='30']
      CALL POMELO_BEFORE_DROP_PRIMARY_KEY(NULL, 'Grant');
      ALTER TABLE `Grant` DROP PRIMARY KEY;
Unhandled exception. MySqlConnector.MySqlException (0x80004005): Can't DROP INDEX `PRIMARY`; check that it exists
   at MySqlConnector.Core.ResultSet.ReadResultSetHeaderAsync(IOBehavior ioBehavior) in /_/src/MySqlConnector/Core/ResultSet.cs:line 43
   at MySqlConnector.MySqlDataReader.ActivateResultSet(CancellationToken cancellationToken) in /_/src/MySqlConnector/MySqlDataReader.cs:line 130
   at MySqlConnector.MySqlDataReader.CreateAsync(CommandListPosition commandListPosition, ICommandPayloadCreator payloadCreator, IDictionary`2 cachedProcedures, IMySqlCommand command, CommandBehavior behavior, Activity activity, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/MySqlDataReader.cs:line 468
   at MySqlConnector.Core.CommandExecutor.ExecuteReaderAsync(IReadOnlyList`1 commands, ICommandPayloadCreator payloadCreator, CommandBehavior behavior, Activity activity, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/CommandExecutor.cs:line 56
   at MySqlConnector.MySqlCommand.ExecuteNonQueryAsync(IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/MySqlCommand.cs:line 296
   at MySqlConnector.MySqlCommand.ExecuteNonQuery() in /_/src/MySqlConnector/MySqlCommand.cs:line 107
   at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteNonQuery(RelationalCommandParameterObject parameterObject)
   at Microsoft.EntityFrameworkCore.Migrations.MigrationCommand.ExecuteNonQuery(IRelationalConnection connection, IReadOnlyDictionary`2 parameterValues)
   at Microsoft.EntityFrameworkCore.Migrations.Internal.MigrationCommandExecutor.ExecuteNonQuery(IEnumerable`1 migrationCommands, IRelationalConnection connection)
   at Microsoft.EntityFrameworkCore.Migrations.Internal.Migrator.Migrate(String targetMigration)
   at Microsoft.EntityFrameworkCore.RelationalDatabaseFacadeExtensions.Migrate(DatabaseFacade databaseFacade)
   at Bit.MySqlMigrations.MySqlDbMigrator.MigrateDatabase(Boolean enableLogging, CancellationToken cancellationToken) in /source/util/MySqlMigrations/MySqlDbMigrator.cs:line 30
   at Bit.Admin.HostedServices.DatabaseMigrationHostedService.StartAsync(CancellationToken cancellationToken) in /source/src/Admin/HostedServices/DatabaseMigrationHostedService.cs:line 29
   at Microsoft.Extensions.Hosting.Internal.Host.StartAsync(CancellationToken cancellationToken)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.Run(IHost host)
   at Bit.Admin.Program.Main(String[] args) in /source/src/Admin/Program.cs:line 9

mitchellvanbijleveld · 2024-01-11T21:43:56Z

And here is another example of docker compose logs:

bitwarden-bitwarden-1  | 2024-01-11 21:41:54,225 INFO spawned: 'admin' with pid 486
bitwarden-bitwarden-1  | 2024-01-11 21:42:09,244 INFO success: admin entered RUNNING state, process has stayed up for > than 15 seconds (startsecs)
bitwarden-bitwarden-1  | 2024-01-11 21:42:17,969 INFO exited: admin (terminated by SIGABRT (core dumped); not expected)
bitwarden-bitwarden-1  | 2024-01-11 21:42:18,972 INFO spawned: 'admin' with pid 507
bitwarden-bitwarden-1  | 2024-01-11 21:42:33,990 INFO success: admin entered RUNNING state, process has stayed up for > than 15 seconds (startsecs)
bitwarden-bitwarden-1  | 2024-01-11 21:42:42,636 INFO exited: admin (terminated by SIGABRT (core dumped); not expected)
bitwarden-bitwarden-1  | 2024-01-11 21:42:43,638 INFO spawned: 'admin' with pid 527
bitwarden-bitwarden-1  | 2024-01-11 21:42:58,657 INFO success: admin entered RUNNING state, process has stayed up for > than 15 seconds (startsecs)
bitwarden-bitwarden-1  | 2024-01-11 21:43:07,314 INFO exited: admin (terminated by SIGABRT (core dumped); not expected)
bitwarden-bitwarden-1  | 2024-01-11 21:43:08,316 INFO spawned: 'admin' with pid 550

withinfocus · 2024-01-11T22:57:53Z

You don't seem to have the latest is all -- I was referring to code merges in this repo, specifically commit db4d7aa60932d867b5c18a42ac3ca567a68dca32.

Nicolaj-H · 2024-01-12T09:26:58Z

Having the same issue with self-host:beta and self-host:2024.1.0-beta, so tried pulling the self-host:dev that have been updated 11 hours ago and the issue is still there.

for now i reverted back to running self-host:2023.12.0-beta

admin.log:

  Failed executing DbCommand (3ms) [Parameters=[], CommandType='Text', CommandTimeout='30']
      INSERT INTO "ef_temp_Grant" ("Id", "ClientId", "ConsumedDate", "CreationDate", "Data", "Description", "ExpirationDate", "Key", "SessionId", "SubjectId", "Type")
      SELECT "Id", IFNULL("ClientId", ''), "ConsumedDate", "CreationDate", IFNULL("Data", ''), "Description", "ExpirationDate", "Key", "SessionId", "SubjectId", IFNULL("Type", '')
      FROM "Grant";
Unhandled exception. Microsoft.Data.Sqlite.SqliteException (0x80004005): SQLite Error 19: 'UNIQUE constraint failed: ef_temp_Grant.Id'.
   at Microsoft.Data.Sqlite.SqliteException.ThrowExceptionForRC(Int32 rc, sqlite3 db)
   at Microsoft.Data.Sqlite.SqliteDataReader.NextResult()
   at Microsoft.Data.Sqlite.SqliteCommand.ExecuteReader(CommandBehavior behavior)
   at Microsoft.Data.Sqlite.SqliteCommand.ExecuteReader()
   at Microsoft.Data.Sqlite.SqliteCommand.ExecuteNonQuery()
   at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteNonQuery(RelationalCommandParameterObject parameterObject)
   at Microsoft.EntityFrameworkCore.Migrations.MigrationCommand.ExecuteNonQuery(IRelationalConnection connection, IReadOnlyDictionary`2 parameterValues)
   at Microsoft.EntityFrameworkCore.Migrations.Internal.MigrationCommandExecutor.ExecuteNonQuery(IEnumerable`1 migrationCommands, IRelationalConnection connection)
   at Microsoft.EntityFrameworkCore.Migrations.Internal.Migrator.Migrate(String targetMigration)
   at Microsoft.EntityFrameworkCore.RelationalDatabaseFacadeExtensions.Migrate(DatabaseFacade databaseFacade)
   at Bit.SqliteMigrations.SqliteDbMigrator.MigrateDatabase(Boolean enableLogging, CancellationToken cancellationToken) in /source/util/SqliteMigrations/SqliteDbMigrator.cs:line 30
   at Bit.Admin.HostedServices.DatabaseMigrationHostedService.StartAsync(CancellationToken cancellationToken) in /source/src/Admin/HostedServices/DatabaseMigrationHostedService.cs:line 29
   at Microsoft.Extensions.Hosting.Internal.Host.StartAsync(CancellationToken cancellationToken)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.Run(IHost host)
   at Bit.Admin.Program.Main(String[] args) in /source/src/Admin/Program.cs:line 9

juanmanuelbc · 2024-01-12T09:30:34Z

for now i reverted back to running self-host:2023.12.0-beta

Yes, this is the same one I'm using too until testing each new version with positive results...

Nicolaj-H · 2024-01-17T11:32:25Z

This seems to be fixed with commit "[PM-5519 PM-5526 PM-5624 PM-5600 More Grant SQL fixes"

I'm running self-host:beta again, version number says 2024.1.2

juanmanuelbc · 2024-01-17T12:44:19Z

This seems to be fixed with commit "[PM-5519 PM-5526 PM-5624 PM-5600 More Grant SQL fixes"

Yes, it's working again!!!

I'm running self-host:beta again, version number says 2024.1.2

I'm running self-host:beta too, but my version number says 2024.1.1...

mitchellvanbijleveld · 2024-01-17T13:52:18Z

Great! Let me check this afternoon :).

karolszklarski · 2024-01-17T17:42:12Z

self-host:beta 2024.1.2 dca8d00-dirty still gives me the same error

withinfocus · 2024-01-17T17:47:43Z

You wouldn't get the same error as the MySQL migration changed. It now runs what you can see at https://github.com/bitwarden/server/blob/main/util/MySqlMigrations/Migrations/20231214162533_GrantIdWithIndexes.cs#L75-L90 that will delete and re-add the new Id column. You won't see those same Pomelo errors that caused the initial issue.

mitchellvanbijleveld added bug bw-unified-deploy An Issue related to Bitwarden unified deployment labels Jan 2, 2024

karolszklarski mentioned this issue Jan 5, 2024

Admin terminated by SIGABRT - Can't DROP INDEX PRIMARY #3637

Closed

1 task

withinfocus assigned withinfocus and unassigned withinfocus Jan 5, 2024

withinfocus mentioned this issue Jan 8, 2024

Primary key drop when data present PomeloFoundation/Pomelo.EntityFrameworkCore.MySql#1828

Closed

withinfocus mentioned this issue Jan 11, 2024

[PM-5519] [PM-5526] [PM-5624] [PM-5600] Tweak EF settings for MySQL grant auto-increment #3662

Merged

withinfocus closed this as completed Jan 22, 2024

withinfocus self-assigned this Jan 22, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Visiting admin page shows 502 Bad Gateway nginx #3635

Visiting admin page shows 502 Bad Gateway nginx #3635

mitchellvanbijleveld commented Jan 2, 2024 •

edited

juanmanuelbc commented Jan 3, 2024

karolszklarski commented Jan 3, 2024

karolszklarski commented Jan 3, 2024

Krychaz commented Jan 3, 2024

DuckThom commented Jan 10, 2024

withinfocus commented Jan 11, 2024

mitchellvanbijleveld commented Jan 11, 2024

mitchellvanbijleveld commented Jan 11, 2024

mitchellvanbijleveld commented Jan 11, 2024

withinfocus commented Jan 11, 2024

Nicolaj-H commented Jan 12, 2024

juanmanuelbc commented Jan 12, 2024

Nicolaj-H commented Jan 17, 2024 •

edited by jira bot

juanmanuelbc commented Jan 17, 2024 •

edited by jira bot

mitchellvanbijleveld commented Jan 17, 2024

karolszklarski commented Jan 17, 2024

withinfocus commented Jan 17, 2024

Visiting admin page shows 502 Bad Gateway nginx #3635

Visiting admin page shows 502 Bad Gateway nginx #3635

Comments

mitchellvanbijleveld commented Jan 2, 2024 • edited

Steps To Reproduce

Expected Result

Actual Result

Screenshots or Videos

Additional Context

Githash Version

Environment Details

Database Image

Issue-Link

Issue Tracking Info

juanmanuelbc commented Jan 3, 2024

karolszklarski commented Jan 3, 2024

karolszklarski commented Jan 3, 2024

Krychaz commented Jan 3, 2024

DuckThom commented Jan 10, 2024

(Warning for anyone attempting this manual "fix": these commands are destructive and can cause data loss if you're not careful. Make sure you know what you are doing!)

withinfocus commented Jan 11, 2024

mitchellvanbijleveld commented Jan 11, 2024

mitchellvanbijleveld commented Jan 11, 2024

mitchellvanbijleveld commented Jan 11, 2024

withinfocus commented Jan 11, 2024

Nicolaj-H commented Jan 12, 2024

juanmanuelbc commented Jan 12, 2024

Nicolaj-H commented Jan 17, 2024 • edited by jira bot

juanmanuelbc commented Jan 17, 2024 • edited by jira bot

mitchellvanbijleveld commented Jan 17, 2024

karolszklarski commented Jan 17, 2024

withinfocus commented Jan 17, 2024

mitchellvanbijleveld commented Jan 2, 2024 •

edited

Nicolaj-H commented Jan 17, 2024 •

edited by jira bot

juanmanuelbc commented Jan 17, 2024 •

edited by jira bot