Alembic fails creating a database file?

[hermsdorfkonrad]

Hi folks,

I am tinkering on this quite a while now but I don't find a solution. I am using the Dockerized version of ARM as recommended.
So when I start the ARM container it keeps restarting forever. The alembic database creation fails and thus the container restarts.

I used Ubuntu 22.04 and Fedora 39 and the container behaves identical. I wrote an ansible playbook that basically does everything the ARM installation scripts do plus my own important steps such as mounting the network drives and editing the start_arm_container.sh. With my ansible playbook I was able to try lots of different configurations but unfortunately I couldn't get it up and running after all.

I tried

• passing my mounted (cifs) network drives (owned by arm:arm) to the arm container
• passing only local (in /home/arm/...) directories (owned by arm:arm) to the arm container (no mounted network drives)
• running the container on Fedora
• running start_arm_container.sh with and without sudo (user arm in group "docker" as well)
• using the apt docker.io package instead of the get.docker.com package
• mounting the network drives without the nobrl option

None of the things I did different really changed the behaviour. So now that I have tried to change the environment in many ways, my next step would be creating my own docker image with i.e. an older version of SQLAlchemy. But before I go down this road, I kindly ask if there is someone who knows where my issues originate. Because I do not believe this is a general problem with fresh installations of ARM and more likely a special problem of my environment.

The Environment: TrueNAS Core

• Linux VM (using bhyve, the BSD Hypervisor Software) as host for ARM (Ubuntu 22.04 or Fedora 39)
• SMB-shares for Music, Movies and ARM Home Directory
• PCI-Passthrough of the entire SATA-Controller, that connects the Optical Drive, to the Linux VM
• 4Core/16GiB allocated for the Linux VM

*** Running /etc/my_init.d/10_syslog-ng.init...
Apr 26 18:56:11 47a0d4f2b5ba syslog-ng[13]: syslog-ng starting up; version='3.25.1'
*** Running /etc/my_init.d/arm_user_files_setup.sh...
Updating arm user id from 1000 to 1001...
Updating arm group id from 1000 to 1001...
Adding arm user to 'render' group
Checking ownership of /home/arm
[OK]: ARM UID and GID set correctly, ARM has access to '/home/arm' using 1001:1001
Removing any link between music and Music
Checking ownership of /etc/arm/config
[OK]: ARM UID and GID set correctly, ARM has access to '/etc/arm/config' using 1001:1001
Checking location of abcde configuration files
/etc/abcde.conf link doesnt exist
*** Running /etc/my_init.d/start_udev.sh...
/etc/my_init.d/start_udev.sh: 0: can't access tty; job control turned off
Trying to start udev
udev Started successfully
# # # # # # 
*** Booting runit daemon...
*** Runit started as PID 104
Starting web ui
Apr 26 18:56:59 47a0d4f2b5ba cron[107]: (CRON) INFO (pidfile fd = 3)
Apr 26 18:56:59 47a0d4f2b5ba cron[107]: (CRON) INFO (Running @reboot jobs)
[2024-04-26 18:57:01,361] DEBUG ARM: __init__.<module> Debugging pin: 12345
[2024-04-26 18:57:01,433] DEBUG ARM: utils.arm_alembic_get Alembic Head is: b326a3663939
[2024-04-26 18:57:01,433] DEBUG ARM: utils.arm_db_check Database file is not present: /home/arm/db/arm.db
[2024-04-26 18:57:01,433] DEBUG ARM: utils.arm_db_cfg No armui cfg setup
[2024-04-26 18:57:01,433] INFO ARM: utils.check_db_version No database found.  Initializing arm.db...
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 145, in __init__
    self._dbapi_connection = engine.raw_connection()
  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 3288, in raw_connection
    return self.pool.connect()
  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py", line 452, in connect
    return _ConnectionFairy._checkout(self)
  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py", line 1267, in _checkout
    fairy = _ConnectionRecord.checkout(pool)
  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py", line 716, in checkout
    rec = pool._do_get()
  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/impl.py", line 284, in _do_get
    return self._create_connection()
  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py", line 393, in _create_connection
    return _ConnectionRecord(self)
  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py", line 678, in __init__
    self.__connect()
  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py", line 903, in __connect
    pool.logger.debug("Error on connect(): %s", e)
  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 147, in __exit__
    raise exc_value.with_traceback(exc_tb)
  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py", line 898, in __connect
    self.dbapi_connection = connection = pool._invoke_creator(self)
  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/create.py", line 637, in connect
    return dialect.connect(*cargs, **cparams)
  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 615, in connect
    return self.loaded_dbapi.connect(*cargs, **cparams)
sqlite3.OperationalError: unable to open database file

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/arm/arm/runui.py", line 8, in <module>
    import arm.config.config as cfg  # noqa E402
  File "/opt/arm/arm/../arm/__init__.py", line 2, in <module>
    import arm.ripper
  File "/opt/arm/arm/../arm/ripper/__init__.py", line 3, in <module>
    from arm.ripper import logger, utils, makemkv, handbrake, identify, ARMInfo # noqa F401
  File "/opt/arm/arm/../arm/ripper/utils.py", line 24, in <module>
    from arm.ui import db  # needs to be imported before models
  File "/opt/arm/arm/../arm/ui/__init__.py", line 66, in <module>
    from arm.ui.database.database import route_database  # noqa: E402,F811
  File "/opt/arm/arm/../arm/ui/database/database.py", line 28, in <module>
    armui_cfg = ui_utils.arm_db_cfg()
  File "/opt/arm/arm/../arm/ui/utils.py", line 206, in arm_db_cfg
    check_db_version(cfg.arm_config['INSTALLPATH'], cfg.arm_config['DBFILE'])
  File "/opt/arm/arm/../arm/ui/utils.py", line 88, in check_db_version
    flask_migrate.upgrade(mig_dir)
  File "/usr/local/lib/python3.8/dist-packages/flask_migrate/__init__.py", line 111, in wrapped
    f(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/flask_migrate/__init__.py", line 200, in upgrade
    command.upgrade(config, revision, sql=sql, tag=tag)
  File "/usr/local/lib/python3.8/dist-packages/alembic/command.py", line 399, in upgrade
    script.run_env()
  File "/usr/local/lib/python3.8/dist-packages/alembic/script/base.py", line 578, in run_env
    util.load_python_file(self.dir, "env.py")
  File "/usr/local/lib/python3.8/dist-packages/alembic/util/pyfiles.py", line 93, in load_python_file
    module = load_module_py(module_id, path)
  File "/usr/local/lib/python3.8/dist-packages/alembic/util/pyfiles.py", line 109, in load_module_py
    spec.loader.exec_module(module)  # type: ignore
  File "/opt/arm/arm/migrations/env.py", line 89, in <module>
    run_migrations_online()
  File "/opt/arm/arm/migrations/env.py", line 73, in run_migrations_online
    connection = engine.connect()
  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 3264, in connect
    return self._connection_cls(self)
  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 147, in __init__
    Connection._handle_dbapi_exception_noconnection(
  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2426, in _handle_dbapi_exception_noconnection
    raise sqlalchemy_exception.with_traceback(exc_info[2]) from e
  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 145, in __init__
    self._dbapi_connection = engine.raw_connection()
  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 3288, in raw_connection
    return self.pool.connect()
  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py", line 452, in connect
    return _ConnectionFairy._checkout(self)
  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py", line 1267, in _checkout
    fairy = _ConnectionRecord.checkout(pool)
  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py", line 716, in checkout
    rec = pool._do_get()
  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/impl.py", line 284, in _do_get
    return self._create_connection()
  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py", line 393, in _create_connection
    return _ConnectionRecord(self)
  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py", line 678, in __init__
    self.__connect()
  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py", line 903, in __connect
    pool.logger.debug("Error on connect(): %s", e)
  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 147, in __exit__
    raise exc_value.with_traceback(exc_tb)
  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py", line 898, in __connect
    self.dbapi_connection = connection = pool._invoke_creator(self)
  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/create.py", line 637, in connect
    return dialect.connect(*cargs, **cparams)
  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 615, in connect
    return self.loaded_dbapi.connect(*cargs, **cparams)
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) unable to open database file
(Background on this error at: https://sqlalche.me/e/20/e3q8)
Starting web ui
[2024-04-26 18:57:03,938] DEBUG ARM: __init__.<module> Debugging pin: 12345
[2024-04-26 18:57:04,961] DEBUG ARM: utils.arm_alembic_get Alembic Head is: b326a3663939
[2024-04-26 18:57:04,961] DEBUG ARM: utils.arm_db_check Database file is not present: /home/arm/db/arm.db
[2024-04-26 18:57:04,961] DEBUG ARM: utils.arm_db_cfg No armui cfg setup
[2024-04-26 18:57:04,961] INFO ARM: utils.check_db_version No database found.  Initializing arm.db...
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 145, in __init__
    self._dbapi_connection = engine.raw_connection()
  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 3288, in raw_connection
    return self.pool.connect()

Also, as soon as I got everything up and running properly, I'll share the ansible Playbook for everyone else who likes my approach of installing the entire thing in one go.

[microtechno9000]

Yeh nice, could add the playbook to the wiki.

It looks like the permissions are ok when ARM creates the folders, but is failing to create a file. This normally happens due to permissions, so to double check

Can you post the permissions of /home/arm and also try touch in the db folder as user arm

[hermsdorfkonrad]

This is it! I never checked the ownership of the directories inside the arm-owned home-directory. I kept double checking the directories, that I myself created for ARM.
So being more clear, the db, logs etc. directories created by ARM were owned by root so arm user had no permissions in there.
Still, I am slightly confused that these automatically created directories have wrong permissions.

I'll add some changes to the Playbook and contribute as soon as it's working as expected.

[hermsdorfkonrad]

Thanks a million for your help!

[microtechno9000]

no worries

Its a common issue at the moment because we removed chown from the install script. Noting docker shouldn't change ownership outside its container as good practice.

There are some updates being worked to help fix this with the UI and report better status to the user.

[DZaremba14600]

I've been getting back to ARM recently (I'm re-doing my Blu-Ray library) and I was absolutely going bonkers wondering why the folder and file creation on even startup worked fine a few months ago, but was now causing me some issues. This explains it. Thanks for the information about this.