Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Server fatal exception: KeyError: 'projects' #1027

Open
gst opened this issue Feb 20, 2024 · 15 comments
Open

Server fatal exception: KeyError: 'projects' #1027

gst opened this issue Feb 20, 2024 · 15 comments

Comments

@gst
Copy link

gst commented Feb 20, 2024 •
edited
Loading

OS + python/pip versions

ubuntu@ip-172-31-21-101:~$ docker exec -it devpi-devpi-1 cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

ubuntu@ip-172-31-21-101:~$ docker exec -it devpi-devpi-1 pip -V
pip 23.1.2 from /usr/local/lib/python3.10/site-packages/pip (python 3.10)

pip check + pip list:

ubuntu@ip-172-31-21-101:~$ docker exec -it devpi-devpi-1 pip check
No broken requirements found.

ubuntu@ip-172-31-21-101:~$ docker exec -it devpi-devpi-1 pip list
Package               Version
--------------------- ----------
aiohttp               3.8.4
aiosignal             1.3.1
argon2-cffi           21.3.0
argon2-cffi-bindings  21.2.0
async-timeout         4.0.2
attrs                 23.1.0
beautifulsoup4        4.12.2
bleach                6.0.0
build                 0.10.0
certifi               2023.5.7
cffi                  1.15.1
Chameleon             4.0.0
charset-normalizer    3.1.0
check-manifest        0.49
cmarkgfm              2022.10.27
defusedxml            0.7.1
devpi-client          6.0.4
devpi-common          3.7.2
devpi-findlinks       3.0.0
devpi-private-mirrors 0.0.3
devpi-semantic-ui     0.2.2
devpi-server          6.8.0
devpi-web             4.2.0
docutils              0.20.1
frozenlist            1.3.3
hupper                1.12
idna                  3.4
iniconfig             2.0.0
itsdangerous          2.1.2
lazy                  1.5
multidict             6.0.4
packaging             21.3
passlib               1.7.4
PasteDeploy           3.0.1
pep517                0.13.0
pip                   23.1.2
pkginfo               1.9.6
plaster               1.1.2
plaster-pastedeploy   1.0.1
platformdirs          3.5.1
pluggy                1.0.0
py                    1.11.0
pycparser             2.21
Pygments              2.15.1
pyparsing             3.0.9
pyproject_hooks       1.0.0
pyramid               2.0.1
pyramid-chameleon     0.3
python-dateutil       2.8.2
readme-renderer       37.3
repoze.lru            0.7
requests              2.30.0
ruamel.yaml           0.17.26
ruamel.yaml.clib      0.2.7
setuptools            67.7.2
six                   1.16.0
soupsieve             2.4.1
strictyaml            1.7.3
tomli                 2.0.1
translationstring     1.4
urllib3               2.0.2
venusian              3.0.0
waitress              2.1.2
webencodings          0.5.1
WebOb                 1.8.7
wheel                 0.40.0
Whoosh                2.7.4
yarl                  1.9.2
zope.deprecation      5.0
zope.interface        6.0

Issue:

have a devpi server instance running since ~1-2y .. last friday it went down for some reason.
today after restart of the host machine, I see in logs it get a "fatal" exception which makes the server to stop (and being restart thx to docker container restart policy) :

2024-02-20 20:33:20,541 INFO  [IDX] Committing 2500 new documents to search index.
2024-02-20 20:33:28,125 INFO  [IDX] Indexer queue size ~ 5
2024-02-20 20:33:30,791 INFO  [req2] GET /+changelog/25877-
2024-02-20 20:34:06,932 INFO  [req3] GET /+changelog/25877-
2024-02-20 20:36:14,372 INFO  [req4] GET /+changelog/25877-
2024-02-20 20:36:45,680 INFO  [req5] GET /+changelog/25877-
2024-02-20 20:36:55,187 INFO  [req6] GET /+changelog/25877
2024-02-20 20:37:15,929 INFO  [req7] GET /+changelog/25877-
2024-02-20 20:37:30,931 INFO  [req8] GET /+changelog/25877-
2024-02-20 20:37:46,555 INFO  [req9] GET /+changelog/25877-
2024-02-20 20:38:06,509 INFO  [req10] GET /+changelog/25877-
2024-02-20 20:38:17,348 INFO  [req11] GET /+changelog/25877-
2024-02-20 20:38:42,783 INFO  [req12] GET /+changelog/25877-
2024-02-20 20:38:47,614 INFO  [req13] GET /+changelog/25877-
2024-02-20 20:39:07,461 ERROR [IDXQ] Exception in thread 'InitialQueueThread'
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/devpi_server/mythread.py", line 40, in run
    result = threading.Thread.run(self)
  File "/usr/local/lib/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.10/site-packages/devpi_web/whoosh_index.py", line 482, in thread_run
    self.shared_data.queue_projects(
  File "/usr/local/lib/python3.10/site-packages/devpi_web/whoosh_index.py", line 286, in queue_projects
    for processed, project in enumerate(projects, start=1):
  File "/usr/local/lib/python3.10/site-packages/devpi_web/indexing.py", line 99, in iter_projects
    names = stage.list_projects_perstage()
  File "/usr/local/lib/python3.10/site-packages/devpi_server/mirror.py", line 447, in list_projects_perstage
    return ensure_deeply_readonly(self._list_projects_perstage())
  File "/usr/local/lib/python3.10/site-packages/devpi_server/mirror.py", line 440, in _list_projects_perstage
    projects = self._update_projects()
  File "/usr/local/lib/python3.10/site-packages/devpi_server/mirror.py", line 394, in _update_projects
    (projects, etag) = self._get_remote_projects()
  File "/usr/local/lib/python3.10/site-packages/devpi_server/mirror.py", line 381, in _get_remote_projects
    parser.feed(response.json())
  File "/usr/local/lib/python3.10/site-packages/devpi_server/mirror.py", line 102, in feed
    self.projects = set(x['name'] for x in data['projects'])
KeyError: 'projects'
2024-02-20 20:39:07,468 INFO  [ASYN] The asyncio event loop stopped
2024-02-20 20:39:07,469 INFO  [ASYN] Thread 'AsyncioLoopThread' ended
Traceback (most recent call last):
  File "/usr/local/bin/devpi-server", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.10/site-packages/devpi_server/main.py", line 77, in main
    return _main(pluginmanager, argv=argv)
  File "/usr/local/lib/python3.10/site-packages/devpi_server/main.py", line 132, in _main
    return xom.main()
  File "/usr/local/lib/python3.10/site-packages/devpi_server/main.py", line 332, in main
    return xom.thread_pool.run(wsgi_run, xom, app)
  File "/usr/local/lib/python3.10/contextlib.py", line 281, in helper
    return _GeneratorContextManager(func, args, kwds)
  File "/usr/local/lib/python3.10/contextlib.py", line 103, in __init__
    self.gen = func(*args, **kwds)
  File "/usr/local/lib/python3.10/site-packages/devpi_server/mythread.py", line 103, in run
    raise self._fatal_exc
  File "/usr/local/lib/python3.10/site-packages/devpi_server/mythread.py", line 40, in run
    result = threading.Thread.run(self)
  File "/usr/local/lib/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.10/site-packages/devpi_web/whoosh_index.py", line 482, in thread_run
    self.shared_data.queue_projects(
  File "/usr/local/lib/python3.10/site-packages/devpi_web/whoosh_index.py", line 286, in queue_projects
    for processed, project in enumerate(projects, start=1):
  File "/usr/local/lib/python3.10/site-packages/devpi_web/indexing.py", line 99, in iter_projects
    names = stage.list_projects_perstage()
  File "/usr/local/lib/python3.10/site-packages/devpi_server/mirror.py", line 447, in list_projects_perstage
    return ensure_deeply_readonly(self._list_projects_perstage())
  File "/usr/local/lib/python3.10/site-packages/devpi_server/mirror.py", line 440, in _list_projects_perstage
    projects = self._update_projects()
  File "/usr/local/lib/python3.10/site-packages/devpi_server/mirror.py", line 394, in _update_projects
    (projects, etag) = self._get_remote_projects()
  File "/usr/local/lib/python3.10/site-packages/devpi_server/mirror.py", line 381, in _get_remote_projects
    parser.feed(response.json())
  File "/usr/local/lib/python3.10/site-packages/devpi_server/mirror.py", line 102, in feed
    self.projects = set(x['name'] for x in data['projects'])
KeyError: 'projects'

and that's re-occurring after few minutes after the server restart (docker container restart always).. after few minutes of such GET /+changelog/25877- (with sometimes some get from my clients).

what can be done ?

Thank you.

Extra infos:

ubuntu@ip-172-31-21-101:~/devpi/data/devpi-master$ ls -ltah server
total 4.6G
drwxr-xr-x 3 ubuntu root 4.0K Feb 20 21:11 .indices
drwxr-xr-x 4 ubuntu root 4.0K Feb 20 21:07 .
-rw------- 1 root   root   68 Feb 20 21:07 .nodeinfo
-rw-r--r-- 1 root   root    5 Feb 20 17:39 .event_serial
-rw-r--r-- 1 ubuntu root 4.6G Feb 20 17:39 .sqlite
drwxr-xr-x 4 ubuntu root 4.0K May  3  2023 +files
drwxr-xr-x 3 ubuntu root 4.0K May  2  2023 ..
-rw-r--r-- 1 ubuntu root    1 May  2  2023 .serverversion
ubuntu@ip-172-31-21-101:~/devpi/data/devpi-master$ sudo more server/.nodeinfo
{
  "role": "master",
  "uuid": "664115d193f8492883523ca4e54e097e"
}

ubuntu@ip-172-31-21-101:~/devpi/data/devpi-master$ sudo more server/.event_serial
25877

ubuntu@ip-172-31-21-101:~/devpi/data/devpi-master$ sudo more server/.serverversion
4
@gst
Copy link
Author

gst commented Feb 20, 2024 •
edited
Loading

just re-launched with docker image re-built to get latest devpi* components version:

devpi-client          7.0.2
devpi-common          4.0.3
devpi-findlinks       3.0.0
devpi-private-mirrors 0.0.3
devpi-semantic-ui     0.2.2
devpi-server          6.10.0
devpi-web             4.2.1

but again looping on :

...
+ devpi-server --role master --serverdir /devpi-server/server --secretfile /devpi-server/secret --host 0.0.0.0 --port 3141 --keyfs-cache-size 32768 --mirror-cache-expiry 3600 --replica-max-retries 3 --max-request-body-size 2684354560 --theme /usr/local/lib/python3.10/site-packages/devpi_semantic_ui
2024-02-20 21:39:20,486 INFO  NOCTX Loading node info from /devpi-server/server/.nodeinfo
2024-02-20 21:39:20,487 INFO  NOCTX wrote nodeinfo to: /devpi-server/server/.nodeinfo
2024-02-20 21:39:20,517 INFO  NOCTX running with role 'master'
2024-02-20 21:39:20,522 INFO  NOCTX Using secret file '/devpi-server/secret'.
2024-02-20 21:39:23,130 INFO  NOCTX Found plugin devpi-web-4.2.1.
2024-02-20 21:39:23,132 INFO  NOCTX Found plugin devpi-private-mirrors-0.0.3.
2024-02-20 21:39:23,138 INFO  NOCTX Found plugin devpi-semantic-ui-0.2.2.
2024-02-20 21:39:23,139 INFO  NOCTX Found plugin devpi-findlinks-3.0.0.
2024-02-20 21:39:23,236 INFO  NOCTX Using /devpi-server/server/.indices for Whoosh index files.
2024-02-20 21:39:23,267 INFO  [ASYN] Starting asyncio event loop
2024-02-20 21:39:23,277 INFO  NOCTX devpi-server version: 6.10.0
2024-02-20 21:39:23,277 INFO  NOCTX serverdir: /devpi-server/server
2024-02-20 21:39:23,277 INFO  NOCTX uuid: 664115d193f8492883523ca4e54e097e
2024-02-20 21:39:23,277 INFO  NOCTX serving at url: http://0.0.0.0:3141 (might be http://[0.0.0.0]:3141 for IPv6)
2024-02-20 21:39:23,277 INFO  NOCTX using 50 threads
2024-02-20 21:39:23,277 INFO  NOCTX bug tracker: https://github.com/devpi/devpi/issues
2024-02-20 21:39:23,278 INFO  NOCTX Hit Ctrl-C to quit.
2024-02-20 21:39:23,290 INFO  Serving on http://0.0.0.0:3141
2024-02-20 21:39:28,447 INFO  [req0] GET /+changelog/25877-
2024-02-20 21:39:28,672 INFO  [IDX] Indexer queue size ~ 26
2024-02-20 21:39:33,689 INFO  [IDX] Indexer queue size ~ 13
2024-02-20 21:40:02,135 INFO  [req1] GET /+changelog/25877-
2024-02-20 21:40:32,922 INFO  [req2] GET /+changelog/25877-
2024-02-20 21:40:34,002 INFO  [IDX] Indexer queue size ~ 1
2024-02-20 21:41:03,203 INFO  [req3] GET /+changelog/25877-
2024-02-20 21:41:33,724 INFO  [req4] GET /+changelog/25877-

EDIT:

and still crashing the same way:

# many many (with different [reqxxx]) :

2024-02-20 22:00:39,133 INFO  [req365] GET /+changelog/25877-

# then, after some/few minutes of running:

2024-02-20 22:00:54,722 ERROR [IDXQ] Exception in thread 'InitialQueueThread'
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/devpi_server/mythread.py", line 40, in run
    result = threading.Thread.run(self)
  File "/usr/local/lib/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.10/site-packages/devpi_web/whoosh_index.py", line 476, in thread_run
    self.shared_data.queue_projects(
  File "/usr/local/lib/python3.10/site-packages/devpi_web/whoosh_index.py", line 280, in queue_projects
    for processed, project in enumerate(projects, start=1):
  File "/usr/local/lib/python3.10/site-packages/devpi_web/indexing.py", line 99, in iter_projects
    names = stage.list_projects_perstage()
  File "/usr/local/lib/python3.10/site-packages/devpi_server/mirror.py", line 477, in list_projects_perstage
    return ensure_deeply_readonly(self._list_projects_perstage())
  File "/usr/local/lib/python3.10/site-packages/devpi_server/mirror.py", line 470, in _list_projects_perstage
    projects = self._update_projects()
  File "/usr/local/lib/python3.10/site-packages/devpi_server/mirror.py", line 421, in _update_projects
    (projects, etag) = self._get_remote_projects()
  File "/usr/local/lib/python3.10/site-packages/devpi_server/mirror.py", line 408, in _get_remote_projects
    parser.feed(response.json())
  File "/usr/local/lib/python3.10/site-packages/devpi_server/mirror.py", line 104, in feed
    self.projects = set(x['name'] for x in data['projects'])
KeyError: 'projects'
2024-02-20 22:00:54,734 INFO  [ASYN] The asyncio event loop stopped
2024-02-20 22:00:54,734 INFO  [ASYN] Thread 'AsyncioLoopThread' ended
2024-02-20 22:00:54,794 INFO  [req373] GET /root/pypi/+f/09a/42662837af144/bokeh-3.0.0-py3-none-any.whl
Traceback (most recent call last):
  File "/usr/local/bin/devpi-server", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.10/site-packages/devpi_server/main.py", line 123, in main
    return _main(runner.pluginmanager, argv=argv)
  File "/usr/local/lib/python3.10/site-packages/devpi_server/main.py", line 179, in _main
    return xom.main()
  File "/usr/local/lib/python3.10/site-packages/devpi_server/main.py", line 385, in main
    return xom.thread_pool.run(wsgi_run, xom, app)
  File "/usr/local/lib/python3.10/contextlib.py", line 281, in helper
    return _GeneratorContextManager(func, args, kwds)
  File "/usr/local/lib/python3.10/contextlib.py", line 103, in __init__
    self.gen = func(*args, **kwds)
  File "/usr/local/lib/python3.10/site-packages/devpi_server/mythread.py", line 103, in run
    raise self._fatal_exc
  File "/usr/local/lib/python3.10/site-packages/devpi_server/mythread.py", line 40, in run
    result = threading.Thread.run(self)
  File "/usr/local/lib/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.10/site-packages/devpi_web/whoosh_index.py", line 476, in thread_run
    self.shared_data.queue_projects(
  File "/usr/local/lib/python3.10/site-packages/devpi_web/whoosh_index.py", line 280, in queue_projects
    for processed, project in enumerate(projects, start=1):
  File "/usr/local/lib/python3.10/site-packages/devpi_web/indexing.py", line 99, in iter_projects
    names = stage.list_projects_perstage()
  File "/usr/local/lib/python3.10/site-packages/devpi_server/mirror.py", line 477, in list_projects_perstage
    return ensure_deeply_readonly(self._list_projects_perstage())
  File "/usr/local/lib/python3.10/site-packages/devpi_server/mirror.py", line 470, in _list_projects_perstage
    projects = self._update_projects()
  File "/usr/local/lib/python3.10/site-packages/devpi_server/mirror.py", line 421, in _update_projects
    (projects, etag) = self._get_remote_projects()
  File "/usr/local/lib/python3.10/site-packages/devpi_server/mirror.py", line 408, in _get_remote_projects
    parser.feed(response.json())
  File "/usr/local/lib/python3.10/site-packages/devpi_server/mirror.py", line 104, in feed
    self.projects = set(x['name'] for x in data['projects'])
KeyError: 'projects'

@gst
Copy link
Author

gst commented Feb 20, 2024

Also extra-extra information:

I'm using 2 replica instances replicating that master server role.

@gst
Copy link
Author

gst commented Feb 20, 2024 •
edited
Loading

well, after an nth restart after such same crash/error, now I see:

2024-02-20 22:29:00,806 INFO  [req100] GET /+changelog/25879-

has this solved by itself ? (but I restarted 1 of the replica meanwhile fwiw)

EDIT

unfortunately same key error / stacktrace just after I posted this.

but now:

2024-02-20 22:32:38,435 INFO  [req14] GET /+changelog/25880-

maybe at some point it will recover completely ?

@fschulze
Copy link
Contributor

The replicas are polling on the latest serial using +changelog.

The traceback is unrelated to that. It is triggered by the search indexing. If you don't use the search functionality, you could disable it with --indexer-backend=null as a workaround to get a bit further.

It seems like the root/pypi mirror has problems reading responses when updating. Can you access /root/pypi/+simple/bokeh/? Do you get a traceback if you do so? Can you access the configured mirror URL from the docker image (https://pypi.org/simple/bokeh/ by default for that project)?

@gst
Copy link
Author

gst commented Feb 21, 2024

The traceback is unrelated to that. It is triggered by the search indexing. If you don't use the search functionality, you could disable it with --indexer-backend=null as a workaround to get a bit further.

put that on my master (and restarted it). correct for master ? it's not on the replica(s) I have to do this ? or both ?

It seems like the root/pypi mirror has problems reading responses when updating. Can you access /root/pypi/+simple/bokeh/? Do you get a traceback if you do so? Can you access the configured mirror URL from the docker image (https://pypi.org/simple/bokeh/ by default for that project)?

can access this url on all my instances: master + the 2 replicas..

but, on the replicas, if I click/try refresh button/devpi refresh => I then get in logs of the related replica:

2024-02-21 11:43:36,060 INFO  [req391] GET /root/pypi/+simple/bokeh/
2024-02-21 11:43:41,109 WARNI [req391] [Rtx25882] serving stale links for 'bokeh', getting data timed out after 5 seconds
2024-02-21 11:43:41,181 INFO  [req392] GET /favicon.ico
2024-02-21 11:43:41,182 ERROR [req392] [Rtx25882] while handling http://localhost:3141/favicon.ico:
no user 'favicon.ico'
2024-02-21 11:43:41,579 INFO  HTTP Request: GET https://devpi.southb.net/root/pypi/+simple/bokeh/ "HTTP/1.1 200 OK"
2024-02-21 11:43:43,510 INFO  [req393] POST /root/pypi/+simple/bokeh/refresh
2024-02-21 11:43:43,511 INFO  [req393] [Rtx25882] relaying: POST https://devpi.southb.net/root/pypi/+simple/bokeh/refresh
2024-02-21 11:43:44,406 INFO  [REP] fetching https://devpi.southb.net/+changelog/25883-
2024-02-21 11:43:48,993 INFO  [req393] [Rtx25882] FIN: relaying: POST https://devpi.southb.net/root/pypi/+simple/bokeh/refresh
2024-02-21 11:43:48,994 ERROR Exception while serving /root/pypi/+simple/bokeh/refresh
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/waitress/channel.py", line 428, in service
    task.service()
  File "/usr/local/lib/python3.10/site-packages/waitress/task.py", line 168, in service
    self.execute()
  File "/usr/local/lib/python3.10/site-packages/waitress/task.py", line 460, in execute
    first_chunk_len = len(chunk)
TypeError: object of type 'NoneType' has no len()

@gst
Copy link
Author

gst commented Feb 21, 2024 •
edited
Loading

put (and restarted) --indexer-backend=null on one of my replica : but same above traceback if I try the refresh after.

EDIT:

atm/since I've put --index-backend=null on the master (and restarted it with that) : it apparently does not crash anymore.. :) already good :)

@gst
Copy link
Author

gst commented Feb 21, 2024

Can you access the configured mirror URL from the docker image (https://pypi.org/simple/bokeh/ by default for that project)?

sorry forgot answer this one : but yes I can access/download pypi.org/that specific url without any issue from within my master container.

@fschulze
Copy link
Contributor

That all looks really weird. The traceback in waitress seems to indicated that there is some kind of connection issue. I can't tell in which direction though, it could be between replica and primary or between the replica and the client. Currently I have no clue what is going on there.

@gst
Copy link
Author

gst commented Feb 22, 2024

hi @fschulze , for info:

my master instance is now ok : no more recurring crash/exception at all and also no more stacktrace/exception if/when I do refresh the (root/pypi/) bokeh indice/project (either via the web-ui page or via devpi refresh cli command). it seems it just succeed.

but from my 2 replicas I still get the TypeError: object of type 'NoneType' has no len() if I try the refresh (still of root/pypi/bokeh).

root@26960be67dbe:/devpi-server# devpi refresh --index root/pypi bokeh
Couldn't refresh bokeh via http://localhost:3141/root/pypi/+simple/bokeh/refresh: 500 Internal Server Error
Refreshed bokeh

replica log:

2024-02-22 14:05:19,681 INFO  [req12] GET /+api
2024-02-22 14:05:23,818 INFO  [req13] POST /+login
2024-02-22 14:05:23,819 INFO  [req13] [Rtx26020] relaying: POST https://devpi.southb.net/+login
2024-02-22 14:05:29,307 INFO  [req13] [Rtx26020] FIN: relaying: POST https://devpi.southb.net/+login
2024-02-22 14:05:40,848 INFO  [REP] fetching https://devpi.southb.net/+changelog/26021-
2024-02-22 14:05:45,212 INFO  [req14] GET /root/pypi/+api
2024-02-22 14:05:45,216 INFO  [req15] POST /root/pypi/+simple/bokeh/refresh
2024-02-22 14:05:45,217 INFO  [req15] [Rtx26020] relaying: POST https://devpi.southb.net/root/pypi/+simple/bokeh/refresh
2024-02-22 14:05:45,426 INFO  [req15] [Rtx26020] FIN: relaying: POST https://devpi.southb.net/root/pypi/+simple/bokeh/refresh
2024-02-22 14:05:45,428 ERROR Exception while serving /root/pypi/+simple/bokeh/refresh
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/waitress/channel.py", line 428, in service
    task.service()
  File "/usr/local/lib/python3.10/site-packages/waitress/task.py", line 168, in service
    self.execute()
  File "/usr/local/lib/python3.10/site-packages/waitress/task.py", line 460, in execute
    first_chunk_len = len(chunk)
TypeError: object of type 'NoneType' has no len()

@fschulze
Copy link
Contributor

You updated the versions in the meantime, could you give me the full current list including waitress? It might be some resent change somewhere. The replicas forward the POST to the primary, so maybe something is going on in the communication between the replica and the master for just that case.

@gst
Copy link
Author

gst commented Feb 22, 2024

replica pip list:

root@26960be67dbe:/devpi-server# pip list
Package               Version
--------------------- ---------
anyio                 4.3.0
argon2-cffi           23.1.0
argon2-cffi-bindings  21.2.0
attrs                 23.2.0
beautifulsoup4        4.12.3
build                 1.0.3
certifi               2024.2.2
cffi                  1.16.0
Chameleon             4.5.2
charset-normalizer    3.3.2
check-manifest        0.49
cmarkgfm              2024.1.14
defusedxml            0.7.1
devpi-client          7.0.2
devpi-common          4.0.3
devpi-findlinks       3.0.0
devpi-private-mirrors 0.0.3
devpi-semantic-ui     0.2.2
devpi-server          6.10.0
devpi-web             4.2.1
docutils              0.20.1
exceptiongroup        1.2.0
h11                   0.14.0
httpcore              1.0.3
httpx                 0.26.0
hupper                1.12.1
idna                  3.6
iniconfig             2.0.0
itsdangerous          2.1.2
lazy                  1.6
nh3                   0.2.15
packaging             23.2
passlib               1.7.4
PasteDeploy           3.1.0
pip                   24.0
pkginfo               1.9.6
plaster               1.1.2
plaster-pastedeploy   1.0.1
platformdirs          4.2.0
pluggy                1.4.0
py                    1.11.0
pycparser             2.21
Pygments              2.17.2
pyproject_hooks       1.0.0
pyramid               2.0.2
pyramid_chameleon     0.3
python-dateutil       2.8.2
readme-renderer       42.0
repoze.lru            0.7
requests              2.31.0
ruamel.yaml           0.18.6
ruamel.yaml.clib      0.2.8
setuptools            69.1.0
six                   1.16.0
sniffio               1.3.0
soupsieve             2.5
strictyaml            1.7.3
tomli                 2.0.1
translationstring     1.4
typing_extensions     4.9.0
urllib3               2.2.1
venusian              3.1.0
waitress              3.0.0
WebOb                 1.8.7
wheel                 0.42.0
Whoosh                2.7.4
zope.deprecation      5.0
zope.interface        6.2

@gst
Copy link
Author

gst commented Feb 22, 2024 

root@26960be67dbe:/devpi-server/server# more .serverversion .nodeinfo .event_serial 
4
{
  "role": "replica",
  "masterurl": "https://devpi.southb.net",
  "uuid": "dfc29598dfd64054a12db1af429c0109",
  "master-uuid": "664115d193f8492883523ca4e54e097e"
}
26043

@fschulze
Copy link
Contributor

So, there was a waitress 3.0.0 release at the beginning of February with some changes which might be related. I still have to investigate further, but you could try to install the last version before that in the meantime.

@gst
Copy link
Author

gst commented Feb 23, 2024

@fschulze but I've the last version of everything in/with above replica pip list.. :/

well, atm :

root@26960be67dbe:/devpi-server# pip list -o
Package    Version Latest Type
---------- ------- ------ -----
httpcore   1.0.3   1.0.4  wheel
httpx      0.26.0  0.27.0 wheel
setuptools 69.1.0  69.1.1 wheel

the second replica has few more outdated packages, but waitress is not part of them : I've 3.0.0 on all my nodes.

second replica:

root@4bb49a6dda4a:/devpi-server# pip list -o
Package        Version Latest Type
-------------- ------- ------ -----
anyio          4.2.0   4.3.0  wheel
httpcore       1.0.2   1.0.4  wheel
httpx          0.26.0  0.27.0 wheel
pip            23.3.2  24.0   wheel
ruamel.yaml    0.18.5  0.18.6 wheel
setuptools     69.0.3  69.1.1 wheel
urllib3        2.2.0   2.2.1  wheel
zope.interface 6.1     6.2    wheel

and my master has same outdated pkgs than my first replica here.

I gave a try with previous waitress version (2.1.2) on my first replica but same exception/traceback if I try refresh root/pypi/bokeh.

@gst
Copy link
Author

gst commented Feb 24, 2024

Extra info:

I spawned a totally fresh/new replica (which fully synced with the master) : and I tried refresh bokeh on it and I still get the same stacktrace/exception as before :/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@gst @fschulze