Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] Kubernetes deployment doesn't consume config.yaml #244

Open
twharding opened this issue Oct 27, 2023 · 6 comments
Open

[BUG] Kubernetes deployment doesn't consume config.yaml #244

twharding opened this issue Oct 27, 2023 · 6 comments
Labels
bug Something isn't working good first issue Good for newcomers

Comments

@twharding
Copy link

twharding commented Oct 27, 2023 •
edited

Describe the bug

Modifying values in config.yaml such as enabling auth or disabling summarizing does not affect the Zep server.

To Reproduce
Modify some values in config.yaml:

  messages:
    summarizer:
      enabled: false
...
auth:
  Set to true to enable authentication
  required: true

create configmap with:
kubectl create configmap zep-config --from-file=config.yaml -n zep

Deploy with:
kubectl apply -f zep-k8-deployment.yaml.

(Reboot deployment if already running)

The JSON printout in the web GUI is unchanged, and

Expected behavior
Zep will now require an API key and summarization will be disabled.

Logs

Environment variable ZEP_NLP_SERVER_HOSTPORT is not set. If running on Render.com, please ensure you are passing the NLP server's hostport as an environment variable.
time="2023-10-27T20:00:47Z" level=warning msg=".env file not found or unable to load. This warning can be ignored if Zep is run using docker compose with env_file defined or you are passing ENV variables."
time="2023-10-27T20:00:47Z" level=info msg="Log level set to: info"
time="2023-10-27T20:00:47Z" level=info msg="Starting Zep server version 0.16.1-04d4698 (2023-10-19T18:37:41+0000)"
time="2023-10-27T20:00:48Z" level=info msg="vector extension version is >= 0.5.0. hnsw indexing available"
time="2023-10-27T20:00:48Z" level=warning msg="message embedding dimensions are 1536, expected 384.\n migrating message embedding column width to 384. this may result in loss of existing embedding vectors"
time="2023-10-27T20:00:48Z" level=info msg="creating hnsw index on message_embedding.embedding if it does not exist"
time="2023-10-27T20:00:48Z" level=info msg="created hnsw index successfully on message_embedding.embedding if it did not exist"
time="2023-10-27T20:00:48Z" level=info msg="migrated to group #1 (20230822100800_add_user_uuid_to_session_schema, 2023092800_add_index_type_to_collection_schema)\n"
time="2023-10-27T20:00:48Z" level=info msg="started document embedding processor"
time="2023-10-27T20:00:48Z" level=info msg="Using memory store: postgres"
time="2023-10-27T20:00:48Z" level=info msg="Initializing extractors"
time="2023-10-27T20:00:48Z" level=info msg="SummaryExtractor attached to memory store"
time="2023-10-27T20:00:48Z" level=info msg="EmbeddingExtractor attached to memory store"
time="2023-10-27T20:00:48Z" level=info msg="TokenCountExtractor attached to memory store"
time="2023-10-27T20:00:48Z" level=info msg="EntityExtractor attached to memory store"
time="2023-10-27T20:00:48Z" level=info msg="IntentExtractor attached to memory store"
time="2023-10-27T20:00:48Z" level=info msg="starting document embedding updater"
time="2023-10-27T20:00:48Z" level=info msg="started document embedding updater"
time="2023-10-27T20:00:48Z" level=info msg="Starting purge delete processor. Purging every 1h0m0s"
time="2023-10-27T20:00:48Z" level=info msg="Web interface enabled"
time="2023-10-27T20:00:48Z" level=info msg="Listening on: 0.0.0.0:8000"
time="2023-10-27T20:00:48Z" level=info msg="Web UI available at: 0.0.0.0:8000/admin"
time="2023-10-27T20:00:48Z" level=info msg="completed purging store"
time="2023-10-27T20:00:57Z" level=info msg="http://192.168.79.148:8000/healthz" bytes=1 category=router duration=36880 duration_display="37.301µs" method=GET proto=HTTP/1.1 remote_ip=192.168.67.246 status_code=200
time="2023-10-27T20:01:02Z" level=info msg="http://192.168.79.148:8000/healthz" bytes=1 category=router duration=28076 duration_display="28.415µs" method=GET proto=HTTP/1.1 remote_ip=192.168.67.246 status_code=200
time="2023-10-27T20:01:07Z" level=info msg="http://192.168.79.148:8000/healthz" bytes=1 category=router duration=29320 duration_display="29.606µs" method=GET proto=HTTP/1.1 remote_ip=192.168.67.246 status_code=200
time="2023-10-27T20:01:11Z" level=info msg="http://**************************************.ca-central-1.elb.amazonaws.com:8000/admin/settings" bytes=6327 category=router duration=7946868 duration_display=7.947225ms method=GET proto=HTTP/1.1 remote_ip=192.168.15.213 status_code=200
time="2023-10-27T20:01:12Z" level=info msg="http://**************************************.ca-central-1.elb.amazonaws.com:8000/static/output.css" bytes=7306 category=router duration=995123 duration_display="995.318µs" method=GET proto=HTTP/1.1 remote_ip=192.168.15.213 status_code=200
time="2023-10-27T20:01:12Z" level=info msg="http://**************************************.ca-central-1.elb.amazonaws.com:8000/static/js/dark-mode.js" bytes=2373 category=router duration=85632 duration_display="85.809µs" method=GET proto=HTTP/1.1 remote_ip=192.168.15.213 status_code=200
time="2023-10-27T20:01:12Z" level=info msg="http://**************************************.ca-central-1.elb.amazonaws.com:8000/static/js/htmx.min.js" bytes=44717 category=router duration=115789 duration_display="115.979µs" method=GET proto=HTTP/1.1 remote_ip=192.168.15.213 status_code=200
time="2023-10-27T20:01:12Z" level=info msg="http://**************************************.ca-central-1.elb.amazonaws.com:8000/static/js/alpinejs-3.13.0.min.js" bytes=42476 category=router duration=341264 duration_display="341.44µs" method=GET proto=HTTP/1.1 remote_ip=192.168.15.213 status_code=200
time="2023-10-27T20:01:12Z" level=info msg="http://**************************************.ca-central-1.elb.amazonaws.com:8000/static/zep-name-logo.svg" bytes=201823 category=router duration=21291993 duration_display=21.292279ms method=GET proto=HTTP/1.1 remote_ip=192.168.15.213 status_code=200
time="2023-10-27T20:01:12Z" level=info msg="http://192.168.79.148:8000/healthz" bytes=1 category=router duration=47067 duration_display="47.34µs" method=GET proto=HTTP/1.1 remote_ip=192.168.67.246 status_code=200
time="2023-10-27T20:01:12Z" level=info msg="http://**************************************.ca-central-1.elb.amazonaws.com:8000/static/preline/preline.js" bytes=59714 category=router duration=180811 duration_display="181.022µs" method=GET proto=HTTP/1.1 remote_ip=192.168.15.213 status_code=200
time="2023-10-27T20:01:13Z" level=info msg="http://**************************************.ca-central-1.elb.amazonaws.com:8000/static/favicon.png" bytes=3324 category=router duration=162840 duration_display="163.011µs" method=GET proto=HTTP/1.1 remote_ip=192.168.15.213 status_code=200
time="2023-10-27T20:01:16Z" level=info msg="http://**************************************.ca-central-1.elb.amazonaws.com:8000/admin/settings" bytes=6327 category=router duration=3732891 duration_display=3.73318ms method=GET proto=HTTP/1.1 remote_ip=192.168.15.213 status_code=200
time="2023-10-27T20:01:16Z" level=info msg="http://**************************************.ca-central-1.elb.amazonaws.com:8000/static/output.css" bytes=7306 category=router duration=889309 duration_display="889.569µs" method=GET proto=HTTP/1.1 remote_ip=192.168.15.213 status_code=200
time="2023-10-27T20:01:16Z" level=info msg="http://**************************************.ca-central-1.elb.amazonaws.com:8000/static/js/dark-mode.js" bytes=2373 category=router duration=65364 duration_display="65.526µs" method=GET proto=HTTP/1.1 remote_ip=192.168.15.213 status_code=200
time="2023-10-27T20:01:16Z" level=info msg="http://**************************************.ca-central-1.elb.amazonaws.com:8000/static/js/htmx.min.js" bytes=44717 category=router duration=169567 duration_display="169.771µs" method=GET proto=HTTP/1.1 remote_ip=192.168.15.213 status_code=200
time="2023-10-27T20:01:16Z" level=info msg="http://**************************************.ca-central-1.elb.amazonaws.com:8000/static/js/alpinejs-3.13.0.min.js" bytes=42476 category=router duration=145679 duration_display="145.859µs" method=GET proto=HTTP/1.1 remote_ip=192.168.15.213 status_code=200
time="2023-10-27T20:01:16Z" level=info msg="http://**************************************.ca-central-1.elb.amazonaws.com:8000/static/preline/preline.js" bytes=59714 category=router duration=202691 duration_display="202.957µs" method=GET proto=HTTP/1.1 remote_ip=192.168.15.213 status_code=200
time="2023-10-27T20:01:16Z" level=info msg="http://**************************************.ca-central-1.elb.amazonaws.com:8000/static/zep-name-logo.svg" bytes=201823 category=router duration=21086070 duration_display=21.086356ms method=GET proto=HTTP/1.1 remote_ip=192.168.15.213 status_code=200
time="2023-10-27T20:01:17Z" level=info msg="http://**************************************.ca-central-1.elb.amazonaws.com:8000/static/favicon.png" bytes=3324 category=router duration=146105 duration_display="146.273µs" method=GET proto=HTTP/1.1 remote_ip=192.168.15.213 status_code=200
time="2023-10-27T20:01:17Z" level=info msg="http://192.168.79.148:8000/healthz" bytes=1 category=router duration=45080 duration_display="45.403µs" method=GET proto=HTTP/1.1 remote_ip=192.168.67.246 status_code=200
time="2023-10-27T20:01:18Z" level=info msg="http://**************************************.ca-central-1.elb.amazonaws.com:8000/admin" bytes=6267 category=router duration=897160 duration_display="897.392µs" method=GET proto=HTTP/1.1 remote_ip=192.168.15.213 status_code=200
time="2023-10-27T20:01:20Z" level=info msg="http://**************************************.ca-central-1.elb.amazonaws.com:8000/admin" bytes=6544 category=router duration=2107329 duration_display=2.107595ms method=GET proto=HTTP/1.1 remote_ip=192.168.15.213 status_code=200
time="2023-10-27T20:01:20Z" level=info msg="http://**************************************.ca-central-1.elb.amazonaws.com:8000/static/output.css" bytes=7306 category=router duration=964809 duration_display="965.074µs" method=GET proto=HTTP/1.1 remote_ip=192.168.15.213 status_code=200
time="2023-10-27T20:01:20Z" level=info msg="http://**************************************.ca-central-1.elb.amazonaws.com:8000/static/js/htmx.min.js" bytes=44717 category=router duration=194857 duration_display="195.08µs" method=GET proto=HTTP/1.1 remote_ip=192.168.15.213 status_code=200
time="2023-10-27T20:01:20Z" level=info msg="http://**************************************.ca-central-1.elb.amazonaws.com:8000/static/js/dark-mode.js" bytes=2373 category=router duration=151388 duration_display="151.567µs" method=GET proto=HTTP/1.1 remote_ip=192.168.15.213 status_code=200
time="2023-10-27T20:01:20Z" level=info msg="http://**************************************.ca-central-1.elb.amazonaws.com:8000/static/js/alpinejs-3.13.0.min.js" bytes=42476 category=router duration=275124 duration_display="275.349µs" method=GET proto=HTTP/1.1 remote_ip=192.168.15.213 status_code=200
time="2023-10-27T20:01:20Z" level=info msg="http://**************************************.ca-central-1.elb.amazonaws.com:8000/static/zep-name-logo.svg" bytes=201823 category=router duration=20913855 duration_display=20.914103ms method=GET proto=HTTP/1.1 remote_ip=192.168.15.213 status_code=200
time="2023-10-27T20:01:20Z" level=info msg="http://**************************************.ca-central-1.elb.amazonaws.com:8000/static/preline/preline.js" bytes=59714 category=router duration=150569 duration_display="150.805µs" method=GET proto=HTTP/1.1 remote_ip=192.168.15.213 status_code=200
time="2023-10-27T20:01:21Z" level=info msg="http://**************************************.ca-central-1.elb.amazonaws.com:8000/static/favicon.png" bytes=3324 category=router duration=143921 duration_display="144.09µs" method=GET proto=HTTP/1.1 remote_ip=192.168.15.213 status_code=200
time="2023-10-27T20:01:22Z" level=info msg="http://192.168.79.148:8000/healthz" bytes=1 category=router duration=51498 duration_display="51.779µs" method=GET proto=HTTP/1.1 remote_ip=192.168.67.246 status_code=200
time="2023-10-27T20:01:22Z" level=info msg="http://**************************************.ca-central-1.elb.amazonaws.com:8000/admin/settings" bytes=11101 category=router duration=2409231 duration_display=2.409462ms method=GET proto=HTTP/1.1 remote_ip=192.168.15.213 status_code=200
time="2023-10-27T20:01:27Z" level=info msg="http://192.168.79.148:8000/healthz" bytes=1 category=router duration=43633 duration_display="43.926µs" method=GET proto=HTTP/1.1 remote_ip=192.168.67.246 status_code=200
time="2023-10-27T20:01:32Z" level=info msg="http://192.168.79.148:8000/healthz" bytes=1 category=router duration=33112 duration_display="33.541µs" method=GET proto=HTTP/1.1 remote_ip=192.168.67.246 status_code=200
time="2023-10-27T20:01:34Z" level=info msg="http://**************************************.ca-central-1.elb.amazonaws.com:8000/admin" bytes=6267 category=router duration=847731 duration_display="847.943µs" method=GET proto=HTTP/1.1 remote_ip=192.168.15.213 status_code=200
time="2023-10-27T20:01:37Z" level=info msg="http://192.168.79.148:8000/healthz" bytes=1 category=router duration=31785 duration_display="32.201µs" method=GET proto=HTTP/1.1 remote_ip=192.168.67.246 status_code=200
time="2023-10-27T20:01:39Z" level=info msg="http://**************************************.ca-central-1.elb.amazonaws.com:8000/healthz" bytes=1 category=router duration=26535 duration_display="26.892µs" method=GET proto=HTTP/1.1 remote_ip=192.168.15.213 status_code=200
time="2023-10-27T20:01:40Z" level=info msg="http://**************************************.ca-central-1.elb.amazonaws.com:8000/healthz" bytes=1 category=router duration=28775 duration_display="29.183µs" method=GET proto=HTTP/1.1 remote_ip=192.168.67.246 status_code=200
time="2023-10-27T20:01:42Z" level=info msg="http://192.168.79.148:8000/healthz" bytes=1 category=router duration=39150 duration_display="39.589µs" method=GET proto=HTTP/1.1 remote_ip=192.168.67.246 status_code=200
time="2023-10-27T20:01:46Z" level=info msg="http://**************************************.ca-central-1.elb.amazonaws.com:8000/api/v1/sessions/30915301-9248-423c-bfb7-a754b58b04eb/memory" bytes=10 category=router duration=2650890 duration_display=2.651147ms method=GET proto=HTTP/1.1 remote_ip=192.168.15.213 status_code=404
time="2023-10-27T20:01:47Z" level=info msg="http://192.168.79.148:8000/healthz" bytes=1 category=router duration=26575 duration_display="26.916µs" method=GET proto=HTTP/1.1 remote_ip=192.168.67.246 status_code=200
time="2023-10-27T20:01:47Z" level=info msg="[DEBUG] POST http://zep-nlp:5557/embeddings/message"
time="2023-10-27T20:01:47Z" level=info msg="http://**************************************.ca-central-1.elb.amazonaws.com:8000/api/v1/sessions/30915301-9248-423c-bfb7-a754b58b04eb/search" bytes=10 category=router duration=26322859 duration_display=26.323187ms method=POST proto=HTTP/1.1 remote_ip=192.168.15.213 status_code=404
time="2023-10-27T20:01:52Z" level=info msg="http://192.168.79.148:8000/healthz" bytes=1 category=router duration=36213 duration_display="36.597µs" method=GET proto=HTTP/1.1 remote_ip=192.168.67.246 status_code=200
time="2023-10-27T20:01:53Z" level=info msg="http://**************************************.ca-central-1.elb.amazonaws.com:8000/api/v1/sessions/30915301-9248-423c-bfb7-a754b58b04eb/memory" bytes=10 category=router duration=2329882 duration_display=2.330228ms method=GET proto=HTTP/1.1 remote_ip=192.168.67.246 status_code=404
time="2023-10-27T20:01:57Z" level=info msg="http://192.168.79.148:8000/healthz" bytes=1 category=router duration=27612 duration_display="27.921µs" method=GET proto=HTTP/1.1 remote_ip=192.168.67.246 status_code=200
time="2023-10-27T20:02:02Z" level=info msg="http://**************************************.ca-central-1.elb.amazonaws.com:8000/api/v1/sessions/30915301-9248-423c-bfb7-a754b58b04eb/memory" bytes=2 category=router duration=7072303 duration_display=7.072715ms method=POST proto=HTTP/1.1 remote_ip=192.168.15.213 status_code=200
time="2023-10-27T20:02:02Z" level=info msg="[DEBUG] POST http://zep-nlp:5557/embeddings/message"
time="2023-10-27T20:02:02Z" level=info msg="[DEBUG] POST https://api.openai.com/v1/chat/completions"
time="2023-10-27T20:02:02Z" level=info msg="http://**************************************.ca-central-1.elb.amazonaws.com:8000/api/v1/sessions/30915301-9248-423c-bfb7-a754b58b04eb/memory" bytes=2 category=router duration=27489320 duration_display=27.489918ms method=POST proto=HTTP/1.1 remote_ip=192.168.15.213 status_code=200
time="2023-10-27T20:02:02Z" level=info msg="[DEBUG] POST http://zep-nlp:5557/embeddings/message"
time="2023-10-27T20:02:02Z" level=info msg="http://192.168.79.148:8000/healthz" bytes=1 category=router duration=33456 duration_display="33.791µs" method=GET proto=HTTP/1.1 remote_ip=192.168.67.246 status_code=200
time="2023-10-27T20:02:02Z" level=info msg="[DEBUG] POST https://api.openai.com/v1/chat/completions"
time="2023-10-27T20:02:04Z" level=info msg="http://**************************************.ca-central-1.elb.amazonaws.com:8000/api/v1/sessions/30915301-9248-423c-bfb7-a754b58b04eb/memory" bytes=2863 category=router duration=2647493 duration_display=2.647783ms method=GET proto=HTTP/1.1 remote_ip=192.168.15.213 status_code=200
time="2023-10-27T20:02:07Z" level=info msg="http://192.168.79.148:8000/healthz" bytes=1 category=router duration=33342 duration_display="33.744µs" method=GET proto=HTTP/1.1 remote_ip=192.168.67.246 status_code=200
time="2023-10-27T20:02:09Z" level=info msg="[DEBUG] POST http://zep-nlp:5557/embeddings/message"
time="2023-10-27T20:02:09Z" level=info msg="http://**************************************.ca-central-1.elb.amazonaws.com:8000/api/v1/sessions/30915301-9248-423c-bfb7-a754b58b04eb/search" bytes=2979 category=router duration=21693716 duration_display=21.693983ms method=POST proto=HTTP/1.1 remote_ip=192.168.15.213 status_code=200
time="2023-10-27T20:02:12Z" level=info msg="http://192.168.79.148:8000/healthz" bytes=1 category=router duration=49743 duration_display="50.178µs" method=GET proto=HTTP/1.1 remote_ip=192.168.67.246 status_code=200
time="2023-10-27T20:02:16Z" level=info msg="http://**************************************.ca-central-1.elb.amazonaws.com:8000/admin/sessions" bytes=7612 category=router duration=3164283 duration_display=3.164496ms method=GET proto=HTTP/1.1 remote_ip=192.168.15.213 status_code=200
time="2023-10-27T20:02:16Z" level=info msg="http://**************************************.ca-central-1.elb.amazonaws.com:8000/admin/settings" bytes=11101 category=router duration=2761257 duration_display=2.761455ms method=GET proto=HTTP/1.1 remote_ip=192.168.15.213 status_code=200
time="2023-10-27T20:02:17Z" level=info msg="http://192.168.79.148:8000/healthz" bytes=1 category=router duration=53896 duration_display="54.2µs" method=GET proto=HTTP/1.1 remote_ip=192.168.67.246 status_code=200
time="2023-10-27T20:02:22Z" level=info msg="http://192.168.79.148:8000/healthz" bytes=1 category=router duration=39216 duration_display="39.496µs" method=GET proto=HTTP/1.1 remote_ip=192.168.67.246 status_code=200

Environment (please complete the following information):

  • Zep version: 0.16.1-04d4698 (2023-10-19T18:37:41+0000)
  • Deployment: using kubernetes to AWS EKS
@twharding
Copy link
Author

I seem to have been able to fix this by looking back at previous versions of the deployment yaml and reintroducing the config volume:

      volumeMounts:
        - name: config-volume
          mountPath: /app/config.yaml
          subPath: config.yaml

was this meant to be removed?

@danielchalef danielchalef added bug Something isn't working good first issue Good for newcomers labels Oct 29, 2023
@Elsayed91
Copy link

Have same issue. Playing around with volumes and what not can get it working again but as it is, it does not work.

@danielchalef
Copy link
Member

Unfortunately, we don't have the bandwidth to debug the Kubernetes deployment. We would most certainly appreciate a contribution if you happen to fix this.

@Elsayed91
Copy link

Would a pull request with a working manifest suffice? If so I can get provide one after the holiday season.

@hipnologo
Copy link

hipnologo commented Jan 3, 2024 •
edited

For me, in the Kubernetes, it worked with the following:
volumeMounts: - mountPath: /app/config.yaml name: config-volume subPath: config.yaml

For Docker Swarm, I've done the following:
configs: - source: zep-config target: /app/config.yaml

@mrkhachaturov
Copy link

For me, in the Kubernetes, it worked with the following: volumeMounts: - mountPath: /app/config.yaml name: config-volume subPath: config.yaml

For Docker Swarm, I've done the following: configs: - source: zep-config target: /app/config.yaml

Hey @hipnologo , can you please supply your code for Docker Swarm?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working good first issue Good for newcomers
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@danielchalef @hipnologo @twharding @mrkhachaturov @Elsayed91