Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: run milvus-backup in k8s pod, retrurn "failed to connect \"Addr\": \"milvus:tcp:\" " #197

Open
jackiedlh opened this issue Sep 8, 2023 · 5 comments
Assignees

Comments

@jackiedlh
Copy link

jackiedlh commented Sep 8, 2023

Current Behavior

When I run milvus-backup command on k8s pod, the target milvus address is wrong.

in backup.yaml:

milvus:
address: milvus
port: 19530
authorizationEnabled: false

in log:

[2023/09/08 07:35:33.943 +00:00] [INFO] [logutil/logutil.go:165] ["Log directory"] [configDir=]
[2023/09/08 07:35:33.943 +00:00] [INFO] [logutil/logutil.go:166] ["Set log file to "] [path=logs/backup.log]
[2023/09/08 07:35:33.943 +00:00] [INFO] [core/backup_impl_create_backup.go:25] ["receive CreateBackupRequest"] [requestId=4b980b8b-4e1a-11ee-8398-82b39bdfe663] [backupName=backup] [collections="[]"] [databaseCollections=] [async=false]
[2023/09/08 07:35:34.168 +00:00] [WARN] [grpclog/grpclog.go:46] ["[core][Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {\n "Addr": "milvus:tcp:",\n "ServerName": "milvus:tcp:",\n "Attributes": null,\n "BalancerAttributes": null,\n "Type": 0,\n "Metadata": null\n}. Err: connection error: desc = "transport: Error while dialing dial tcp: address milvus:tcp:: too many colons in address""]

Expected Behavior

expect connect to:

milvus:19530

Steps To Reproduce

1 copy milvus-backup tool to k8s milvus proxy pod
2 copy config files to milvus pod
3 run command:  ./milvus-backup create -n backup

Environment

aws EKS

Anything else?

No response

@wayblink
Copy link
Collaborator

address should be a ip or a domain.

@Archalbc
Copy link

Archalbc commented Sep 18, 2023

address should be a ip or a domain.

"milvus" is correct in a Kubernetes context, it will resolve to the milvus-proxy pod, actually this look like a bug.

In debug mode you can have the proof that "milvus" is correctly resolved but badly interpreted by sub-libraries :

[2023/09/18 10:24:01.536 +00:00] [DEBUG] [core/backup_context.go:58] ["Start Milvus client"] [endpoint=milvus:tcp://10.224.44.247:19530]
[2023/09/18 10:24:01.536 +00:00] [WARN] [grpclog/grpclog.go:46] ["[core][Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {\n  \"Addr\": \"milvus:tcp:\",\n  \"ServerName\": \"milvus:tcp:\",\n  \"Attributes\": null,\n  \"BalancerAttributes\": null,\n  \"Type\": 0,\n  \"Metadata\": null\n}. Err: connection error: desc = \"transport: Error while dialing dial tcp: address milvus:tcp:: too many colons in address\""]

Same issue using : milvus.my-namespace.svc.cluster.local

I guess related code is : https://github.com/xushaoxiao/milvus-backup/blob/80c754eb9bfbb96961f244404ba7a3a5054d3e24/core/backup_context.go#L63C22-L63C50
and
https://github.com/milvus-io/milvus-sdk-go/blob/master/client/client.go

The used func is deprecated as described in comment : https://github.com/milvus-io/milvus-sdk-go/blob/master/client/client.go#L285

No issue with v0.3.2

[2023/09/18 13:01:26.991 +00:00] [DEBUG] [core/backup_context.go:58] ["Start Milvus client"] [endpoint=milvus:443]     

@nightmorning0
Copy link

nightmorning0 commented Sep 29, 2023

It is definitely a bug. Add PORT# at the tail of the address. Something likes this:

milvus:
address: milvus:19530#
port: 19530
authorizationEnabled: false

It will comment :tcp: from milvus:tcp:: in your error as a tricky solution.

@adeelamin15
Copy link

Yup it is definitely a bug. For me workaround is to set the port using MILVUS_PORT environment variable. Apparently the bug is causing garbage value in port field which can be overridden using environment variable.

@wayblink
Copy link
Collaborator

wayblink commented May 9, 2024

/assign

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants