Gateway to S3 - MalformedXML Error

[jhuynh64]

Background: We're currently using S3 as a storage for many of our files and from reading the documentation on Minio (particularly around the Gateway service), we thought we could set up Minio Gateway + Browser as a way for people to browse our S3 files, without having to create IAM roles for everyone.

So as a test environment, I tried setting Minio Gateway on Docker. I followed the instructions here: https://docs.min.io/docs/minio-gateway-for-s3.html

Here's the docker command I ran:

docker run -p 9000:9000 --name minio-s3 -e "MINIO_ROOT_USER=<redacted_aws_access_key>" -e "MINIO_ROOT_PASSWORD=<redacted_aws_secret_key>" -e "MINIO_REGION_NAME=us-west-2" minio/minio gateway s3

Unfortunately, this didn't seem to work. I get a 'malformed XML' error.

From the Browser:

From the Docker env:

API: WebListBuckets()
Time: 22:30:44 UTC 02/16/2021
DeploymentID: 0f7d1fb6-7c5e-488a-914a-e5b19cc351ae
RemoteHost: 172.17.0.1
Host: 127.0.0.1:9000
UserAgent: Mozilla/5.0 (Macintosh; Intel Mac OS X 11_1_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.141 Safari/537.36
Error: XML syntax error on line 112: invalid character entity   (*xml.SyntaxError)
       3: cmd/web-handlers.go:2428:cmd.toWebAPIError()
       2: cmd/web-handlers.go:2295:cmd.toJSONError()
       1: cmd/web-handlers.go:381:cmd.(*webAPIHandlers).ListBuckets()

Does anyone know what I'm doing wrong here? I feel like I'm missing something, but I can't find it in the documentation.

[kannappanr]

@jhuynh64 Not sure where you got the MINIO_REGION_NAME environment variable from.

Can you check if

docker run -p 9000:9000 --name minio-play -e "MINIO_ROOT_USER=Q3AM3UQ867SPQQA43P2F" -e "MINIO_ROOT_PASSWORD=zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG" minio/minio gateway s3 https://play.min.io

[jhuynh64]

I found MINIO_REGION_NAME from some old forum posts somewhere when I was looking for help. I tried with it and without it, both didn't seem to work.

I also tried adding https://s3.us-west-2.amazonaws.com/ as an endpoint as well, that didn't seem to work.

Tried docker command, but got an "Access Denied" error:

API: WebListBuckets()
Time: 17:14:30 UTC 02/18/2021
DeploymentID: d15354d8-bced-43ec-8b6a-5d19cb420386
RemoteHost: 172.17.0.1
Host: 127.0.0.1:9000
UserAgent: Mozilla/5.0 (Macintosh; Intel Mac OS X 11_1_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.150 Safari/537.36
Error: Prefix access is denied: / (cmd.PrefixAccessDenied)
       3: cmd/web-handlers.go:2428:cmd.toWebAPIError()
       2: cmd/web-handlers.go:2295:cmd.toJSONError()
       1: cmd/web-handlers.go:381:cmd.(*webAPIHandlers).ListBuckets()

[klauspost]

As somebody hinted to on slack you are likely hitting a proxy/LB that writes HTML responses.

[jhuynh64]

Hmm.. Is there a "DEBUG" mode, or a log file somewhere where I can see what Minio is getting as a response?

I'm not convinced that there is a proxy getting in the way. My computer can connect to S3 using other tools, like Cyberduck, with the same AWS access and secret keys.

[jhuynh64]

OK, after some more tinkering, it looks like it have something to do with my Docker environment/networking.

I tried installing minio (via brew) natively and that seemed to work.

Thanks again for the help, I will post back here if I've got more updates (in case it can help others).

[recklessop]

I'm seeing the same issue inside of a docker container. I however also see the error when i run the minio binary native on Ubuntu 20.04

API: WebListBuckets()
Time: 21:21:10 UTC 02/18/2021
DeploymentID: 2768fb6f-ab5a-4f3b-b843-7fe64b58c014
RemoteHost: 10.10.1.152
Host: 10.10.1.64:9000
UserAgent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4400.0 Safari/537.36 Edg/90.0.782.0
Error: XML syntax error on line 113: invalid character entity   (*xml.SyntaxError)
5: cmd/api-errors.go:2009:cmd.toAPIErrorCode()
4: cmd/api-errors.go:2034:cmd.toAPIError()
3: cmd/web-handlers.go:2429:cmd.toWebAPIError()
2: cmd/web-handlers.go:2295:cmd.toJSONError()
1: cmd/web-handlers.go:381:cmd.(*webAPIHandlers).ListBuckets()

[recklessop]

Downloaded minio to a windows 10 machine and ran it from there and got the same error in the client again

[recklessop]

also on the play.min.io endpoint i get "Access Denied" when i go to the gateway's web interface.

[recklessop]

OK I figured out what my issue was. The documentation does not say that you need to have 4 environment variables, but you do. Once I added the AWS Access key and AWS Secret Access Key variables it works fine.

I only had minio root user and root password before....

MINIO_ROOT_USER=keyhere
MINIO_ROOT_PASSWORD=secret here
AWS_ACCESS_KEY=keyhere
AWS_SECRET_ACCESS_KEY=secret here

[harshavardhana]

This is because of a bug in minio-go since we changed the ENVs we had forgotten to update the library. You don't need to set both envs.

[harshavardhana]

minio/minio-go#1455 here is the fix

[jhuynh64]

Just to post another update from me..

Our S3 buckets has a policy that only those that that are connected to our VPC (via VPN) are able to access the files. So when I tried to connect via native minio on MacOS, I was successful as I was connected through VPN.

So my initial thought was that Docker for Mac has issues with VPN, I did some digging and found some bug reports to Docker for Mac with regards to VPN connection. However, after trying a few attempts, nothing worked.

I then decided to docker exec -it minio-s3 /bin/bash into the container to debug some more. It turns out, my connection is fine between my container and VPN. I was able to successfully curl one of the files that currently is stored on S3. -- Additionally, I was able to curl another file from another file server behind our firewall.

So.. I'm back to square 1. Hope there are other ideas out there.