Can't delete empty folders

[nicolamori]

My bucket has some empty folders that cannot be removed in any way. For example:

$ mcli ls herd-storage/snapshots/documents-snapshots/data_2024-02-23 
[2024-04-16 10:02:45 CEST]     0B data_2024-02-23/
$ mcli ls herd-storage/snapshots/documents-snapshots/data_2024-02-23/
$ mcli rm herd-storage/snapshots/documents-snapshots/data_2024-02-23/
mcli: <ERROR> Failed to remove `herd-storage/snapshots/documents-snapshots/data_2024-02-23/`. Object does not exist
$ rclone purge herd-storage:snapshots/documents-snapshots/data_2024-02-23/
$ mcli ls herd-storage/snapshots/documents-snapshots/data_2024-02-23
[2024-04-16 10:03:13 CEST]     0B data_2024-02-23/

I also tried to delete via the Minio console but it didn't work. I also tried a mcli admin heal --recursive --remove which resulted in 100% green objects, and still the folder is there.

I really need help since I don't know what to try now. Thanks in advance.

[nicolamori]

Any help/idea/suggestion?

[vipurkumar]

Have you tried with ilm policy to expire ?

[nicolamori]

Hi @vipurkumar, I did some search to understand your suggestion (I'm a Minio noob), and then I did:

$ mcli ilm rule add --expire-days=10 herd-storage/snapshots
Lifecycle configuration rule added with ID `coourfhj0qvbbcn8kqrg` to herd-storage/snapshots.
$ mcli ilm rule ls herd-storage/snapshots
───────────────────────────────────────────────────────────────────┐
│ Expiration for latest version (Expiration)                                            │
├──────────────────────┬─────────┬────────┬──────┬────────────────┬─────────────────────┤
│ ID                   │ STATUS  │ PREFIX │ TAGS │ DAYS TO EXPIRE │ EXPIRE DELETEMARKER │
├──────────────────────┼─────────┼────────┼──────┼────────────────┼─────────────────────┤
│ coourfhj0qvbbcn8kqrg │ Enabled │ -      │ -    │             10 │ false               │
└──────────────────────┴─────────┴────────┴──────┴────────────────┴─────────────────────┘

The problematic folders are still there, e.g.:

$ mcli ls herd-storage/snapshots/documents-snapshots/data_2024-02-23 
[2024-05-01 09:20:54 CEST]     0B data_2024-02-23/
$ mcli rm herd-storage/snapshots/documents-snapshots/data_2024-02-23 
mcli: <ERROR> Failed to remove `herd-storage/snapshots/documents-snapshots/data_2024-02-23`. Object does not exist

So I did some more search and I found that object expiration is tied to the activity of the scanner, so I supposed it could take some time. Then I decided to monitor the scanner, but I get this error:

$ mcli admin scanner info herd-storage/snapshots
Scanner Activity: ∙●∙
mcli: <ERROR> Unable to fetch scanner metrics. Failed to parse server response (unexpected end of JSON input):.

Did what I do correspond to what you suggested? Should I worry about the scanner activity given the error message above?

[klauspost]

First of all, upgrade your minio server. We do not provide support on old releases - and this seems quite old.

Second. You can't do mcli rm on prefixes. documents-snapshots/data_2024-02-23 is a prefix and not an object. Directories does not exist - but you can create objects that look like a directory. You will need to do mcli rm -r -versions the very least, which will remove all objects with that prefix.

[nicolamori]

@klauspost Thank you very much, adding -versions to the rm command I have been able to delete the dangling prefixes.
About the cluster upgrade: I inherited it from the previous maintainer, I'm still a Minio noob, and this is a production cluster; so upgrade will have to wait until I feel comfortable in doing it, but I took good note of your suggestion.