-
-
Notifications
You must be signed in to change notification settings - Fork 95
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[FEAT]: robots.txt per routes #228
Labels
enhancement
New feature or request
Comments
Update: I turned on the “Allow search engines to index your server” feature on Cosmos and cleared my cache in Cloudflare and the correct robots.txt appeared. I turned off that checkmark and cleared my cache again, and the disallow showed back up. This is really a feature that should be by domain, because there are some sites I want accessible and others I don’t. |
I will probably add a per domain checkbox yes |
Good call—thank you! |
azukaar
added
enhancement
New feature or request
and removed
bug
Something isn't working
labels
Apr 15, 2024
azukaar
changed the title
[BUG]: Issue with robots.txt files
[BUG]: robots.txt per routes
Apr 15, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
What happened?
I’m attempting to set a robots.txt on a Ghost server I have running on my domain, but nothing is working, and I can’t figure out why. No matter what I do, it remains
I would like this server to be accessible to search engines, but I cannot set it. I see a recent change to robots.txt was made to Cosmos that produces the same output and want to confirm this is not the cause.
What should have happened?
When I uploaded a robots.txt file to my Ghost theme, it should have loaded, it did not. It may be related to Cosmos, not Ghost.
How to reproduce the bug?
disallow
messageRelevant log output
No response
Other details
I see that a recent feature was added to support robots.txt to prevent the Cosmos server from being visible on search engines, but even after clearing the site’s cache and removing cookies, the old file is still there.
System details
Client:
Server:
The text was updated successfully, but these errors were encountered: