Simple robots.txt template. Keep unwanted robots out (disallow). White lists (allow) legitimate user-agents. Useful for all websites.
search-engine
whitelist
user-agent
seo
crawling
twitterbot
robots-txt
googlebot
crawlers
web-crawling
bingbot
robots-exclusion-standard
blocking-bots
web-robots
search-engine-optimization
baiduspider
-
Updated
Feb 18, 2024