robots.txt: The Complete Practical Guide
Small file, large consequences. How to configure it without accidentally deindexing your site.
Technical SEO
What it does and does not do
robots.txt controls crawling, not indexing. A blocked URL can still appear in results if other sites link to it, showing without a description. To prevent indexing, use a noindex meta tag on a crawlable page.
This distinction causes more accidental deindexing than any other technical misunderstanding.
What to block
Admin areas, internal search results, cart and checkout URLs, filter parameter combinations that generate near-duplicates, and staging environments. Never block CSS or JavaScript files, which prevents proper rendering.
Testing before deploying
Use the robots.txt tester in Search Console. Verify your most important URLs are crawlable. A single misplaced wildcard can block an entire site, and it happens regularly during redesigns.
Key takeaways
- robots.txt controls crawling, not indexing
- Use noindex meta tags to prevent indexing, not robots.txt
- Never block CSS or JavaScript files
- Test before deploying; one wildcard can block everything
Want this handled for you?
We deliver this work for 500+ clients across 36 industries. Month-to-month, published pricing, no contracts.
Related Reading
Technical SEO
Core Web Vitals in 2026: What Actually Matters
LCP, INP, and CLS explained without the jargon, plus the fixes that produce the biggest improvements....
Technical SEO
JavaScript SEO: Making React and Next.js Sites Rank
How Google actually renders JavaScript, where it breaks, and how to structure modern frameworks so content get...
Technical SEO
Schema Markup: Which Types Actually Earn Results
There are hundreds of schema types. These are the ones that produce visible outcomes....