SEO workflow
How to write robots.txt for a small website
Create a simple robots.txt file for crawl guidance while avoiding common privacy and indexing mistakes.
Start with crawl guidance, not secrecy
Robots.txt is for crawler instructions. It is not a password, privacy control, or reliable way to remove a public URL from search results.
For a small website, the safest starting point is often simple: allow normal pages, block obvious internal paths only when needed, and point to the sitemap.
- Decide what crawlers can access.
- Add the sitemap URL.
- Use noindex on public pages that should not be indexed.
Simple workflow
Generate a starter robots.txt file, review every disallow rule, add the sitemap URL, then check page-level robots tags for pages that need noindex.
Do
- Keep rules readable.
- Use full sitemap URLs.
- Review after redesigns or migrations.
Don't
- Do not block important assets by accident.
- Do not put sensitive URLs in robots.txt as a privacy measure.
- Do not confuse disallow with noindex.
FAQs
Can robots.txt hide private content?
No. Use access controls for private content. Robots.txt can reveal paths and does not stop users from opening public URLs.
Should every robots.txt include a sitemap line?
It is often useful for small sites because it gives crawlers a clear sitemap location.