Every website's root directory holds two small files visitors almost never see: robots.txt and sitemap.xml. One tells search engines where not to go, the other which pages exist. Both are only a few lines long, but misconfigured, they can make an entire site invisible on Google.
What does robots.txt do?
robots.txt is a plain text file you can see by adding /robots.txt to the end of your site's address. Search engine bots read it before crawling your site and learn which sections they're asked to stay out of.
On a typical small business site the file is very short: one line addressing all bots, a few rules blocking folders that don't need crawling, like the admin panel or cart, and a line pointing to the sitemap. For example, “User-agent: *” addresses all bots, and “Disallow: /admin/” asks for the admin folder not to be crawled.
One point is often confused here. robots.txt blocks crawling but doesn't reliably block indexing. A page with links from other sites can appear in search results by its address even if it isn't crawled. If you want a page never to appear on Google, the right tool is a “noindex” tag on that page. And for the noindex tag to be read, the page must not be blocked by robots.txt.
What does a sitemap do?
A sitemap (sitemap.xml) is the list of pages on your site you want indexed. It contains each page's address and, optionally, its last updated date.
Google normally finds pages by following links. A sitemap can speed up that discovery and is especially useful for new sites, pages buried deep in a site, or frequently updated content. There's an upper limit on the number of URLs and file size in a single sitemap; small sites are unlikely to come anywhere near it, and you can find the current limit in Google's documentation.
The sitemap should list only working, canonical addresses you want indexed. Redirected addresses, pages returning 404 or pages tagged noindex shouldn't be on it. Most modern platforms generate the sitemap automatically; new pages are added to it on their own.
Once the sitemap exists, add its address to robots.txt and submit it in the Sitemaps section of Search Console.
The most common mistakes
The most dangerous are these two lines: “User-agent: *” and “Disallow: /”. This rule stops all search engines from crawling the entire site. It's often used during development to keep an unfinished site off Google. The problem is forgetting to remove it at launch. The result: weeks with almost no traffic from Google, and nobody understands why. In our post on sites that don't show up on Google, it's one of the first items on the checklist.
Other common mistakes are quieter. Blocking folders that hold CSS and JavaScript files stops Google from rendering the page correctly. Leaving old, redirected or deleted pages in the sitemap wastes Google's time. A misspelled sitemap address in robots.txt or mixed http and https versions are also frequent problems.
Should you allow AI bots?
In the past few years, robots.txt has gained a new question: should AI companies' bots be able to crawl your site?
It helps to split the bots into two groups. One group collects data for training AI models. OpenAI's GPTBot and Google-Extended, the permission Google uses for Gemini training, are examples. The other group reads your page live when answering a user's question and cites it as a source, such as OpenAI's OAI-SearchBot or PerplexityBot. According to Google, blocking Google-Extended doesn't affect your visibility in Google search results.
For most small businesses, we recommend at least allowing the search and answer bots. If you want your site cited by AI assistants, blocking them means shooting yourself in the foot. Allowing training bots is more a matter of preference; blocking them may make more sense for publishers, photographers or anyone selling educational content whose original content is the product.
Since bot names and rules change from time to time, check the current list in each company's documentation. We covered the llms.txt file proposed for introducing your site to AI tools in a separate post.
How to test it
The first test is simple: add /robots.txt and /sitemap.xml to the end of your site's address in a browser. Both should open. In robots.txt, look for a “Disallow: /” line.
Search Console's robots.txt report shows when Google last read your file and whether there were errors. The Sitemaps report shows the status of the sitemap you submitted and how many pages were discovered. To see whether a specific page is blocked, just enter its address in the URL Inspection tool.
Repeat this check after every redesign and every major update. It takes two minutes and can prevent weeks of invisible traffic loss.
Frequently Asked Questions
Does a small site need a sitemap?
It's not mandatory; Google can find a small, well-linked site by following links. But a sitemap can speed up discovery, makes it easier to track indexed pages in Search Console, and since most platforms generate it automatically, there's no reason not to have one.
Can I remove a page from Google with robots.txt?
Not reliably. robots.txt blocks crawling but doesn't guarantee removal from the index. To get a page out of search results, add a noindex tag and don't block it in robots.txt, so Google can read the tag.
Blog posts are for general information only and are not a substitute for legal, financial or technical advice.
