SPIOUNby 6S

Sitemaps and robots.txt

The two files that tell Google what to read and what to leave alone.

Issues on this page

Sitemap with errorsSitemap

What
Google reports errors in this sitemap.
Why
Pages listed after an error may be skipped.
How
Open the sitemap report in Search Console to see the lines. Usually a wrong address or a file that is not XML. Steps for your platform →

Sitemap with warningsSitemap

What
Google reports warnings in this sitemap.
Why
Not blocking, but usually addresses that redirect or are blocked.
How
Open the sitemap report in Search Console and fix the listed addresses. Steps for your platform →

Sitemap pendingSitemap

What
Google has not processed this sitemap yet.
Why
Normal just after submitting.
How
Wait a day. Do not resubmit while it is pending.

Sitemap never read by GoogleSitemap

What
Google has never read this sitemap.
Why
New pages reach Google late or not at all.
How
Check the sitemap address opens in a browser, then press Resubmit. Steps for your platform →

Sitemap not read recentlySitemap

What
Google last read this sitemap more than three weeks ago.
Why
Recent pages may not be known to Google yet.
How
Press Resubmit if you added pages since. Steps for your platform →

robots.txt blocks the whole siterobots.txt

What
robots.txt forbids crawlers from the entire site (Disallow: /).
Why
Google cannot read any page. Pages start dropping out of Google's results.
How
Remove the Disallow: / line. Site builders have a setting for this; in WordPress it is "Discourage search engines" under Settings → Reading. Steps for your platform →

robots.txt answers with an errorrobots.txt

What
robots.txt answers with an error.
Why
When robots.txt fails with a server error, Google may stop crawling the site to be safe.
How
Make sure /robots.txt either exists or answers 404. Ask your hosting if it answers 5xx. Steps for your platform →

robots.txt could not be fetchedrobots.txt

What
SPIOUN could not fetch robots.txt.
Why
Google may have the same trouble.
How
Open yoursite/robots.txt in a browser. If it fails, check hosting and DNS. Steps for your platform →

No robots.txtrobots.txt

What
The site has no robots.txt.
Why
Nothing is blocked; crawlers may visit everything. That is fine.
How
Nothing to do. Optionally add one with a Sitemap: line.

robots.txt without a Sitemap linerobots.txt

What
robots.txt does not mention your sitemap.
Why
Optional. Google finds the sitemap through Search Console anyway.
How
If you edit robots.txt anyway, add a line: Sitemap: https://yoursite/sitemap.xml Steps for your platform →

Words used here

Sitemap
A file, usually at /sitemap.xml, that lists the pages you want Google to know about. Site builders create it for you. You submit it once in Search Console. Glossary
robots.txt
A small text file at the root of your site that tells crawlers which parts they may visit. One wrong line (Disallow: /) can hide the whole site from Google. Glossary