Why sitemaps and robots.txt matter

XML Sitemap: the rules

Include only canonical, indexable URLs. Your sitemap should be a list of the pages you want Google to find and index. Do not include:

• Pages with noindex tags
• 301 redirected URLs (include the destination, not the source)
• Pages that have canonical tags pointing elsewhere
• Admin, login, and functional pages
• Paginated pages (include only the first page of a series)

Keep the sitemap current. A sitemap that still lists pages you deleted three months ago is a quality signal problem. Most CMSs update sitemaps automatically. For static sites, regenerate the sitemap whenever pages are added or removed.

Specify accurately. The date tells Google when the page was last meaningfully updated. Do not set it to today's date for every page this is a manipulation signal Google ignores and discounts. Set it to the actual last-modified date.

Submit the sitemap in Google Search Console. Go to Search Console ? Sitemaps ? enter your sitemap URL ? Submit. Monitor the reported indexed count against your expected count.

Sitemap size limits: Each sitemap file can contain a maximum of 50,000 URLs and be a maximum of 50MB (uncompressed). For sites exceeding this, create a sitemap index file that references multiple sitemap files.

Example of a correct sitemap structure:

`xml https://www.perceptra.in/services/web-design/ 2026-01-15 monthly 0.8 https://www.perceptra.in/blog/technical-seo-and-core-web-vitals/ 2026-03-10 monthly 0.7 `

Robots.txt: the rules

Robots.txt controls what Google crawls not what Google indexes. A page blocked by robots.txt can still be indexed if Google finds a link to it. A robots.txt block does not remove a page from Google's index it just stops Googlebot from reading it. To prevent indexing, use a noindex tag.

The robots.txt file lives at yourdomain.com/robots.txt. There is only one robots.txt file per domain.

What to block in robots.txt:

• Admin and backend URLs: Disallow: /wp-admin/ (allow /wp-admin/admin-ajax.php for WordPress functionality)
• Duplicate or low-value URL patterns: Disallow: /*?sort= (blocks filter parameter combinations)
• Search result pages: Disallow: /search/
• Staging or development directories: Disallow: /staging/

What NOT to block in robots.txt:

• Your CSS and JavaScript files Google needs these to render your pages correctly
• Your images Google Images is a real traffic source
• Any page you want indexed, even if it has thin content use noindex for that, not robots.txt

Example of a correct robots.txt for a WordPress site:

` User-agent: Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Disallow: /?s= Disallow: /cart/ Disallow: /checkout/

Sitemap: https://www.perceptra.in/sitemap.xml `

Frequently asked questions

Should we block CSS and JavaScript in robots.txt? Never. Blocking CSS and JS prevents Google from rendering your pages correctly it sees bare, unstyled HTML and may misinterpret your page structure and content.

Does the sitemap guarantee pages get indexed? No. A sitemap tells Google about pages; it does not force Google to index them. Pages with thin content, duplicate content, or low authority may be crawled from the sitemap but not indexed. The sitemap is a discovery tool, not an indexation guarantee.

How often does Google check robots.txt? Googlebot fetches robots.txt before every crawl session. Changes to robots.txt take effect quickly typically within 24 48 hours for most pages. Use Google Search Console's robots.txt tester to verify your rules are working as expected.