SiteSecura Logo
SiteSecura.
Crawlability & Technical SEO Utility

Robots.txt & Sitemap Checker

Inspect your website's robots.txt against the official RFC 9309 Robots Exclusion Protocol, test crawler access with an interactive path simulator, and validate XML Sitemaps for search engine compliance.

RFC 9309 ProtocolXML Sitemaps.orgInteractive Path SimulatorSSRF-Protected
SSRF Protected • RFC 9309
Examples:
RFC 9309 Compliant Parser
XML Sitemaps.org Protocol
SSRF-Safe Origin Defense
Internet Standard

What is robots.txt & RFC 9309?

The Robots Exclusion Protocol originated in 1994 as an informal consensus among early web crawler developers. For nearly three decades, different search engines implemented slightly divergent interpretations of pattern matching, wildcards, and rule precedence.

In September 2022, the Internet Engineering Task Force (IETF) published RFC 9309, codifying robots.txt into a formal Proposed Standard. RFC 9309 establishes deterministic rules for:

User-Agent Grouping

Directives are grouped under one or more User-agent: lines. Crawlers must match the most specific group matching their product token, or fall back to the wildcard * group.

Longest Match Precedence

When multiple Allow and Disallow rules match a URL path, the rule with the longest matching pattern length in octets takes precedence.

Equal Specificity Allow Precedence

If an Allow: and Disallow: rule match a path with equal pattern length, the Allow directive wins under RFC 9309 §2.2.2.

Empty Disallow Means Allow All

An empty Disallow: line clears all crawl restrictions, explicitly declaring the entire origin open to crawling.

Discovery Architecture

What is an XML Sitemap?

While robots.txt defines boundaries on what crawlers may not fetch, an XML sitemap serves the opposite purpose: it acts as a structured catalog telling search engines which canonical URLs exist, when they were last updated, and how they should be prioritized.

Standard URL Set<urlset>

The root XML tag for standard sitemaps containing up to 50,000 individual <url> nodes with <loc>, <lastmod>, <changefreq>, and <priority>.

Up to 50,000 URLs
Sitemap Index<sitemapindex>

The parent catalog tag for websites with more than 50,000 URLs or multiple sections. Lists child sitemaps inside <sitemap> elements.

Parent Catalog
Host Location Requirements

Where Should Files Be Located?

Robots.txt Location (Strict RFC 9309 Rule)Origin Root Only
https://<your-domain>/robots.txt

robots.txt MUST be hosted at the root path of the target origin. Subdirectory placements (e.g., /blog/robots.txt) are ignored by compliant crawlers.

XML Sitemap Location & DeclarationConventional & Declared
Sitemap: https://<your-domain>/sitemap.xml

Conventionally placed at /sitemap.xml, but may reside anywhere within the same host origin as long as it is explicitly declared inside robots.txt.

Configuration Diagnostics

Common Failures & Syntax Errors

Automated validation regularly uncovers the following critical configuration mistakes:

Missing User-agent Declaration

ROBOTS_SYNTAX_ERROR

Allow or Disallow directives published without an preceding User-agent line are invalid under RFC 9309 and will be discarded by crawlers.

Fix: Always group crawl rules under explicit 'User-agent: *' or specific bot token lines.

UTF-8 BOM Characters

ROBOTS_BOM_PRESENT

Text editors sometimes save robots.txt with a Byte Order Mark (\uFEFF), which can cause older parsers to misread the first User-agent directive.

Fix: Save robots.txt in plain UTF-8 without BOM encoding.

Disallowing URLs with 'noindex'

ROBOTS_BLOCKS_NOINDEX

If a page is blocked in robots.txt, crawlers cannot fetch it to discover the meta noindex tag, causing the URL to remain indexed via external links.

Fix: Remove the Disallow rule and place '<meta name="robots" content="noindex">' in the HTML <head>.

Exceeding Sitemap Limits (50k URLs / 50 MB)

SITEMAP_SIZE_LIMIT_EXCEEDED

A single XML sitemap containing more than 50,000 URLs or exceeding 50 MB uncompressed violates the official Sitemaps.org specification.

Fix: Partition URLs into multiple child sitemaps and link them using a Sitemap Index document (<sitemapindex>).

Invalid ISO 8601 <lastmod> Timestamps

SITEMAP_INVALID_LASTMOD

Using non-standard date formats (e.g. MM/DD/YYYY) instead of W3C Datetime format (YYYY-MM-DD or YYYY-MM-DDTHH:MM:SSZ).

Fix: Format all <lastmod> entries following ISO 8601 / W3C specifications.

Cross-Origin Sitemaps on External Hosts

SITEMAP_EXTERNAL_ORIGIN

Declaring sitemap URLs located on an external domain without cross-submission proof. These are skipped under safe-origin security policies.

Fix: Host sitemaps on the same domain and protocol as the website.
Security Myth Debunked

Does robots.txt Protect Private Pages?

robots.txt is NOT an Access Control Boundary

A common and dangerous mistake is using robots.txt to hide sensitive endpoints such as /admin/, /staging/, or /internal/.

robots.txt is a publicly accessible file. Malicious actors, scrapers, and penetration testers regularly inspect robots.txt as part of initial reconnaissance to discover confidential or unlinked endpoints.

Best Practice: Secure sensitive endpoints using proper authentication, multi-factor authorization, IP allowlists, or firewalls. Never rely on robots.txt for security or privacy.
Frequently Asked Questions

Robots.txt & Sitemap FAQs

Detailed technical guidance on RFC 9309 parsing, sitemap validation, search engine indexing, and safe crawler configuration.

robots.txt is a plain-text file placed in the root directory of a web origin to instruct web crawlers and search engine spiders which parts of the site they may or may not access. In September 2022, the Internet Engineering Task Force (IETF) published RFC 9309 ('Robots Exclusion Protocol'), formally standardizing crawler directives, user-agent grouping, pattern matching, precedence rules, and parsing semantics after 28 years of informal conventions.
SiteSecura Utilities

Explore More Free Security & Web Tools

Inspect other facets of your website’s security, transport integrity, and domain posture:

security.txt Validator

Validate RFC 9116 security disclosure files, Contact directives, and PGP encryption keys.

HTTP Status & Redirect Checker

Inspect HTTP status codes, trace redirect chains, and detect SSL downgrade loops.

DNS Lookup Tool

Query forward DNS records (A, AAAA, MX, NS, TXT, SOA, CAA) with TTL metrics.

SSL/TLS Certificate Checker

Inspect TLS certificate validity, issuer trust chains, expiration dates, and cipher suites.

Security Headers Checker

Audit CSP, HSTS, X-Frame-Options, Permissions-Policy, and CORS configurations.

Website Security Checker

Launch a comprehensive perimeter scan assessing vulnerabilities, SSL, and DNS posture.

Automated Perimeter Defense

Ready for a Complete Security Assessment?

Robots.txt and sitemaps define crawlability, but what about SSL misconfigurations, open ports, missing HTTP security headers, and domain spoofing risks?

Launch Free Full Security ScanBrowse All Resources