How matching actually works
Crawlers do not read robots.txt like a firewall list. For a given URL, they find their group (their own user-agent section, else the * group), test every rule's pattern against the path — where * matches any characters and a trailing $ anchors the end — and the longest pattern decides. That is why "Disallow: /admin/" plus "Allow: /admin/public" blocks the admin area but not the public subfolder. Test the URLs you care about most after every robots.txt change: one stray "Disallow: /" line (left from a staging site) quietly removes a whole site from search.
Frequently Asked Questions
The tester says allowed, but Google does not crawl the page.
Robots permission is only the first gate — crawl budget, server errors, noindex tags and thin content also stop crawling. Allowed simply means robots.txt is not the blocker.
Related Tools
SEO Analyzer — Analyze any page's on-page SEO — title, meta, headings, images, links.
Open Graph Checker — See how your link will look when shared on Facebook, X and chat apps.
Redirect Checker — Trace a URL's redirect chain — every hop and status code.
Meta Tag Generator — Generate title, description, Open Graph and Twitter tags in one go.
SERP Preview Tool — Preview your Google result — desktop and mobile — before you publish.
Robots.txt Generator — Build a robots.txt — user-agent, allow/disallow paths, sitemap line.
XML Sitemap Generator — Turn a list of URLs into a valid sitemap.xml — download included.
Keyword Density Checker — Count single words and 2–3 word phrases in any text, with percentages.
UTM Builder — Build campaign URLs with UTM parameters for clean analytics.
URL Slug Generator — Turn any headline into a clean, SEO-friendly URL slug.
Schema Markup Generator — Generate JSON-LD schema — Article, Product, FAQ, LocalBusiness, WebSite.
Hreflang Tag Generator — Generate hreflang link tags for multi-language and regional pages.
YouTube Keyword Tool — Get live YouTube autocomplete suggestions for any seed keyword.