# robots.txt for ibmathrevision.com # # Private/app pages (admin, dashboards, account, class, trial flows…) are # kept out of Google with — NOT with # Disallow. A Disallow stops Google from fetching the page, so it never sees # the noindex and can still index the bare URL from links ("Indexed, though # blocked by robots.txt"). CSS/JS must stay crawlable for rendering. # # alevelmathrevision.com has its own robots.txt (alevel-frontend/robots.txt) # — a Sitemap: line here for another host is ignored by Google. # # /marketing/*.pdf? : the one-pager PDF is linked with a cache-busting ?v= # that changes on every rebuild, so each version was a new URL to crawl # (Search Console "Duplicate without user-selected canonical", Oct 2026). # Only the ?query forms are blocked; the PDF itself stays crawlable. User-agent: * Allow: / Disallow: /dropbox/ Disallow: /marketing/*.pdf? # AI assistants and AI search (Oct 2026). We WANT ChatGPT, Gemini / Google # AI Overviews, Perplexity, Copilot (Bing), Claude and Apple Intelligence to # read and cite this site, so their crawlers get an explicit Allow below. # (A crawler follows only the most specific group that names it, so this # group repeats the rules of the * group above; keep the two in step.) # A summary of the site for them: https://ibmathrevision.com/llms.txt User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: GPTBot User-agent: Google-Extended User-agent: PerplexityBot User-agent: Perplexity-User User-agent: ClaudeBot User-agent: Claude-SearchBot User-agent: Applebot-Extended User-agent: Bingbot User-agent: CCBot Allow: / Disallow: /dropbox/ Disallow: /marketing/*.pdf? Sitemap: https://ibmathrevision.com/sitemap.xml Sitemap: https://ibmathrevision.com/image-sitemap.xml