# robots.txt for bryanmanio.com — https://www.rfc-editor.org/rfc/rfc9309
#
# The companion to the agent discovery in public/_headers and /llms.txt: those
# two say "here is the machine-readable index of this site", and this file says
# which parts of it are worth fetching at all.
#
# This is a personal blog. Everything on it is meant to be read, by people and
# by machines alike, so it's an allow-list: the only paths closed off hold no
# readable text — build artefacts, ciphertext, and the JSON behind the live
# widgets.
#
#   /pagefind/          Pagefind's search index fragments and vendor JS/CSS.
#                       Hundreds of files, a machine-readable duplicate of
#                       every page's text, and not one of them a page.
#   /protected/         The Plex post's encrypted payload AND its images, which
#                       sit there unencrypted. A soft gate, but no reason to
#                       put the images in an index.
#   /.netlify/functions/ The widget/bookshelf/bookmarks/reviews endpoints. They
#                       feed this site's own pages and several spend a
#                       rate-limited third-party token per call, which is the
#                       same reason _headers advertises no rel="api-catalog".
#   /search             The search page is an empty shell; Pagefind renders the
#                       results client-side, so there is nothing in the HTML.
#                       Written WITHOUT the trailing slash on purpose — it's a
#                       prefix, so it covers both /search (which trailingSlash:
#                       "always" 301s) and the canonical /search/, in one rule.
#                       astro.config.mjs filters that same page out of the
#                       sitemap, since a sitemap listing a roboted-out URL
#                       contradicts itself; its filter matches the EXACT
#                       pathname "/search/", because a substring test would
#                       also drop the post "simhub-stuck-on-searching-for-
#                       device". Add a Disallow here, check that filter too.
#
# Styles, scripts, fonts and images stay open so anything that renders the page
# sees it the way a reader does — block /_astro/ and a renderer gets an
# unstyled page. They're spelled out as Allow rules rather than left to the
# default so a future tightening here has to step over them deliberately.
#
# TWO THINGS TO KNOW BEFORE EDITING:
#
# 1. A crawler obeys exactly ONE group — the most specific User-agent match —
#    and groups do NOT inherit. So the named block below repeats the rules
#    rather than extending them. Change one, change both.
# 2. Rule order is load-bearing for older crawlers. Modern parsers take the
#    longest match, but the original spec is first-match-wins, so the broad
#    "Allow: /" has to come LAST or it would swallow every Disallow above it.
#
# THE SITEMAP LINE POINTS AT /sitemap.xml, WHICH IS NOT THE FILE THE BUILD
# WRITES. @astrojs/sitemap only ever emits an index (`sitemap-index.xml` plus
# `sitemap-0.xml`), but /sitemap.xml is the path crawlers probe unprompted, so
# public/_redirects rewrites it with a 200. Keep those two in step: this line is
# only true because that rewrite exists, and `astro preview` doesn't process
# _redirects, so verify it under `netlify dev`.
#
# It's a full absolute URL because the spec requires one — unlike every path
# below it, Sitemap: is not relative to this file. Change it with the domain.
# And it sits OUTSIDE both groups: Sitemap: belongs to no User-agent group, it
# applies to every crawler wherever it appears.

Sitemap: https://bryanmanio.com/sitemap.xml

User-agent: *
Disallow: /pagefind/
Disallow: /protected/
Disallow: /.netlify/functions/
Disallow: /search
Allow: /_astro/
Allow: /fonts/
Allow: /icons/
Allow: /images/
Allow: /llms.txt
Allow: /rss.xml
Allow: /

# AI and agent crawlers, named explicitly and allowed on the same terms as
# everyone else. This is the deliberate position of a site that ships
# /llms.txt, RFC 8288 Link headers and a DNS-AID record: an agent arriving here
# is a reader, and there's a map at the door for it.
#
# It also matches the Cloudflare side, which is the half of this decision that
# isn't in the repo — the zone's "Block AI Scrapers and Crawlers" WAF rule is
# disabled, and nothing in bot management is blocking. See CLAUDE.md for the
# verified state and the one setting that still disagrees with this file.
#
# Google-Extended and Applebot-Extended are not crawlers — they're training
# opt-out tokens that only read Disallow — so listing them with Allow: / is
# simply not taking the opt-out.
#
# To change your mind for all of them, replace this group's rules with a single
# "Disallow: /". That only binds the well-behaved ones; the enforcement for the
# rest is the Cloudflare rule above, which is a toggle in the dashboard.
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-User
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: meta-externalagent
User-agent: Amazonbot
User-agent: Bytespider
Disallow: /pagefind/
Disallow: /protected/
Disallow: /.netlify/functions/
Disallow: /search
Allow: /_astro/
Allow: /fonts/
Allow: /icons/
Allow: /images/
Allow: /llms.txt
Allow: /rss.xml
Allow: /