feat: add agent-friendly 404 page and llms.txt

[RobbieTheWagner]

Part 6 of 6 of the agent-readiness stack (stacked on #3495).

• Custom 404.astro in the site's design with recovery links (docs, pricing, blog, llms.txt, sitemap, contact). The Vercel build output already routes unmatched paths to it with a real HTTP 404 status.
• llms.txt with an explicit 'When to use Shepherd' section naming best-fit jobs (onboarding walkthroughs, feature announcements, complex-form guidance, in-app training), install/API guidance, what Shepherd is not for, and links to docs and trust pages.

Test plan: e2e tests assert nonexistent paths return HTTP 404 with recovery links and that /llms.txt serves the when-to-use guidance; build-output tests assert 404.html and llms.txt are emitted — full suite of 32 tests passing on this branch.

After the stack lands, live verification:

curl -s -o /dev/null -w "%{http_code}" https://www.shepherdjs.dev/some-path-that-does-not-exist
curl -sI -H "Accept: text/markdown" https://www.shepherdjs.dev/ | grep -iE "content-type|vary"

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
shepherd-docs	Ready	Preview	Aug 25, 2026 3:07am
shepherd-landing	Ready	Preview	Aug 25, 2026 3:07am

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: faa79820-c9fa-4ce8-b97c-e875586c3e09

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[qltysh]

Coverage Impact

This PR will not change total coverage.

🚦 See full report on Qlty Cloud »

🛟 Help

• Diff Coverage: Coverage for added or modified lines of code (excludes deleted files). Learn more.

• Total Coverage: Coverage for the whole repository, calculated as the sum of all File Coverage. Learn more.

• File Coverage: Covered Lines divided by Covered Lines plus Missed Lines. (Excludes non-executable lines including blank lines and comments.)

  • Indirect Changes: Changes to File Coverage for files that were not modified in this PR. Learn more.

[RobbieTheWagner]

Closing in favor of a fresh PR — this one is stuck in the broken stack state left behind when #3494 was declined (GitHub refuses to recompute its merge state or edit its base). The branch is unchanged; the replacement PR has the identical single commit.