chore(guide): add crawl policy [Codex]

[jderochervlk]

The guide needs an explicit crawler policy before launch, but its production domain and sitemap location are not decided yet.

Add an app-local static robots file that permits indexing of all guide routes. Vite copies it into the client build, and the guide's React Router build hook copies that directory into Wrangler's deployed asset directory. The policy deliberately contains no host-specific or sitemap directive.

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-13T14:41:49.055224Z	3a9ac27	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[github-actions]

Guide Cloudflare deployment

⛅️ wrangler 4.114.0
────────────────────
🌀 Building list of assets...
✨ Read 11 files from the assets directory /home/runner/work/rescript-lang.org/rescript-lang.org/apps/guide/out
🌀 Starting asset upload...
🌀 Found 1 new or modified static asset to upload. Proceeding with upload...

• /robots.txt
  Uploaded 1 of 1 asset
  ✨ Success! Uploaded 1 file (9 already uploaded) (0.63 sec)

Total Upload: 0.31 KiB / gzip: 0.22 KiB
Worker Startup Time: 0 ms
Uploaded guide (1.97 sec)
Worker Version ID: 5124d383-5967-47a1-86a7-d5d83f2e26bf
Version Preview URL: https://5124d383-guide.rescript.workers.dev
Version Preview Alias URL: https://chore-guide-robots-guide.rescript.workers.dev

To deploy this version to production traffic use the command wrangler versions deploy

Changes to non-versioned settings (config properties 'logpush' or 'tail_consumers') take effect after your next deployment using the command wrangler versions deploy

Changes to triggers (routes, custom domains, cron schedules, etc) must be applied with the command wrangler triggers deploy

[cloudflare-workers-and-pages]

Deploying with    Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status	Name	Latest Commit	Preview URL	Updated (UTC)
✅ Deployment successful! View logs	guide	3a9ac27	Commit Preview URL Branch Preview URL	Sep 13 2026, 02:40 PM

[github-actions]

Cloudflare deployment

Deployement ID: b595fda1-7434-4be3-ae0e-4b0e3cc437b4
Deployment Environment: preview

⛅️ wrangler 4.114.0
────────────────────
✨ Compiled Worker successfully
Uploading... (8289/8770)
Uploading... (8449/8770)
Uploading... (8609/8770)
Uploading... (8770/8770)
✨ Success! Uploaded 481 files (8289 already uploaded) (3.12 sec)

✨ Uploading _redirects
✨ Uploading Functions bundle
🌎 Deploying...
✨ Deployment complete! Take a peek over at https://b595fda1.rescript-lang.pages.dev
✨ Deployment alias URL: https://chore-guide-robots.rescript-lang.pages.dev