diff --git a/.github/workflows/pr.yml b/.github/workflows/pr.yml index 7296673a8..2abb9d115 100644 --- a/.github/workflows/pr.yml +++ b/.github/workflows/pr.yml @@ -54,7 +54,7 @@ jobs: - run: make ci-check-hardcoded-strings - run: make ci-check-ci-parity - run: make ci-check-locales - # Backticked repo paths in docs/, CLAUDE.md, README.md and skills must exist (#519). + # Backticked repo paths in docs/, AGENTS.md, CLAUDE.md, README.md and skills must exist (#519). - run: make ci-doc-paths # Fixed sleeps in tests are timing bets; poll with simple_module_test.wait_for. - run: make ci-check-test-sleeps diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 000000000..86bf1f3ee --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,513 @@ +# AGENTS.md + +Guidance for coding agents (Claude Code, Codex, Cursor, OpenCode, …) working in this +repository. It holds what changes how code gets written here; history and rationale live +in `docs/`. Claude Code reads it through `CLAUDE.md`, which adds a few Claude-only rules. + +## What this is + +A **modular-monolith framework**: feature modules are Python packages loaded into +one FastAPI app, each with its own SQLModel tables, REST + Inertia endpoints and +React pages. There is no host–module API boundary. + +Stack: Python 3.12 + FastAPI + SQLModel + Alembic; Inertia.js v3 + React + +Tailwind 4 + Vite. `uv` workspaces for Python, `npm` workspaces for JS. +Published as ~23 PyPI packages (`simple_module_*`) and 3 npm packages +(`@simple-module-py/*`) in lockstep (`docs/release.md`). + +Layout: `framework/{core,db,hosting,inertia,testing,cli}` (the framework), +`modules/` (plugin modules), `host/` (the dev host app and all Alembic +migrations), `packages/{ui,i18n,tsconfig}` (shared JS), `skills/` (agent skills +shipped to users via `smpy skills add`), `.claude/skills/` (skills for this repo). + +## Commands + +Everything goes through `make`; the root `Makefile` is the full list +(`docs/reference/make-commands.md` documents the *scaffolded app's* Makefile, not this one). + +| Command | Purpose | +|---|---| +| `make install` | `uv sync --all-packages` + `npm install` | +| `make dev` | Shared Docker stack up (needs the sibling `../dev-services` checkout) + `gen-pages` + API (:8000) and Vite (:5050). Without that stack, run `make dev-api` and `make dev-ui` | +| `make test` / `make test-py` / `make test-js` | Pytest (SQLite, e2e/perf excluded; parallel via `pytest-xdist`, `-n $(PYTEST_WORKERS) --dist loadgroup`, default 4 workers, `PYTEST_WORKERS=0` for serial) then Vitest | +| `make test-py-pg` | Pytest on Postgres (`SM_TEST_PG_URL`, default db `sm_test`); parallel like `test-py`, each xdist worker creates its own `sm_test_gw` database (role needs `CREATEDB`); not in `pr-checks` | +| `make test-e2e` | Playwright smoke tests against a running `make dev` | +| `make lint` | Every CI lint check: Ruff format + Ruff + `ty` + Biome + per-workspace `tsc`, package metadata/README, locale files, doc paths, i18n and API-type freshness, file-size cap, no test sleeps, no double casts, no raw `fetch`, hardcoded strings, CI parity, untranslated strings | +| `make doctor` | Module diagnostics (`SM0xx`) | +| `make migrate` / `make downgrade` / `make migration-history` | Alembic `upgrade heads` / `-1` / history | +| `make migration LABEL= msg="..."` | Autogenerate on the selected branch (`--head @head`); `LABEL` is required | +| `make migrations-roundtrip-pg` | Postgres portability gate (upgrade → downgrade base → upgrade → `alembic check`); CI runs it | +| `make gen-pages` | Regenerate `host/client_app/modules.{manifest.json,generated.ts,generated.css}` (gitignored; needed before `tsc`/Vite) | +| `make gen-i18n` | Regenerate `packages/i18n/src/{keys.generated,generated-resources}.ts` (committed; CI diffs them) | +| `make gen-api-types` | Regenerate each module's `pages/api.generated.ts` from its routes (committed; CI diffs via `ci-api-types-fresh`) | +| `make new-module name=` | Scaffold a module package (see § Adding a module for what it does *not* do) | +| `make openapi` | Export `openapi.json` for typed clients | +| `make worker` / `make beat` | Celery worker / beat for `background_tasks` | + +Single tests: `uv run pytest path/to/test_file.py::test_name` from the repo root +(`asyncio_mode=auto`, `-m 'not e2e and not perf'` are set in `pyproject.toml`; +`-n` is passed by `make test-py`, not by `addopts`, so this runs in-process); +`npx vitest run ` for JS. Entry point: `host/main.py` +(`uv run --project host uvicorn host.main:app --reload`). Alembic runs from the +repo root via `host/alembic.ini`, so it shares `.env`/`SM_DATABASE_URL` with the API. + +### Local environment + +- **No env var is required.** The default DB is SQLite (`./app.db`), so tests and + a plain local run need no services. `make dev` starts the author's *shared* + dev-services stack from `../dev-services/compose.yml` (Postgres on + `localhost:5432`, db `simple_module_python`; Redis on `localhost:6379`, one + `SM_REDIS_URL=redis://localhost:6379/4` seeds broker and backend). Never start + your own Postgres/Redis containers; `make docker-down` only stops this repo's + worker/beat. +- **Sandboxed sessions have no Docker.** Launch API + Vite directly per + `.claude/skills/running-the-stack/SKILL.md` (spare ports, SQLite file DB, + `smpy users create-admin`). `make kill` runs `pkill -f vite` and kills other + projects' servers, so kill by port instead. +- If a repo-root `.env` exists, read it (don't copy values) and move it aside + for `make test-py`, then restore it: the `settings` fixture does not pin + `vite_dev_url`, so a `.env` value leaks into a dashboard test that asserts + Vite on `:5050`. Never export `SM_ENVIRONMENT=production` in a shell that + runs pytest: production validators (localhost broker, placeholder secrets) + abort the test plugin. +- The invalidation tests need `redis-server` on `PATH` or `SM_TEST_REDIS_URL`; + without either they skip locally and **fail** when `CI` is set. + +## Before you push + +Run what CI runs, in this order; a red push costs a review cycle. + +1. `make gen-pages` once per fresh checkout (generated frontend files are + gitignored; `tsc` and Vite fail without them). +2. `make lint`. `ruff format --check .` also formats Python fences inside + `.md` files, so run `uv run ruff format docs/` after editing docs. Every + backticked repo path in `AGENTS.md`, `CLAUDE.md`, `README.md`, `docs/` and + `skills/` must be a tracked file (`scripts/check_doc_paths.py`; exceptions in + `scripts/doc_path_allowlist.txt`). +3. `make test-py` and `make test-js`. +4. Touched any `locales/*.json`? `make gen-i18n` and commit the two generated + files; CI runs the generator and fails on a diff. The generator refuses to + drop a namespace (an unsynced venv looks like a deleted module); + `make gen-i18n ARGS=--allow-removals` is only for a module you really + removed or a deliberate `SM_MODULES_ENABLED` subset. + Changed an endpoint's response shape in a module that has `pages/api.generated.ts`? + `make gen-api-types` and commit it. Don't write `as unknown as` in shipped + TS (`ci-check-no-double-cast`); mark an unavoidable one `// cast-ok: `. +5. Added a module, a table or a migration? Also run `make doctor`, and review + the revision per `docs/module-authoring.md` § Reviewing a revision + autogenerated on SQLite (boolean `server_default` as `sa.true()/sa.false()`, + expression indexes by hand, enum downgrades via `drop_enums_if_postgres`). + `make migrations-roundtrip-pg` proves Postgres portability against a + disposable DB; where no Postgres is available, skip it and let CI run it. +6. User-visible change? Add a `## [Unreleased]` entry to `CHANGELOG.md` (Keep a + Changelog; cite issue numbers and migration revision ids; mark behaviour + changes). + +CI (`.github/workflows/pr.yml`, aggregate job `pr-checks`): python-lint (the +`make lint` checks above that are not JS), docs-build (`make ci-docs-build`, +the VitePress site), python-typecheck, python-tests (Redis service), js-tests, +js-lint (`biome ci --error-on-warnings`, plus the untranslated check), +js-typecheck (plus `make ci-api-types-fresh`), js-build (`make ci-css-build`), +migrations-roundtrip-pg, doctor (`make ci-doctor`: migrate a throwaway SQLite +DB, `alembic check`, then `make doctor`), e2e-smoke, perf-guards, +file-size-check, package-build. CI calls the same `make ci-*` targets as +`make lint` (`make ci-check-ci-parity` enforces it), with +`UV_SYNC_FLAGS=--locked` so a stale `uv.lock` fails: commit `uv.lock` whenever a +`pyproject.toml` changes (`make install` stays unlocked). `make ci-css-build` +and `make ci-docs-build` are in CI but not in `make lint` (slow). +`.github/workflows/postgres-tests.yml` runs `make test-py-pg` nightly, on +demand and on PRs touching `framework/db`, `framework/testing`, +`modules/{tenants,users,background_tasks,file_storage}` or `host/migrations`; +it is not in `pr-checks`. e2e-smoke and perf-guards enable a hardcoded +`SM_MODULES_ENABLED` JSON list; a new module is not exercised there until added. + +## Architecture + +**Discovery.** Modules declare `[project.entry-points.simple_module]` pointing +at a `ModuleBase` subclass with `meta = ModuleMeta(...)`. +`simple_module_core.discovery.discover_modules()` loads them, topologically sorts +by `ModuleMeta.depends_on` (plus `optional_depends_on`, ordering only, may be +absent), and the host calls each `register_*` hook in that order +(`docs/framework/discovery.md`). Outside development a missing `depends_on` +target fails the boot (`MissingDependencyError`, `SM030`); selecting +`keycloak` drops `Users`, so modules needing it must be left out. + +**Two names per module.** `ModuleMeta.name` (scaffold default: PascalCase, +`FeatureFlags`) keys Inertia render paths, `depends_on` and `SM_MODULES_ENABLED`. +The Python package name (`feature_flags`) keys the i18n namespace, +`app.state.` and table prefixes. Don't swap them: `keys.` breaks if +the namespace is the meta name. + +**Module layout.** Every module uses the area layout below +(`host/tests/test_module_layout.py` enforces it; `docs/module-layout.md` has +the details and the checklist for moving a flat module). + +```text +modules/// +├── module.py, models.py (or models/), contracts/ # entry point, tables, public imports +├── core/ # constants.py, settings.py, services.py, errors.py +├── web/ # deps.py, service.py, endpoints/{api,views}.py +├── pages/, locales/ # lookup paths stay at the root +├── ui/ # components/, hooks/, utils/ (when needed) +└── tasks.py, task_config.py, cli.py, theme.css, styles.css # optional +``` + +Other root directories are allowed only as named domain areas, listed per module +in `DOMAIN_AREAS` in the layout test. Area `__init__.py` files hold only a +docstring. Private helpers stay in their area with a leading underscore. Tests +mirror `tests///`, with no `__init__.py`, unique `test__*.py` +basenames, no top-level `test_*.py`, and `conftest.py` at `tests/` root. JS +tests stay in `tests-js/`. + +`gen-pages` writes `modules.generated.css` with an `@import` per CSS file that +exists. A `@theme` inside a cascade layer is inert and an unlayered rule beats +every Tailwind utility, so keep tokens in `theme.css` and rules in `styles.css` +(`SM022`/`SM023`). See `docs/module-authoring.md` § Styling. + +**Lifecycle hooks** on `ModuleBase` (all no-op by default; order and examples in +`docs/framework/lifecycle.md`): `register_settings`, `register_menu_items`, +`register_permissions`, `register_feature_flags`, `register_event_handlers`, +`register_invalidations`, `register_health_checks`, `register_public_routes`, +`register_body_limits`, `register_cors_policies`, `register_rate_limits`, +`register_setup_steps`, `register_csp_sources`, `register_design_packs`, +`register_audit_links`, `register_resource_types`, `register_exception_handlers`, +`register_middleware`, `register_routes(api_router, view_router)`, +`register_admin_routes(admin_router)` (only with `ModuleMeta.admin_view_prefix`), +then async `check_data_compatibility(db)` (read-only upgrade checks before +settings hydration; an `error` refuses boot outside development), then async +`on_startup` / `on_shutdown` (reverse order). Non-hook overrides: `locale_dirs()`, +`db_extensions()`, `template_dirs()`, `static_mounts()`, `spa_mounts()`. + +Rules that go with them: + +- A module gets one view router per prefix. All-admin modules point + `view_prefix` at `/admin/`; modules with both public and admin pages set + `admin_view_prefix` and use `register_admin_routes`. A prefix is a URL + convention, not a permission: guard every route. +- `register_public_routes` exempts routes from `AuthMiddleware`, method-aware + (`registry.add_regex(r"…/tilejson$", methods={"GET"})`): + `docs/framework/public-routes.md`. +- `register_invalidations` handlers may only *forget* (no delivery guarantee), + so every cache keeps its TTL as a floor. Publish from a `session.on_commit` + callback: `await request.app.state.sm.invalidation.publish(channel, key=...)`. + Cross-process delivery needs the Redis transport `background_tasks` installs + (`SM_BG_TASKS_BROADCAST_INVALIDATIONS`). `docs/framework/invalidation.md`. +- `register_csp_sources(registry)` takes fetch directives only, validated at + boot; an origin from DB settings uses `registry.add_provider(fn)`. +- `register_setup_steps`: while a required step is pending, `SetupMiddleware` + serves the wizard at `/setup`. A step's `SetupAction` runs only while that step + is pending. `docs/module-authoring.md` § First-run setup steps. +- Events: dispatch keys on the exact class. Durable delivery: + `bus.subscribe_durable(..., name=...)` (`SM034` without `name=`) + + `await bus.publish_durable(event, db)` inside the request transaction; a + durable handler must not commit `ctx.db`. `docs/framework/events.md`. +- Users: read accounts through `get_user_directory(app)` + (`simple_module_core.user_directory`), never `users.models` outside `users`. + `DirectoryUser.roles is None` means unknown (Keycloak), not none. Recent + activity: `get_activity_feed(app)`. `docs/framework/user-directory.md`. +- Menus: `MenuRegistry.add_provider(fn)` for per-request items; permissions: + `PermissionRegistry.add_source(name, provider)` + `invalidate_source(name)`. + `docs/framework/permissions.md`. + +**Middleware** (Starlette `add_middleware` is LIFO). Execution order per +request: `(ForwardedHost if trusted proxy + allowlist) → (ProxyHeaders if +SM_TRUSTED_PROXY) → (SentryContext if Sentry is enabled) → CorrelationId → +RequestLogging → (ServerTiming, development only) → CORS → BodyLimit → GZip → +SecurityHeaders → Session → InertiaMiddleware → → (Tenant: +multi_tenant, or fixed to default_tenant) → Locale → InertiaLayoutData → +RateLimit → Setup → Maintenance → CommitBeforeResponse → app` +(`framework/hosting/tests/test_middleware_order.py` pins it). Rules that follow: + +- Modules at the same dependency tier: the one that sorts later wraps outermost. + Use `depends_on` to order, never names. +- Any `X-Inertia` response is forced `private, no-store`; to cache a public + page, set `Cache-Control`/ETag on the *document* response only. +- `RateLimit` runs after auth, so by default it throttles only anonymous + traffic to public routes. Per-route budgets: `register_rate_limits` rules + (`RateLimitRule(rate, key="user"|"ip"|"user_or_ip"|"tenant")`) or the + `RateLimit("10/minute", key="user")` dependency from + `simple_module_hosting.rate_limit_deps`. +- Request bodies over `max_request_body_bytes` (10 MiB) are refused before any + module sees them; per-path overrides via `register_body_limits`. +- CORS is opt-in per path (`register_cors_policies`, most specific match wins). + `docs/framework/cors.md`. + +Detail: `docs/framework/middleware.md`, `docs/framework/request-guards.md`. + +**Admin section.** Admin screens live under `/admin/*`, register into +`MenuSection.ADMIN_SIDEBAR` and render in `AdminLayout`, all three together. +`SidebarLayout` renders the menu its `menuKey` names, so a page left on +`AuthenticatedLayout` after its item moved shows a sidebar without it. +`group=` sub-clusters inside a sidebar (`Access`, `Appearance`, `System`). +`/admin` itself ships with `simple_module_hosting`; old URLs 301 from +`host/routes_legacy.py`. `/api/*` paths never move. + +**Inertia.** `inertia.render("/")` maps to +`modules///pages/.tsx`; host pages under +`host/client_app/pages/` use the bare name. Mismatches fire `SM003`/`SM004`. +Use `InertiaDep` from `simple_module_hosting.inertia_deps`; `InertiaResponse` +and prop factories (`optional`, `always`, `defer`, `merge`, `once`, `scroll`) +come from `simple_module_inertia`. Shared props `auth`, `menus`, `i18n` are set +by `InertiaLayoutDataMiddleware`; `auth.user` is null unless +`app.state.principal_serializer` is set (i18n caching via `X-SM-I18n`: +`docs/frontend/shared-props.md`). Inertia +`router.post/patch/put/delete` must target view routes, never `/api/*` +(`SM018`). Fast pages: `docs/frontend/performance.md` and the +`simple-module-page-perf` skill. + +**CSRF.** Baseline is `SameSite=Lax` on the session cookie, so a same-site +request needs no token. Defence in depth: `RequiresCsrf` router dependency + +`get_csrf_token(request)` as a view prop, echoed as `X-CSRF-Token` +(`simple_module_hosting.csrf`). The session cookie is `Secure` outside +development and testing. + +## Database + +- Per-module `Base` via `create_module_base("")`; every module owns its + `MetaData` but all tables share one schema. Prefix `__tablename__` with the + module name (`orders_order`); the scaffold does it, nothing enforces it. +- Mixins in `simple_module_db.mixins`: `AuditMixin`, `SoftDeleteMixin`, + `MultiTenantMixin`, `VersionedMixin`. Soft-delete and tenant filters cover + every statement shape. Bypass with `execution_options(include_deleted=True)`; + purge with `hard_delete(session, obj)`. `docs/database/mixins.md`. +- A bare `datetime` field already maps to a UTC-aware type. When you declare + an explicit column type, use `sa_type=AwareDateTime()` + (`simple_module_db.types`), never `DateTime(timezone=True)`, which reads back + naive on SQLite. +- SQLite runs with WAL (file DBs), `busy_timeout` and **`foreign_keys=ON`**: + both sides of an FK need the same column type (`sa.Uuid` vs fastapi-users' + `GUID` differ on SQLite) and parents must be flushed before children. +- **Sessions.** The request session (`get_db`) commits in + `CommitBeforeResponseMiddleware` only if something was written (ORM flush or + `session.execute(update/insert/delete)`); otherwise it rolls back. Raw + `text("UPDATE …")` needs `mark_written(session)`. Service and request code + must not call `session.commit()`; flush when you need DB-assigned values. Use + `session.on_commit(cb)` for post-commit side effects. `docs/database/sessions.md`. +- **Tenancy fails closed.** With `multi_tenant` on, a query or insert on a + `MultiTenantMixin` model with no current tenant raises `TenantIsolationError`. + Cross-tenant code says so with `all_tenants()` / + `execution_options(all_tenants=True)`; jobs act for one tenant with + `tenant_context(id)` (both in `simple_module_db.tenancy`). Off, rows are + stamped `default_tenant` when set, else `DEFAULT_TENANT_ID` (`"default"`). + Unique keys on tenant tables must include `tenant_id` (`SM024`). Tenant-level + routes act on the *active* tenant, never a tenant id from the URL. The `tenants` + module owns memberships and `app.state.tenant_resolver` (`SM025` if missing). Map tenant + roles with `tenant_role(TenantRole.MEMBER)` from `simple_module_core.tenancy`, + never by importing `tenants`. `docs/framework/multi-tenancy.md`. +- **Migrations** live in `host/migrations/versions/`, never in module packages. + `host/migrations/env.py` builds metadata from every installed module. The + history has several roots and heads (`host/tests/test_migration_graph.py` + pins them; update it with any new root or head), so apply with + `upgrade heads` (`make migrate`) and autogenerate on a named branch: + + ``` + uv run --project host alembic -c host/alembic.ini revision --autogenerate \ + -m "" --head @head + ``` + + A module's first revision sets `branch_labels`. Labels are named targets, not + per-module rollbacks: `downgrade