Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion architecture/key-sequence-flows.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ Related: `POST /v1/mobile/games/quiz/generate` (and `/wordpass/generate`) is als

## SF-02 Backoffice operator changes the runtime AI target (UC-09)

There are two independent targets. This flow covers the llama target that `ai-engine-api` uses for generation. The gateway keeps its own ai-engine API/stats target, changed with `PUT /internal/admin/ai-engine/target` (guarded by `API_GATEWAY_ADMIN_TOKEN` when set); it is not changed by this flow.
This flow covers the llama target that `ai-engine-api` uses for generation. The gateway keeps no routing state: its ai-engine API/stats upstreams come only from environment variables.

```mermaid
sequenceDiagram
Expand Down
33 changes: 13 additions & 20 deletions guides/capabilities/edge/public-edge-routing.md
Original file line number Diff line number Diff line change
@@ -1,17 +1,17 @@
# Public Edge Routing

Last updated: 2026-05-03.
Last updated: 2026-11-13.

## Purpose

This capability provides the stable ingress boundary for mobile and backoffice clients while keeping downstream services private, replaceable, and operationally reroutable.
This capability provides the stable ingress boundary for mobile and backoffice clients while keeping downstream services private and replaceable.

## Functional objective

- accept public traffic through one edge service
- apply edge auth, CORS, timeout, and forwarding policy before reaching BFFs
- expose internal AI proxy routes without making ai-engine public
- preserve a stable public surface even when runtime targets change underneath
- preserve a stable public surface when downstream targets change

## Ownership and boundaries

Expand All @@ -33,8 +33,8 @@ This capability provides the stable ingress boundary for mobile and backoffice c

### Explicit boundaries

- owns public ingress, route registration, forwarding, and AI proxy target resolution in `api-gateway/src/app/routes/proxy.ts`
- owns persisted AI target overrides in `api-gateway/src/app/services/routingStateStore.ts`
- owns public ingress and the route table in `api-gateway/src/app/routes/routeTable.ts`, with forwarding and AI proxy target resolution in `proxy.ts`
- resolves the `/internal/ai-engine/*` upstreams only from `AI_ENGINE_API_URL` and `AI_ENGINE_STATS_URL`; it holds no persisted routing state (the runtime llama target lives in `ai-engine-api`, see [Runtime routing and service targeting](../../../operations/runtime-routing-and-service-targeting.md))
- delegates business orchestration to BFFs and AI execution to ai-engine
- does not own domain validation, persistence, or operator UI semantics

Expand All @@ -48,20 +48,17 @@ This capability provides the stable ingress boundary for mobile and backoffice c
| HTTP route | `/v1/mobile/*` | public ingress toward `bff-mobile` |
| HTTP route | `/v1/backoffice/*` | public ingress toward `bff-backoffice` |
| HTTP route | `/internal/ai-engine/*` | internal proxy for generation, ingest, catalogs, health, and stats |
| HTTP route | `/internal/admin/ai-engine/target` | privileged runtime target read/update/delete |

### Inputs

- public HTTP requests from mobile and backoffice clients
- edge auth tokens and correlation metadata
- environment defaults for BFF and AI upstream URLs
- persisted AI routing override loaded from `GATEWAY_ROUTING_STATE_FILE`

### Outputs

- forwarded public responses for mobile and backoffice clients
- forwarded internal responses for ai-engine generation, ingest, health, and stats
- persisted side effects when operators mutate the AI target through admin routes

### Contract sources

Expand All @@ -74,27 +71,25 @@ This capability provides the stable ingress boundary for mobile and backoffice c

1. A public client reaches `api-gateway` on a `/v1/mobile/*` or `/v1/backoffice/*` route.
2. The gateway applies edge auth and request policy in `server.ts`, including health-route exemptions and route classification for rate limiting.
3. `proxy.ts` resolves the correct upstream target, using persisted AI overrides for `/internal/ai-engine/*` and static BFF URLs for public BFF paths.
3. `proxy.ts` resolves the upstream from the route table, using the environment-configured AI targets for `/internal/ai-engine/*` and static BFF URLs for public BFF paths.
4. The gateway forwards the request through shared HTTP helpers and route-specific circuit breakers.
5. The response is returned to the caller while preserving edge-visible status and correlation context.

## Persistence and runtime state

### Owned persistence

- gateway routing state persisted by `RoutingStateStore`
- none

### Runtime state

- default upstream URLs from `api-gateway/src/app/config.ts`
- effective AI API and stats targets computed from env defaults plus persisted overrides
- upstream URLs from `api-gateway/src/app/config.ts` (BFF URLs, `AI_ENGINE_API_URL`, `AI_ENGINE_STATS_URL`)
- per-upstream circuit-breaker state inside the gateway process

### State risks

- stale AI target after restart or redeploy because the routing state file survives
- gateway target drift relative to BFF or ai-engine runtime state
- public route availability diverging from operator expectations when the gateway lags behind downstream route additions
- gateway upstream URLs drifting from the real BFF or ai-engine addresses after a redeploy
- public route availability diverging from operator expectations when the gateway lags behind downstream route additions or removals (the route table is the single list to update)

## Observability evidence

Expand All @@ -105,27 +100,25 @@ This capability provides the stable ingress boundary for mobile and backoffice c
### Operational signals

- gateway logs and upstream timeout/failure traces
- admin readback from `/internal/admin/ai-engine/target`
- route-aware rate-limiter classification in `api-gateway/src/app/services/rateLimiter.ts`

### Verification evidence

- `api-gateway/src/tests/proxy.test.ts`
- `api-gateway/src/tests/routingStateStore.test.ts`
- `api-gateway/src/tests/proxyRouteEquivalence.test.ts`
- focused health-route tests in `api-gateway/src/tests/health.test.ts`

## Failure modes and risks

| Failure mode | Likely cause | User/operator impact | Detection path | Recovery path |
|---|---|---|---|---|
| Forwarding failure | BFF unreachable, timeout, or broken proxy route | client-visible outage or edge 404/5xx | proxy tests, logs, smoke requests | restore upstream or add/fix the missing proxy route |
| Wrong AI target | stale persisted override or bad admin update | internal AI routes fail while public gateway stays healthy | `/internal/admin/ai-engine/target`, AI smoke checks | clear or replace the persisted override |
| Wrong AI target | `AI_ENGINE_API_URL` or `AI_ENGINE_STATS_URL` points at the wrong host | internal AI routes fail while public gateway stays healthy | AI smoke checks | fix the env value and redeploy |
| Partial route exposure | downstream route exists but gateway route is missing | feature appears broken from public edge only | end-to-end probe through gateway | add the corresponding gateway proxy route and retest |

## Security and access model

- public edge access is separated from privileged admin routing control
- `/internal/admin/ai-engine/target` is protected by the gateway admin token path, not public client auth
- the gateway exposes no admin routing control; the runtime llama target is changed through the authenticated Backoffice surface and lives in `ai-engine-api`
- internal AI proxy routes stay private to the platform and are not directly exposed on public client flows

## Deployment and rollback implications
Expand Down
153 changes: 39 additions & 114 deletions guides/capabilities/operations/runtime-routing-control.md
Original file line number Diff line number Diff line change
@@ -1,139 +1,70 @@
# Runtime Routing Control

Last updated: 2026-05-03.
Last updated: 2026-11-13.

## Purpose

This capability lets operators change effective service and AI routing at runtime, persist those overrides, and return safely to defaults without rebuilding the platform.

## Functional objective

- persist and apply runtime target overrides and presets
- expose the effective routing picture to operators
- support controlled rollback from overrides back to default topology
This capability lets operators point `ai-engine-api` at a different llama runtime without a redeploy, persist that override, and return to the environment default. It is the only runtime routing control that remains; the full flow is in [Runtime routing and service targeting](../../../operations/runtime-routing-and-service-targeting.md).

## Ownership and boundaries

### Main owners

- `bff-backoffice`
- `api-gateway`
- `ai-engine`

### Upstream actors

- backoffice operators

### Explicit boundaries
| Owner | Responsibility | Code |
|---|---|---|
| `Backoffice` | llama target card (`AiEngineTargetControl`) in AI diagnostics | `Backoffice/src/src/ui/components/AiEngineTargetControl.tsx` |
| `api-gateway` | plain forwarding of `/v1/backoffice/*` to the BFF | `api-gateway/src/app/routes/routeTable.ts` |
| `bff-backoffice` | validation, host allowlist, probe, sync to ai-engine | `bff-backoffice/src/app/routes/backoffice.ts` |
| `ai-engine-api` | owns and persists the override | `ai-engine/src/ai_engine/api/app.py` |

- owns BFF-side service target and AI preset management in `bff-backoffice/src/app/routes/backoffice.ts`
- owns gateway-side AI target persistence in `api-gateway/src/app/services/routingStateStore.ts`
- owns ai-engine local llama target override in `ai-engine`
- does not replace deploy-time configuration management in `platform-infra`, but can override runtime behavior after deploy
Out of scope: deploy-time configuration (`platform-infra`) and any BFF-side routing state (none exists).

## Entrypoints and contracts

### Main entrypoints

| Type | Location | Notes |
| Type | Route | Notes |
|---|---|---|
| HTTP route | `GET/PUT/DELETE /v1/backoffice/ai-engine/target` | BFF AI target inspection and mutation |
| HTTP route | `GET/POST/PUT/DELETE /v1/backoffice/ai-engine/presets*` | shared AI preset management |
| HTTP route | `/v1/backoffice/service-targets/:service` | runtime service override control |
| HTTP route | `GET/PUT/DELETE /internal/admin/ai-engine/target` | gateway-owned AI target control |
| HTTP route | ai-engine internal llama target routes | ai-engine local runtime override |
| HTTP | `GET/PUT/DELETE /v1/backoffice/ai-engine/target` | read, apply, clear |
| HTTP | `POST /v1/backoffice/ai-engine/probe` | non-mutating reachability check of `<target>/v1/models` |
| HTTP | `GET /v1/backoffice/ai-engine/plugin/capabilities` | effective plugin runtime capabilities |
| HTTP (private) | `GET/PUT/DELETE /internal/admin/llama-target` on `ai-engine-api` | called by the BFF with `x-api-key` |

### Inputs

- operator target changes and preset payloads validated by Zod schemas
- current runtime state from BFF routing store, gateway routing store, and ai-engine target store
- service identifiers such as `ai-engine-api`, `ai-engine-stats`, and other BFF-managed upstreams

### Outputs

- persisted override files and preset state
- effective runtime target changes
- reset-to-default side effects when overrides are cleared

### Contract sources

- `bff-backoffice/src/app/routes/backoffice.ts`
- `bff-backoffice/src/app/services/routingStateStore.ts`
- `api-gateway/src/app/services/routingStateStore.ts`
- `api-gateway/src/app/routes/proxy.ts`
Payload for `PUT` and `POST probe`: `host`, `protocol` (`http` or `https`), `port`, optional `label`, validated with Zod in the BFF.

## Main functional flow

1. An operator changes a target or preset in backoffice.
2. `bff-backoffice` validates the payload and persists the change in its routing-state store.
3. When the change affects AI execution, the BFF synchronizes runtime state toward `api-gateway` and ai-engine as needed.
4. Subsequent requests use the new effective target without a redeploy.
5. Operators verify health and can clear the override to fall back to defaults.

## Persistence and runtime state

### Owned persistence
1. The operator edits the llama target card and optionally probes it.
2. The BFF checks the host against `ALLOWED_ROUTING_TARGET_HOSTS` and calls `ai-engine-api`.
3. `ai-engine-api` persists the override in `AI_ENGINE_LLAMA_TARGET_STATE_FILE` and applies it.
4. The BFF clears its upstream cache and breakers and returns the read-back target.
5. `DELETE` restores the environment default.

- BFF service-target overrides and AI preset records
- gateway AI target override file
- ai-engine local llama target override file
## Persistence and risks

### Runtime state
- State lives only in `ai-engine-api` (file on mounted storage); it survives restarts and redeploys.
- A stale override after a rollout leaves generation pointing at an unexpected runtime while pods look healthy.
- Rolling back images does not roll back the override.

- effective topology is computed from environment defaults plus one or more persisted overrides
- overrides survive restart and redeploy unless explicitly cleared
- BFF, gateway, and ai-engine can each hold part of the routing truth
## Failure modes

### State risks

- stale override after a healthy rollout leaves runtime using an unexpected target
- sync-order mistakes across BFF, gateway, and ai-engine create split routing state
- UI can display one topology while the actual traffic path uses another

## Observability evidence

### Health signals

- health probes executed after target changes

### Operational signals

- target and preset readback routes in the BFF and gateway
- operator probe route `POST /v1/backoffice/ai-engine/probe`
- runtime-target comparison in backoffice diagnostics

### Verification evidence

- `bff-backoffice/src/tests/routingStateStore.test.ts`
- `bff-backoffice/src/tests/backofficeRouteGuards.test.ts`
- `api-gateway/src/tests/routingStateStore.test.ts`
- `api-gateway/src/tests/proxy.test.ts`

## Failure modes and risks

| Failure mode | Likely cause | User/operator impact | Detection path | Recovery path |
|---|---|---|---|---|
| Wrong target configured | operator error or stale preset | runtime requests fail while pods stay healthy | target read routes and smoke requests | reset to default or apply a known-good preset |
| State mismatch across owners | failed sync or concurrent edits | UI shows one topology while runtime uses another | compare BFF, gateway, and ai-engine target reads | resynchronize holders and retest |
| Override survives too long | runtime holder not cleared after rollout | traffic keeps pointing to a temporary host | post-deploy routing inspection | explicitly clear override and verify |
| Failure mode | Likely cause | Detection | Recovery |
|---|---|---|---|
| `400` on `PUT` | host not allowed or invalid payload | response message | fix the payload or `ALLOWED_ROUTING_TARGET_HOSTS` |
| Probe not reachable | wrong host/port or runtime down | probe result | correct the target |
| Target set but generation fails | llama runtime unreachable or warming up | smoke request, `ai-engine-api` health | `DELETE` the override or fix the runtime |
| Admin call times out | `ai-engine-api` hung | BFF error citing the timeout (`AI_ENGINE_ADMIN_TIMEOUT_MS`) | restore `ai-engine-api` |

## Security and access model

- routing mutation paths are high-impact admin operations and should remain role-gated and auditable
- preset and target payloads can reveal sensitive internal topology and must stay on protected surfaces
- route guards in BFF and gateway are part of the control boundary, not optional UI behavior

## Deployment and rollback implications
- Mutating routes are high-impact admin operations behind the backoffice authentication path.
- Targets are restricted by `ALLOWED_ROUTING_TARGET_HOSTS` (denied outside development when empty).
- `ai-engine-api` internal admin routes are not public.

### Deployment impact
## Observability and verification

- routing overrides can outlive container rollouts and therefore change the effective behavior of an otherwise healthy deployment
- gateway, BFF, and ai-engine changes must preserve routing-state compatibility across restarts
- Readback via `GET /v1/backoffice/ai-engine/target`; reachability via the probe route.
- Tests: `bff-backoffice/src/tests/backoffice.test.ts`, `backofficeBranches.test.ts`, `routingAllowlist.test.ts`; `ai-engine/src/tests/test_api/test_llama_target_store.py`; `Backoffice/src/src/test/ai-diagnostics-panel.integration.test.tsx`.

### Rollback impact
## History

- rolling back binaries does not automatically roll back persisted routing state
- safe rollback often requires both image rollback and explicit override reset or reapplication
Service-target overrides, shared presets and connections, the Kubernetes overview, routing history, and the matching Backoffice tab and presets selector were removed (`bff-backoffice` PR #9, `Backoffice` PR #7). The matching dead forwarding entries and the gateway's own `/internal/admin/ai-engine/target` store were removed from `api-gateway` (PR #8).

## Related capabilities

Expand All @@ -142,13 +73,7 @@ This capability lets operators change effective service and AI routing at runtim
- [AI observability and diagnostics](../ai/ai-observability-and-diagnostics.md)
- [Backoffice operational diagnostics](./backoffice-operational-diagnostics.md)

## Open questions

- whether all routing transitions are audited consistently enough across BFF, gateway, and ai-engine holders
- whether concurrent operator edits need stronger conflict semantics and UI signaling

## References

- [Platform capability matrix](../../foundations/platform-capability-matrix.md)
- [Platform functional overview](../../foundations/platform-functional-overview.md)
- [Backoffice operational diagnostics](./backoffice-operational-diagnostics.md)
Loading
Loading