From bff87ec5ecd47d87083d4a3a22ce6fa63b50221d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Sebasti=C3=A1n=20Ramiro=20Entrerrios=20Garc=C3=ADa?= <100527906+Sebas1705@users.noreply.github.com> Date: Sat, 14 Nov 2026 02:27:20 +0100 Subject: [PATCH] docs: remove remaining routing-plane and auto-deploy mentions --- AGENTS.md | 2 +- README.md | 2 +- architecture/c4-views.md | 18 ++++++------ .../0001-single-public-edge.md | 2 ++ .../0005-runtime-control-plane.md | 2 ++ architecture/domain-model.md | 21 +++++++------- architecture/glossary.md | 4 +-- architecture/repository-map.md | 19 ++++++------- architecture/target-architecture.md | 28 +++++++++---------- architecture/use-cases.md | 6 ++-- .../capabilities/ai/ai-content-generation.md | 22 +++++++-------- .../ai/ai-observability-and-diagnostics.md | 14 +++++----- .../capabilities/edge/public-edge-routing.md | 2 +- .../backoffice-operational-diagnostics.md | 26 ++++++++--------- .../deployment-and-rollout-automation.md | 2 +- .../communication/local-edge-integration.md | 3 +- guides/foundations/ai-plugin-architecture.md | 2 +- .../foundations/platform-capability-matrix.md | 12 ++++---- .../platform-functional-overview.md | 26 ++++++++--------- .../repository-standards-adoption.md | 4 ++- .../critical-workflow-smoke-matrix.md | 4 +-- .../primary-workflows-validation.md | 2 +- operations/branch-protection-policy.md | 4 ++- operations/cicd-workflow-map.md | 8 +++--- operations/deployment-strategy.md | 6 ++-- operations/environments-and-secrets.md | 4 +-- operations/functionality-review-baseline.md | 8 +++--- operations/repository-minimum-standards.md | 4 ++- operations/threat-model.md | 2 +- 29 files changed, 126 insertions(+), 133 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 8acc172..8fa4a57 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -14,7 +14,7 @@ Central architecture and operations documentation for AxiomNode. ## CI/CD notes - Docs now have a markdown link validation workflow; keep relative links resolvable. -- Docs should track real workflow behavior (service CI -> platform-infra -> stg deploy by default, prod by manual promotion). +- Docs should track real workflow behavior (service CI dispatches the `platform-infra` image build on merge; deploys are manual-only via `deploy.yaml` `workflow_dispatch`, and no staging host exists today). ## LLM editing rules - Keep content concise, actionable, and in English. diff --git a/README.md b/README.md index f5df592..96379d8 100644 --- a/README.md +++ b/README.md @@ -107,7 +107,7 @@ Backoffice access is role-gated through `microservice-users`. The current role m ### Current runtime reality to document explicitly - The deployed Kubernetes topology and the effective runtime topology are not always identical. -- `api-gateway`, `bff-backoffice`, and `ai-engine-api` can all hold persisted runtime targeting state. +- Only `ai-engine-api` holds persisted runtime targeting state (the llama target); `api-gateway` and `bff-backoffice` hold none. - Staging may run `ai-engine-api` and `ai-engine-stats` in-cluster while the llama runtime remains external, or it may render the optional full in-cluster AI overlay for controlled diagnostics. - Documentation must distinguish environment defaults, persisted overrides, and live effective targets. diff --git a/architecture/c4-views.md b/architecture/c4-views.md index e730177..f94d85d 100644 --- a/architecture/c4-views.md +++ b/architecture/c4-views.md @@ -1,6 +1,6 @@ # C4 Views -Last updated: 2026-04-22. +Last updated: 2026-11-14. This document captures the AxiomNode platform using the C4 model: System Context (Level 1), Containers (Level 2), and Components (Level 3) for the most relevant containers. Level 4 (code) is intentionally omitted; it lives in each repository. @@ -25,7 +25,7 @@ Rel(player, axiom, "Plays games / consults stats", "HTTPS") Rel(operator, axiom, "Operates and monitors", "HTTPS") Rel(platformEng, github, "Pushes code, dispatches builds") Rel(github, ghcr, "Publishes images") -Rel(github, axiom, "Deploys to k3s") +Rel(github, axiom, "Manual deploy (workflow_dispatch; no staging host today)") Rel(axiom, llama, "Generation requests", "HTTP") Rel(player, firebase, "Auth / push", "HTTPS") ``` @@ -40,7 +40,7 @@ Person(player, "Mobile Player") Person(operator, "Backoffice Operator") System_Boundary(edge, "Edge") { - Container(gateway, "api-gateway", "Node/Express", "Single public ingress, auth, CORS, routing, persists active ai-engine target") + Container(gateway, "api-gateway", "Node/Express", "Single public ingress, auth, CORS, routing; stateless (no routing state)") } System_Boundary(experience, "Experience / BFF") { @@ -78,7 +78,7 @@ Rel(operator, spa, "HTTPS") Rel(spa, gateway, "HTTPS") Rel(gateway, bffMobile, "HTTP") Rel(gateway, bffBack, "HTTP") -Rel(gateway, aiApi, "HTTP (live target)") +Rel(gateway, aiApi, "HTTP (/internal/ai-engine, upstream from env)") Rel(bffMobile, quiz, "HTTP") Rel(bffMobile, wp, "HTTP") Rel(bffMobile, users, "HTTP") @@ -86,7 +86,7 @@ Rel(bffBack, users, "HTTP") Rel(bffBack, quiz, "HTTP") Rel(bffBack, wp, "HTTP") Rel(bffBack, aiStats, "HTTP") -Rel(bffBack, gateway, "HTTP (control plane mutations)") +Rel(bffBack, aiApi, "HTTP (llama target, plugin capabilities)") Rel(quiz, gateway, "HTTP (generation, /internal/ai-engine)") Rel(wp, gateway, "HTTP (generation, /internal/ai-engine)") Rel(aiApi, llama, "HTTP") @@ -108,14 +108,12 @@ Rel(sdk, bffBack, "imports") - `RouteRegistry` — declarative route map, version-aware. - `Forwarder` (`proxy` SDK module) — `forwardHttp` with structured upstream errors. - `ContractGuards` — Zod-based validation for shared public query contracts. -- `RuntimeTargetStore` — persists active ai-engine API/stats target on mounted PV. - `EdgePolicy` — auth, CORS, rate-limit hooks. ### bff-backoffice - `OperatorAuth` — session and role resolution against `microservice-users`. -- `ServiceTargetOverrideStore` — persists per-route upstream overrides. -- `AIPresetStore` — reusable ai-engine destination presets. +- `LlamaTargetControl` — proxies llama target read/set/reset/probe to `ai-engine-api` `/internal/admin/llama-target`; keeps no routing state. - `MetricsAggregator` — adapts upstream metrics caches with normalized TTL keys. - `RowsQueryEngine` — UI-side filter with precomputed lowercase row strings. @@ -124,7 +122,7 @@ Rel(sdk, bffBack, "imports") - `RAGRetriever` — multilingual MiniLM embeddings + metadata-aware oversample/rerank. - `Optimizer` — query enrichment with category name, model, corpus signature. - `CacheManager` — keyed by category, embedding model, corpus signature. -- `LlamaClient` — points to active llama target (persisted). +- `LlamaClient` — points to active llama target (persisted in `AI_ENGINE_LLAMA_TARGET_STATE_FILE`). - `Diagnostics` — structured counters surfaced via FastAPI hooks. ### microservice-quizz / wordpass @@ -137,7 +135,7 @@ Rel(sdk, bffBack, "imports") ### platform-infra - `BuildPushWorkflow` — receives dispatch, builds, publishes `:sha`/`:stg`/`:prod`. -- `DeployWorkflow` — `kubectl apply -k overlays/{env}` with smoke checks. +- `DeployWorkflow` — manual-only (`workflow_dispatch`): `kubectl apply -k overlays/{env}` with smoke checks; no staging host exists today. - `Overlays` — `dev`, `stg`, `prod` (+ optional full-AI overlay). - `SealedSecrets` — per-overlay sealed material. diff --git a/architecture/decision-records/0001-single-public-edge.md b/architecture/decision-records/0001-single-public-edge.md index eb632a1..04a1104 100644 --- a/architecture/decision-records/0001-single-public-edge.md +++ b/architecture/decision-records/0001-single-public-edge.md @@ -23,6 +23,8 @@ Run a single public ingress, `api-gateway`, as the only externally reachable HTT - `api-gateway` becomes a routing-state holder (live AI target). - Operational dependency: gateway downtime equals platform downtime; mitigated by k3s replicas and probes. +> **Superseded in part (2026-11-14).** The gateway no longer holds a live AI target: its persisted routing state was removed (`api-gateway` PR #8) and it resolves ai-engine upstreams from `AI_ENGINE_API_URL` and `AI_ENGINE_STATS_URL`. See [Runtime routing and service targeting](../../operations/runtime-routing-and-service-targeting.md). + ## Related - [Repository map](../repository-map.md) diff --git a/architecture/decision-records/0005-runtime-control-plane.md b/architecture/decision-records/0005-runtime-control-plane.md index 33dc51a..ed94cdf 100644 --- a/architecture/decision-records/0005-runtime-control-plane.md +++ b/architecture/decision-records/0005-runtime-control-plane.md @@ -3,6 +3,8 @@ - Status: Accepted - Date: 2026-04-22 +> **Superseded in part (2026-11-14).** Only the third bullet of the decision still holds: `ai-engine-api` persists the active llama target, and operators change it through `bff-backoffice`. The `bff-backoffice` service-target overrides and shared presets and the `api-gateway` persisted ai-engine target were removed (`bff-backoffice` PR #9, `Backoffice` PR #7, `api-gateway` PR #8); the gateway and the BFF now hold no routing state. The text below is kept as the original record. Current behavior: [Runtime routing and service targeting](../../operations/runtime-routing-and-service-targeting.md). + ## Context We need to change AI upstream routing without redeploying the platform (e.g., point `api-gateway` to a different `ai-engine-api`, or `ai-engine-api` to a different llama target). GitOps alone forces a build/deploy cycle that is too slow for AI recovery scenarios. diff --git a/architecture/domain-model.md b/architecture/domain-model.md index 3b48ead..1430c94 100644 --- a/architecture/domain-model.md +++ b/architecture/domain-model.md @@ -1,6 +1,6 @@ # Domain Model -Last updated: 2026-04-22. +Last updated: 2026-11-14. High-level domain model of the AxiomNode platform. Each bounded context is owned by exactly one microservice; cross-context references travel by ID and never by foreign-key joins across databases. @@ -12,7 +12,7 @@ High-level domain model of the AxiomNode platform. Each bounded context is owned | Quiz Gameplay | `microservice-quizz` | PostgreSQL (quizz DB) | Indirect | | Wordpass Gameplay | `microservice-wordpass` | PostgreSQL (wordpass DB) | Indirect | | AI Generation | `ai-engine` | Redis cache + vector store + on-disk corpus | Internal only | -| Runtime Routing State | `api-gateway`, `bff-backoffice`, `ai-engine-api` | Mounted PV (per service) | Operator only | +| Runtime Routing State (llama target) | `ai-engine-api` | Mounted PV | Operator only | | Contracts | `contracts-and-schemas` | Git | n/a | ## Entities (logical view) @@ -70,11 +70,9 @@ classDiagram +latencyMs: int +cacheHit: bool } - class RoutingTarget { - +scope: enum (gateway-ai, bff-override, ai-llama) - +targetUrl: string - +presetId: string? - +updatedBy: string + class LlamaTarget { + +url: string + +label: string? +updatedAt: datetime } @@ -113,12 +111,13 @@ classDiagram - `cacheKey` uniquely identifies a generation request shape. - `corpusSignature` change invalidates dependent cached entries logically. -### RoutingTarget (control-plane state, distributed) +### LlamaTarget (control-plane state, owned by `ai-engine-api`) -- Root: `RoutingTarget` keyed by `scope`. +- Root: `LlamaTarget`, persisted as a single JSON file (`AI_ENGINE_LLAMA_TARGET_STATE_FILE`, fields `url`, `label`, `updatedAt`). - Invariants: - - At most one active target per scope per service instance share the mounted PV. - - Drift between scopes is allowed but observable (operators can see effective vs declared). + - At most one active override; deleting it falls back to the environment URL. + - The override can differ from the declared manifests; operators read the effective value through `GET /v1/backoffice/ai-engine/target`. + - The gateway and `bff-backoffice` keep no routing state (service-target overrides and presets were removed). ## Cross-context contracts diff --git a/architecture/glossary.md b/architecture/glossary.md index 96379a9..809b305 100644 --- a/architecture/glossary.md +++ b/architecture/glossary.md @@ -15,8 +15,8 @@ Common terms used across AxiomNode documentation. | Full in-cluster AI | Optional overlay that runs llama inside k3s for diagnostics. | | Effective topology | The routing actually in use, including persisted runtime overrides. | | Declared topology | The topology described by Kubernetes manifests in `platform-infra`. | -| Routing target | Persisted runtime URL for an upstream (gateway-ai, bff-override, ai-llama). | -| Control plane (runtime) | Backoffice-driven set of routing mutations that survive pod restart. | +| Routing target | Persisted runtime URL for the ai-engine llama upstream (the only runtime routing target). | +| Control plane (runtime) | Backoffice-driven llama target mutation (set, reset) that survives pod restart. | | Cache key | Composite key `(category_id, embedding_model, corpus_signature, request_shape)` used by `ai-engine-api`. | | Corpus signature | Stable hash of the curated corpus used for retrieval; participates in cache keys. | | GHCR | GitHub Container Registry, where service images are published. | diff --git a/architecture/repository-map.md b/architecture/repository-map.md index 5ccb45c..0525cf4 100644 --- a/architecture/repository-map.md +++ b/architecture/repository-map.md @@ -1,6 +1,6 @@ # Repository Map -Last updated: 2026-04-19. +Last updated: 2026-11-14. ## Runtime repositories @@ -19,8 +19,8 @@ Last updated: 2026-04-19. | Repository | Persisted runtime state | Why it matters | |---|---|---| -| `api-gateway` | Active ai-engine API/stats target | Determines where live AI traffic is forwarded | -| `bff-backoffice` | Service target overrides and shared ai-engine presets | Shapes operator-visible effective topology | +| `api-gateway` | None (upstreams come from `AI_ENGINE_API_URL` and `AI_ENGINE_STATS_URL`) | Stateless edge; no routing store | +| `bff-backoffice` | None (only an audit trail of admin actions) | Proxies the llama target control to `ai-engine-api` | | `ai-engine` | Active llama target inside `ai-engine-api` | Controls where generation calls reach the model runtime | ## Client and operator entry points @@ -30,7 +30,7 @@ Last updated: 2026-04-19. | Mobile app | `api-gateway` | `api-gateway` -> `bff-mobile` -> quiz/word-pass services | | Backoffice SPA | `api-gateway` | `api-gateway` -> `bff-backoffice` -> users/services/ai diagnostics | | Internal AI producers | service-local clients | quiz/word-pass -> `ai-engine-api` | -| Operators changing AI routing | `backoffice` through `bff-backoffice` | backoffice -> `bff-backoffice` -> `api-gateway` / `ai-engine-api` | +| Operators changing the llama target | `backoffice` through `bff-backoffice` | backoffice -> `api-gateway` -> `bff-backoffice` -> `ai-engine-api` | ## Delivery-chain classification @@ -48,9 +48,9 @@ These repositories participate in the current automatic image publication and st - `microservice-quizz` - `microservice-wordpass` -### Outside the current automatic staging chain +### Outside the image-build chain -- `mobile-app`: validated in its own CI, but not published as a k3s runtime image through `platform-infra` +- `mobile-app`: validated in its own CI, but not published as a runtime image through `platform-infra` - external llama runtime: still operationally external in the split staging topology unless the optional full in-cluster AI overlay is rendered deliberately ## Platform repositories @@ -79,10 +79,9 @@ For AI-related flows, the effective runtime topology may differ from the deploye Reason: -- `api-gateway` can hold the active ai-engine target at runtime -- `bff-backoffice` can persist shared ai-engine presets and service-target overrides -- `ai-engine-api` can hold the active llama target at runtime -- staging can therefore operate in split-AI mode even while the rest of the services remain auto-deployed on k3s +- `ai-engine-api` can hold the active llama target at runtime (the only runtime routing state) +- `api-gateway` and `bff-backoffice` hold none +- a deployment can therefore run in split-AI mode, with the llama runtime outside the cluster, even though the manifests describe in-cluster services ## Current CI/CD automation map diff --git a/architecture/target-architecture.md b/architecture/target-architecture.md index 4ef5da5..ac3f285 100644 --- a/architecture/target-architecture.md +++ b/architecture/target-architecture.md @@ -143,24 +143,22 @@ That distinction is important: deployment topology and effective runtime topolog The current architecture contains a narrow operational state plane outside static manifests: -- `bff-backoffice` persists service-target overrides and shared ai-engine presets -- `api-gateway` persists the live ai-engine API and stats target used by forwarded runtime traffic -- `ai-engine-api` persists the active llama target used by generation requests +- `ai-engine-api` persists the active llama target used by generation requests (`AI_ENGINE_LLAMA_TARGET_STATE_FILE`) -These states survive pod recreation through mounted storage and must be documented as part of the effective topology. +`api-gateway` and `bff-backoffice` hold no routing state: the gateway resolves ai-engine upstreams from `AI_ENGINE_API_URL` and `AI_ENGINE_STATS_URL`. Service-target overrides, shared presets and connections, and the gateway live target were removed. + +The llama target survives pod recreation through mounted storage and must be documented as part of the effective topology. ### Runtime routing control plane -Backoffice operational tooling now acts as a lightweight runtime control plane for non-static upstream selection. +Backoffice operational tooling acts as a lightweight runtime control plane for exactly one setting, the `ai-engine-api` llama target. Current capabilities: -- `bff-backoffice` can persist shared upstream routing overrides -- `api-gateway` can persist the live `ai-engine` target used by the platform -- backoffice operators can select and manage reusable ai-engine destination presets -- routing state survives pod recreation because it is stored on mounted persistent volumes +- backoffice operators can read, probe, set and reset the llama target through `bff-backoffice` (`GET/PUT/DELETE /v1/backoffice/ai-engine/target`, `POST /v1/backoffice/ai-engine/probe`) +- `ai-engine-api` persists the override on a mounted volume, so it survives pod recreation -This control-plane behavior is intentionally narrow in scope: it does not replace GitOps or declarative infrastructure, but it provides pragmatic runtime flexibility for operational AI connectivity. +This control-plane behavior is intentionally narrow in scope: it does not replace GitOps or declarative infrastructure. See [Runtime routing and service targeting](../operations/runtime-routing-and-service-targeting.md). ### Delivery architecture @@ -196,13 +194,13 @@ This preserves local service autonomy without fragmenting the deployment contrac ### Traceability -- automatic staging deploys are tied to immutable image SHAs +- images are tagged with immutable short SHAs; deploys are manual-only, so the tag a deploy uses is chosen by the operator - service CI, central packaging, and deployment workflows form a clear audit chain ### Operability - smoke tests and rollout checks are embedded in deployment workflows -- runtime overrides for AI connectivity can be changed without rebuilding or redeploying the entire stack +- the llama target can be changed without rebuilding or redeploying the stack ### Evolvability @@ -211,9 +209,9 @@ This preserves local service autonomy without fragmenting the deployment contrac ### Current architectural constraints -- staging automatic deploys are pinned to immutable short-SHA artifacts, but manual recovery paths can still use mutable environment tags -- runtime overrides introduce a second operational state plane that must be documented and observed -- not every repository in the workspace is part of the GHCR-to-k3s delivery chain +- deploys are manual-only (`deploy.yaml` `workflow_dispatch`), so image tags are chosen by the operator; the thesis-era automatic path pinned immutable short-SHA artifacts +- the llama target override introduces a second operational state plane that must be documented and observed +- not every repository in the workspace is part of the GHCR image-build chain - AI runtime behavior in staging must always be described in terms of both in-cluster services and external llama reachability ### Documentation implications diff --git a/architecture/use-cases.md b/architecture/use-cases.md index 6d64147..6b74ca1 100644 --- a/architecture/use-cases.md +++ b/architecture/use-cases.md @@ -69,10 +69,10 @@ This document enumerates the primary use cases supported by the AxiomNode platfo ### UC-09 Change runtime AI target (Backoffice Operator) -- Trigger: Operator selects a new ai-engine-api/stats target or llama upstream from the backoffice control plane. -- Path: `backoffice -> bff-backoffice -> api-gateway` (persists active target on PV) and/or `bff-backoffice -> ai-engine-api` (persists active llama target). +- Trigger: Operator selects a new llama upstream from the backoffice control plane. +- Path: `backoffice -> api-gateway -> bff-backoffice -> ai-engine-api` (persists active llama target on a mounted volume). - Post-conditions: Effective routing diverges from declarative manifests; persisted on mounted volume; survives pod recreation. -- Risk: state drift versus manifests; must be observable via stats endpoints and runbook. +- Risk: state drift versus manifests; must be observable via `GET /v1/backoffice/ai-engine/target` and the runbook. ### UC-10 Generate quiz/wordpass via internal service (Internal Service) diff --git a/guides/capabilities/ai/ai-content-generation.md b/guides/capabilities/ai/ai-content-generation.md index 8915fc1..36257c6 100644 --- a/guides/capabilities/ai/ai-content-generation.md +++ b/guides/capabilities/ai/ai-content-generation.md @@ -1,6 +1,6 @@ # AI Content Generation -Last updated: 2026-05-03. +Last updated: 2026-11-14. ## Purpose @@ -46,7 +46,7 @@ This capability turns internal generation requests into validated game payloads, | HTTP route | `POST /internal/ai-engine/generate/word-pass` in api-gateway | internal proxy to `POST /generate/word-pass` in ai-engine | | HTTP route | `POST /internal/ai-engine/ingest/quiz` and `POST /internal/ai-engine/ingest/word-pass` in api-gateway | bridge-only ingest into ai-engine RAG | | HTTP route | `GET /internal/ai-engine/catalogs` and `GET /internal/ai-engine/health` in api-gateway | catalog discovery and liveness for internal consumers | -| HTTP route | `GET/PUT/DELETE /v1/backoffice/ai-engine/target`, `GET/POST/PUT/DELETE /v1/backoffice/ai-engine/presets*`, `POST /v1/backoffice/ai-engine/probe` in bff-backoffice | operator control plane around the same runtime | +| HTTP route | `GET/PUT/DELETE /v1/backoffice/ai-engine/target`, `POST /v1/backoffice/ai-engine/probe` in bff-backoffice | operator control for the llama target of the same runtime | | Service method | `GenerationService.generateAndStore*()` in quiz and word-pass microservices | main domain-side consumer, dedup, persistence, batch orchestration | ### Inputs @@ -55,7 +55,7 @@ This capability turns internal generation requests into validated game payloads, - `X-API-Key` with games or bridge scope depending on route family - `X-Game-Language` and `X-Difficulty-Percentage` headers resolved by ai-engine generation routes - ingest document batches with optional `X-Ingest-Source` override for telemetry -- operator-provided target/preset payloads from backoffice for llama target changes and connectivity probes +- operator-provided llama target payloads from backoffice for llama target changes and connectivity probes ### Outputs @@ -76,7 +76,7 @@ This capability turns internal generation requests into validated game payloads, 1. A domain service builds a request in `GenerationService.generateAndStoreWithResult()`, adding `query`, `item_count`, fixed `max_tokens`, and optional `difficulty_percentage`. 2. The shared `AiEngineClient` sends the request to the internal ai-engine base URL, applies retry/backoff, and caches catalog lookups for 60 seconds by default. -3. api-gateway forwards `/internal/ai-engine/*` to ai-engine API or ai-engine stats, using persisted target overrides when present. +3. api-gateway forwards `/internal/ai-engine/*` to ai-engine API or ai-engine stats, resolving the upstreams from `AI_ENGINE_API_URL` and `AI_ENGINE_STATS_URL`. 4. ai-engine generation routes normalize inputs through query params plus `X-Game-Language` and `X-Difficulty-Percentage`, then call `_execute_generate()`. 5. `_execute_generate()` enforces rate and capacity limits, resolves context through the RAG pipeline when available, invokes the wrapped generator, validates the structured payload, records a local event, and forwards observability data to ai-stats on a best-effort basis. 6. The quiz and word-pass microservices canonicalize the response again, compute a content-based uniqueness key, reject duplicates, and persist only valid new models. @@ -97,11 +97,11 @@ This capability turns internal generation requests into validated game payloads, - generation capacity limiter tracks in-flight and queued work; health surfaces that dependency status explicitly - microservice auth circuit state tracks failure streak, cooldown, open-until timestamp, and total openings - catalogs in the microservices can come from `ai-engine` or fall back locally when remote lookup fails -- api-gateway and bff-backoffice may repoint ai-engine API and stats targets without changing domain service code +- operators may repoint the llama target (held by `ai-engine-api`) without changing domain service code; the gateway and the BFF hold no routing state ### State risks -- target drift between ai-engine internal llama target and gateway/BFF routing overrides can produce confusing partial failures +- drift between the persisted llama target override and the environment default can produce confusing partial failures after a redeploy - stale category caches in consumers can hide upstream recovery or degradation for up to the configured TTL - RAG diagnostics are cached and explicitly invalidated on ingest; broken invalidation would surface stale health data - the auth circuit in domain services can intentionally block generation even after ai-engine recovers until cooldown expires @@ -118,7 +118,7 @@ This capability turns internal generation requests into validated game payloads, - ai-engine emits local collector events and forwards them to ai-stats via `_publish_event_to_stats()` when a stats URL is configured - `GET /internal/ai-engine/stats` in api-gateway points to ai-engine stats rather than the main FastAPI app - `GET /internal/cache/stats` and `POST /internal/cache/reset` expose cache runtime controls for monitoring bridges -- backoffice target, preset, and probe routes provide operator-level checks before switching runtime targets +- backoffice target and probe routes provide operator-level checks before switching runtime targets ### Verification evidence @@ -130,7 +130,7 @@ This capability turns internal generation requests into validated game payloads, | Failure mode | Likely cause | User/operator impact | Detection path | Recovery path | |---|---|---|---|---| -| Unreachable ai-engine API or llama target | bad target override, gateway routing mismatch, runtime outage | generation fails or backoffice probes return invalid target errors | ai-engine `/health`, backoffice probe, gateway proxy failures, outbound request metrics | reset target, restore runtime, or reapply a known-good preset | +| Unreachable ai-engine API or llama target | bad llama target override, gateway upstream misconfiguration, runtime outage | generation fails or backoffice probes return invalid target errors | ai-engine `/health`, backoffice probe, gateway proxy failures, outbound request metrics | reset the target (`DELETE`), restore the runtime, or reapply a known-good target | | Auth circuit opens in game services | repeated `401/403` or auth failures from ai-engine | quiz or word-pass generation is blocked before new upstream calls are attempted | `GenerationService` circuit snapshot, observer callbacks, test-covered error string `AI auth circuit open until ...` | fix API keys, wait for cooldown, or restart the affected microservice if operationally needed | | Invalid structured output | model drift, prompt regression, or schema mismatch | payload is rejected or not stored downstream | canonicalization/validation failures, model-failed callbacks, history validation errors | adjust prompts or validators and retry with smoke checks | | Stale diagnostics or catalogs | cache reuse without timely refresh | operators see outdated health/catalog state | RAG stats cache behavior, catalog timestamps, backoffice catalog view | force refresh via consumer call paths or invalidate cache through internal controls | @@ -139,7 +139,7 @@ This capability turns internal generation requests into validated game payloads, ## Security and access model - ai-engine keeps `/health` and `/ready` public for probes, but generation routes require a games or bridge key and ingest routes require the bridge key specifically -- backoffice target and preset mutations are validated with Zod schemas in bff-backoffice before they can affect runtime routing +- backoffice target mutations are validated with Zod schemas in bff-backoffice before they can affect runtime routing - api-gateway exposes ai-engine generation only on internal routes; the public edge does not call ai-engine directly - prompt inputs, ingest documents, and generated payloads are normalized and canonicalized before persistence to reduce malformed-content and duplicate-content risks @@ -148,7 +148,7 @@ This capability turns internal generation requests into validated game payloads, ### Deployment impact - ai-engine API, ai-engine stats, and llama runtime are separable runtime units, so a healthy API deployment does not guarantee a healthy model target -- gateway routing overrides and BFF-managed presets can redirect traffic without redeploying the game services +- the llama target override (held by `ai-engine-api`) can redirect generation traffic without redeploying the game services - changes to shared client retries, validation helpers, or prompt shaping affect both quiz and word-pass behavior immediately after rollout ### Rollback impact @@ -168,7 +168,7 @@ This capability turns internal generation requests into validated game payloads, - the quiz tests still assert effective language as English on model-specific generation paths even when the route defaults use `X-Game-Language=es`; that contract should be clarified explicitly at the platform level - true-false generation and ingest routes exist in ai-engine, but no microservice, BFF route or mobile game uses them (checked on `origin/main`, 2026-11-12) and the gateway does not proxy them; the platform only plays quiz and word-pass -- the operational boundary between ai-engine local llama-target state and gateway/BFF ai-engine target overrides is functional but still easy to misread during incident response +- the persisted llama target override outlives pod restarts and image rollbacks, so it is easy to forget during incident response; reset it explicitly ## References diff --git a/guides/capabilities/ai/ai-observability-and-diagnostics.md b/guides/capabilities/ai/ai-observability-and-diagnostics.md index 27e05f1..2ac0b8f 100644 --- a/guides/capabilities/ai/ai-observability-and-diagnostics.md +++ b/guides/capabilities/ai/ai-observability-and-diagnostics.md @@ -1,6 +1,6 @@ # AI Observability And Diagnostics -Last updated: 2026-05-03. +Last updated: 2026-11-14. ## Purpose @@ -24,7 +24,7 @@ This capability exposes the operational picture of AI behavior to operators by c - owns operator-facing AI diagnostics and aggregated AI visibility - consumes generation/runtime signals from the AI content-generation capability -- reads routing and preset state but does not own all runtime-target mutation paths itself +- reads the llama target state but does not own the runtime-target mutation path itself (`ai-engine-api` persists it) ## Entrypoints and contracts @@ -33,14 +33,14 @@ This capability exposes the operational picture of AI behavior to operators by c | Type | Location | Notes | |---|---|---| | HTTP route | ai-engine stats endpoints | aggregate AI runtime and event signals | -| HTTP route | BFF AI target, preset, and diagnostics routes | operator aggregation and enrichment path | +| HTTP route | BFF AI target, probe, and diagnostics routes | operator aggregation and enrichment path | | UI flow | `AIDiagnosticsPanel.tsx` and related backoffice screens | interactive operator surface | ### Inputs - AI event streams and stats queries emitted by ai-engine - cache runtime data from ai-engine internal endpoints -- routing and preset state from BFF routing stores +- the effective llama target read back from `ai-engine-api` through the BFF - operator requests initiated from the backoffice UI ### Outputs @@ -60,7 +60,7 @@ This capability exposes the operational picture of AI behavior to operators by c 1. ai-engine records local events and forwards them to ai-stats on a best-effort basis. 2. ai-engine stats exposes aggregate runtime information for internal consumers. -3. `bff-backoffice` reads AI target, preset, probe, and diagnostics data and enriches it with routing context. +3. `bff-backoffice` reads AI target, probe, and diagnostics data and enriches it with routing context. 4. `backoffice` fetches those summaries and renders diagnostic panels for operators. 5. Operators compare health, routing, and runtime evidence before taking action. @@ -69,7 +69,7 @@ This capability exposes the operational picture of AI behavior to operators by c ### Owned persistence - collector state in ai-stats -- routing and preset state surfaced by `bff-backoffice` +- none for routing: the BFF keeps no routing state (the llama target is persisted by `ai-engine-api`) ### Runtime state @@ -95,7 +95,7 @@ This capability exposes the operational picture of AI behavior to operators by c - `AIDiagnosticsPanel.tsx` - `fetchServiceOperationalSummary()` and related backoffice operational summary calls -- BFF AI target, preset, and probe endpoints +- BFF AI target and probe endpoints ### Verification evidence diff --git a/guides/capabilities/edge/public-edge-routing.md b/guides/capabilities/edge/public-edge-routing.md index 9feac90..73e1553 100644 --- a/guides/capabilities/edge/public-edge-routing.md +++ b/guides/capabilities/edge/public-edge-routing.md @@ -126,7 +126,7 @@ This capability provides the stable ingress boundary for mobile and backoffice c ### Deployment impact - gateway deployment can change public behavior independently of BFF or ai-engine deploys -- AI behavior at the edge depends on both deployed gateway code and persisted target state +- AI behavior at the edge depends on deployed gateway code and the environment-provided upstream URLs; the gateway keeps no persisted target state ### Rollback impact diff --git a/guides/capabilities/operations/backoffice-operational-diagnostics.md b/guides/capabilities/operations/backoffice-operational-diagnostics.md index 6a60c99..87b256d 100644 --- a/guides/capabilities/operations/backoffice-operational-diagnostics.md +++ b/guides/capabilities/operations/backoffice-operational-diagnostics.md @@ -1,10 +1,10 @@ # Backoffice Operational Diagnostics -Last updated: 2026-05-03. +Last updated: 2026-11-14. ## Purpose -This capability gives operators a single UI and BFF-backed surface to inspect service health, logs, metrics, the packaged backoffice version, Kubernetes state, and AI runtime context. +This capability gives operators a single UI and BFF-backed surface to inspect service health, logs, metrics, the packaged backoffice version, and AI runtime context. ## Functional objective @@ -26,13 +26,12 @@ This capability gives operators a single UI and BFF-backed surface to inspect se - `microservice-wordpass` - `ai-engine` - `api-gateway` -- Kubernetes observability inputs ### Explicit boundaries - owns operator-facing visualization in backoffice panels such as `ServiceOverviewPanel.tsx` and `ServiceConsolePanel.tsx` - owns BFF aggregation logic in `bff-backoffice/src/app/routes/backoffice.ts` -- reads Kubernetes state and exposes the packaged backoffice version, but does not own the health of the underlying services it displays +- exposes the packaged backoffice version, but does not own the health of the underlying services it displays - overlaps with runtime routing control only where diagnostics must reflect current effective targets ## Entrypoints and contracts @@ -42,21 +41,20 @@ This capability gives operators a single UI and BFF-backed surface to inspect se | Type | Location | Notes | |---|---|---| | UI flow | backoffice dashboards and operational panels | operator entry surface | -| HTTP route | BFF metrics, logs, data, and Kubernetes routes | aggregated diagnostics data | +| HTTP route | BFF metrics, logs, and data routes | aggregated diagnostics data | | Service call | backoffice `operationalSummary.ts` | front-end composition of diagnostics calls | ### Inputs - operator auth/session state from backoffice - downstream service metrics, logs, and data endpoints -- Kubernetes overview data from `kubernetesObservability.ts` - effective target state and AI probe results where relevant ### Outputs - service-overview panels and drill-down console views - enriched diagnostics summaries through the BFF -- operator-visible packaged version and Kubernetes overview payloads +- operator-visible packaged version ### Contract sources @@ -69,34 +67,32 @@ This capability gives operators a single UI and BFF-backed surface to inspect se 1. An operator opens a diagnostics screen in backoffice. 2. The UI calls gateway-exposed backoffice routes that land in `bff-backoffice`. -3. The BFF queries downstream services and Kubernetes observability helpers. -4. The UI merges those responses with the packaged version into operator panels for health, logs, metrics, deployments, and runtime topology. -5. Operators use the same surface to correlate a visible issue with the current effective routing and deployment state. +3. The BFF queries downstream services. +4. The UI merges those responses with the packaged version into operator panels for health, logs, metrics, and the llama target. +5. Operators use the same surface to correlate a visible issue with the current llama target. ## Persistence and runtime state ### Owned persistence -- UI screen state and BFF-managed routing/preset state used by diagnostics panels +- UI screen state only: the BFF keeps no routing state ### Runtime state - downstream availability directly affects what panels can render -- Kubernetes overview can become stale relative to the actual effective routing path - operator screen state may lag behind new probes or target changes until refreshed ### State risks - partial downstream outage makes the diagnostics picture incomplete or inconsistent - missing proxy exposure at gateway level breaks whole panels despite healthy downstream code -- stale deployment or routing context leads to wrong incident conclusions +- stale llama target context leads to wrong incident conclusions ## Observability evidence ### Health signals - backoffice and BFF health -- Kubernetes overview route availability ### Operational signals @@ -116,7 +112,7 @@ This capability gives operators a single UI and BFF-backed surface to inspect se |---|---|---|---|---| | Missing proxy or route exposure | gateway or BFF route drift | diagnostics page fails even if the underlying service is healthy | UI error state, edge 404, route smoke checks | restore the route chain and revalidate | | Partial downstream visibility | one downstream data source unavailable | operator sees incomplete runtime picture | missing panels and failed BFF calls | degrade clearly and restore the missing source | -| Stale deployment or routing context | cached or unrefreshed panel state | wrong operational action | compare diagnostics output with direct target reads | refresh and reconcile state | +| Stale llama target context | cached or unrefreshed panel state | wrong operational action | compare diagnostics output with direct target reads | refresh and reconcile state | ## Security and access model diff --git a/guides/capabilities/platform/deployment-and-rollout-automation.md b/guides/capabilities/platform/deployment-and-rollout-automation.md index 97c7d1a..d549ecd 100644 --- a/guides/capabilities/platform/deployment-and-rollout-automation.md +++ b/guides/capabilities/platform/deployment-and-rollout-automation.md @@ -76,7 +76,7 @@ This capability packages runtime services, publishes images, and rolls environme - deployed cluster state after rollout - rollout behavior depends on current image, secret, and cluster conditions -- runtime-state holders such as gateway/BFF overrides can change effective behavior after deploy +- the persisted llama target override in `ai-engine-api` can change effective behavior after deploy ### State risks diff --git a/guides/communication/local-edge-integration.md b/guides/communication/local-edge-integration.md index b2c55cc..a138cb4 100644 --- a/guides/communication/local-edge-integration.md +++ b/guides/communication/local-edge-integration.md @@ -59,8 +59,7 @@ The smoke now covers: - authenticated mobile profile read/update/read round-trip with persisted `playerId`, `displayName`, `preferredLanguage`, timestamp presence, and zero-state stats semantics - deferred gameplay event sync with converged profile stats and duplicate-batch deduplication - direct quiz AI generation through `microservice-quizz:/games/generate` and `microservice-quizz:/games/generate/process/wait`, followed by observable history growth and inventory selection through `/games/models/random` -- runtime service-target override for `microservice-quiz`, followed by an affected catalogs request and rollback to `env` -- dedicated `ai-engine target` override/readback/reset through backoffice routing controls +- `ai-engine target` override/readback/reset through backoffice routing controls - mobile quiz and word-pass discovery routes - selected mobile generate routes with invalid-payload rejection and playable response-envelope checks - selected backoffice leaderboard and monitoring routes diff --git a/guides/foundations/ai-plugin-architecture.md b/guides/foundations/ai-plugin-architecture.md index 2401353..87afcb6 100644 --- a/guides/foundations/ai-plugin-architecture.md +++ b/guides/foundations/ai-plugin-architecture.md @@ -97,7 +97,7 @@ Best fit: ```text VPS platform - -> gateway/BFF target override + -> llama target override (ai-engine-api) -> plugin ai-engine-api -> plugin ai-engine-stats -> plugin vector store diff --git a/guides/foundations/platform-capability-matrix.md b/guides/foundations/platform-capability-matrix.md index 61a741e..ffa6b3c 100644 --- a/guides/foundations/platform-capability-matrix.md +++ b/guides/foundations/platform-capability-matrix.md @@ -1,6 +1,6 @@ # Platform Capability Matrix -Last updated: 2026-05-03. +Last updated: 2026-11-14. ## Purpose @@ -12,17 +12,17 @@ Use this document when the goal is to analyze a capability in terms of inputs, o | Capability | Main owners | Typical inputs | Main outputs | Owned persistence/state | Main observability signals | Main risks/failure modes | |---|---|---|---|---|---|---| -| Public edge routing | `api-gateway` | mobile/backoffice HTTP requests, auth headers, correlation IDs | stable public responses, forwarded traffic to BFFs, internal AI proxy calls | gateway routing state for active AI target | gateway health, edge logs, forwarding errors, correlation-id traces | auth rejection, bad upstream target, timeout to BFF, route drift | +| Public edge routing | `api-gateway` | mobile/backoffice HTTP requests, auth headers, correlation IDs | stable public responses, forwarded traffic to BFFs, internal AI proxy calls | none (the gateway keeps no routing state) | gateway health, edge logs, forwarding errors, correlation-id traces | auth rejection, bad upstream target, timeout to BFF, route drift | | Mobile gameplay orchestration | `bff-mobile`, `mobile-app` | player actions, bearer token, category/language filters | playable quiz and word-pass sessions, profile reads/updates, synced events | mobile local state, BFF request shaping, player continuity on client | mobile app logs, gateway/BFF traces, sync responses | contract mismatch, offline sync divergence, invalid payload shaping | | Player identity and profile | `microservice-users`, `bff-mobile`, `bff-backoffice` | Firebase tokens, profile updates, gameplay event batches | player profile, stats snapshots, session-linked user identity | PostgreSQL user/profile/event data | auth/session logs, leaderboard reads, stats endpoints | invalid auth, duplicate events, inconsistent profile aggregation | | Quiz inventory and generation | `microservice-quizz`, `ai-engine` | generation requests, category/language filters, ingest payloads | quiz models, curated/random quiz inventory, generation results | PostgreSQL quiz inventory/history | health, private docs, monitor stats, generation logs | invalid generated payload, persistence failure, empty inventory, AI timeout | | Word-pass inventory and generation | `microservice-wordpass`, `ai-engine` | generation requests, language/category filters, ingest payloads | word-pass models, curated/random inventory, generation results | PostgreSQL word-pass inventory/history | health, private docs, monitor stats, generation logs | malformed hints/answers, persistence failure, AI timeout, stale data | | AI content generation | `ai-engine-api`, llama runtime | generation prompts, ingest requests, internal API keys, active model target | validated AI payloads, generation artifacts, ingest results | active llama target, cache state, model/runtime state | `/health`, stats API, AI event collector, cache/runtime metrics | unreachable llama target, invalid JSON output, resource pressure, cache degradation | -| AI observability and diagnostics | `ai-engine-stats`, `bff-backoffice`, `backoffice` | AI event streams, stats queries, operator diagnostics requests | aggregated AI stats, diagnostics views, operator summaries | collector state, persisted presets and operator-visible runtime state | stats endpoints, dashboards, backoffice diagnostic views | partial stats visibility, stale routing picture, split-topology confusion | -| Runtime routing control | `bff-backoffice`, `api-gateway`, `ai-engine` | operator target changes, presets, service override updates | effective target changes, synced routing state, updated runtime topology | persisted gateway AI target, backoffice service-target overrides/presets, ai-engine model target | backoffice routing views, gateway target APIs, health checks after change | effective topology differs from manifests, wrong host configured, restart-state mismatch | -| Backoffice operational diagnostics | `backoffice`, `bff-backoffice` | operator auth/session, metrics/logs/data queries, deployment inspection requests | dashboards, service metrics/logs, packaged backoffice version, Kubernetes overview | UI state, BFF routing state, shared AI presets | service metrics/logs endpoints, Kubernetes overview, backoffice dashboards | missing proxy route, partial downstream visibility, role-gating mistakes | +| AI observability and diagnostics | `ai-engine-stats`, `bff-backoffice`, `backoffice` | AI event streams, stats queries, operator diagnostics requests | aggregated AI stats, diagnostics views, operator summaries | collector state (no routing state in the BFF) | stats endpoints, dashboards, backoffice diagnostic views | partial stats visibility, stale routing picture, split-topology confusion | +| Runtime routing control | `bff-backoffice`, `ai-engine` | operator llama target changes (set, probe, reset) | effective llama target, updated runtime topology | ai-engine persisted llama target (`AI_ENGINE_LLAMA_TARGET_STATE_FILE`) | `GET /v1/backoffice/ai-engine/target`, probe route, health checks after change | effective topology differs from manifests, wrong host configured, restart-state mismatch | +| Backoffice operational diagnostics | `backoffice`, `bff-backoffice` | operator auth/session, metrics/logs/data queries | dashboards, service metrics/logs, packaged backoffice version | UI state | service metrics/logs endpoints, backoffice dashboards | missing proxy route, partial downstream visibility, role-gating mistakes | | Contract governance | `contracts-and-schemas`, `shared-sdk-client` | API design changes, schema updates, examples, CI validation inputs | OpenAPI specs, JSON schemas, event contracts, generated SDK artifacts | versioned schema files and generated SDK outputs | CI contract validation, SDK generation jobs | contract drift, SDK lag, incompatible changes between repos | -| Deployment and rollout automation | `platform-infra` | service-repo dispatches, manifests, image tags, environment overlays | GHCR images, staging rollouts, deployment receipts | manifests, overlays, workflow state, release metadata | GitHub Actions runs, rollout logs, smoke checks | build failure, deploy failure, overlay mismatch, staged image not healthy | +| Deployment and rollout automation | `platform-infra` | service-repo dispatches, manifests, image tags, environment overlays | GHCR images, manual deploys (`deploy.yaml` `workflow_dispatch`; no staging host today), deployment receipts | manifests, overlays, workflow state, release metadata | GitHub Actions runs, rollout logs, smoke checks | build failure, deploy failure, overlay mismatch, staged image not healthy | | Monitoring and alerting | `observability-platform` | service metrics, logs, traces, alert definitions | dashboards, alert rules, monitoring stack assets | dashboard JSON/YAML, alert rules, stack configuration | Prometheus/Grafana/alert outputs, dashboard panels | missing telemetry, noisy alerts, blind spots in runtime coverage | | Secrets governance | `secrets` | environment secret values, placeholder definitions, audit scripts | prepared secret bundles, validated placeholders, distribution material | secret inventories and environment-specific secret sources | audit scripts, placeholder validation outputs, workflow checks | drift between placeholders and real secrets, accidental exposure, broken environment bootstrap | | Documentation and engineering guidance | `docs` | architecture changes, workflow updates, repository facts | architecture guides, runbooks, governance rules, cross-repo summaries | versioned Markdown docs and ADRs | markdown-link checks, contributor usage | stale guidance, undocumented cross-repo changes, conflicting repo-local docs | diff --git a/guides/foundations/platform-functional-overview.md b/guides/foundations/platform-functional-overview.md index 3ff8237..a2980f7 100644 --- a/guides/foundations/platform-functional-overview.md +++ b/guides/foundations/platform-functional-overview.md @@ -1,6 +1,6 @@ # Platform Functional Overview -Last updated: 2026-05-03. +Last updated: 2026-11-14. ## Purpose @@ -42,7 +42,7 @@ The workspace contains four broad kinds of assets: - operational dashboards for service health and traffic - runtime inspection of services, metrics, logs, and data inventories -- control of effective AI routing and service-target overrides +- control of the ai-engine llama target (read, probe, set, reset) - management of generation workflows and curated content inventories ### Platform-facing capabilities @@ -64,7 +64,7 @@ The workspace contains four broad kinds of assets: - applies edge concerns such as auth, CORS, tracing, and forwarding policy - routes traffic to `bff-mobile` and `bff-backoffice` - proxies internal AI generation, ingest, catalogs, health, and stats traffic -- persists the live `ai-engine` upstream target used at runtime +- resolves the `ai-engine` upstreams from environment variables only; it holds no runtime override or persisted routing state **Main dependencies:** `bff-mobile`, `bff-backoffice`, active `ai-engine-api`, active `ai-engine-stats` @@ -118,10 +118,8 @@ The workspace contains four broad kinds of assets: - aggregates and normalizes data from internal services for backoffice screens - exposes admin APIs for auth, monitoring, service inspection, and generation flows -- persists service-target overrides and shared AI presets -- synchronizes effective AI target changes toward `api-gateway` -- exposes effective runtime topology, not only environment defaults -- provides Kubernetes and AI observability surfaces for operators +- proxies the ai-engine llama target control (read, probe, set, reset) to `ai-engine-api`; it keeps no routing state +- provides AI observability surfaces for operators **Main dependencies:** `microservice-users`, `microservice-quizz`, `microservice-wordpass`, `ai-engine`, `api-gateway` @@ -218,12 +216,12 @@ The workspace contains four broad kinds of assets: - owns Kubernetes manifests and overlays per environment - builds and publishes runtime images to GHCR -- drives automatic staging rollout for covered services +- builds and publishes images for covered services on dispatch; deploys are manual-only (`deploy.yaml` `workflow_dispatch`) - validates deployment inputs and rollout steps - supports local full-stack or partial-stack development tooling - provides optional environment overlays for alternative runtime topologies -**Main consumers:** all auto-deployed runtime repositories +**Main consumers:** all runtime repositories that publish images ### `observability-platform` @@ -311,8 +309,8 @@ The individual capability dossiers are indexed in [../capabilities/README.md](.. 1. `backoffice` calls `api-gateway`. 2. `api-gateway` routes to `bff-backoffice`. -3. `bff-backoffice` aggregates health, logs, metrics, inventory, routing, and AI diagnostics. -4. Runtime control changes may propagate to `api-gateway` and affect effective AI routing. +3. `bff-backoffice` aggregates health, logs, metrics, inventory, and AI diagnostics. +4. Llama target changes are forwarded to `ai-engine-api` and affect effective AI generation routing. ### Delivery flow @@ -320,15 +318,13 @@ The individual capability dossiers are indexed in [../capabilities/README.md](.. 2. Shared integration code is generated or updated in `shared-sdk-client`. 3. Runtime repositories consume those artifacts. 4. Pushes to covered runtime repos dispatch builds to `platform-infra`. -5. `platform-infra` publishes images and rolls out staging. +5. `platform-infra` publishes images to GHCR; deployment is a manual `deploy.yaml` dispatch (no staging host exists today). ## Repositories with persisted runtime influence Some repositories do more than serve requests; they also hold runtime state that changes effective behavior: -- `api-gateway`: active AI upstream target -- `bff-backoffice`: service-target overrides and shared AI presets -- `ai-engine`: active llama/model target +- `ai-engine`: active llama/model target (the only one; `api-gateway` and `bff-backoffice` hold no routing state) This matters because the effective runtime topology can differ from the static Kubernetes manifests. diff --git a/guides/governance/repository-standards-adoption.md b/guides/governance/repository-standards-adoption.md index 6afd21c..b373e78 100644 --- a/guides/governance/repository-standards-adoption.md +++ b/guides/governance/repository-standards-adoption.md @@ -1,6 +1,8 @@ # Repository Standards Adoption Guide -Last updated: 2026-04-22. +Last updated: 2026-11-14. + +> **Status (2026-11-14):** the k3s staging host no longer exists and `platform-infra` `deploy.yaml` is manual-only (`workflow_dispatch`), so no automatic staging deployment runs today. Where this document describes an automatic staging chain, it records the thesis-era policy; the image build on dispatch is the only automatic step. ## Purpose diff --git a/guides/validation/critical-workflow-smoke-matrix.md b/guides/validation/critical-workflow-smoke-matrix.md index 6e659df..ac0cbf3 100644 --- a/guides/validation/critical-workflow-smoke-matrix.md +++ b/guides/validation/critical-workflow-smoke-matrix.md @@ -27,7 +27,7 @@ This document complements [Primary validation workflows](./primary-workflows-val | `WF-03` Mobile game start from inventory-backed generate routes | prove mobile generate routes remain inventory-backed and do not fall through to runtime AI generation | `platform-infra/environments/dev/scripts/smoke-edge.sh` and `secrets/scripts/run-nightly-edge-smoke.sh` now reject invalid generate payloads with `400` and require valid generate responses to return `{ gameType, generated }` for both quiz and word-pass; repository tests still prove that `bff-mobile` calls `/games/models/random` in `bff-mobile/src/tests/mobile.test.ts` | Partial | The smoke now discriminates route validation and playable envelope shape, but it still does not prove from runtime logs or upstream traces that the request stayed on `/games/models/random` without any hidden live-generation fallback. | | `WF-04` Offline play and deferred event sync | prove continuity offline and reconciliation on reconnect | `platform-infra/environments/dev/scripts/smoke-edge.sh` and `secrets/scripts/run-nightly-edge-smoke.sh` now post a deferred event batch to `/v1/mobile/games/events`, verify converged profile stats, and replay the same batch to confirm deduplication; mobile unit tests still cover offline cache behavior in `mobile-app/App/composeApp/src/commonTest/kotlin/es/sebas1705/axiomnode/data/repositories/GamesRepositoryOfflineFallbackTest.kt` and `GamesRepositoryCatalogCacheTest.kt` | Automated | The smoke now proves reconciliation and deduplication through the edge path, but it still does not exercise the actual client-side offline banner, local queue persistence, or reconnect trigger UX. | | `WF-05` AI content production for catalog refresh | prove AI-generated content enters quiz/word-pass inventory and becomes selectable by mobile-facing routes | `platform-infra/environments/dev/scripts/smoke-edge.sh` and `secrets/scripts/run-nightly-edge-smoke.sh` now require `ai-engine-api` health, execute both `microservice-quizz:/games/generate` and `microservice-quizz:/games/generate/process/wait`, verify `task.created >= 1`, confirm `GET /games/history` grows by that amount, and re-check selectable inventory via `GET /games/models/random`; `platform-infra/environments/dev/scripts/smoke-wf05-quiz.sh` now narrows `random` to the exact process window and requires the returned item id to belong to freshly persisted quiz rows; `platform-infra/environments/dev/scripts/smoke-wf05-wordpass.sh` offers the equivalent word-pass chain when `microservice-wordpass` is started with `BATCH_GENERATION_ENABLED=false`; `ai-engine/src/scripts/install/smoke_test.sh` and repository tests still cover segmented AI auth, catalog refresh, and generation internals | Partial | The narrow smokes now prove direct quiz and word-pass AI generation, observable persistence, and exact random traceability, but the full edge smoke still does not cover word-pass refresh and the workflow still does not replay the refreshed inventory through a full mobile chain in the same run. | -| `WF-06` Runtime routing diagnostics and rollback | prove operator override, smoke use of the affected route, and rollback to defaults | `platform-infra/environments/dev/scripts/smoke-edge.sh` and `secrets/scripts/run-nightly-edge-smoke.sh` now cover both variants: service-target override via `/v1/backoffice/service-targets/:service` plus affected catalogs request, and dedicated AI-target override via `/v1/backoffice/ai-engine/target` with readback and reset to `env`; repository tests still cover local state holders in `api-gateway`, `bff-backoffice`, and `ai-engine` | Automated | The smoke now covers both operator-facing override families, but it still does not detect split-brain mismatches across BFF, gateway, and ai-engine holders after partial sync failures. | +| `WF-06` Runtime routing diagnostics and rollback | prove operator override, smoke use of the affected route, and rollback to defaults | `platform-infra/environments/dev/scripts/smoke-edge.sh` and `secrets/scripts/run-nightly-edge-smoke.sh` cover the llama target override via `/v1/backoffice/ai-engine/target` with readback and reset to `env` (the service-target override variant was removed with the routing plane); repository tests still cover the state holder in `ai-engine` | Automated | The smoke covers the one remaining operator-facing override, but it does not check that a smoke request actually reaches the overridden llama runtime. | ## Existing automation surfaces @@ -69,7 +69,7 @@ These tests do not replace a cross-repository smoke, but they narrow the uncerta 1. Add one reconnect-oriented executable flow for `WF-04` that proves event upload and stats convergence. 2. Tighten `WF-03` and `WF-05` assertions so they distinguish healthy inventory-backed generation from degraded or AI-routed fallback paths. -3. If routing incidents continue, add a stronger mismatch-detection check that compares BFF, gateway, and ai-engine readback instead of only verifying the operator-facing success path. +3. If routing incidents continue, add a stronger check that a generation request reaches the overridden llama target instead of only verifying the operator-facing readback. 4. Keep `WF-01` focused on edge-owned profile semantics and avoid coupling it to optional upstream identity fields unless a regression history justifies that tighter binding. ## Related documents diff --git a/guides/validation/primary-workflows-validation.md b/guides/validation/primary-workflows-validation.md index e99a28d..559dbd6 100644 --- a/guides/validation/primary-workflows-validation.md +++ b/guides/validation/primary-workflows-validation.md @@ -210,7 +210,7 @@ Validate operator control of runtime targets and safe return to defaults. ### Evidence - Diagnostic panel screenshots for override and rollback states. -- Gateway/BFF logs for applied target and smoke request outcome. +- BFF and `ai-engine-api` logs for the applied target and smoke request outcome. ## Exit criteria diff --git a/operations/branch-protection-policy.md b/operations/branch-protection-policy.md index 2f463bf..fc1ac3a 100644 --- a/operations/branch-protection-policy.md +++ b/operations/branch-protection-policy.md @@ -1,6 +1,8 @@ # Branch Protection Policy -Last updated: 2026-05-03. +Last updated: 2026-11-14. + +> **Status (2026-11-14):** the k3s staging host no longer exists and `platform-infra` `deploy.yaml` is manual-only (`workflow_dispatch`), so no automatic staging deployment runs today. Where this document describes an automatic staging chain, it records the thesis-era policy; the image build on dispatch is the only automatic step. ## Purpose diff --git a/operations/cicd-workflow-map.md b/operations/cicd-workflow-map.md index 3d765e2..f8101c3 100644 --- a/operations/cicd-workflow-map.md +++ b/operations/cicd-workflow-map.md @@ -6,7 +6,7 @@ Last updated: 2026-11-12. ## Purpose -This document describes the current production-grade CI/CD chain across the AxiomNode workspace. It focuses on the services that publish container images and can trigger automatic rollout to the staging VPS. +This document describes the current production-grade CI/CD chain across the AxiomNode workspace. It focuses on the services that publish container images and could trigger automatic rollout to the staging VPS (thesis-era; deploys are manual-only today). ## When to use this @@ -169,7 +169,7 @@ Expected behavior: ### Use case 4: staging with externalized ai-engine -The service deployment chain continues to auto-update the cluster services even when the effective AI model runtime is external to the cluster. In the current split topology, `ai-engine-api` and `ai-engine-stats` may still auto-deploy in-cluster while runtime wiring for the llama target is controlled through persisted routing state. +Thesis-era behavior: the deployment chain auto-updated the cluster services even when the effective AI model runtime was external to the cluster. Today deploys are manual-only; in a split topology `ai-engine-api` and `ai-engine-stats` run in-cluster while the llama target is controlled through the persisted override in `ai-engine-api`. ### Use case 5: full in-cluster AI verification @@ -223,14 +223,14 @@ Each failure domain stops promotion at its own level and should be diagnosed in ## Governance notes - Deployment policy is centralized in `platform-infra`, not duplicated across service repositories. -- Mutable environment tags exist for operations convenience, but automatic staging deploys use immutable short-SHA image references. +- Mutable environment tags exist for operations convenience; the thesis-era automatic staging deploys used immutable short-SHA image references (disabled; deploys are manual-only today). - Documentation that still mentions automatic `dev` deployment is obsolete and should be considered superseded by this document. ## Risks and cautions - split AI runtime means a successful cluster rollout does not prove end-to-end AI reachability on its own - dispatch-token or cross-repo-read drift can break the chain before packaging even starts -- mutable tags are operationally useful but should never be treated as the authoritative record of automatic staging state +- mutable tags are operationally useful but should never be treated as the authoritative record of what a deploy applied ## Related documents diff --git a/operations/deployment-strategy.md b/operations/deployment-strategy.md index fa7b1d5..b41d5b1 100644 --- a/operations/deployment-strategy.md +++ b/operations/deployment-strategy.md @@ -22,7 +22,7 @@ This document defines how AxiomNode promotes runtime changes across environments - service repositories validate service correctness - `platform-infra` owns packaging and deployment policy -- automatic deployment is allowed only where rollback cost and operational blast radius are acceptable +- automatic deployment is allowed only where rollback cost and operational blast radius are acceptable (none is enabled today) - production promotion must be explicit, inspectable, and reversible - mutable image tags may exist for operator convenience, but automated staging rollout must be pinned to immutable artifacts @@ -131,7 +131,7 @@ Expected outcome: Expected outcome: -- covered services still auto-deploy to staging +- covered services have their images built on dispatch; deploys are manual-only (`deploy.yaml` `workflow_dispatch`) and no staging host exists today - `ai-engine-api` and `ai-engine-stats` may still be deployed in-cluster - llama reachability remains a routing concern controlled through persisted runtime state @@ -179,7 +179,7 @@ Preferred approach: ## Operational constraints - split or externalized AI runtime means not every end-to-end behavior in staging is driven only by Kubernetes manifests -- routing overrides in `bff-backoffice`, `api-gateway`, and `ai-engine-api` can influence effective runtime topology +- the llama target override in `ai-engine-api` can influence effective runtime topology - deployment documentation must therefore distinguish between cluster deployment state and effective runtime routing state ## Recommended release criteria diff --git a/operations/environments-and-secrets.md b/operations/environments-and-secrets.md index 1d1c922..cce5e74 100644 --- a/operations/environments-and-secrets.md +++ b/operations/environments-and-secrets.md @@ -52,9 +52,7 @@ Some operational behaviors depend on persisted runtime state rather than only on Important examples: -- gateway ai-engine target override state -- backoffice service-target and preset state -- ai-engine llama target override state +- ai-engine llama target override state (the only persisted runtime state; the gateway and the BFF keep none) These are not secret values, but they change effective runtime behavior and must be considered alongside environment configuration during incident analysis. diff --git a/operations/functionality-review-baseline.md b/operations/functionality-review-baseline.md index affa274..1a8cf09 100644 --- a/operations/functionality-review-baseline.md +++ b/operations/functionality-review-baseline.md @@ -43,8 +43,8 @@ How easy it is to prove that the capability works through tests, smoke checks, r | Word-pass inventory and generation | `microservice-wordpass`, `ai-engine`, `runtime-seed` | BFF/backoffice/tooling -> word-pass service -> `ai-engine` or repair path | `generationService.test.ts`, `gamesRoutes.test.ts`, `modelGenerationJob.test.ts`, workflow `WF-05` | High | Medium | Low | Coverage verification is currently blocked; repair and post-generation maintenance are important but still weakly formalized for dry-run and audit evidence. | | AI content generation | `ai-engine`, `microservice-quizz`, `microservice-wordpass`, `api-gateway` | internal service or operator path -> `ai-engine-api` -> llama runtime | `ai-engine/src/tests/test_api/test_app.py`, `test_games/test_generator.py`, `test_api/test_llama_target_store.py`, large pytest tree | High | Medium | Medium | Strong unit coverage exists, but split-runtime state makes operational proof harder; true-false generation still lacks a clearly documented consumer path. | | AI observability and diagnostics | `ai-engine-stats`, `bff-backoffice`, `backoffice` | operator -> backoffice -> `bff-backoffice` -> ai stats and service probes | `backoffice` diagnostics panel tests, `bff-backoffice/src/tests/backoffice.test.ts`, `ai-engine` observability tests | Medium | Medium | Medium | Partial-data and degraded-data labeling rules are still not explicit enough; metric ownership across raw telemetry and UI panels remains diffuse. | -| Runtime routing control | `bff-backoffice`, `api-gateway`, `ai-engine` | operator -> backoffice -> `bff-backoffice` -> gateway / ai-engine | `test_llama_target_store.py` and the BFF target tests, workflow `WF-06` | Medium | Low | Medium | The llama target state in `ai-engine-api` is the only runtime state holder affecting topology; audit-trail consistency and concurrent-edit semantics are not yet clear enough. | -| Backoffice operational diagnostics | `backoffice`, `bff-backoffice`, `api-gateway` | operator UI -> gateway -> `bff-backoffice` -> services/k8s/AI | `service-overview-panel.integration.test.tsx`, `service-console-panel.integration.test.tsx`, `bff-backoffice/src/tests/backoffice.test.ts` | Medium | Medium | Medium | The UI is well covered locally, but there is still no single explicit smoke matrix proving the whole diagnostics chain through gateway, BFF, and downstream services. | +| Runtime routing control | `bff-backoffice`, `ai-engine` | operator -> backoffice -> `bff-backoffice` -> ai-engine | `test_llama_target_store.py` and the BFF target tests, workflow `WF-06` | Medium | Low | Medium | The llama target state in `ai-engine-api` is the only runtime state holder affecting topology; audit-trail consistency and concurrent-edit semantics are not yet clear enough. | +| Backoffice operational diagnostics | `backoffice`, `bff-backoffice`, `api-gateway` | operator UI -> gateway -> `bff-backoffice` -> services/AI | `service-overview-panel.integration.test.tsx`, `service-console-panel.integration.test.tsx`, `bff-backoffice/src/tests/backoffice.test.ts` | Medium | Medium | Medium | The UI is well covered locally, but there is still no single explicit smoke matrix proving the whole diagnostics chain through gateway, BFF, and downstream services. | | Contract governance | `contracts-and-schemas`, `shared-sdk-client` | schema change -> validation -> SDK generation -> consumer repos | `scripts/validate_contracts.py`, TypeScript generation builds, CI validations | Medium | Medium | Low | `shared-sdk-client/typescript` still has build-only verification and no test or coverage script; some effective contracts remain implicitly service-owned instead of centrally versioned. | | Deployment and rollout automation | `platform-infra`, covered runtime repos | service repo CI -> `platform-infra` build/push -> deploy | service CI workflows, `platform-infra` validation, smoke scripts such as `smoke-stg-edge.sh` | High | Medium | Medium | Covered services deploy well, but post-deploy smoke depth is uneven by capability and split-AI topologies complicate equivalent validation. | | Monitoring and alerting | `observability-platform`, runtime repos, `backoffice` | services export telemetry -> dashboards / alerts / operator views | alert validation, service metrics tests in Node services, observability asset repo | Medium | Medium | Low | Critical-path alert coverage is not yet mapped capability by capability; blind spots are still possible even with dashboards and rules present. | @@ -69,7 +69,7 @@ How easy it is to prove that the capability works through tests, smoke checks, r ### Capabilities with duplicated or layered state -- runtime routing control spans `bff-backoffice`, `api-gateway`, and `ai-engine`, so effective topology can diverge from static manifests +- the llama target override held by `ai-engine` means effective topology can diverge from static manifests - player-facing continuity spans mobile local persistence, BFF-side player snapshot storage, and `microservice-users` authoritative persistence - diagnostics surfaces combine UI state, BFF state, runtime routing state, and live downstream health, which makes stale or partial views more likely without explicit labeling @@ -79,7 +79,7 @@ How easy it is to prove that the capability works through tests, smoke checks, r - the smoke matrix now exists, but several critical workflows still rely on manual evidence or partial automation across gateway, BFF, and downstream services - mobile auth support is not equally usable across Android, iOS, and JVM targets -- routing state is split across multiple repositories and needs stronger auditability and recovery semantics +- the llama target override needs stronger auditability and recovery semantics ### P1 corroborability gaps diff --git a/operations/repository-minimum-standards.md b/operations/repository-minimum-standards.md index 6523bd2..87031c0 100644 --- a/operations/repository-minimum-standards.md +++ b/operations/repository-minimum-standards.md @@ -1,6 +1,8 @@ # Repository Minimum Standards -Last updated: 2026-05-03. +Last updated: 2026-11-14. + +> **Status (2026-11-14):** the k3s staging host no longer exists and `platform-infra` `deploy.yaml` is manual-only (`workflow_dispatch`), so no automatic staging deployment runs today. Where this document describes an automatic staging chain, it records the thesis-era policy; the image build on dispatch is the only automatic step. ## Purpose diff --git a/operations/threat-model.md b/operations/threat-model.md index 4382cce..bde8feb 100644 --- a/operations/threat-model.md +++ b/operations/threat-model.md @@ -83,7 +83,7 @@ Critical boundaries: | Locally loaded LLM models | Confidential | Risk of exfiltration via internal endpoints | | Prompts, RAG and eval datasets | Internal | May contain product IP | | Logs and Prometheus metrics | Internal | May leak PII if not sanitized | -| Persisted runtime configuration (routes, presets) | Internal | Changes behavior without restart | +| Persisted runtime configuration (ai-engine llama target override) | Internal | Changes behavior without restart | ## STRIDE summary per component