Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@ All notable changes to this project are documented in this file. The format is b
- The quickstart matches what a new app does: an anonymous `curl /api/orders` answers 401 (the scaffold guards the API with `orders.*` permissions), the single-file test example names `tests/web/crud/test_orders_crud.py`, and `smpy users create-admin` is shown without `--password`, so it prompts instead of putting a weak password in shell history (#919).
- A `smpy new` app stays installed under plain uv commands. The workspace root now depends on the host (`<app>-host`, a `{ workspace = true }` source), so `uv sync` at the root installs the app instead of uninstalling it. `smpy add … --path host` inside an app, where the host is a workspace member, syncs the whole workspace from the root (`uv sync --all-packages`, as `make install` does) instead of syncing `host/` alone, which removed the root's dev tools and other modules. `smpy create-module` inside an app prints the app's steps (`smpy add`, `make install`, `make migration`, `make migrate`, `make test`, restart `make dev`) instead of `uv sync --extra dev` in the module, which uninstalled the rest of the app; `--no-install` prints `make install`. An existing app can add `dependencies = ["<app>-host"]` and `[tool.uv.sources] <app>-host = { workspace = true }` to its root `pyproject.toml` (#916).
- A module installed while `make dev` runs is reported as such. The reload keeps the import path the server started with, so its package cannot be imported until a restart; discovery used to log `No module named '<x>.module'` with a traceback, as if the module were broken. It now logs that the entry point is installed but its package cannot be imported in this process, and to restart `make dev` (`strict` discovery still refuses it). A missing third-party import inside a module is still reported with its traceback (#915).
- The SQLite expression-index warning is no longer printed on every migration. The revision hook now re-emits the indexes of new tables first and then names only expression indexes that the database does not have and the revision does not create (SQLite records index names even though it cannot reflect their definitions), so a fresh app's initial schema (which creates `ix_users_user_email_lower`) and an unrelated revision print nothing, while an expression index added to an existing table is still reported. The warning, `SM026` and the `migrate` note point to `make migrations-roundtrip-pg` only where the Makefile has it; elsewhere they say to run `make migrate` and `alembic check` against a PostgreSQL `SM_DATABASE_URL` and link the docs. `smpy new` shows the output of `uv sync`, `npm install` and alembic only when a step fails, stops after a failed initial migration, and prints "Next steps" once, at the end, listing the steps still to run (#917).

## [0.0.45] — 2026-11-11

Expand Down
18 changes: 12 additions & 6 deletions docs/module-authoring.md
Original file line number Diff line number Diff line change
Expand Up @@ -362,13 +362,19 @@ generated file against this checklist before treating it as portable
scaffolded `env.py`) rewrite these for you; check any revision written
without them, and always write `server_default=sa.false()` on the model.
2. **Expression indexes.** SQLite cannot reflect `Index(..., text("lower(email)"))`,
so autogenerate skips it and `alembic check` cannot see it. Autogenerate on
SQLite logs a banner naming every expression index in your models, and
`make doctor` reports `SM026`. Add each one to a revision by hand
so autogenerate skips it and `alembic check` cannot see it. The revision hook
re-emits the expression indexes of every table a revision creates, so an
initial schema needs nothing. When the models declare an expression index the
database does not have (one added to an existing table), autogenerate on
SQLite logs a banner naming it, and `make doctor` reports `SM026` for every
expression index. Add it to the revision by hand
(`op.create_index(..., [sa.text("lower(email)")], if_not_exists=True)`) and
confirm with the PostgreSQL round trip below. The warning is deliberately not
an error: it fires on every autogenerate, including unrelated revisions, and
the Postgres round trip is the actual gate. Prefer writing an ordered index column as
confirm with the PostgreSQL round trip below; in an app without that make
target, point `SM_DATABASE_URL` at a PostgreSQL database, run `make migrate`,
then `uv run alembic -c host/alembic.ini check`, which reports a missing
index. The warning is deliberately not an error:
SQLite can neither confirm nor refute the drift, and PostgreSQL is the actual
gate. Prefer writing an ordered index column as
`sa.desc(sa.column("col"))` over `sa.text("col DESC")` or `sa.desc("col")`:
the latter two are expressions to Alembic, and while the host's include filter
suppresses the false SQLite change when only the ordering differs, the column
Expand Down
2 changes: 1 addition & 1 deletion docs/reference/diagnostic-codes.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ The framework runs a set of static checks over installed modules at app boot. Th
| `SM023` | WARNING | A module's `theme.css` contains an unlayered plain rule (anything but an at-rule or a `:root`-style selector). Unlayered CSS outranks every Tailwind utility. | Move the rule to the module's `styles.css`, which is imported into `layer(components)` so utilities still win. |
| `SM024` | WARNING | A unique column, constraint or index on a `MultiTenantMixin` table does not include `tenant_id`, so the first tenant to claim a value locks every other tenant out of it. | Make the key per tenant: add `tenant_id` to it (`Index(..., "tenant_id", "slug", unique=True)`). |
| `SM025` | WARNING | `multi_tenant` is on but no module registered `app.state.tenant_resolver`, so only the principal's `tenant_id` claim can bind a tenant — with most auth providers every tenant-scoped query then fails closed. Checked at boot (it needs the built app), in every environment, not by `make doctor`. | Install the `tenants` module, or register your own `async (Request) -> str \| None` resolver on `app.state.tenant_resolver`. |
| `SM026` | INFO | The configured database is SQLite and a module model declares an expression index (e.g. `lower(email)`). SQLite cannot reflect those, so autogenerate and `alembic check` cannot verify them and a clean diff says nothing about them. | Add the index to a revision by hand and verify with `make migrations-roundtrip-pg` (PostgreSQL). See [Migrations](/module-authoring#reviewing-a-revision-autogenerated-on-sqlite). |
| `SM026` | INFO | The configured database is SQLite and a module model declares an expression index (e.g. `lower(email)`). SQLite cannot reflect those, so autogenerate and `alembic check` cannot verify them and a clean diff says nothing about them. | Add the index to a revision by hand and verify on PostgreSQL: `make migrations-roundtrip-pg` in this repository, `make migrate` then `alembic check` against a PostgreSQL `SM_DATABASE_URL` in an app. See [Migrations](/module-authoring#reviewing-a-revision-autogenerated-on-sqlite). |
| `SM027` | WARNING | A key in `host/locales/overrides/<locale>.json` matches no catalog key, so the override was skipped. Reported by `make doctor` and at dev boot. | Fix the key: overrides replace an existing key by its full dotted path (e.g. `users.login.aside_heading`). See [i18n](/framework/i18n#overriding-module-copy). |
| `SM028` | WARNING | A module's `tasks.py` imports `torch`, `tensorflow`, `ultralytics`, `osgeo`, `rasterio`, `cv2` or `sklearn` at module level, so every process that loads the module's tasks (web, each worker, beat) pays for it and a queue-restricted worker cannot avoid it. | Import the package inside the task function. See [background_tasks](/modules/background_tasks#running-a-worker-for-a-subset-of-modules). |
| `SM029` | INFO | A module overrides `db_extensions()` (Postgres extensions such as `ltree`) and the configured database is SQLite, which has none. `ensure_db_extensions` raises `UnsupportedDatabaseError` when migrations run. | Point `SM_DATABASE_URL` at PostgreSQL. See [Postgres extensions and column types](/database/extensions). |
Expand Down
80 changes: 51 additions & 29 deletions framework/cli/simple_module_cli/new.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@

import shutil
import subprocess
from collections.abc import Sequence
from enum import StrEnum
from pathlib import Path
from typing import Annotated
Expand Down Expand Up @@ -174,19 +175,12 @@ def new_project(
rel = path.relative_to(target) if path.is_relative_to(target) else path
typer.echo(f" {rel}")
typer.echo("\nGenerated secrets were written to .env (gitignored), not .env.example.")
typer.echo("\nNext steps:")
typer.echo(f" cd {target}")
manual_setup = ["make install", 'make migration msg="initial schema"', "make migrate"]
if no_install:
typer.echo(" make install")
typer.echo(' make migration msg="initial schema"')
typer.echo(" make migrate")
typer.echo(" make dev")
typer.echo(" make docker-up # or run the full stack in containers")
if "background_tasks" in resolved:
typer.echo(" docker compose up -d redis worker beat # background jobs")
_echo_next_steps(target, manual_setup, resolved)
return

typer.echo("Installing dependencies...")
typer.echo("\nInstalling dependencies...")
# What `make install` runs: a workspace syncs every member, not just the root.
uv_sync = ["uv", "sync"] if flat else ["uv", "sync", "--all-packages"]
for cmd in (uv_sync, ["npm", "install"]):
Expand All @@ -196,40 +190,68 @@ def new_project(
"Install it and finish setup manually.",
err=True,
)
_echo_next_steps(target, manual_setup, resolved)
return
result = subprocess.run(cmd, cwd=target, check=False)
if result.returncode != 0:
typer.echo(
f"WARNING: {' '.join(cmd)} failed (exit {result.returncode}); "
"finish setup manually.",
err=True,
)
if not _run_quietly(cmd, target):
_echo_next_steps(target, manual_setup, resolved)
return

alembic = _alembic_argv(target, host_dir)
_bootstrap_initial_migration(target, host_dir, alembic)
typer.echo("Creating the database...")
if not _bootstrap_initial_migration(target, host_dir, alembic):
_echo_next_steps(target, manual_setup[1:], resolved)
return
# `heads` (plural) applies every per-module branch head; `head` (singular)
# errors once a second module ships its own migration branch label.
subprocess.run([*alembic, "upgrade", "heads"], cwd=target, check=False)
typer.echo("\nSetup complete. Run `make dev` in the new directory.")
typer.echo("To run the full stack in containers instead: make docker-up")
if not _run_quietly([*alembic, "upgrade", "heads"], target):
# The revision exists now; regenerating it would create a duplicate.
_echo_next_steps(target, manual_setup[2:], resolved)
return
typer.echo("\nSetup complete.")
_echo_next_steps(target, [], resolved)


def _run_quietly(cmd: list[str], cwd: Path) -> bool:
"""Run a setup step, showing its output only when it fails (GH #917)."""
try:
result = subprocess.run(
cmd, cwd=cwd, check=False, capture_output=True, text=True, errors="replace"
)
except OSError as exc: # the executable is missing
typer.echo(
f"WARNING: {' '.join(cmd)} could not run ({exc}); finish setup manually.", err=True
)
return False
if result.returncode == 0:
return True
typer.echo(f"{result.stdout}{result.stderr}", err=True)
typer.echo(
f"WARNING: {' '.join(cmd)} failed (exit {result.returncode}); finish setup manually.",
err=True,
)
return False


def _echo_next_steps(target: Path, setup: list[str], resolved: Sequence[str]) -> None:
typer.echo("\nNext steps:")
typer.echo(f" cd {target}")
for step in [*setup, "make dev"]:
typer.echo(f" {step}")
typer.echo(" make docker-up # or run the full stack in containers")
if "background_tasks" in resolved:
typer.echo("For background jobs, also run: docker compose up -d redis worker beat")
typer.echo(" docker compose up -d redis worker beat # background jobs")


def _bootstrap_initial_migration(target: Path, host_dir: Path, alembic: list[str]) -> None:
def _bootstrap_initial_migration(target: Path, host_dir: Path, alembic: list[str]) -> bool:
"""Autogenerate the baseline migration if the scaffold ships none.

Without a real revision, ``alembic upgrade head`` is a silent no-op
against an empty schema — the bundled modules' tables never exist.

Runs from ``target`` (the project root) — see :func:`_alembic_argv`.
Returns False when the autogenerate failed.
"""
versions_dir = host_dir / "migrations" / "versions"
if any(p.name != "__init__.py" for p in versions_dir.glob("*.py")):
return
subprocess.run(
[*alembic, "revision", "--autogenerate", "-m", "initial schema"],
cwd=target,
check=False,
)
return True
return _run_quietly([*alembic, "revision", "--autogenerate", "-m", "initial schema"], target)
5 changes: 3 additions & 2 deletions framework/cli/tests/test_cli_new_migration_bootstrap.py
Original file line number Diff line number Diff line change
Expand Up @@ -22,12 +22,13 @@ def test_bootstrap_initial_migration_runs_autogenerate_when_versions_empty(

calls: list[tuple[list[str], Path]] = []

def fake_run(cmd, *, cwd, check):
del check
def fake_run(cmd, *, cwd, check, **kwargs):
del check, kwargs
calls.append((list(cmd), Path(cwd)))

class _Result:
returncode = 0
stdout = stderr = ""

return _Result()

Expand Down
82 changes: 82 additions & 0 deletions framework/cli/tests/test_cli_new_quiet_install.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,82 @@
"""`smpy new` keeps tool chatter out of a successful run (GH #917).

A first run printed ~150 lines (uv's package list, alembic's INFO lines) and
"Next steps: cd <app>" before the install had even started.
"""

from __future__ import annotations

import subprocess
from pathlib import Path

from simple_module_cli import new as new_mod
from simple_module_cli.cli import app
from typer.testing import CliRunner


def _invoke(tmp_path: Path, monkeypatch, *, fail: str | None = None):
calls: list[list[str]] = []

def fake_run(cmd, *, cwd, check, capture_output=False, text=False, errors=None):
del cwd, check, errors
calls.append(list(cmd))
assert capture_output and text, f"{cmd} must not stream to the terminal"
failed = fail is not None and fail in cmd
return subprocess.CompletedProcess(
cmd,
1 if failed else 0,
stdout=f"chatter from {cmd[0]}\n",
stderr="boom\n" if failed else "",
)

monkeypatch.setattr(new_mod.shutil, "which", lambda name: f"/usr/bin/{name}")
monkeypatch.setattr(new_mod.subprocess, "run", fake_run)
result = CliRunner().invoke(
app, ["new", "demo", "--yes", "--db", "sqlite", "--dest", str(tmp_path / "demo")]
)
return result, calls


def test_successful_install_is_quiet_and_ends_with_next_steps(tmp_path, monkeypatch) -> None:
result, calls = _invoke(tmp_path, monkeypatch)

assert result.exit_code == 0, result.output
assert any("upgrade" in c for c in calls)
assert "chatter from" not in result.output
assert result.output.count("Next steps") == 1
assert result.output.index("Next steps") > result.output.index("Setup complete")
assert "make dev" in result.output.split("Next steps", 1)[1]


def test_a_failing_step_shows_its_output(tmp_path, monkeypatch) -> None:
result, _ = _invoke(tmp_path, monkeypatch, fail="npm")

assert "chatter from npm" in result.output
assert "boom" in result.output
assert "failed" in result.output


def test_a_failed_initial_migration_stops_and_prints_the_manual_steps(
tmp_path, monkeypatch
) -> None:
result, calls = _invoke(tmp_path, monkeypatch, fail="--autogenerate")

assert not any("upgrade" in c for c in calls)
steps = result.output.split("Next steps", 1)[1]
assert "make migration" in steps
assert "make migrate" in steps
assert "Setup complete" not in result.output


def test_a_missing_executable_prints_the_manual_steps(tmp_path, monkeypatch) -> None:
def missing(cmd, **kwargs):
raise FileNotFoundError(cmd[0])

monkeypatch.setattr(new_mod.shutil, "which", lambda name: f"/usr/bin/{name}")
monkeypatch.setattr(new_mod.subprocess, "run", missing)
result = CliRunner().invoke(
app, ["new", "demo", "--yes", "--db", "sqlite", "--dest", str(tmp_path / "demo")]
)
assert result.exit_code == 0, result.output
assert "could not run" in result.output
assert "make install" in result.output.split("Next steps", 1)[1]
34 changes: 30 additions & 4 deletions framework/core/simple_module_core/diagnostics/_expression_index.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,9 @@

import importlib
import logging
import re
from collections.abc import Iterable
from pathlib import Path
from typing import TYPE_CHECKING, Any

from simple_module_core.diagnostics._types import Diagnostic, DiagnosticLevel
Expand Down Expand Up @@ -94,6 +96,32 @@ def module_all_tables(mod: ModuleBase) -> list[Any]:
return list(seen.values())


_ROUNDTRIP_TARGET = re.compile(r"^migrations-roundtrip-pg:", re.MULTILINE)
_DOCS_URL = "https://docs.py.simplemodule.dev/module-authoring"


def expression_index_advice(root: Path | None = None) -> str:
"""How to verify expression indexes, in terms of what this project has.

The framework repository has a Postgres round-trip make target; an app made
by ``smpy new`` does not, so pointing it there sends users nowhere (GH #917).
``root`` defaults to the working directory, where make, alembic and doctor run.
"""
try:
makefile = (root or Path.cwd()) / "Makefile"
has_target = _ROUNDTRIP_TARGET.search(makefile.read_text(encoding="utf-8")) is not None
except OSError:
has_target = False
if has_target:
return "verify them on PostgreSQL with `make migrations-roundtrip-pg`"
return (
"verify them on PostgreSQL: point `SM_DATABASE_URL` at a PostgreSQL database, "
"run `make migrate`, then `alembic check` with the app's alembic.ini "
"(`uv run alembic -c host/alembic.ini check` in a `smpy new` app), which "
f"reports a missing index. See {_DOCS_URL}"
)


def check_expression_indexes_unverifiable(
tables: Iterable[Any], dialect: str | None, module_name: str
) -> list[Diagnostic]:
Expand All @@ -114,16 +142,14 @@ def check_expression_indexes_unverifiable(
f"about them"
),
module_name=module_name,
suggestion=(
"Add them to a revision by hand and verify on Postgres: "
"`make migrations-roundtrip-pg`"
),
suggestion=f"Add them to a revision by hand and {expression_index_advice()}",
)
]


__all__ = [
"check_expression_indexes_unverifiable",
"expression_index_advice",
"find_expression_indexes",
"index_is_expression_based",
"module_all_tables",
Expand Down
5 changes: 3 additions & 2 deletions framework/db/simple_module_db/_sqlite_index_compare.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,9 @@
``make_include_object`` can skip it on SQLite. Anything else still reaches
Alembic: a different column list, a uniqueness change, or a real expression
such as ``lower(email)``. A change of ordering alone (``DESC`` to ``ASC``) is not
reported on SQLite, which cannot reflect ordering at all; the Postgres round
trip (``make migrations-roundtrip-pg``) is what catches it.
reported on SQLite, which cannot reflect ordering at all; a PostgreSQL
round trip (``make migrations-roundtrip-pg`` in the framework repository) is what
catches it.
"""

from __future__ import annotations
Expand Down
Loading
Loading