Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,24 @@

## Unreleased

- Add the `singleOwner` option for a process that is the only writer of its
SQLite database. With the option, an idle runtime sends no query. Each role
waits for an in-process wake-up or for the next due time that the database
reports, instead of a fixed polling interval. Each role writes its process
row at start, at shutdown, and when it claims work. The periodic dead-process
cleanup is replaced by one cleanup at start, and the actor worker stops
waking for leases when it holds no activation. A watch of the wall clock
wakes every role after the computer sleeps. `run()` refuses to start with the
new `SingleOwnerConflict` error when another process row is alive. The option
refuses a database other than SQLite and a wake-up that crosses processes.
The default is `false`, and the default behavior does not change. In an idle
minute, a SQLite runtime with every role sent 2,466 statements by default and
none with the option.
- Fix an abort-listener leak. A role that waited with the shared shutdown
signal added a listener on every pass and removed it only at shutdown. The
redrive scheduler added one each idle second, which grew memory and printed
listener-limit warnings in a long-running process.

## 1.0.6 - 2026-11-08

- Refresh the benchmark evidence against the released v1.0.5 source, with
Expand Down
8 changes: 5 additions & 3 deletions docs/api.md
Original file line number Diff line number Diff line change
Expand Up @@ -793,11 +793,13 @@ The root exports `SolidObjectsError` and its supported subclasses:
`ScheduleWithoutOperation`, `StateMigrationError`, `ApplicationWriteForbidden`, `UnknownEffect`, and
`UnknownCommitAction`;
- operational failures: `LostActivation`, `DatabaseDeadlineExceeded`,
`UnknownDeadLetter`, `UnknownReminder`, `ReminderNotPaused`, and
`UnsupportedDatabase`.
`UnknownDeadLetter`, `UnknownReminder`, `ReminderNotPaused`,
`UnsupportedDatabase`, and `SingleOwnerConflict`.

`NonRetryableError` is the application subclassing point. `SyncTimeoutDetails`
and `SyncTimeoutWaitingOn` type timeout diagnostics. See
and `SyncTimeoutWaitingOn` type timeout diagnostics. `SingleOwnerConflict`
carries one `SingleOwnerConflictProcess` for each live process row that stopped
a `singleOwner` runtime from starting. See
[Errors and recovery](errors-and-recovery.md) before deciding what to catch.

## `solid-objects/database/sqlite`
Expand Down
107 changes: 92 additions & 15 deletions docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -65,19 +65,20 @@ as an instruction to divide the actor. See

## Runtime roles and supervision

| Option | Default | Contract |
| ------------------------------------------- | --------------: | ----------------------------------------------------------------- |
| `workerCount` | `1` | Non-negative actor workers. |
| `effectWorkerCount` | `1` | Non-negative effect workers. |
| `broadcastWorkerCount` | `1` | Non-negative broadcast workers when realtime is enabled. |
| `reminderSchedulerCount` | `1` | Non-negative reminder schedulers. |
| `processHeartbeatIntervalMilliseconds` | `15_000` | Positive persisted heartbeat cadence. |
| `processAliveThresholdMilliseconds` | `60_000` | Positive age after which an owner is stale. |
| `shutdownTimeoutMilliseconds` | `15_000` | Positive shared graceful-shutdown budget. |
| `supervisorRestartDelayMilliseconds` | `100` | Positive initial failed-role replacement delay. |
| `supervisorMaximumRestartDelayMilliseconds` | `10_000` | Positive cap no smaller than the initial delay. |
| `wakeUp` | `"automatic"` | A name or an adapter implementing `watch`, `notify`, and `close`. |
| `logger` | console methods | Structured `debug`, `info`, `warn`, and `error` sink. |
| Option | Default | Contract |
| ------------------------------------------- | --------------: | ------------------------------------------------------------------- |
| `workerCount` | `1` | Non-negative actor workers. |
| `effectWorkerCount` | `1` | Non-negative effect workers. |
| `broadcastWorkerCount` | `1` | Non-negative broadcast workers when realtime is enabled. |
| `reminderSchedulerCount` | `1` | Non-negative reminder schedulers. |
| `processHeartbeatIntervalMilliseconds` | `15_000` | Positive persisted heartbeat cadence. |
| `processAliveThresholdMilliseconds` | `60_000` | Positive age after which an owner is stale. |
| `shutdownTimeoutMilliseconds` | `15_000` | Positive shared graceful-shutdown budget. |
| `supervisorRestartDelayMilliseconds` | `100` | Positive initial failed-role replacement delay. |
| `supervisorMaximumRestartDelayMilliseconds` | `10_000` | Positive cap no smaller than the initial delay. |
| `wakeUp` | `"automatic"` | A name or an adapter implementing `watch`, `notify`, and `close`. |
| `singleOwner` | `false` | `true` when this process is the only writer of its SQLite database. |
| `logger` | console methods | Structured `debug`, `info`, `warn`, and `error` sink. |

Counts may be zero, but the complete configuration must leave at least one
runtime role enabled. Broadcast workers are started only when `broadcast` or
Expand All @@ -90,14 +91,89 @@ Consecutive empty passes double it up to
fixed cadence. A custom wake-up adapter must return `true` for a notification
and `false` for a timeout. An older adapter that returns `void` remains
compatible and keeps the fast cadence. Wake-ups reduce latency, while database
polling remains the correctness path.
polling remains the correctness path. The one exception is
[`singleOwner`](#single-owner), which replaces polling with wake-ups and due
times.

Busy SQL workers and testing drains give the event loop a turn between passes.
In Node this lets timers and input run even with synchronous SQLite calls.
An instrumentation callback can abort `runtime.run()` before its next pass.
This does not interrupt a synchronous handler: responsiveness is bounded by
one pass, or by a single overlong handler that must return first.

## Single owner

Set `singleOwner: true` when one process is the only writer of its SQLite
database, for example a daemon that embeds the runtime and holds its database
file. With the option, an idle runtime sends no query. The default is `false`,
and the default behavior does not change.

```typescript
const runtime = createRuntime({
database: sqlite({ path: "app.sqlite" }),
singleOwner: true,
})
```

With `singleOwner: true`, the runtime changes these behaviors:

- Each role waits for an in-process wake-up or for the next due time that the
database reports. A due time is, for example, the time of the next ready
message, effect, broadcast, reminder, or retry, the end of a lease, or the
time when the owner of a claim becomes stale. A role with nothing due waits
up to 24 hours and then reads its due time again.
- A write in this process that adds work or moves it earlier wakes the roles
that must react. A role that waits for a later due time then reads the new
due time.
- The actor worker renews a lease only while it holds a cached activation.
When idle deactivation releases the last activation, the worker stops waking
for leases.
- Each role writes its process row once at start and once at shutdown. A claim
also writes the row of the worker that claims, in the claim transaction, so
that running work always has a fresh owner. An empty pass writes nothing.
- No periodic dead-process cleanup runs, and
`deadProcessCleanupIntervalMilliseconds` has no effect. `run()` releases
stale process rows once, at start.
- A watch compares the wall clock with the process clock every 10 seconds.
Node timers stop while the computer sleeps, so after a sleep, or after a jump
of one second or more, every role reads its due time again. The watch sends
no query.
- When a due time is now or has passed and a claim still finds nothing, the
role tries again after 1 ms. A timer can end a few milliseconds before the
database clock reaches the due time, and this retry takes the work at once.
Each further retry doubles the wait, up to the polling interval, so stuck
work cannot spin. `pollingIntervalMilliseconds` and
`idlePollingIntervalMilliseconds` set only this limit.

Work that is active runs as before. A turn renews its lease, and an effect or a
broadcast sends its process heartbeat while it runs. Automatic retention still
runs at start and every `retentionIntervalMilliseconds`. Set that interval to
`0` to stop it, and prune with `runtime.retention.prune()` instead.

`run()` refuses to start with `SingleOwnerConflict` when another process row is
alive by the usual rule: the row is not stopped, and its heartbeat is newer
than `processAliveThresholdMilliseconds`. The error names each row and the time
at which the row becomes stale. The rows of the runtime's own synchronous
callers do not count.

`createRuntime()` refuses the option in these cases:

- The database is not SQLite. PostgreSQL and MySQL are servers that other
processes can reach. The runtime cannot know that it is the only writer, and
without polling a write from another process can wait up to 24 hours.
- `wakeUp` is `"postgresql"`, `"redis"`, or an adapter. A notification over a
connection can be lost when the connection fails, and the runtime cannot
prove that a custom adapter delivers every signal. Without polling, a lost
signal can delay work for up to 24 hours. With `"automatic"` or
`"in_process"`, the runtime uses in-process signalling and ignores
`SOLID_OBJECTS_REDIS_URL`.

Do not use the option with `sharedSqliteWasm`, because each tab runs its own
runtime against the shared database. See
[Correctness and delivery semantics](correctness.md#single-owner) for the
guarantee and its limits, and [Operations](operations.md#single-owner) for
monitoring.

## Retention and cleanup

| Option | Default | Contract |
Expand Down Expand Up @@ -206,7 +282,8 @@ the files only when no process has the database open. An in-memory database

`wakeUp` takes a name or an adapter, as `config.cache_store` does in Rails.
`"automatic"` is the default. It prefers `SOLID_OBJECTS_REDIS_URL`, then
PostgreSQL notifications, then polling. `"in_process"` opts out, `"postgresql"`
PostgreSQL notifications, then polling. With `singleOwner`, the runtime runs no
selection and uses in-process signalling. `"in_process"` opts out, `"postgresql"`
and `"redis"` force one, and an unknown name throws rather than polls quietly.

A requested adapter that the environment cannot provide polls instead and says
Expand Down
57 changes: 54 additions & 3 deletions docs/correctness.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,9 +36,10 @@
- Observable outbox rows are claimed in actor revision order. Subscription
sessions and browser clients reject duplicate or stale revisions within an
incarnation, while a recreated actor starts a new revision sequence.
- Wake-up notifications happen only after commit and never replace polling.
Workers watch before claiming, so an in-process notification between an empty
claim and the following wait is retained.
- Wake-up notifications happen only after commit and never replace polling,
except with [`singleOwner`](#single-owner). Workers watch before claiming, so
an in-process notification between an empty claim and the following wait is
retained.
- Activation passes are bounded. Yielding changes ready-membership polling
order only; it neither changes durable message sequence nor makes future work
due early.
Expand All @@ -65,6 +66,56 @@
delivery. A state change on an actor with payloads creates a revision
broadcast, even when that actor declares no scalar observables.

## Single owner

With `singleOwner: true`, an idle runtime sends no query, and it still runs all
work when the work arrives or comes due. This guarantee holds only while one
process writes the database.

The guarantee rests on these rules:

- Every write that adds work or moves it earlier happens in this process. After
the commit, the write notifies an in-process signal for each role that must
react. A host that enqueues with
`runtime.enqueueInternalMessageInTransaction()` must call
`runtime.announceInternalMessage()` after its commit, as it must without the
option. Without that call, the message waits for the next wake-up.
- A role watches the signal before it claims, so it cannot lose a notification
that arrives during a pass.
- After an empty pass, a role reads the earliest time at which a claim can take
something. The read uses the same conditions as the claims: the head of each
mailbox, lease ends, pending effects, broadcasts, and reminders, claims whose
owners become stale, and recovery timeouts. The role waits until that time,
or for at most 24 hours.
- A claim writes the row of the claiming worker in the claim transaction. A
worker that was idle for longer than `processAliveThresholdMilliseconds`
therefore never holds work under a stale row, and the alive rule protects
running work as it does without the option.

Fencing and at-least-once delivery do not change. Claims, activation leases,
the commit fence, retries, dead letters, and effect recovery use the same rows
and the same conditions.

The guarantee has these limits:

- One process only. A write from another process, for example a second
runtime, a script, or a command, does not wake this runtime. That work waits
for the next local wake-up or due time, for at most 24 hours.
- The start check reads the process rows once. It sees another writer only
while that writer's row is fresh, and an idle single owner does not refresh
its row. Two processes that start at the same moment can both pass the
check. Only an operating system lock, such as SQLite's exclusive locking
mode, keeps a second process out.
- A due time runs on a Node timer. A long synchronous handler delays the timer,
as it delays polling. Node timers stop while the computer sleeps. A watch
sees the jump of the wall clock within 10 seconds and makes every role read
its due time again, so work that came due during a sleep runs at most about
10 seconds after the computer wakes.
- When a due time is now or has passed and a claim still finds nothing, for
example after an actor setup failure, the role retries after 1 ms and then
doubles the wait up to its polling interval. Stuck work is then retried at
the polling interval, as it is without the option.

## Limitations and non-goals

- At-least-once execution means actor code may begin more than once. State and
Expand Down
4 changes: 4 additions & 0 deletions docs/errors-and-recovery.md
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,10 @@ stable public code and does not classify the operation as failed.
lookup found no record.
- `ReminderNotPaused` means only a completed reminder remains; schedule it
again through its actor instead of resuming it.
- `SingleOwnerConflict` means a `singleOwner` runtime found another live
process row at start. Its `processes` list names each row and the time at
which the row becomes stale. Stop the other process, or wait until that time,
and start again.

Realtime protocol validation uses `TypeError` for malformed subscription or
browser envelopes. Server delivery failures are isolated to the session and
Expand Down
50 changes: 45 additions & 5 deletions docs/operations.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,8 @@ and work enqueued with `send` stays ready until some process runs the roles.
A direct call or an explicit `sync` needs no running role, because the caller's
own path executes it.

Runtime roles use durable polling as the correctness fallback. Consecutive
Runtime roles use durable polling as the correctness fallback, unless the
runtime runs as a [single owner](#single-owner). Consecutive
empty passes double each role's wait from `pollingIntervalMilliseconds` to
`idlePollingIntervalMilliseconds`, which defaults to one second. Processed
work and wake-up notifications reset the role to the fast interval. Actor
Expand Down Expand Up @@ -149,10 +150,10 @@ state from reconciliation code.
The runtime automatically prunes expired message and stopped-process history
once at startup and every `retentionIntervalMilliseconds`, which defaults to
one hour. Stale process ownership is recovered independently every
`deadProcessCleanupIntervalMilliseconds`, which defaults to one minute. Set an
interval to zero to disable its scheduler. Failed passes emit metadata-only
events and retry with bounded exponential backoff without stopping other
runtime roles.
`deadProcessCleanupIntervalMilliseconds`, which defaults to one minute. A
single owner recovers it once, at start. Set an interval to zero to disable its
scheduler. Failed passes emit metadata-only events and retry with bounded
exponential backoff without stopping other runtime roles.

Operators can also use `runtime.retention.preview()` and
`runtime.retention.prune()`. Both calls require administration authorization;
Expand Down Expand Up @@ -228,6 +229,45 @@ Scheduling an existing actor operation for a different time emits
`solid_objects.reminder.replaced` after the fenced actor commit. Scheduling it
for the same time emits nothing.

## Single owner

With [`singleOwner: true`](configuration.md#single-owner), an idle runtime sends
no query. Its roles wake for an in-process notification, for a due time that
the database reports, or after the wall clock jumps. Expect these differences
from the default:

- `runtime.administration.processes()` reports the rows of an idle single owner
as `stale`, because a role writes its row only at start, at shutdown, and
when it claims work. The `runtime` doctor check can report no live roles for
the same reason. Both are normal while the runtime is idle.
- The `wakeUp` doctor check passes with in-process signalling, because no other
process writes the database.
- `cleanup()` can mark the rows of an idle single owner `stopped`. The next
claim of each role writes its row again. An idle role holds no claim, so the
cleanup releases nothing.
- `currentPollingIntervalMilliseconds` and the
`solid_objects.polling.interval_changed` events describe only the retry of
due work that a claim cannot take.

`run()` refuses to start with `SingleOwnerConflict` when another process row is
alive. A process that stopped without cleanup stays alive until its heartbeat
is older than `processAliveThresholdMilliseconds`. The error gives that time
for each row. Stop the other process, or wait until that time, and start again.

The start check reads the process rows once. It sees another writer only while
that writer's row is fresh, and an idle single owner does not refresh its row.
For a guarantee that the operating system enforces, take an exclusive lock on
the file before the runtime writes to it:

```typescript
const database = sqlite({ path: "app.sqlite" })
await database.connection((connection) => connection.run("PRAGMA locking_mode = EXCLUSIVE"))
```

SQLite then holds the lock from the first write until the database closes. A
second process, including the `solid-objects` command for the same file, gets
`database is locked`.

## Dead letters, retry, and redrive

A handler failure that exhausts its attempts becomes a dead letter.
Expand Down
Loading
Loading