Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
68 changes: 0 additions & 68 deletions .changelog/adapter-aware-workers-and-schedules.md

This file was deleted.

30 changes: 0 additions & 30 deletions .changelog/consistent-job-runtime.md

This file was deleted.

44 changes: 0 additions & 44 deletions .changelog/hot-reloading-jobs.md

This file was deleted.

34 changes: 34 additions & 0 deletions .changelog/redis-schedule-due-index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
# Indexed Redis Schedule Claims

## Performance Improvement

The Redis adapter now maintains a `schedules::due` sorted-set index scored by each schedule's
`next_run_at`. Claiming the next due schedule uses this index instead of scanning every stored
schedule, so polling no longer grows linearly with the total schedule count.

Schedule hashes remain the source of truth. Creating, updating, pausing, resuming, deleting, and
claiming schedules maintain the derived index, while claiming repairs stale entries when the hash
and index disagree.

Schedule hash and index writes are atomic, and index consistency is preserved across concurrent
lifecycle changes. Cron finalization now rejects stale calculations when a schedule is paused,
deleted, or reconfigured while its next run is being calculated. Claiming also discards malformed
due scores instead of allowing one corrupt entry to block later schedules.

## Upgrade Notes

This change requires an explicit migration for existing Redis schedules. The `Adapter` contract now
includes an idempotent `migrate()` method; built-in adapters without migrations implement it as a
no-op, and custom adapters must implement it as well.

Run the migration once during deployment, before starting workers or any process that creates or
updates schedules:

```typescript
await QueueManager.init(config)
await QueueManager.use('redis').migrate()
```

Existing Redis schedules will not fire through the new index until the migration has completed.
The migration scans all schedules and should remain an explicit deployment step rather than run in
the worker polling loop.
20 changes: 20 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -334,6 +334,26 @@ const connection = new Redis({ host: 'localhost' })
const adapter = redis(connection)
```

#### Migrating Redis schedules after an upgrade

The Redis adapter uses a `schedules::due` sorted-set index to find due schedules. When upgrading
from a version that predates this index, run the adapter migration once during deployment, before
starting any workers:

```typescript
import { QueueManager, Worker } from '@boringnode/queue'

await QueueManager.init(config)
await QueueManager.use('redis').migrate()

const worker = new Worker(config)
await worker.start(['default'])
```

The migration is idempotent and rebuilds the derived index from the canonical schedule hashes.
Existing Redis schedules will not fire through the new index until it has run. Do not run the
`O(number of schedules)` migration from the worker polling loop.

### Knex (PostgreSQL, MySQL, SQLite)

```typescript
Expand Down
10 changes: 10 additions & 0 deletions src/contracts/adapter.ts
Original file line number Diff line number Diff line change
Expand Up @@ -223,6 +223,16 @@ export interface Adapter {
*/
destroy(): Promise<void>

/**
* Run adapter-specific migrations needed after a major version upgrade.
*
* This method is idempotent — it is always safe to call multiple times.
* Adapters that have no pending migrations return immediately.
*
* Call this once during your deployment process before starting workers.
*/
migrate(): Promise<void>

/**
* Create or update a schedule.
*
Expand Down
4 changes: 4 additions & 0 deletions src/drivers/fake_adapter.ts
Original file line number Diff line number Diff line change
Expand Up @@ -404,6 +404,10 @@ export class FakeAdapter implements Adapter {
return Promise.resolve()
}

migrate(): Promise<void> {
return Promise.resolve()
}

async upsertSchedule(config: ScheduleConfig): Promise<string> {
const id = config.id ?? randomUUID()
const existing = this.#schedules.get(id)
Expand Down
2 changes: 2 additions & 0 deletions src/drivers/knex_adapter.ts
Original file line number Diff line number Diff line change
Expand Up @@ -88,6 +88,8 @@ export class KnexAdapter implements Adapter {
}
}

async migrate(): Promise<void> {}

async pop(): Promise<AcquiredJob | null> {
return this.popFrom('default')
}
Expand Down
2 changes: 2 additions & 0 deletions src/drivers/kysely_adapter.ts
Original file line number Diff line number Diff line change
Expand Up @@ -122,6 +122,8 @@ export class KyselyAdapter<DB = QueueDatabase> implements Adapter {
// The Kysely instance is always owned by the application.
}

async migrate(): Promise<void> {}

async pop(): Promise<AcquiredJob | null> {
return this.popFrom('default')
}
Expand Down
Loading
Loading