[18.0][ADD] queue_job: per DB channel managers with UI configuration and hot reload

[guewen]

Implements #765 (comment)

Introduction of new configuration parameters

With this in the config file, the job runner uses the static server-side configuration as before, with no hot-reload (single, shared channel manager):

[queue_job]
channels = root:10,root.p1:2

With this in the config file, the job runner uses the channels configured in the database (the job runner builds one channel manager per database):

[queue_job]
max_capacity = 10

When capacity or other options of a channel (or channels are created/deleted, paused) change, the channel manager for the concerned database is hot reloaded.

Capacity can be constrained by database:

[queue_job]
max_capacity = 10
db_max_capacity = 2

Or with patterns

[queue_job]
max_capacity = 10
db_max_capacity = prod_*:6,staging:3,*:1

When db_max_capacity is not configured, each database gets a max capacity of the global max_capacity.
When there is more jobs pending than max_capacity across the databases, a round-robin allows each database to enqueue jobs in turns.

The channels option for server-side configuration takes precedence over the new configuration.

What does this bring?

Using this, I can, for instance

• Each database has its own channel tree, which can vary depending of installed addons
• Pause and resume a channel
• Create a new channel, assign a job function this channel, set a capacity at 1 (or anything) or sequential, all from the UI and applied live by the job runner for a single database

Migration path

Since the server-side configuration (ODOO_QUEUE_JOB_CHANNELS or channels in config file) is used by default, updating the addon has no effect. It allows to configure the channels capacity and other options on the UI / by script of the databases, then the server-side configuration can be replaced by the parameters max_capacity and db_max_capacity (or their environment variables counterparts).

Caveat

When using per DB channel managers, we cannot have a shared channel for several databases since each database has its own channels (use case mentioned in #765 (comment)). It could be possible to implement it using a single channel manager that coexists with the db channel managers, yet to define the specifics, e.g. use another root (shared:3, shared.foo:2) or a new option shared_channels = root.shared where the subchannels defined in this option are excluded from the DB channels.
In the meantime, using the server-side channels instead of per-db channels is a trade-off to accept if this use case is essential for a server.

[OCA-git-bot]

Hi @sbidoul,
some modules you are maintaining are being modified, check this out!

[guewen]

Hi @sbidoul , can I have your thoughts on this, before I start completing the tests and docs?

[OCA-git-bot]

Hi @sbidoul,
some modules you are maintaining are being modified, check this out!

[sbidoul]

Hi Guewen,

I have not had time to do a proper review, but I think I'm on board with this.

Great solution!

I have some doubts about the round robin rule, but on the other hand I don't have serious multi-db use cases myself so this can be addressed later.

[sbidoul]

Handle the new default subchannel capacity too?

[amh-mw]

It looks like the subchannel pull request (#767) has a merge conflict. I'll take a look at it today.

[sbidoul]

Ah I thought that was merged already.

[guewen]

I'm writing a benchmark script to have some comparison points (I'll open another PR for the script at some point). It also demonstrates an interesting effect of the round-robin (nothing surprising but nice to see it with numbers).

In the current state of my benchmark, the script enqueues 5k jobs in db0, then 5k in db1, then 5k in db2.

With the global channel manager, the jobrunner executes the jobs strictly by creation order, so the 5k jobs of db0 are executed, then those of db1 and finally those of db2.

The round-robin (yet to test in real situation, which I do not have as we have a single DB in production), might help in situations where a large DB clutters the queue with a large quantity of jobs and starve other databases.

Do not give much value to the number otherwise, it is only a single run of each.

global channel manager

The throughput per database is much higher than the round-robin version: this is because the results are based on when the 5000 jobs are started and when they are all done, and all jobs the bench0 are done before the others, the 5k jobs are done much earlier.

[bench0] jobs:             5000 (5000 done, 0 failed)
[bench0] wall time:        32.6s
[bench0] throughput:       9196 jobs/minute
[bench0] avg exec_time:    0.0063s
[bench0] p95 exec_time:    0.0100s
[bench0] max connections:  10
[bench1] jobs:             5000 (5000 done, 0 failed)
[bench1] wall time:        50.6s
[bench1] throughput:       5930 jobs/minute
[bench1] avg exec_time:    0.0098s
[bench1] p95 exec_time:    0.0184s
[bench1] max connections:  11
[bench2] jobs:             5000 (5000 done, 0 failed)
[bench2] wall time:        55.1s
[bench2] throughput:       5445 jobs/minute
[bench2] avg exec_time:    0.0108s
[bench2] p95 exec_time:    0.0165s
[bench2] max connections:  8
## summary
jobs:             15000 (15000 done, 0 failed)
wall time:        138.3s
throughput:       6506 jobs/minute
max connections (peak per db): {'bench0': 10, 'bench1': 11, 'bench2': 8}

round-robin

In the round-robin version, the throughput is equal for each database, because their jobs are all starting and ending at about the same points.

[bench0] jobs:             5000 (5000 done, 0 failed)
[bench0] wall time:        124.5s
[bench0] throughput:       2410 jobs/minute
[bench0] avg exec_time:    0.0077s
[bench0] p95 exec_time:    0.0142s
[bench0] max connections:  5
[bench1] jobs:             5000 (5000 done, 0 failed)
[bench1] wall time:        124.5s
[bench1] throughput:       2410 jobs/minute
[bench1] avg exec_time:    0.0075s
[bench1] p95 exec_time:    0.0137s
[bench1] max connections:  7
[bench2] jobs:             5000 (5000 done, 0 failed)
[bench2] wall time:        124.5s
[bench2] throughput:       2410 jobs/minute
[bench2] avg exec_time:    0.0076s
[bench2] p95 exec_time:    0.0139s
[bench2] max connections:  7
## summary
jobs:             15000 (15000 done, 0 failed)
wall time:        124.5s
throughput:       7229 jobs/minute
max connections (peak per db): {'bench0': 5, 'bench1': 7, 'bench2': 7}

memo for myself on the benchmark

odoo -c odoorc -d testqueue18 \                
    --load=web,queue_job --workers=0 --log-level=warn -i queue_job --stop-after-init
dropdb --if-exists -p 6432 -hlocalhost bench0
dropdb --if-exists -p 6432 -hlocalhost bench1
dropdb --if-exists -p 6432 -hlocalhost bench2
createdb -p 6432 -hlocalhost -T testqueue18 bench0
createdb -p 6432 -hlocalhost -T testqueue18 bench1
createdb -p 6432 -hlocalhost -T testqueue18 bench2

BENCHMARK_DATABASES=bench0,bench1,bench2 BENCHMARK_JOBS=5000 \
    odoo shell -c odoorc --log-level=warn < queue/queue_job/scripts/benchmark_jobs.py 

BENCHMARK_RUN='benchmark-2026-08-31-15:04:50' BENCHMARK_DATABASES=bench0,bench1,bench2 \
  odoo shell -c odoorc --log-level=warn < queue_job/scripts/benchmark_jobs.py

ODOO_QUEUE_JOB_CHANNELS=root:8 \
  odoo -c odoorc -d bench0,bench1,bench2 \
  --load=web,queue_job --workers=0 --log-level=warn

# or

ODOO_QUEUE_JOB_MAX_CAPACITY=8 \
  odoo -c odoorc -d bench0,bench1,bench2 \
  --load=web,queue_job --workers=0 --log-level=warn

[amh-mw]

With the current global channel manager, the jobrunner executes the jobs strictly by creation order, so the 5k jobs of db0 are executed, then those of db1 and finally those of db2.

A single database running multiple companies can also find itself at the mercy of strict creation order.

[guewen]

A single database running multiple companies can also find itself at the mercy of strict creation order.

This one won't be solved though