Gitea Version
1.27.2, and 1.27.1 before it. services/actions/job_emitter.go is unchanged on main @ 475e51c.
What happened?
If a repository is deleted while one of its Actions runs still has a pending job update in actions_ready_job, that queue item can never be handled, and it is retried forever.
services/actions/job_emitter.go on main:
if err := checkJobsByRunID(ctx, update.RunID); err != nil {
log.Error("check run %d: %v", update.RunID, err)
ret = append(ret, update) // returned as unhandled, so the queue re-pushes it
}
func checkJobsByRunID(ctx context.Context, runID int64) error {
run, exist, err := db.GetByID[actions_model.ActionRun](ctx, runID)
if !exist {
return fmt.Errorf("run %d does not exist", runID) // never becomes true again
}
The run is gone, so the error is permanent, but it is reported as unhandled and pushed back. There is no retry bound and no dead letter, so the only way out is to stop Gitea and delete the queue store.
This has happened four times on our instance. The two I still have logs for:
- run 12144, 08-09 to 08-10: 89,450 log lines, 9.3% of everything Gitea logged in those days.
- run 16919, from 08-19 05:44:03 until we deleted the store the next day: about 49,000 lines in 22 hours, one every 1.6 seconds.
For 16919 the cause is in the same second:
05:44:03 router: completed DELETE /api/v1/repos/<owner>/<scratch-repo> ... 204 No Content
05:44:03 jobEmitterQueueHandler [E] check run 16919: run 16919 does not exist
It also crowds out the rest of the log, which is its own problem when you are trying to investigate something else.
To reproduce: delete a repository while one of its runs still has jobs that have not settled.
A deleted run has nothing to emit, so I think the item should be treated as handled rather than returned. Happy to send a PR if you would like one.
Related, but not the same problem: #33882
How are you running Gitea?
Docker gitea/gitea:1.27.2 on Docker Swarm, PostgreSQL 16.
Written by Claude Code (Opus 5).
Gitea Version
1.27.2, and 1.27.1 before it.
services/actions/job_emitter.gois unchanged on main @ 475e51c.What happened?
If a repository is deleted while one of its Actions runs still has a pending job update in
actions_ready_job, that queue item can never be handled, and it is retried forever.services/actions/job_emitter.goon main:The run is gone, so the error is permanent, but it is reported as unhandled and pushed back. There is no retry bound and no dead letter, so the only way out is to stop Gitea and delete the queue store.
This has happened four times on our instance. The two I still have logs for:
For 16919 the cause is in the same second:
It also crowds out the rest of the log, which is its own problem when you are trying to investigate something else.
To reproduce: delete a repository while one of its runs still has jobs that have not settled.
A deleted run has nothing to emit, so I think the item should be treated as handled rather than returned. Happy to send a PR if you would like one.
Related, but not the same problem: #33882
How are you running Gitea?
Docker
gitea/gitea:1.27.2on Docker Swarm, PostgreSQL 16.Written by Claude Code (Opus 5).