fix(actions): reduce DB load from many polling runners and fix stalled task pickup

[bircni]

Problem

On instances with many global/instance-scoped Actions runners (we saw it with 100+ runners against a 32-core server backed by MariaDB), the server sits at ~100% CPU with heavy SQL traffic whenever all runners are active, and a single bad job can stop Actions entirely. Three issues compound:

1. A write on every poll. Each FetchTask poll unconditionally runs UPDATE action_runner SET last_online=? (interceptor.go), regardless of whether there's any work — so N runners × poll-rate = a constant stream of indexed writes.

2. A global-version thundering herd over an unbounded scan. IncreaseTaskVersion always bumps the global scope (0,0) on every job state change anywhere in the instance. Since global runners all read that scope, a single queued job invalidates all of them, and on their next poll they all enter the full PickTask → CreateTaskForRunner transaction at once. Each one scans every waiting job (unbounded), matches labels in memory, and only one wins each job via the optimistic task_id=0 update — the other 99 do wasted work. At high throughput the version-skip optimization barely fires.

3. One unpreparable job stalls everything (#37586). Assignment picks the oldest waiting job and returns an error if it can't be prepared — e.g. its run was deleted out from under it, or its payload won't parse. Because that job stays at the head of the queue, every runner's poll hits it, fails, and never advances. Actions stop instance-wide until the job is manually cleared.

Changes

• Debounce last_online. New RunnerHeartbeatInterval (30s) + ShouldPersistLastOnline helper; the interceptor persists last_online only when it's stale enough to matter, skipping the UpdateRunner call entirely otherwise. last_active is unchanged (still written on UpdateTask/UpdateLog). Safe because the offline threshold is 60s.

• Throttle concurrent task assignments. New MAX_CONCURRENT_TASK_PICKS setting (default 16) backs a process-wide non-blocking semaphore via TryPickTask. When the limit is hit, the pick is skipped and — crucially — the response echoes the runner's request tasks version instead of latestVersion, so the runner retries on its next poll rather than sleeping until the next bump (which would stall the backlog).

• Match labels in SQL instead of scanning in memory. Runner labels are now matched against a normalized action_run_job_label table (one row per runs_on label) via a correlated NOT EXISTS, so CreateTaskForRunner selects the oldest matchable waiting job directly (LIMIT 1) regardless of backlog size. This removes the unbounded scan + in-memory filtering entirely — without the head-of-line starvation a simple row cap would introduce. The table is backfilled for waiting/blocked jobs and kept in sync through a single InsertActionRunJob entry point.

• Add a composite (status, updated) index so the "oldest waiting job" lookup is an index seek rather than a sort of the whole waiting backlog.

• Make the pick resilient (fixes gitea actions stopped working #37586). A job that can't be prepared is marked failed and skipped (bounded per poll) so the next candidate is tried, instead of erroring out and stalling every poll. The failure marking uses a direct row update because the normal status path aggregates the run, which may no longer exist.

Behavior / compatibility

• No runner-visible semantic change: offline/idle detection, oldest-first ordering, and label-match semantics (a runner must cover all of a job's labels) are preserved — label matching just moves from memory into SQL.
• MAX_CONCURRENT_TASK_PICKS defaults to 16; lower it to protect a small DB, raise it for very high job throughput.
• One migration adds the action_run_job_label table (backfilled) and the (status, updated) index.

[wxiaoguang]

The existing indices/SQLs/logic are wrong

• Improve the performance for runner of picking up task #37163 (review)
• related: gitea actions stopped working gitea actions stopped working #37586

But they just don't want to or don't know how to fix it.

[wxiaoguang]

Is it right? I don't understand the details of the Actions runner, but the code below checks the jobs against the labels.

What if the first 100 jobs don't match but the 101 job matches? And the first 100 jobs just get stuck?

	// TODO: a more efficient way to filter labels
	var job *ActionRunJob
	log.Trace("runner labels: %v", runner.AgentLabels)
	for _, v := range jobs {
		if runner.CanMatchLabels(v.RunsOn) {
			job = v
			break
		}
	}
	if job == nil {
		return nil, false, nil
	}

[bircni]

this needs a better fix - working on it

[bircni]

@wxiaoguang could you check again please?

[wxiaoguang]

It needs to be "cancelable" if ctx is canceled.

[wxiaoguang]

@wxiaoguang could you check again please?

Better to ask real Actions users to check.