CI/CD pipelines
The job DAG, scheduling and dispatch, execution isolation, secrets, environments, and built-in DORA metrics.
The job graph#
A pipeline is a directed acyclic graph of jobs, each with:
| Field | Meaning |
|---|---|
needs | Names of jobs that must reach a terminal state before this job is eligible to run — this is what makes it a DAG rather than a flat list. |
runWhen | ALWAYS, ON_FAILURE, or the default (on success) — controls whether the job runs given its dependencies' outcomes; a job whose condition isn't met is marked SKIPPED, not silently omitted. |
requiredTags (tags:) | A runner only claims this job if it advertises every tag listed — e.g. tags: [gpu] restricts a job to runners started with RUNNER_TAGS=gpu. An untagged job runs on any runner. |
cacheKey | Jobs sharing a cache key share a persistent workspace subdirectory across runs (e.g. for node_modules), avoiding a full reinstall every pipeline. |
approvalStatus (approvers) | When set, the job is held at PENDING regardless of its dependencies until a permitted approver acts — used to gate production deploy jobs behind a manual sign-off. |
| downstream triggers | On success, a job can enqueue a pipeline run in a different repository — the mechanism behind multi-repository release trains. |
Build this graph visually in Pipelines → Editor, or author .khub/ci.yml directly; the editor and the YAML parser both compile down to the same internal job-graph representation, so either path produces identical runtime behavior.
Scheduling and dispatch#
Pipeline creation inserts every job row as PENDING. services/runner computes, after each job reaches a terminal state, which sibling jobs just became eligible (all of needs terminal, condition satisfied, not still gated on approval) and enqueues them onto a Redis-backed BullMQ queue named ci-jobs.
A runner that dequeues a job whose requiredTags it doesn't satisfy declines it by throwing rather than executing — BullMQ's retry/backoff (up to 20 attempts, exponential backoff starting at 5s) then gives every other runner in the fleet a turn to claim it before the job is ever reported as unschedulable. This "decline and retry" pattern is what lets a heterogeneous runner fleet (some tagged gpu, some not) share one queue without a separate scheduler process making placement decisions.
Execution isolation#
Each job runs in its own Docker container, created fresh per job (never reused across jobs or pipelines):
CapDrop: ["ALL"]andSecurityOpt: ["no-new-privileges"]— the job process gets no Linux capabilities beyond an unprivileged user's defaults, and cannot regain privileges even via a setuid binary in its image.- Memory and memory+swap are both capped at the runner's configured
jobMemoryMb; CPU is capped viaNanoCpus; process count is capped at 512 (PidsLimit) to bound fork-bombs. - Networking is a private bridge network per container — jobs can reach the public internet (to
npm install, pull images, etc.) but not the host's other containers or the Docker socket. - The job's workspace and cache directory are mounted in via a volume subpath (
VolumeOptions.Subpath) rather than a bind mount from the host filesystem, so a job cannot traverse outside the directory scoped to it even with an escaped container. - Job stdout/stderr is streamed to a per-job log file and, in the same pass, run through a secret masker that redacts any resolved secret value it can match — a job that does
echo $DEPLOY_TOKENgets a masked log line, not a leaked credential, without needing the job author to remember to suppress it.
Secrets#
Secrets are scoped to ORGANIZATION, PROJECT, or REPOSITORY and cascade — a job's environment is the union of all three scopes that apply to it, with more specific scopes able to shadow a same-named broader one. Values are stored encrypted at rest and decrypted only inside the runner process immediately before injecting them into the job's container environment. A secret can additionally be marked protected, in which case it's withheld entirely from jobs running on a non-protected branch — so a production database credential can be made unavailable to a pipeline run on an arbitrary feature branch, even one opened by a maintainer.
Environments & deployments#
An Environment (e.g. staging, production) tracks the currently-deployed pipeline/commit and its deployment history. Redeploy re-runs the exact same deploy job against the same artifact; rollback targets a specific prior successful deployment — both are ordinary pipeline job executions, not a separate deployment engine, which is why they show up in the same job logs and status model as everything else.
Pipeline status#
A pipeline's overall status is derived, not stored independently: computed from the set of its jobs' statuses every time a job reaches a terminal state (PASSED, FAILED, CANCELED, SKIPPED). Once every job is terminal, the runner notifies the API over an internal endpoint so it can fire the PIPELINE_COMPLETED webhook event and update DORA metrics.
Engineering insights (DORA metrics)#
An organization's Insights page computes the four DORA metrics directly from pipeline and deployment history — no separate instrumentation or opt-in is required:
- Deployment frequency — count of successful deployments to production-tagged environments over time.
- Lead time for changes — elapsed time from a commit landing on the default branch to its deployment.
- Change failure rate — proportion of deployments followed by a rollback or a failed pipeline against the same environment within a short window.
- Mean time to recovery — elapsed time between a failed deployment and the next successful one to the same environment.