How a build runs

A build is driven by the engine over a recipe, in order:

1. Resolve every component’s source, and plan the order over them. Once per run, whatever it targets: neither a source nor the order over the sources depends on an architecture.
2. Build each component in that order, and record what happened. Once per architecture the recipe names, in the order it names them.

Splitting there is what makes a run targeting two architectures a build of the same commits at the same stamped versions — a pair of separate runs cannot promise that, since a git-ref naming a branch may move between them. See Cross-architecture builds.

1. Resolve

Every component’s tree is assembled under the work directory, from its source, then any packaging overlay, then any patch series. A git source is cloned or updated, its ref checked out, and its submodules initialized — so a submodule superproject such as cosmic-epoch resolves its members. A source.path tree is copied under the work directory as it stands, afresh each run, so nothing writes into the tree the recipe named. Either way the result is a source tree that holds the component’s debian/ directory, and it is one src2deb owns.

Every component is resolved, whatever the run was asked to build, because the build order derives from all of them: which component produces a package is read from that component’s own debian/control, so the graph is complete only once every control file has been read. A --only run over a cold work directory therefore clones the whole recipe once; later runs fetch.

A component whose source will not resolve is a failure of that component, not of the run — the same as one whose build fails. Without --keep-going the run still stops at the first; with it, the run goes on and the component is recorded as failed. A component outside the run’s selection is weaker still: the run was never going to build it, so its source failing is reported and passed over whatever --keep-going says.

Any Git LFS content in a resolved tree is then fetched. A repository may hold large assets outside itself, leaving a short text pointer in their place, and a checkout made without LFS support writes those pointers as ordinary files. They are valid files, so a build embeds or installs one and succeeds; the substitution only surfaces when the installed program reads the asset and finds a stub. Resolve therefore fetches the real content and then verifies no pointer survives in the tree it is about to build. A surviving pointer fails the component, on the same terms as any other resolve failure.

The check is scoped twice over. It covers the subdirectory actually built, so an unrelated component’s assets elsewhere in a superproject do not concern it; and within that, the files git tracks, which is the only place a pointer can come from. That second bound also keeps it off a previous build’s vendored crates, which stay in the tree between runs and are not the component’s assets.

A source.path tree is scanned on the same terms but never fetched for: a pointer there fails the component with the command that fixes it, and src2deb leaves the tree alone. A path that is not part of a git working tree carries no pointers, and is passed over.

A component that declares a packaging overlay then has one applied. Its source is resolved by the same two origins the component’s own source is, and its debian/ directory replaces whatever the source tree held. Only debian/ is taken, so a packaging repository that also carries a copy of the upstream tree contributes its packaging and not its idea of the source.

Last, a component’s declared patch series is applied over the tree, in the order the recipe lists it — after any overlay, so a patch may fix packaging the recipe did not write. All of this happens inside resolve, before any debian/control is read, so an overlay may supply what a component build-depends on and a patch may change it, and the build order follows the assembled file. A patch that does not apply fails the component; both an overlay and a series are inputs of the component’s fingerprint, so changing either rebuilds. See Packaging overlays and Patches.

2. Plan

Each component’s debian/control is read for two things: what it build-depends on, and the binary packages it produces (its Package: stanzas). An edge runs from a producer to each consumer, and a topological sort of that graph yields the build order. Build-dependencies the recipe’s own components do not produce are left to archive and pool resolution at provision time, so the components may be listed in any order.

Only the first alternative of an a | b build-dependency group is considered, so an in-set build edge must be a direct dependency rather than a later alternative. The COSMIC recipe’s single edge is direct, so this bounds how a future recipe may express an edge rather than affecting the current build.

3. Build each architecture

Each architecture the recipe names is built in turn, against the one resolution above. What it settles for itself is its build roots, the pool it publishes into, its output tree, and its manifest — all keyed by suite and architecture, so the architectures of a run never write over one another.

It also settles which of the recipe’s binary packages to build: an architecture that does not own the recipe’s Architecture: all output builds only the architecture-dependent half, and skips a component that has no other half. See Who builds the Architecture: all packages.

For each component, in order:

• A build root is provisioned — the base system, the toolchain, and that component’s build-dependencies, resolved against the archive sources and the local pool. The archive is read once for it: a bootstrap installs the plan that resolve produced rather than resolving again, so the packages that land are the ones whose digests were verified against the release. See Build roots, and Provisioning progress below for what it reports while it works.
• The component is built in two cage passes, for packages that vendor Rust crates, as COSMIC’s do. The vendor pass runs debian/rules clean in a cage with the host network, which triggers the component’s own vendoring and leaves a vendor.tar in the tree. The build pass runs dpkg-buildpackage -nc in a cage with an isolated network, building offline from that vendor.tar; -nc keeps it from re-triggering vendoring. Before it builds, the build pass prepends a debian/changelog entry stamping the build’s version — see Package versions — to its own copy of the tree, leaving the resolved checkout as upstream wrote it. Both passes carry SOURCE_GIT_HASH, the resolved commit, which packaging reads to stamp a revision into the built binary. Output is assembled into lines and streamed to the terminal, and the artifacts are read from the .changes file the build writes.
• The finished packages are published to the local pool, so the next component resolves against them.

A component’s failure is recorded rather than propagated: by default the run stops at the first failure, and with --keep-going it continues to the next component. Either way the run finishes with a report of what built and what failed.

A run that stops accounts for the components it never began as not reached, so built + failed + skipped covers every component the run set out to build. They are recorded with what their source resolved to, and the manifest keeps whatever it already said of them — a component a prior run built stays recorded as built.

The same rule decides whether the next architecture starts. Without --keep-going a failure stops the run where it is, so the architectures after it are never begun and the summary names them; with it, every architecture is attempted and the summary tallies them all.

The vendor pass, and the one source that skips it

The vendor pass is the only step in a run that executes upstream’s own code with the host network. Everything that produces packages happens in the build pass, which is isolated. That split is deliberate: the build is hermetic, and acquiring the dependencies to build it is not.

A component built from a Debian source package skips the vendor pass entirely. A source package carries everything its build needs — that is what makes it a source package — so there is nothing for the pass to fetch, and the component is built start to finish inside an isolated cage.

Every other source runs it. Whether a checkout, a tree on disk, or a release archive actually needs anything fetched is the packaging’s to decide by what its debian/rules clean target does; for a component that vendors nothing, the pass runs and does nothing.

What ends a run outright is what leaves it nothing coherent to do: a selection naming a component the recipe does not have, a dependency cycle, a target suite with no version tag, a selection that leaves out a producer of a selected component’s build-dependencies, and a shared base that will not bootstrap. Each of those is settled before the build phase, so the run prints the error alone and writes no manifest — there is nothing yet to record.

Building in parallel

With --jobs N, up to N components build concurrently — components within one architecture, not architectures alongside each other. Two emulated builds running at once contend for the same cores and the same package cache without finishing sooner. A scheduler over the dependency graph releases each component the moment the components producing its build-dependencies have published, so the build fans out across the graph’s independent components while still ordering each consumer after its producers. Every phase runs in parallel, including the two that touch shared state. The package cache stages each download under a name unique to its writer, so two components fetching the same package both succeed; and the pool serializes publishes internally while making each one visible to a reader all at once, so a component may resolve against the pool while another publishes into it. A single job (the default) reproduces the sequential order exactly.

What a build root sees of the pool follows from when it was provisioned. A component whose build-dependencies the recipe produces is released only after its producers have published, so it always resolves against their packages — that is what the scheduler is for. A component that depends on none of them is released as soon as a worker is free, so whether a sibling’s packages had reached the pool by then depends on how the run happened to be scheduled.

That is visible only where the pool holds a package name the archive also carries — a rebuild of a package the archive ships, whose default stamp outranks the archive’s own copy wherever both are read. A recipe holding one may resolve its other components differently between two otherwise identical parallel runs, and may miss the build-root cache for the same reason, since a root’s plan key names every package it resolved. --jobs 1 builds in a fixed order and so resolves against a fixed pool.

4. Record

Each architecture writes a provenance manifest to <work>/manifests/<recipe>/<suite>/<architecture>.toml as it finishes, mapping every component to what its source resolved to, the .buildinfo its build wrote, and the package versions it produced. A component whose source never resolved is recorded as failed in every architecture’s manifest: it failed once, for the run, and no architecture has it. See The provenance manifest.

Provisioning progress

Provisioning is the longest stretch of a run: a cold shared base resolves, fetches, and unpacks several hundred packages before the first build starts. It reports as it goes, labeled with the root each event belongs to — base for the shared base, and the component’s own name for a per-component root — so a concurrent run’s interleaved output stays attributable.

At the default verbosity a run announces each root as it starts on it, says how many packages the shared base will install, then counts the packages it downloads and unpacks:

src2deb: provisioning the shared base
src2deb: base: 163 package(s) to install
src2deb: base: downloading 66/163
src2deb: base: unpacking 66/163
src2deb: base: installing the rustup 1.97.0 toolchain
src2deb: building cosmic-comp (1/5)
src2deb: provisioning the build root for cosmic-comp

The base’s package count reports by default because it is the largest single commitment a run makes, and a cold work directory has no other way of saying so before the counters start. Each component’s own count stays behind -v, where it is one line among many rather than the run’s headline cost.

The toolchain line appears only when the recipe pins one and the root was actually provisioned; a root reused from a prior run already carries it. Its own output is captured rather than printed, so a failure reports what the installer wrote and a successful install says nothing further.

On a terminal, and with a single job, the counter is one line rewritten in place. With --jobs N, or with stderr redirected to a file, it becomes a line each time the count passes a tenth of the total — several workers rewriting one row is unreadable, and a redirected stream would collect carriage returns rather than lines.

A package the shared cache already holds is not downloaded, so a warm cache counts only what it has to fetch, and the unpack counter carries the run.

-v replaces the counters with a line per package, and adds each URL fetched and the size of each resolved package set. -q prints no provisioning progress at all.

Notes that report whatever the verbosity

Two things report at every verbosity above -q, because they change what the run guarantees rather than what it is doing:

• A foreign-architecture target, which runs every compiler invocation under emulation. See Cross-architecture builds.
• No unprivileged overlay, which drops the run to full reprovisioning, the weaker of the two isolation guarantees.

The layered default stays behind -v, as the stronger strategy and the ordinary case. Both notes are shown in Troubleshooting.

In-cage build output

Each line a build writes is passed through indented by two spaces, from both its standard output and its standard error, unchanged otherwise. The two streams render alike because Debian’s build tooling uses the choice of stream to separate its output rather than to signal severity — dpkg-buildpackage writes its ordinary progress to stderr. With --jobs N, each line also carries its component, since several builds’ output interleaves.

Cancelling a run

Ctrl-C — or SIGTERM — stops a run at the next point where stopping leaves a coherent state behind, rather than killing the process mid-provision. The run prints cancelled; stopping, winds down, writes its manifest, and exits 130.

What “the next point” means depends on what the run is doing:

A build is stopped with SIGTERM first, so dpkg-buildpackage can finish the file it is writing, and killed if it has not exited within five seconds. Every build sandbox is also tied to this process’s lifetime, so even a src2deb that is killed outright leaves no in-cage build running.

A second Ctrl-C exits immediately, for the cases in the table above where the first has to wait.

What a cancelled run leaves behind

Everything the next run can pick up from:

• Components that finished are built, published to the pool, and recorded in the manifest as built. A later --skip-published run skips them.
• The component the cancel interrupted, and any it never reached, are recorded as skipped, with what their source resolved to. Nothing claims they were built.
• The architectures the run had not started have no manifest written for them and nothing published, so nothing was recorded that a later run would have to undo. The summary names them.
• A partly-provisioned build root is removed rather than left half-made, so it can never be mistaken for a usable one. The next run provisions it from clean.
• The work directory lock is released, unless the run was ended by a second Ctrl-C or killed outright. Then <work>/.lock survives, and the next run reports it as locked and names the file to remove.

The run’s exit status is 130 whenever it was cancelled, including when a component had already failed: the run did not finish, so nothing can be concluded about the components it never reached.

The local pool

The pool is a dists/-structured .deb archive, trusted without a signature, that carries build-dependencies from one component to the next. It is written up front as a valid empty pool, so the first component can declare it as a repository; each build then adds its packages and regenerates the index. A component that build-depends on an earlier one resolves against the packages src2deb just built.

A pool lives at <work>/pool/<suite>/<architecture>/ and belongs to that pair. Any number of recipes may publish into one pool — that is what sharing a work directory is for — but a run targeting another suite or architecture publishes into a pool of its own.

Scoping by architecture is a requirement. An Architecture: all package’s file name carries no architecture, and its stamped version is identical however the package was built, so the same component built for amd64 and for arm64 yields one file name for two different files. Sharing a pool would mean the second publish overwriting the first and leaving the earlier architecture’s Packages naming a checksum that no longer matches — a hash mismatch at the next apt update.

Scoping by suite is a choice. A rebuild for another suite differs in the deb13/deb14 field of its version, so its file names differ and it could share the pool. It gets one of its own so that a pool is a single servable archive — one that can be signed, mirrored, or discarded without reference to another suite’s builds, and whose index accounts for every file beside it. That is a departure from Debian’s layout, where one pool/ serves every distribution in the archive.

Publishing is incremental and forward-only

A publish merges into whatever the pool already indexes rather than replacing it, and keeps the highest version of each package name. Two consequences are worth knowing before serving a pool to anything but the next build:

• A lower version does not publish. Its .deb is copied into the pool, but the index keeps the higher version it already recorded. Correcting a package’s version downward therefore needs a pool that never saw the higher one.
• There is no unpublish. Nothing removes a package from the index, and deleting a .deb by hand leaves the index naming a file that is no longer there, which a client reports as a failed fetch. To stop shipping a package, stop anything from depending on it rather than trying to take it out of the pool.

Both follow from the index being merged under the pool’s lock, which is what lets one component resolve against the pool while another publishes into it.

The pool directory grows until it is pruned

The index keeps only the highest version of each package name, but nothing removes the file a higher version superseded. Because every build carries the build date in its version, each run writes a fresh set of .deb files and leaves the previous set on disk, indexed by nothing. A recipe rebuilt daily accretes its full artifact set per day.

That is the pool’s design: a publish is additive and takes no view on what came before. Removing what it superseded is a separate step — src2deb prune, or --keep N on a build, which runs it once the run has finished. Watch the directory’s size rather than the index’s, which stays one entry per package however many versions sit behind it.

The pool is not the largest thing in a work directory, though: the shared base and the package cache usually are. See The work directory.

Signing follows a run, never precedes it

A run publishes an empty set into its pool before building, so the first component has a valid Release to declare as a repository. A publish replaces that Release, and a signature only covers the Release it was made over, so publishing discards any signature the pool carried.

Sign a pool after the run that finished it, and re-sign after every run that publishes into it. A run with nothing to build does not touch the pool at all, so a signed pool stays signed across a fully-skipped re-run.