dowel

Implementation status

90-roadmap.md is the phase-level plan. This document records what is implemented. Where the two disagree, this document describes the current state.

The command reference is 60-cli.md; task-oriented how-tos are in 63-guides.md.

Approach: connect a minimal build end to end first

The roadmap orders Phase 1 (core) to completion before Phase 2 (generation), but the implementation deviated from that order exactly once: parser → evaluation → target graph → action graph → ninja generation → actual C compilation were connected first, each in minimal form.

Two reasons:

The incremental query engine and the persistent store were then plugged in afterward. The insertion point is confined to dowel_model::session::Session; both are in place (below).

Crate layout

Crate Responsibility
dowel-support spans, source maps, diagnostics, structured logging, JSON output
dowel-syntax lexing, lossless CST, error-tolerant parser
dowel-query memoization, dependency tracking, early cutoff, durability layers, cancellation
dowel-store the on-disk store: append-only value log, fixed-length index, single writer
dowel-eval typed values with provenance, expression evaluation, schema and merge semantics, configuration specialization, value serialization
dowel-model package loading, targets, the dependency graph, interface merging, why
dowel-build glob expansion, the action graph, the backend layer (ninja / direct / make / graph), compile_commands.json, execution
dowel-lsp the language server: JSON-RPC framing, diagnostics publishing, hover
dowel-cli the dowel binary
dowel-up the dowelup binary: acquiring, pinning, and switching dowel itself

Real-world test material lives in tests/projects/ (realistic fixtures) and examples/ (the documented examples).

Implemented

Syntax (dowel-syntax)

Incrementality (dowel-query)

Session reads files through this engine (dowel_model::query). Session::reload re-reads the disk, but files whose contents did not change are not re-lexed. The degree of reuse is observable via Session::query_stats.

Early cutoff cannot help on per-file queries in principle (values contain spans, so any change to the text changes the value). Where it does apply is the per-target derivations (interface and compile_env), whose fingerprints come from span-free summaries (ADR-0011). A comment-only edit never reaches merging. The path that displays provenance spans (dowel why) bypasses the memo and redoes the merge on the spot.

Persistence (dowel-store)

.dowel/cache/<format-version>/ holds lock / values / index.

Session records the files it read into the store, and the next process checks against that record to judge changes. The verdicts (UnchangedByStat / UnchangedByContent / Changed) appear at --log-level=trace.

Only evaluation results (Evaluated) are stored; per-target derivations stay in the in-process memo. The reasons are in ADR-0012. Storage is limited to files that evaluated without emitting a single diagnostic.

Restoration is keyed on a matching content fingerprint. On a match, the file is neither lexed, parsed, nor evaluated. A per-run summary appears at --log-level=debug:

store: wrote 1 values, restored 3, skipped 0 with diagnostics

FileId is the hash of the normalized path, so the same file has the same identifier across processes (ADR-0009). Restoring stored values requires no renumbering pass, and a restored document carries its own FileId, so key collisions are detectable on the value side.

Serialization lives in dowel_eval::codec: length-prefixed bytes covering every type in Document. A failed restore returns None, and the store treats unreadable values as absent. Whether the format version mismatches, the file was truncated, or something external rewrote it — the result never changes, only the speed.

Evaluation (dowel-eval)

Model (dowel-model)

Build (dowel-build)

Language server (dowel-lsp)

dowel lsp speaks LSP on stdin/stdout. The editor is the starting party, which distinguishes it from the resident daemon rejected by ADR-0002.

Documents inside a package are diagnosed to the same depth as check (ADR-0010): a workspace model is built per change — the open buffers overlay the disk (the buffer is the source of truth), and the model is loaded from every open manifest’s directory, so a document edited as someone’s dependency gets its diagnostics (e.g. its half of a merge conflict) from the dependent’s model — and then the plan stage runs over it, producing glob-expansion, path-resolution, and toolchain-existence diagnostics (empty-glob / unresolved-path / invalid-source / no-sources / missing-toolchain) from real file-system scans. Everything is read-only: the editor session never touches the network (git checkouts are reused, not fetched), never reads or writes the store, starts no external processes, and is created and dropped per change — it is not a daemon. What remains excluded — fetching, --target triggered checks, and system-package resolution — is listed with reasons in dowel_lsp::UNSUPPORTED.

VS Code extension (editors/vscode)

Starts dowel lsp and relays diagnostics and hover to the editor, with syntax highlighting for dowel.build (a TextMate grammar). Zero runtime dependencies; framing and JSON-RPC correlation are in-house (the design is in editors/vscode/README.md). Development happens inside the container via editors/vscode/dev.sh, and the checks include an integration test that talks to the real dowel lsp. It is not yet published to the marketplace.

Acquisition (dowel-up)

dowelup acquires dowel itself and pins a version per project (ADR-0013; usage in 61-acquisition.md).

Migration (dowel-build)

Scaffolding (dowel-cli)

Diagnostics and logging

What --log-level=trace shows (the material for tracing “why did this argument end up like this” when debugging):

Source Contents
session files read and their sizes, tables and key values evaluated, properties assigned to targets
input per-input verdicts against the previous run
query input changes and version advancement, files parsed/evaluated, memo validation vs recomputation
graph edge resolution, topological order
interface per-property counts of arriving values and merge results (both interface and compile_env)
specialize which arm match chose, which elements when dropped
glob files scanned with match/no-match, directories pruned, match counts
plan resolved sources, includes, defines, flags; the full command line of every action
exec why something was judged fresh; why something re-ran (which input was newer)
test the list of tests to launch (before launching), their working directories and commands
runner the declared wrappers and the command chosen for the configuration

Verification

One entry point; local runs and CI execute the same thing. What each layer answers, and where a new test belongs, is in 51-testing.md.

make verify      # run every stage, leaving results in .work/verify/

A mid-run failure does not stop the run; it proceeds to the end and fails afterward. Results land in summary.md (for humans and the GitHub summary), results.json (machine-readable), and logs/<stage>.log. CI (.github/workflows/verify.yml) stores these as artifacts and prints the summary into the job summary. Details in 50-development.md section 3.1.

Current breakdown (477 tests):

Stage Contents Count
fmt / clippy formatting check and lints (-D warnings)
unit-* per-crate unit tests 290
syntax-robustness no panics and losslessness on broken input 5
model-integration manifest loading through interface merging 10
model-incremental counting what a reload did not recompute 10
e2e compile real C and C++, run it, check the output 104
scenario operation sequences over time (edit and rebuild, configuration switches, cross-process change detection and restore) 24
fixture real-shaped projects (tests/projects/) end to end 11
diagnostics diagnostics reaching the CLI (55 cases), applying fix suggestions, location presence, check scope, coverage tracking 12
example build the real examples/hello and run its tests 3
up dowelup resolution, acquisition, and switching against an upstream fixture 3
docs link resolution and index consistency 5
startup startup-time measurement (informational; machine noise does not fail the run)

The scenario / fixture / diagnostics layers were added later. Their first runs surfaced the following four defects, none of which could appear in the pre-existing layers:

Defect Why the existing layers could not catch it
merging deduplicated by relative path only, dropping another package’s include/ once dependencies exceeded two levels the synthetic project has only one dependency level
the direct backend omitted the command line from freshness, missing flag changes a single run never sees the second execution
a directory in sources surfaced as the linker’s input file unused invalid-source had never been reached
a nonexistent source surfaced as ninja’s no known rule unresolved-path had never been reached

Measurements

The startup budget is under 10ms with nothing to do (docs/20-architecture.md 5.4). Release build, a 2-package / 2-target configuration, min/median of 20 runs. make measure produces these on their own.

Run Min Median
dowel --version 1.6ms 1.7ms
dowel check 2.3ms 2.5ms
dowel graph --format=json 2.0ms 2.2ms

Binary 1.2MB; 4 dynamic links (libc and friends). Currently inside budget. The previous figures (--version 1.2/1.4ms, check 1.5/1.7ms, graph 1.4/1.6ms) were taken on a different machine and are not directly comparable. The same-machine change in check is recorded in ADR-0010.

The effect of storing evaluation results

check min/median measured on one machine before and after ADR-0012, separated into runs without manifest changes (where restores happen) and with changes (where stores happen).

Subject unchanged, before unchanged, after changed, before changed, after
examples/hello 2.35/2.70ms 2.24/2.44ms 2.36/2.53ms 5.21/7.22ms
tests/projects/layered 3.48/3.71ms 3.28/3.60ms 3.70/4.22ms 7.48/9.35ms

At the current fixture sizes, restoring saves 0.1–0.2ms: manifests are a few hundred bytes, and lexing + parsing + evaluation together are small next to the fixed startup cost.

Runs that store gain 3–5ms, all of it the two sync_data calls in Writer::commit (measuring without sync makes the increase vanish). The cost falls only on runs that changed a manifest — the runs that proceed to a build anyway. Unchanged runs have nothing to write and skip the sync.

The savings scale with size; the sync cost does not. The break-even size cannot be measured with the current fixtures; the scale fixture (51-testing.md, “Future”) is needed.

Not implemented (deliberately deferred)

Item Standing
mmap-ing the index (currently read whole) Phase 1; reading whole suffices up to thousands of records
making loading and name resolution queries (Declared / Deps as derivations) Phase 1; today Session assembles them and passes them as inputs
the probe-fact DB Phase 2
the bench / template / toolchain kinds Phase 2 / 4
Meson introspect import Phase 3 backlog; CMake File API import and migrate verify are implemented
dowel debug Phase 4
language-server diagnostics that need fetching, --target, or external processes the editor session is read-only and host-targeted by design; the remaining exclusions are listed with reasons in dowel_lsp::UNSUPPORTED
cleaning up artifacts left on target machines; skipping redundant transfers Phase 4; transfers run every time
a native registry / tarball dependency source Phase 5; version deps delegate to pkg-config (ADR-0015) and dowel.lock records their resolutions — a dowel-run registry, if ever wanted, is a separate future decision
prebuilt acquisition for dowelup Q10; today source builds only
automatic ABI label computation Phase 6; today only must_equal verification of a hand-written abi. Nothing verifies that a surface declaring abi = "c" really is extern "C" — the claim is narrower and more checkable than a language label, and is what an IDL or a header scan would confirm (ADR-0019)
automatic composition of the ABI label from its components Q2; c_std / cxx_std are now typed values the label can read (ADR-0016), but which components make up the label, and at what granularity, is still open

Divergences from the design documents

Points where the implementation departed from the documents, made explicit. Whether to amend the documents is decided separately.

Where Document Implementation Reason
the consequence in ADR-0003 “there will be two parsers” one parser; dowel.toml strictness is imposed by validation the ADR’s rationale (third-party tools read it without a custom parser) is equally satisfied by validation, and a single tree keeps provenance and diagnostics paths simpler
types defines : Map<Ident, Val> Val implemented as a type the document’s notation was adopted as-is
abi ABI labels are computed currently a hand-written string computation is Phase 6; only the must_equal path is wired up
30-devexp.md section 1 args = ["-L", sysroot()] args : List<Str>; sysroot() cannot be written sysroot-based paths are Phase 4 (unimplemented-path-base); strings work first, widening to List<Val> when bases land
50-development.md section 3 CI runs in a --network none container built from dotfiles GitHub Actions runners (staying so for now) the path for evaluating the dotfiles flake from this repository’s CI is not set up, and there is no present need. The checks are defined solely in scripts/verify.sh, so a migration later swaps only the workflow’s internals