siftwise v0.3 adds risk-based test ordering and time budgets for CI

siftwise, an open-source, self-hosted tool that narrows CI test runs to the tests a code change can actually affect — and labels each failure as new, already broken on main, or flaky — has released version 0.3.0. According to the announcement on dev.to, the theme is smarter selection: tests now run in order of failure likelihood, runs can be constrained by a time budget, and selection support extends to JavaScript and Java builds.

Riskiest tests first

Previously siftwise chose the right tests but ran them in path order, so a failure in the last test still cost a full wait. In v0.3 every selected test gets a score, a rank and a reason, and the riskiest run first.

The score is a fixed, documented formula with no machine learning: a link term worth 6 points when the change reaches the test directly, 3 through other files, or 0 by history only, plus three times a recency factor that halves every seven days since the test last failed, plus two times its failure rate, with recency and rate discounted for flaky tests, and a flat 2 points for new tests. Ties go to the faster test.

Ordering is on by default and never changes which tests run, only when; order settings on the CLI, the GitHub Action, GitLab and MCP restore the old behaviour. How far the order is respected depends on the runner: pytest follows the given arguments unless a plugin such as pytest-xdist reorders them, Go packages run one after another riskiest first, Jest treats the order as a hint because it runs files in parallel, and Maven modules follow the reactor's dependency order.

Replaying real CI history, with each run evaluated using only prior data, the maintainer reports that the median share of a run passing before the first failure dropped from 37.3% to 1.4% on RDFLib/rdflib (Python, 17 runs) and from 42.0% to 25.6% on agent-substrate/substrate (Go, 19 runs). Recall did not change, since the same tests ran in a different order. A cost-aware variant, ordering by risk per estimated second, was not clearly better on both repos and did not ship.

The post is also candid about a regression. A new import-depth rule, initially applied to every language, made substrate worse — the median went from 23.3% to 34.4% by test count — because 83% of Go's import-linked packages sat at depth one and were promoted above the end-to-end suites a depends rule selects. The fix excludes Go packages from depth-based promotion while keeping it for Python, Jest and Java test files, which restored substrate's numbers. The maintainer claims no speed-up from import depth itself, since rdflib's medians did not move.

Time budgets

A new budget mode answers the question of which tests fit in, say, five minutes. siftwise select --budget 5m — or 90s, or 1h30m — runs the riskiest tests whose estimated durations fit, with equivalents on the GitHub Action (which outputs budget-limited and budget-left-out flags), GitLab, MCP and the API. The mode is off by default.

Two caveats are spelled out. The budget is an estimate built from each test's mean historical duration, not a hard cap, so a real CI timeout should stay in place. And a budget-limited run is not a complete check: pull request comments state how many tests were left out, and the full selection should run before merging. In shadow mode, a failure the budget omitted is recorded as a miss with the cause budget, letting teams measure what a budget costs on their own repositories.

Jest, Java and fixes

Jest selection now uses a real JS/TS import graph, with no Node.js required, including computed imports such as import.meta.glob. Java selection covers Maven and Gradle, including multi-module builds, with build-file scoping per module. Unplanned additions include Jest support without addFileAttribute, re-runs for JUnit 5 DisplayName tests, Go workspaces and nested modules, a GitLab Tests tab, a change so go list no longer edits go.mod, and six bug fixes — three of which had caused siftwise to silently miss tests.

Upgrading

The release carries two additive database migrations that run automatically when the server starts, and the maintainer recommends updating server and clients together. A new CLI against a 0.2 server degrades gracefully except for budgets, which cannot be dropped silently: the Action, GitLab and MCP fall back to the full suite with a warning, and the CLI exits with an error. An old CLI works with a new server unchanged. The project now also runs siftwise on its own CI on every pull request and push, and it gained its first two outside contributors.

Why it matters

Time to first failure is the delay developers actually feel. Cutting it from roughly a third of a run to under 2% on the replayed rdflib history — without changing which tests run — is a meaningful efficiency win for self-hosted CI. The fixed, documented scoring formula keeps behaviour auditable rather than opaque, and publishing the numbers that went the wrong way, from the import-depth regression to the unshipped cost-aware order, is the kind of evidence that makes efficiency claims credible.