flaky-test-tracker

flaky-tracker

Local pass/fail history tracker for your test suite. Record test run results over time, rank tests by flip rate to spot flaky ones, and quarantine them without touching your test framework's config.

Zero runtime dependencies. Node.js >= 18.3.0.

Install

npm install -g flaky-tracker

Usage

# record a run result (call this from your CI after each test)
flaky-tracker record my.test.spec pass
flaky-tracker record my.test.spec fail

# see which tests flip the most
flaky-tracker rank
flaky-tracker rank --window 20   # only consider the last 20 runs

# inspect one test
flaky-tracker show my.test.spec

# quarantine a known-flaky test
flaky-tracker quarantine my.test.spec

# export the quarantine list for your CI to exclude
flaky-tracker export-quarantine
flaky-tracker export-quarantine --format csv

flaky-tracker unquarantine my.test.spec

All commands accept --file <path> (or --file=<path>) to point at a custom data file. Default: ./.flaky-tracker/data.json in the current directory.

Flip rate

For a test's recent N runs (all runs by default, or the last N with --window N):

flipRate = flipCount / max(1, observations - 1)

A "flip" is any adjacent pair of runs where the status changed (pass→fail or fail→pass). Fewer than 2 recorded runs always gives a flip rate of 0.

Quarantine

export-quarantine prints quarantined test ids (one per line, or comma-separated with --format csv). Wire the output into your test runner's exclude flag, e.g.:

flaky-tracker export-quarantine --format csv
# pytest -k "not ($(flaky-tracker export-quarantine --format csv | tr ',' '|'))"

flaky-tracker does not parse test framework output or modify test configs — you (or your CI script) call record after each run, and consume export-quarantine for exclusion.

License

MIT