yamltest

YAMLTest

Declarative YAML-based test runner for HTTP endpoints, shell commands, and Kubernetes resources.

Define tests in YAML, run them from the CLI or import them programmatically. No test framework boilerplate required.

---

Installation

From npm (once published):

npm install -g yamltest

From a local clone (before publishing):

# Install globally from the repo directory
npm install -g /path/to/YAMLTest

# Or run directly without installing
node /path/to/YAMLTest/src/cli.js -f your-tests.yaml

# Or link it for development (makes YAMLTest available system-wide, auto-updates)
cd /path/to/YAMLTest && npm link

As a dev dependency in another project:

npm install --save-dev /path/to/YAMLTest

---

Quick start

YAMLTest -f - <<EOF
- name: httpbin returns 200
  http:
    url: "https://httpbin.org"
    method: GET
    path: "/get"
  source:
    type: local
  expect:
    statusCode: 200
    bodyContains: "httpbin.org"
EOF

Output:

  ✓ httpbin returns 200 312ms

  1 passed | 1 total

---

CLI

USAGE
  YAMLTest -f <file.yaml>
  YAMLTest -f -              # read from stdin (heredoc)

OPTIONS
  -f, --file <path|->   YAML file to run, or - for stdin
  --retries <n>         Force the retry count for every test, overriding the
                        per-test `retries` and the default. Use --retries 0 to
                        run once when debugging a failure.
  --check               Validate YAML structure only; do not run tests
  -h, --help            Show this help

ENVIRONMENT
  DEBUG_MODE=true       Enable verbose debug logging
  NO_COLOR=1            Disable ANSI colour output

When a test fails and you want to debug it, run the same file with --retries 0 so it executes exactly once instead of retrying up to the default 120 times:

yamltest -f test.yaml --retries 0

Exit codes: 0 = all passed, 1 = one or more failed.

---

Input validation

All test definitions are validated against a strict schema before any test executes. Invalid input produces clear error messages pointing to the exact location of the problem:

Validation failed:

  Test #1 ("login") /http/method: must be one of: GET, POST, PUT, DELETE, PATCH
  Test #3: must define exactly one of: http, command, wait, httpBodyComparison
  Test #5 ("check pods") /source: missing required property "selector"

Validation checks include:

• Exactly one test type per definition (http, command, wait, or httpBodyComparison)
• Required fields per test type (e.g., command.command, wait.target)
• Value types and allowed enums (e.g., source.type must be local or pod)
• Conditional requirements (e.g., source.type: pod requires selector; setVars requires expect for HTTP/command tests)
• Unknown property detection on sub-objects (catches typos like htpp instead of http)

---

Programmatic API

const { runTests, executeTest, validateTestDefinitions } = require('yamltest');

// Run one or more tests from a YAML string (array or single object)
const result = await runTests(yamlString);
console.log(result.passed, result.failed, result.skipped, result.total);
// result.results → [{name, passed, error, durationMs, attempts}]

// Run a single test (low-level)
await executeTest(yamlString); // returns true or throws

// Validate test definitions without executing them
const yaml = require('js-yaml');
const definitions = Array.isArray(yaml.load(yamlString))
  ? yaml.load(yamlString)
  : [yaml.load(yamlString)];
validateTestDefinitions(definitions); // throws on validation errors

---

Test format

Tests are defined as flat YAML objects (or an array of them). All fields except the test type key (http, command, wait, httpBodyComparison) are optional.

- name: my-test          # optional display name
  retries: 3             # retry up to N times on failure (default: 0)
  http: ...              # ← test type
  source:
    type: local          # local | pod
  expect:
    statusCode: 200

---

Test types

HTTP test

Test any HTTP endpoint locally or from within a Kubernetes pod.

- name: health check
  http:
    url: "https://api.example.com"   # required (or auto-discovered for Service selectors)
    method: GET                       # GET (default) | POST | PUT | DELETE | PATCH
    path: "/health"                   # default: /
    headers:
      Authorization: "Bearer ${API_TOKEN}"
      Content-Type: application/json
    params:
      key: value                      # query parameters
    body: '{"foo":"bar"}'             # request body (string or object)
    skipSslVerification: true         # disable TLS verification
    maxRedirects: 0                   # redirects to follow (default: 0)
    cert: /path/to/cert.pem           # mTLS client certificate
    key:  /path/to/key.pem
    ca:   /path/to/ca.pem
  source:
    type: local
  expect:
    statusCode: 200                   # or [200, 201, 202]
    body: "exact body match"
    bodyContains: "substring"         # or array, or {value, negate, matchword}
    bodyRegex: "pattern.*"            # or {value, negate, caseInsensitive}
    bodyJsonPath:
      - path: "$.user.id"
        comparator: equals
        value: 42
    headers:
      - name: content-type
        comparator: contains
        value: application/json

Environment variable substitution

Any $VAR or ${VAR} in the url, headers, or body fields is resolved from the environment:

http:
  url: "${API_BASE_URL}"
  headers:
    Authorization: "Bearer ${API_TOKEN}"
  body: '{"id": "${REQUEST_ID}"}'

Pod-based HTTP test

Execute the HTTP request from inside a Kubernetes pod (useful for internal service testing):

source:
  type: pod
  selector:
    kind: Pod
    metadata:
      namespace: production
      labels:
        app: test-client
    context: my-cluster          # optional kubectl context
  container: my-container        # optional
  usePortForward: true           # use kubectl port-forward instead of debug pod
  usePodExec: true               # use kubectl exec + curl

Auto-discovery for Kubernetes Services

Omit http.url when source.selector.kind is Service and the IP/port are discovered automatically from the LoadBalancer status:

- name: auto-discover service
  http:
    method: GET
    path: /health
    scheme: https                # optional, defaults to http
    port: 443                    # optional port name/number/index
  source:
    type: local
    selector:
      kind: Service
      metadata:
        namespace: production
        name: my-service
  expect:
    statusCode: 200

---

Command test

Run any shell command and validate its output.

- name: check kubectl version
  command:
    command: "kubectl version -short"
    parseJson: false              # parse stdout as JSON (default: false)
    env:
      MY_VAR: value               # extra environment variables
    workingDir: /tmp              # working directory
  source:
    type: local                   # or pod (uses kubectl exec)
  expect:
    exitCode: 0
    stdout:
      contains: "Client Version"  # or: equals, matches/regex, negate
    stderr:
      contains: ""

Multiple stdout expectations (all must pass):

expect:
  exitCode: 0
  stdout:
    - contains: "Running"
    - matches: "\\d+ pods"

JSON output validation:

- name: cluster info JSON
  command:
    command: "kubectl cluster-info --output=json"
    parseJson: true
  source:
    type: local
  expect:
    exitCode: 0
    jsonPath:
      - path: "$.Kubernetes"
        comparator: exists

---

Wait test

Poll a Kubernetes resource until a condition is met (or timeout).

- name: wait for deployment
  wait:
    target:
      kind: Deployment
      metadata:
        namespace: default
        name: my-app
      context: my-cluster        # optional
    jsonPath: "$.status.readyReplicas"
    jsonPathExpectation:
      comparator: greaterThan
      value: 0
    polling:
      timeoutSeconds: 120        # default: 60
      intervalSeconds: 5         # default: 2
      maxRetries: 24             # optional upper bound
  setVars:                       # optional: capture the extracted value
    READY_REPLICAS:
      value: true

To use the captured value in subsequent tests, see the setVars section below.

Selector by labels:

wait:
  target:
    kind: Pod
    metadata:
      namespace: production
      labels:
        app: web-server
        version: v1.2.0
  jsonPath: "$.status.phase"
  jsonPathExpectation:
    comparator: equals
    value: Running

---

HTTP body comparison test

Compare the response bodies of two HTTP calls and assert they are identical (useful for canary / shadow traffic validation).

- name: compare two backends
  httpBodyComparison:
    request1:
      http:
        url: "http://service-v1"
        method: GET
        path: /api/data
      source:
        type: local
    request2:
      http:
        url: "http://service-v2"
        method: GET
        path: /api/data
      source:
        type: local
    parseAsJson: true            # parse bodies as JSON before comparing
    delaySeconds: 1              # wait between requests
    removeJsonPaths:             # ignore dynamic fields
      - "$.timestamp"
      - "$.requestId"

---

Expectation operators

All comparators can be negated with negate: true.

Comparator	Description	Types
equals	Deep equality	any
contains	Substring / JSON-stringified search	string, object
matches	Regular expression test	string
exists	Value is not null/undefined	any
greaterThan	Numeric >	number
lessThan	Numeric <	number

Negation example

expect:
  bodyContains:
    value: "error"
    negate: true        # assert the body does NOT contain "error"

Word-boundary match

expect:
  bodyContains:
    value: "ok"
    matchword: true     # uses \bok\b regex (whole word only)

---

Advanced features

Run control: retries, consecutive, timeout, maxtime

Each test definition accepts four optional run-control knobs. All have defaults tuned for waiting on eventually-consistent systems, and all defaults are overridable per run via environment variables (so you never have to edit the test to tune them):

Field	Meaning	Default	Env override
retries	Max retry attempts after the first	120	YAMLTEST_RETRIES
consecutive	Successful runs required per attempt (all must pass in a row)	1	—
timeout	Per-attempt cap, in ms (a hung attempt is abandoned and retried)	10000	YAMLTEST_TIMEOUT_MS
maxtime	Wall-clock cap on the whole retry loop — the circuit breaker	max(180000, timeout + 60000)	YAMLTEST_MAXTIME_MS (floor)

maxtime accepts either a number of milliseconds or a duration string ("500ms", "30s", "3m", "1h"). The retry loop stops at whichever comes first: retries exhausted or maxtime elapsed. The pause between retries defaults to 1000ms (YAMLTEST_RETRY_INTERVAL_MS).

- name: flaky service
  retries: 5            # retry up to 5 times after the first attempt
  http:
    url: "http://flaky-service"
    method: GET
    path: /api
  source:
    type: local
  expect:
    statusCode: 200

- name: routing is stable, not just lucky
  consecutive: 3        # must return the expected provider 3 times in a row
  retries: 10
  http: { url: "http://gw", method: POST, path: /failover }
  source: { type: local }
  expect:
    bodyJsonPath:
      - path: "$.model"
        comparator: contains
        value: "gemini"

- name: long browser/OAuth flow
  timeout: 360000       # one attempt may take up to 6 minutes
  maxtime: "8m"         # cap the whole retry loop at 8 minutes
  command: { command: "./run-oauth-flow.sh" }
  source: { type: local }
  expect: { exitCode: 0 }

On failure, the CLI prints the observed request/response (or command result) from the last attempt inline — no need to re-run under DEBUG_MODE to see why a test failed. Bodies and outputs are truncated so a large response can't flood the log.

Multiple tests in one file

Tests run sequentially and stop at the first failure (fail-fast).

- name: first test
  http: { url: "http://svc", method: GET, path: /ready }
  source: { type: local }
  expect: { statusCode: 200 }

- name: second test
  command: { command: "kubectl get pods -n default" }
  source: { type: local }
  expect: { exitCode: 0, stdout: { contains: "Running" } }

Environment variables in url, headers, and body

http:
  url: "$API_BASE_URL"          # $VAR or ${VAR}
  headers:
    Authorization: "Bearer ${API_TOKEN}"
  body: '{"id": "${REQUEST_ID}"}'

setVars — variable passing between steps

Extract values from a test response and store them for use in subsequent steps via ${VAR_NAME} syntax. setVars requires expect to be present on the test — variables are only captured after all assertions pass.

HTTP extraction sources

- name: login
  http:
    url: "http://api.example.com"
    method: POST
    path: /login
    body: '{"user":"admin","pass":"secret"}'
  source:
    type: local
  expect:
    statusCode: 200
  setVars:
    AUTH_TOKEN:
      jsonPath: "$.token"           # extract from JSON body via JSONPath
    SESSION_ID:
      header: "x-session-id"       # extract a response header (case-insensitive)
    STATUS_CODE:
      statusCode: true             # capture the HTTP status code
    RAW_BODY:
      body: true                   # capture the full response body
    CSRF_TOKEN:
      regex:                       # extract via regex capture group from body
        pattern: 'name="csrf" value="([^"]+)"'
        group: 1                   # capture group index (default: 1)

Command extraction sources

- name: read config
  command:
    command: "cat config.json"
    parseJson: true                # required for jsonPath extraction
  source:
    type: local
  expect:
    exitCode: 0
  setVars:
    DB_HOST:
      jsonPath: "$.database.host"  # extract from parsed JSON stdout
    FULL_OUTPUT:
      stdout: true                 # capture full stdout
    ERR_OUTPUT:
      stderr: true                 # capture full stderr
    EXIT:
      exitCode: true               # capture exit code
    PID:
      regex:                       # extract via regex from stdout or stderr
        source: stdout             # "stdout" (default) or "stderr"
        pattern: "PID (\\d+)"
        group: 1

Wait extraction source

- name: wait for deployment
  wait:
    target:
      kind: Deployment
      metadata:
        namespace: default
        name: my-app
    jsonPath: "$.status.readyReplicas"
    jsonPathExpectation:
      comparator: greaterThan
      value: 0
  setVars:
    READY_REPLICAS:
      value: true                  # capture the jsonPath-extracted value

Chaining example: login then access protected endpoint

- name: login
  http:
    url: "http://localhost:3000"
    method: POST
    path: /login
    body: '{"user":"admin","pass":"secret"}'
  source:
    type: local
  expect:
    statusCode: 200
  setVars:
    AUTH_TOKEN:
      jsonPath: "$.token"

- name: access protected endpoint
  http:
    url: "http://localhost:3000"
    method: GET
    path: /protected
    headers:
      Authorization: "Bearer ${AUTH_TOKEN}"
  source:
    type: local
  expect:
    statusCode: 200

Shell heredoc gotcha: if you feed YAML via a heredoc (<<EOF), the shell expands $AUTH_TOKEN in the heredoc body to its value in the parent shell (usually empty) before YAMLTest ever sees the text. Use one of these patterns to prevent that:

# 1. Escape the dollar sign
YAMLTest -f - <<EOF
    command: echo \$AUTH_TOKEN
EOF

# 2. Quote the heredoc delimiter (disables all expansion)
YAMLTest -f - <<'EOF'
    command: echo $AUTH_TOKEN
EOF

# 3. Use a file — no shell expansion at all (recommended)
YAMLTest -f tests.yaml

---

Debug logging

DEBUG_MODE=true YAMLTest -f tests.yaml

Prints full request/response details, comparison results, and kubectl commands.

---

Project structure

src/
  core.js       # Test execution engine (HTTP, command, wait, comparison)
  runner.js     # Multi-test orchestration (YAML parsing, validation, fail-fast, retry)
  validate.js   # JSON Schema validation (Ajv)
  index.js      # Public API
  cli.js        # YAMLTest binary entry point
test/
  unit/         # Pure function tests (compareValue, filterJson, parseCurl, ...)
  integration/  # Real HTTP server + real shell command tests
  e2e/          # CLI binary spawned end-to-end

---

Running the test suite

npm test                    # all tests
npm run test:unit           # unit tests only
npm run test:integration    # integration tests only
npm run test:e2e            # end-to-end CLI tests only
npm run test:coverage       # with coverage report

---

CI/CD

# .github/workflows/test.yml
name: Tests
on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '20'
      - run: npm install
      - run: npm test

---

License

MIT