Docs/GitHub Actions

GitHub Actions Integration

Automatically detect flaky tests on every push, pull request, and on a schedule. Get FlakeScore results in your GitHub Actions summary and PR comments.

Quick Start

Get DeFlaky running in GitHub Actions in under 5 minutes. Copy this workflow to .github/workflows/deflaky.yml in your repository:

.github/workflows/deflaky.yml

name: DeFlaky - Flaky Test Detection


on:

push:

branches: [main]

pull_request:

branches: [main]

schedule:

- cron: '0 2 * * 1'


jobs:

deflaky:

runs-on: ubuntu-latest

steps:

- uses: actions/checkout@v4


- uses: actions/setup-node@v4

with:

node-version:'20'


- run: npm ci


- run: npm install -g deflaky-cli


- name: Run DeFlaky

run:deflaky -c "npx playwright test" -r 3 --push --token ${{ secrets.DEFLAKY_TOKEN }}

env:

DEFLAKY_TOKEN: ${{ secrets.DEFLAKY_TOKEN }}

Download the complete example workflow with PR comments, step outputs, and multi-framework support.

How It Works

DeFlaky wraps your existing test command and runs it multiple times. After all runs complete, it compares results across runs to identify tests that sometimes pass and sometimes fail -- these are your flaky tests.

  1. The workflow checks out your code and installs dependencies.
  2. DeFlaky runs your test command N times (configured with -r).
  3. Results are compared across runs. A test that passes in some runs but fails in others is flagged as flaky.
  4. A FlakeScore is calculated: (stable tests / total tests) x 100.
  5. Results are pushed to the DeFlaky Dashboard via --push.

Step 1: Get Your API Token

  1. Go to the DeFlaky Dashboard and sign in.
  2. Click New Project and enter a name for your repository.
  3. Copy the generated API token. It uses the format df_<uuid>.

Step 2: Add GitHub Secret

Add your DeFlaky token as an encrypted secret in your GitHub repository. Never commit tokens to source control.

  1. Open your repository on GitHub.
  2. Navigate to Settings > Secrets and variables > Actions.
  3. Click New repository secret.
  4. Set the name to DEFLAKY_TOKEN and paste your token.
  5. Click Add secret.

Step 3: Create the Workflow

Create .github/workflows/deflaky.yml in your repository. The workflow installs your dependencies, installs the DeFlaky CLI, and runs your test command multiple times.

See the Quick Start section above for the full workflow, or scroll down for framework-specific examples.

CLI Flags

These flags control how DeFlaky runs in your GitHub Actions workflow:

FlagDescriptionDefault
-c, --commandTest command to runrequired
-r, --runsNumber of test iterations5
--pushPush results to the DeFlaky dashboardfalse
--tokenAPI token (or set DEFLAKY_TOKEN env var)-
--fail-thresholdFail if FlakeScore is below this % (0-100)disabled
--verboseShow detailed output per runfalse
--formatReport format: json, junit, autoauto

Failing CI on Low FlakeScore

Use --fail-threshold to enforce a minimum FlakeScore. If the score drops below the threshold, DeFlaky exits with code 1 and the GitHub Actions check fails:

Terminal

$ deflaky -c "npx playwright test" -r 5 --push --token $DEFLAKY_TOKEN --fail-threshold 90

This prevents PRs that introduce flaky tests from merging. Combine with branch protection rules for maximum enforcement.

Scheduling Runs

Use the GitHub Actions schedule trigger for automated weekly detection:

Cron Examples

# Every Monday at 2am UTC

cron:'0 2 * * 1'


# Every day at midnight UTC

cron:'0 0 * * *'


# Monday and Thursday at 3am UTC

cron:'0 3 * * 1,4'

Playwright

Playwright requires browser binaries to be installed. Add the install step before running DeFlaky:

.github/workflows/deflaky.yml

steps:

- uses: actions/checkout@v4

- uses: actions/setup-node@v4

with:

node-version:'20'

- run: npm ci

- run: npx playwright install --with-deps

- run: npm install -g deflaky-cli

- name: Run DeFlaky

run:deflaky -c "npx playwright test" -r 3 --push --token ${{ secrets.DEFLAKY_TOKEN }}

Cypress

Cypress runs in headless mode by default with cypress run. No additional browser installation step is needed:

.github/workflows/deflaky.yml

steps:

- uses: actions/checkout@v4

- uses: actions/setup-node@v4

with:

node-version:'20'

- run: npm ci

- run: npm install -g deflaky-cli

- name: Run DeFlaky

run:deflaky -c "npx cypress run" -r 3 --push --token ${{ secrets.DEFLAKY_TOKEN }}

Jest

Jest works out of the box. Use --ci for CI-optimized behavior:

.github/workflows/deflaky.yml

steps:

- uses: actions/checkout@v4

- uses: actions/setup-node@v4

with:

node-version:'20'

- run: npm ci

- run: npm install -g deflaky-cli

- name: Run DeFlaky

run:deflaky -c "npx jest --ci" -r 3 --push --token ${{ secrets.DEFLAKY_TOKEN }}

Pytest

For Python projects, set up Python and install your dependencies first. DeFlaky still runs via Node.js:

.github/workflows/deflaky.yml

steps:

- uses: actions/checkout@v4

- uses: actions/setup-python@v5

with:

python-version: '3.12'

- uses: actions/setup-node@v4

with:

node-version:'20'

- run: pip install -r requirements.txt

- run: npm install -g deflaky-cli

- name: Run DeFlaky

run:deflaky -c "pytest" -r 3 --push --token ${{ secrets.DEFLAKY_TOKEN }}

Selenium (Java / Maven)

For Java-based Selenium projects, set up Java and Maven. DeFlaky parses Surefire XML reports automatically:

.github/workflows/deflaky.yml

steps:

- uses: actions/checkout@v4

- uses: actions/setup-java@v4

with:

distribution: 'temurin'

java-version: '17'

- uses: actions/setup-node@v4

with:

node-version:'20'

- run: npm install -g deflaky-cli

- name: Run DeFlaky

run:deflaky -c "mvn test" -r 3 --push --token ${{ secrets.DEFLAKY_TOKEN }}

PR Comments with Results

Post DeFlaky results as a comment on every pull request. This uses actions/github-script to read step outputs and create a formatted comment:

.github/workflows/deflaky.yml (PR comment step)

- name: Run DeFlaky

id: deflaky

run: |

deflaky -c "npx playwright test" -r 3 --push --token ${{ secrets.DEFLAKY_TOKEN }} | tee output.txt

echo "flake-score=$(grep -oP 'FlakeScore:\s*\K[\d.]+' output.txt || echo '100')" >> $GITHUB_OUTPUT

echo "flaky-count=$(grep -oP 'Flaky tests:\s*\K\d+' output.txt || echo '0')" >> $GITHUB_OUTPUT


- name: Comment on PR

if: github.event_name == 'pull_request'

uses: actions/github-script@v7

with:

script: |

// See full example at

// deflaky.com/examples/github-actions-workflow.yml

Download the complete example workflow for the full PR comment implementation.

Reusable Workflow

DeFlaky provides a reusable workflow that you can reference from your own repository. This simplifies setup -- you only need to pass your test command and token:

.github/workflows/deflaky.yml (in your repo)

name: Flaky Test Detection

on: [push, pull_request]


jobs:

deflaky:

uses: PramodDutta/deflaky/.github/workflows/deflaky.yml@main

with:

test-command: "npx playwright test"

runs: 3

fail-threshold: 90

secrets:

deflaky-token: ${{ secrets.DEFLAKY_TOKEN }}

Composite Action

For more control, use the DeFlaky composite action as a step in your existing workflow:

.github/workflows/your-tests.yml

steps:

- uses: actions/checkout@v4

- run: npm ci


- name: Detect Flaky Tests

uses: PramodDutta/deflaky/.github/actions/deflaky@main

with:

command:"npx playwright test"

runs: 3

token: ${{ secrets.DEFLAKY_TOKEN }}

fail-threshold: 90

Troubleshooting

"deflaky: command not found"

Install the CLI before running it. Add `npm install -g deflaky-cli` as a step before the DeFlaky run step. Alternatively, use `npx deflaky-cli` instead of `deflaky`.

Token not working / push fails

Verify the GitHub secret name is exactly `DEFLAKY_TOKEN`. Make sure you copied the full token (format: `df_<uuid>`) from the dashboard. Note: secrets are not available in workflows triggered by forked repositories.

Tests timing out

DeFlaky runs your tests N times, so total runtime is roughly N x single-run duration. Set an explicit `timeout-minutes` on the job. For example, if tests take 10 minutes and you run 3 times, set `timeout-minutes: 45`.

Playwright browsers not found

Always run `npx playwright install --with-deps` before the DeFlaky step. The `--with-deps` flag installs required system libraries on Ubuntu runners.

Results not appearing on dashboard

Make sure you include both `--push` and `--token` flags. Check that the token has not expired. Verify network connectivity from the runner (the push endpoint is `https://deflaky.com/api/push`).

Workflow not triggering on schedule

GitHub Actions only runs scheduled workflows on the default branch (usually `main`). Make sure the workflow file exists on the default branch. Note that GitHub may delay or skip scheduled runs during periods of high load.

Related Resources

Blog: GitHub Actions Setup Guide

Step-by-step walkthrough with screenshots and tips.

Blog: Flaky Tests in GitHub Actions

Deep dive into GitHub Actions-specific flakiness patterns.

Full CLI Documentation

All commands, flags, and configuration options.

Example Workflow File

Ready-to-copy workflow with PR comments.