Pull request comments

E2E.GitHub is a reporter that posts the test results as a pull request comment from GitHub Actions, and writes them to the job summary. A rerun updates the same comment. It ports upstream’s @e2e-dev/github (docs/github.mdx).

Setup

dotnet add package E2E.GitHub

List the reporter in the reporters of e2e.config.json. The run fixture calls it when the last test is done (E2ERunFixture of E2E.NUnit or E2E.XUnit.V3), so add the fixture too: without it no reporter runs.

{ "reporters": ["E2E.GitHub.GitHubReporter, E2E.GitHub"] }

Give the workflow pull-requests: write. Pass GITHUB_TOKEN to the test step:

permissions:
  contents: read
  pull-requests: write

jobs:
  e2e:
    steps:
      # ...checkout, setup-dotnet, build
      - name: Run e2e
        run: dotnet test
        env:
          GITHUB_TOKEN: ${{ github.token }}
      - name: Upload artifacts
        if: ${{ !cancelled() }}
        uses: actions/upload-artifact@v4
        with:
          name: e2e-artifacts
          path: .e2e/results

Upload the artifacts also when a retry passes: each failure’s evidence links open the run’s Artifacts section. Keep path: .e2e/results so the download has the same per-test directories as the report.

Source links start at the nearest directory with .git inside GITHUB_WORKSPACE. A project below it, such as packages/e2e-tests, gets links under that path. actions/checkout with a path input works too.

The comment

The comment is the page that MarkdownReport.RenderMarkdownReport writes (see Rendering the report as markdown): the headline with the counts, the spend line, a block for each failed test with its step and the line to look at, the flaky tests folded, and every test as a table. The first line is a hidden marker, <!-- e2e-github project=... workflow=... job=... -->, which the reporter finds its comment by.

Run-level errors, when there are any, come before the failures. A run whose test filters left tests out says so in the headline, such as e2e: 7 passed, 35 left out by filters, so it never reads as the whole suite.

The reporter shortens a large report to fit GitHub’s comment limit. App text that GitHub would turn into a mention, an issue link, or a URL shows as code, so a screen that reads @someone notifies no one.

Matrix jobs

The reporter finds its comment by project, workflow, and job, which matrix jobs share. Give each matrix entry its own key. Set E2E_GITHUB_KEY in the step’s env:

env:
  GITHUB_TOKEN: ${{ github.token }}
  E2E_GITHUB_KEY: ${{ matrix.browser }}

Or set key in the options of the config entry:

{
  "reporters": [
    { "type": "E2E.GitHub.GitHubReporter, E2E.GitHub", "options": { "key": "chromium" } }
  ]
}

key is the only option. An unknown option or a key that is not a string is INVALID_CONFIG, and the message never holds a value. The key shows in the headline, such as e2e chromium: 77 passed. Keep it stable across runs so a rerun updates the existing comment. The options and the environment variable are .NET-only: upstream calls github({ key }) in its config module, and a JSON config holds no code. When key is not set, the reporter reads E2E_GITHUB_KEY.

Explorations

An e2e explore run posts a comment of its own, under a marker with kind=explore, so a job can run the tests and then explore. Give the exploration its own --output, and upload that directory too. Otherwise it replaces the test run’s results, and the evidence links in the test run’s comment break.

When nothing is posted

The reporter never changes the run’s status or exit code. When it can’t post, one row under the summary gives the reason:

GitHub  not posted: set GITHUB_TOKEN in the step's env (GITHUB_TOKEN: ${{ github.token }}); written to the job summary

Situation	What happens
Not on GitHub Actions	Nothing is written. The row says so.
A push, schedule, or workflow_dispatch run	The job summary is written. There is no pull request to comment on.
No GITHUB_TOKEN in the step’s env	The job summary is written. The row shows the line to add. The reporter also reads GH_TOKEN.
A pull_request event from a fork	The token is read-only, so posting fails. The row says why.
An issue_comment event	On a pull request’s thread, the comment has no source links, because the checked-out commit is on the default branch.
Missing pull-requests: write	Posting fails, and the reporter names the permission.
GitHub API rate limit	Posting stops and reports throttling. Wait for the reset or Retry-After before rerunning the job.

The reporter reads the token when the run finishes, and the report doesn’t include it. The token never appears in a message the reporter prints. On GitHub Enterprise Server, the reporter uses GITHUB_SERVER_URL and GITHUB_API_URL.

After a --last-failed rerun

A rerun (E2E_LAST_FAILED=1, see Test filters) runs only what the last run did not pass. The reporter folds it into that run (FinishedRun.LastRun), so the comment shows the whole suite and not only the rerun. See COMPATIBILITY.md.