Runner
This page ports the explore() and openHost() parts of upstream’s docs/reference/runner.mdx (tester-army/e2e, 17a2e46, #1711). The port has no runner: tests run under dotnet test, so upstream’s run() and list() entries have no counterpart here (see Compatibility). The calls that have one are Explore.ExploreAsync, which e2e explore runs, and E2EHost.OpenAsync, upstream’s openHost, which opens hosted sessions for a runner of your own.
var outcome = await Explore.ExploreAsync(new ExploreOptions{ Goal = "Explore the login flow", MaxSteps = 4,});Console.WriteLine($"{outcome.Status} {outcome.Explore.Ended} {outcome.Explore.Findings.Count}");Options
Section titled “Options”See also the generated API reference for ExploreOptions.
| Option | Example | Type | Default |
|---|---|---|---|
Goal |
"Explore the login flow" |
string |
"Explore the app and find bugs"; with ChangedSince, "Explore what this change affects and find bugs in it" |
Target |
"web" |
string |
the first configured target |
Agent |
"smart" |
string |
agents.default |
ChangedSince |
"origin/main" |
string |
none |
MaxSteps |
4 |
int |
8 |
Timeout |
TimeSpan.FromMinutes(5) |
TimeSpan |
10 minutes |
Cwd |
"apps/web" |
string |
the working directory |
ConfigPath |
"e2e.ci.config.json" |
string |
the nearest e2e.config.json |
RawConfig |
new E2EConfig(...) |
E2EConfig |
none: a config built in code, with Cwd as the project root |
Session |
"loggedIn" |
string |
none |
Headed |
true |
bool |
false |
Output |
"out" |
string |
output in the config, else .e2e |
Agents |
new Dictionary<string, AgentOptions>() |
IReadOnlyDictionary<string, AgentOptions> |
none: agents given in code replace the config’s entry of the same name |
Credentials |
new Dictionary<string, UserCredential>() |
IReadOnlyDictionary<string, UserCredential> |
none: accounts the config does not declare |
Notice |
n => Console.Error.WriteLine(n) |
Action<string> |
none |
Trace, Video |
RecordingMode.On |
RecordingMode |
the target’s mode |
Debug, AiTrace |
true |
bool |
false (E2E_DEBUG=1 and E2E_AI_TRACE=1 turn them on) |
Engine |
new WinAppEngine(...) |
IEngine |
the target’s platform picks it |
OnProgress |
p => Console.WriteLine(p) |
callback | none |
Sessionnames a session the host already restored intoEngine: the port has no setup runner to run it.Goalis at most 2,000 characters.ChangedSinceexplores the change since that ref, as--changed-sincedoes.MaxStepsis from 1 to 100.Timeoutis from 3 minutes to 2 hours.OnProgressreceives each moment as it happens:Started,Planning,StepStarted,FindingReported,StepFinished, andFinished.
Outcome
Section titled “Outcome”ExploreOutcome holds Status, ExitCode, Error, and Explore, the record report.run.explore also holds.
| Field | Meaning |
|---|---|
Explore.Goal, Explore.Budgets |
The goal, with secret values redacted, and the step and time budgets it ran with. |
Explore.Ended |
Why exploration stopped: Finished, Skipped (with ChangedSince, the planner judged the change has nothing a user can see; the run passes), StepLimit, Time, Stuck, or Aborted. |
Explore.Summary |
The agent’s closing assessment, when it gave one. |
Explore.Change |
With ChangedSince, the change it planned from: Base, the merge base, and Files, how many changed. |
Explore.Steps |
Each step the agent planned and ran, with its status: Passed, Failed, Blocked, or Exhausted. |
Explore.Findings |
Each finding: kind, severity from 1 to 5, title, expected, actual, and reproduction. An issue fails the run; a warning does not. |
Errors
Section titled “Errors”These reject first, as the matching e2e explore flags do. Recognize them with E2EException.Code.
| Code | When |
|---|---|
INVALID_CONFIG |
Goal, MaxSteps, or Timeout is out of range, or Agent names no configured agent. |
UNKNOWN_TARGET |
Target names no configured target. |
MODEL_UNAVAILABLE |
The agent has no model. |
INVALID_CHANGED_SINCE |
ChangedSince is empty, starts with -, or names a ref git can’t measure a change from. |
Hosted sessions
Section titled “Hosted sessions”E2EHost.OpenAsync(OpenHostOptions? options, CancellationToken) is upstream’s openHost. It prepares e2e for a runner of your own and returns an E2EHost. C# has no free function, so the type it returns owns the call.
await using var host = await E2EHost.OpenAsync(new OpenHostOptions { Target = "web" });await using var session = await host.OpenSessionAsync(new OpenSessionOptions { Name = "checkout" });await session.App.OpenAsync("/checkout");- Your runner owns collection, retries, and reporting.
- The host prepares the target’s engine, starts app processes, and initializes the engine for each session. Use a host for the worker and a session for each test.
- Each
host.OpenSessionAsynccreates a test-lifetime attempt with fresh fixtures, a deadline, steps, and artifacts. - If opening fails, e2e tears down what started before it throws.
OpenHostOptions
Section titled “OpenHostOptions”| Option | Example | Type | Default |
|---|---|---|---|
ConfigPath |
"e2e.config.json" |
string |
nearest e2e.config.json |
Config |
E2EConfig.Load(...) |
E2EConfig |
- |
Cwd |
"apps/web" |
string |
the current directory |
Target |
"android" |
string |
the only target |
Env |
env |
IReadOnlyDictionary<string, string?> |
the process environment |
the CancellationToken argument |
cts.Token |
CancellationToken |
- |
Headed |
true |
bool |
false |
OnNotice |
(target, message) => { } |
Action<string, string> |
- |
EngineFactory |
target => new WebEngine() |
Func<TargetConfig, IEngine> |
the engine of the target’s platform |
ConfigPathis a config file path relative toCwd.Configis anE2EConfigvalue, whoseProjectRootis the project root. Upstream’sconfigtakes either; C# has no union, so there are two members, and passing both isINVALID_CONFIG. Omit both to find the config fromCwd.Targetis required when the config declares several targets.Envis the environment the config, engines, and providers read.- The token cancels host provisioning, a pending session open, or live fixture calls.
Headedruns a browser engine headed.OnNoticereceives progress outside a step, such as provisioning or an app process. It must not throw.EngineFactoryis .NET-only.e2e.config.jsoncannot hold a live engine, so a device, Windows, or test engine comes from here. A factory builds a fresh engine for each host.
| Throws | When |
|---|---|
UNKNOWN_TARGET |
Target names no configured target. |
TARGET_REQUIRED |
The config declares several targets and Target names none. |
INVALID_CONFIG |
Both Config and ConfigPath are set. |
Opening failures are classified, redacted errors. Match them with E2EException.Code.
E2EHost
Section titled “E2EHost”| Member | Returns |
|---|---|
RunId, Target |
string |
OpenSessionAsync(OpenSessionOptions, CancellationToken) |
Task<E2ESession> |
CloseAsync() |
Task<IReadOnlyList<Exception>> |
RunId and Target are the identity the host’s sessions share.
Only one hosted attempt may be opening, live, or closing per host. Await session.CloseAsync() before opening the next session. A session whose engine cleanup failed closes the host for good, since its isolation is uncertain; open a new host. A launch that fails or is cancelled leaves the host usable.
| Throws | When |
|---|---|
WORKER_PROTOCOL |
An overlapping hosted attempt, or a call on a closed host. |
host.OpenSessionAsync
Section titled “host.OpenSessionAsync”Opens a fresh session on the host’s target. A session is one test attempt with no retry loop. A retry opens a new session on the same host.
var session = await host.OpenSessionAsync(new OpenSessionOptions{ Name = "checkout", Timeout = TimeSpan.FromSeconds(60), // default: the config's timeout OnStep = step => { if (step.Phase == StepProgressPhase.End) { Console.WriteLine($"{step.Api} {step.Label}: {step.Status}"); } },});| Option | Example | Type | Default |
|---|---|---|---|
Name |
"checkout" |
string |
required |
Timeout |
TimeSpan.FromSeconds(60) |
TimeSpan? |
the config’s timeout |
Agent |
"smart" |
string |
the default agent |
OnStep |
step => { } |
Action<StepProgress> |
- |
Nameis the test name for step and trace reporting.Timeoutis the deadline for the session’s operations, positive and finite, counted from when the session starts opening. It also cuts a session launch that runs past it, but it does not bound host provisioning, and it does not close the session.- The
CancellationTokenargument cancels opening or live fixture calls. The host’s token also applies. Provisioning happens inOpenAsync, so only the host’s token cancels it. Agentis the agent the session’sAgentuses when a call names none. Omitted, it is the config’s selected agent:default, unless the config was loaded withE2E_AGENTnaming others, and then the first name it lists. Declaration order in the config file never selects it.OnStepreceives liveStepProgress, the same feed reporters receive. It must not block or throw. ItsPhaseisStart,Event,Activity,Replay, orEnd.
| Throws | When |
|---|---|
INVALID_CONFIG |
Name is blank, or Timeout is not positive and finite. |
| cancellation | The session’s or the host’s token was cancelled before or during opening. |
WORKER_PROTOCOL |
Another hosted attempt is opening, live, or closing, or the host is closed. |
host.CloseAsync
Section titled “host.CloseAsync”Cancels any active or pending session, waits for it to close, then disposes the engine, stops app processes, and releases provisioning.
var failures = await host.CloseAsync();if (failures.Count > 0){ throw new AggregateException("e2e host close failed", failures);}- Idempotent: it disposes and releases resources once.
- A host is
IAsyncDisposable:await using var host = await E2EHost.OpenAsync()closes it at scope exit and throws anAggregateExceptionwhen closing reports failures, even for one.
E2ESession
Section titled “E2ESession”E2ESession is what host.OpenSessionAsync returns.
| Member | Returns |
|---|---|
Fixtures |
TestContext |
RunId, AttemptId, Target |
string |
Steps() |
IReadOnlyList<ReportStep> |
Report() |
Report? |
ArtifactsDir |
string |
TracePath |
string |
CloseAsync() |
Task<IReadOnlyList<Exception>> |
Fixturesis the session’sTestContext. The port has notest.extend, so it has no fixtures type argument.RunIdandTargetare the host’s run and target;AttemptIdis unique to this attempt.Steps()returns the steps run so far, as the report records them. Upstream’sStepRecordisReportStephere.Report()returns the sessionReportthe trace is rendered from, after close and a successful trace write; otherwisenull. Its status describes the SDK session, not your runner’s test verdict.ArtifactsDiris the absolute directory for this attempt’s screenshots and recordings. The trace is not in it: readTracePath.TracePathis the absolute path totrace.md; attach or read it afterCloseAsync()settles.
session.CloseAsync
Section titled “session.CloseAsync”Cancels remaining operations, ends the attempt, and writes its trace. It returns soft expectation failures (one ASSERTION_FAILED), abandoned operations (STEP_NOT_AWAITED), and cleanup failures instead of throwing them. Upstream returns SerializedError[]; the port returns the exceptions, and E2EException.Code carries the code.
var failures = await session.CloseAsync();if (failures.Count > 0){ throw new AggregateException("e2e session close failed", failures);}- Idempotent: later calls return the same failures.
- The host stays open for the next test.
- A session is
IAsyncDisposable:await using var session = await host.OpenSessionAsync(...)closes it at scope exit and throws anAggregateExceptionof the returned failures.
Deadline and soft failures
Section titled “Deadline and soft failures”Soft expectation failures are collected for the session’s whole lifetime. Check CloseAsync()’s result to fail your runner’s test.
- When the deadline passes, a poll or operation that was watching it reports its own cut. Everything else still running, including an operation an engine or a model never answered, fails with
TEST_TIMEOUT. - Either way the session’s outcome is timed out and
CloseAsync()returns the timeout. Later operations fail withTEST_TIMEOUT; close the session.
Artifacts
Section titled “Artifacts”At close, e2e writes trace.md to the test’s directory (<results>/<test>, beside the attempt’s directory) through the same redacted formatting reports use. A hosted session writes its trace, not report.json. For your runner’s reporting, use OnStep, Steps(), Report(), and CloseAsync()’s failures. Your runner’s test-body failures are not passed to e2e.
Differences from upstream: the attempt’s directory is <results>/<test>/attempt-1 and trace.md sits beside it, in <results>/<test>; read TracePath for its location. Upstream’s directory is results/host/<segment>. Two sessions with one name share it; RUNNER_BUSY is not raised, since the port has no suite runner. See Compatibility.
| Type | Describes |
|---|---|
OpenHostOptions, E2EHost |
E2EHost.OpenAsync’s options and the host |
OpenSessionOptions, E2ESession |
host.OpenSessionAsync’s options and the session |
StepProgress, ReportStep |
One OnStep progress event, and one step Steps() returns (upstream’s StepRecord) |
E2EException |
One failure CloseAsync() returns; its Code is the error code (upstream’s SerializedError, and isE2EError is a type test for it) |
Examples
Section titled “Examples”A host for each test class and a session for each test
Section titled “A host for each test class and a session for each test”A host for the worker and a session for each test, in NUnit:
[TestFixture]public class SettingsTests{ private E2EHost _host = null!;
[OneTimeSetUp] public async Task OpenHost() => _host = await E2EHost.OpenAsync(new OpenHostOptions { Target = "web" });
[OneTimeTearDown] public async Task CloseHost() { var failures = await _host.CloseAsync(); if (failures.Count > 0) { throw new AggregateException("e2e host close failed", failures); } }
[Test] public async Task Turns_on_dark_mode() { var session = await _host.OpenSessionAsync(new OpenSessionOptions { Name = TestContext.CurrentContext.Test.FullName, OnStep = step => { if (step.Phase == StepProgressPhase.End) { Console.WriteLine($"{step.Api} {step.Label}: {step.Status}"); } }, }); try { await session.Agent.ActAsync("open settings and turn on dark mode"); await session.App.ScreenshotAsync(); } finally { var failures = await session.CloseAsync(); TestContext.AddTestAttachment(session.TracePath, "e2e trace"); if (failures.Count > 0) { throw new AggregateException("e2e session close failed", failures); } } }}