Skip to content

Runner

This page ports the explore() and openHost() parts of upstream’s docs/reference/runner.mdx (tester-army/e2e, 17a2e46, #1711). The port has no runner: tests run under dotnet test, so upstream’s run() and list() entries have no counterpart here (see Compatibility). The calls that have one are Explore.ExploreAsync, which e2e explore runs, and E2EHost.OpenAsync, upstream’s openHost, which opens hosted sessions for a runner of your own.

var outcome = await Explore.ExploreAsync(new ExploreOptions
{
Goal = "Explore the login flow",
MaxSteps = 4,
});
Console.WriteLine($"{outcome.Status} {outcome.Explore.Ended} {outcome.Explore.Findings.Count}");

See also the generated API reference for ExploreOptions.

Option Example Type Default
Goal "Explore the login flow" string "Explore the app and find bugs"; with ChangedSince, "Explore what this change affects and find bugs in it"
Target "web" string the first configured target
Agent "smart" string agents.default
ChangedSince "origin/main" string none
MaxSteps 4 int 8
Timeout TimeSpan.FromMinutes(5) TimeSpan 10 minutes
Cwd "apps/web" string the working directory
ConfigPath "e2e.ci.config.json" string the nearest e2e.config.json
RawConfig new E2EConfig(...) E2EConfig none: a config built in code, with Cwd as the project root
Session "loggedIn" string none
Headed true bool false
Output "out" string output in the config, else .e2e
Agents new Dictionary<string, AgentOptions>() IReadOnlyDictionary<string, AgentOptions> none: agents given in code replace the config’s entry of the same name
Credentials new Dictionary<string, UserCredential>() IReadOnlyDictionary<string, UserCredential> none: accounts the config does not declare
Notice n => Console.Error.WriteLine(n) Action<string> none
Trace, Video RecordingMode.On RecordingMode the target’s mode
Debug, AiTrace true bool false (E2E_DEBUG=1 and E2E_AI_TRACE=1 turn them on)
Engine new WinAppEngine(...) IEngine the target’s platform picks it
OnProgress p => Console.WriteLine(p) callback none
  • Session names a session the host already restored into Engine: the port has no setup runner to run it.
  • Goal is at most 2,000 characters.
  • ChangedSince explores the change since that ref, as --changed-since does.
  • MaxSteps is from 1 to 100. Timeout is from 3 minutes to 2 hours.
  • OnProgress receives each moment as it happens: Started, Planning, StepStarted, FindingReported, StepFinished, and Finished.

ExploreOutcome holds Status, ExitCode, Error, and Explore, the record report.run.explore also holds.

Field Meaning
Explore.Goal, Explore.Budgets The goal, with secret values redacted, and the step and time budgets it ran with.
Explore.Ended Why exploration stopped: Finished, Skipped (with ChangedSince, the planner judged the change has nothing a user can see; the run passes), StepLimit, Time, Stuck, or Aborted.
Explore.Summary The agent’s closing assessment, when it gave one.
Explore.Change With ChangedSince, the change it planned from: Base, the merge base, and Files, how many changed.
Explore.Steps Each step the agent planned and ran, with its status: Passed, Failed, Blocked, or Exhausted.
Explore.Findings Each finding: kind, severity from 1 to 5, title, expected, actual, and reproduction. An issue fails the run; a warning does not.

These reject first, as the matching e2e explore flags do. Recognize them with E2EException.Code.

Code When
INVALID_CONFIG Goal, MaxSteps, or Timeout is out of range, or Agent names no configured agent.
UNKNOWN_TARGET Target names no configured target.
MODEL_UNAVAILABLE The agent has no model.
INVALID_CHANGED_SINCE ChangedSince is empty, starts with -, or names a ref git can’t measure a change from.

E2EHost.OpenAsync(OpenHostOptions? options, CancellationToken) is upstream’s openHost. It prepares e2e for a runner of your own and returns an E2EHost. C# has no free function, so the type it returns owns the call.

await using var host = await E2EHost.OpenAsync(new OpenHostOptions { Target = "web" });
await using var session = await host.OpenSessionAsync(new OpenSessionOptions { Name = "checkout" });
await session.App.OpenAsync("/checkout");
  • Your runner owns collection, retries, and reporting.
  • The host prepares the target’s engine, starts app processes, and initializes the engine for each session. Use a host for the worker and a session for each test.
  • Each host.OpenSessionAsync creates a test-lifetime attempt with fresh fixtures, a deadline, steps, and artifacts.
  • If opening fails, e2e tears down what started before it throws.
Option Example Type Default
ConfigPath "e2e.config.json" string nearest e2e.config.json
Config E2EConfig.Load(...) E2EConfig -
Cwd "apps/web" string the current directory
Target "android" string the only target
Env env IReadOnlyDictionary<string, string?> the process environment
the CancellationToken argument cts.Token CancellationToken -
Headed true bool false
OnNotice (target, message) => { } Action<string, string> -
EngineFactory target => new WebEngine() Func<TargetConfig, IEngine> the engine of the target’s platform
  • ConfigPath is a config file path relative to Cwd. Config is an E2EConfig value, whose ProjectRoot is the project root. Upstream’s config takes either; C# has no union, so there are two members, and passing both is INVALID_CONFIG. Omit both to find the config from Cwd.
  • Target is required when the config declares several targets.
  • Env is the environment the config, engines, and providers read.
  • The token cancels host provisioning, a pending session open, or live fixture calls.
  • Headed runs a browser engine headed.
  • OnNotice receives progress outside a step, such as provisioning or an app process. It must not throw.
  • EngineFactory is .NET-only. e2e.config.json cannot hold a live engine, so a device, Windows, or test engine comes from here. A factory builds a fresh engine for each host.
Throws When
UNKNOWN_TARGET Target names no configured target.
TARGET_REQUIRED The config declares several targets and Target names none.
INVALID_CONFIG Both Config and ConfigPath are set.

Opening failures are classified, redacted errors. Match them with E2EException.Code.

Member Returns
RunId, Target string
OpenSessionAsync(OpenSessionOptions, CancellationToken) Task<E2ESession>
CloseAsync() Task<IReadOnlyList<Exception>>

RunId and Target are the identity the host’s sessions share.

Only one hosted attempt may be opening, live, or closing per host. Await session.CloseAsync() before opening the next session. A session whose engine cleanup failed closes the host for good, since its isolation is uncertain; open a new host. A launch that fails or is cancelled leaves the host usable.

Throws When
WORKER_PROTOCOL An overlapping hosted attempt, or a call on a closed host.

Opens a fresh session on the host’s target. A session is one test attempt with no retry loop. A retry opens a new session on the same host.

var session = await host.OpenSessionAsync(new OpenSessionOptions
{
Name = "checkout",
Timeout = TimeSpan.FromSeconds(60), // default: the config's timeout
OnStep = step =>
{
if (step.Phase == StepProgressPhase.End)
{
Console.WriteLine($"{step.Api} {step.Label}: {step.Status}");
}
},
});
Option Example Type Default
Name "checkout" string required
Timeout TimeSpan.FromSeconds(60) TimeSpan? the config’s timeout
Agent "smart" string the default agent
OnStep step => { } Action<StepProgress> -
  • Name is the test name for step and trace reporting.
  • Timeout is the deadline for the session’s operations, positive and finite, counted from when the session starts opening. It also cuts a session launch that runs past it, but it does not bound host provisioning, and it does not close the session.
  • The CancellationToken argument cancels opening or live fixture calls. The host’s token also applies. Provisioning happens in OpenAsync, so only the host’s token cancels it.
  • Agent is the agent the session’s Agent uses when a call names none. Omitted, it is the config’s selected agent: default, unless the config was loaded with E2E_AGENT naming others, and then the first name it lists. Declaration order in the config file never selects it.
  • OnStep receives live StepProgress, the same feed reporters receive. It must not block or throw. Its Phase is Start, Event, Activity, Replay, or End.
Throws When
INVALID_CONFIG Name is blank, or Timeout is not positive and finite.
cancellation The session’s or the host’s token was cancelled before or during opening.
WORKER_PROTOCOL Another hosted attempt is opening, live, or closing, or the host is closed.

Cancels any active or pending session, waits for it to close, then disposes the engine, stops app processes, and releases provisioning.

var failures = await host.CloseAsync();
if (failures.Count > 0)
{
throw new AggregateException("e2e host close failed", failures);
}
  • Idempotent: it disposes and releases resources once.
  • A host is IAsyncDisposable: await using var host = await E2EHost.OpenAsync() closes it at scope exit and throws an AggregateException when closing reports failures, even for one.

E2ESession is what host.OpenSessionAsync returns.

Member Returns
Fixtures TestContext
RunId, AttemptId, Target string
Steps() IReadOnlyList<ReportStep>
Report() Report?
ArtifactsDir string
TracePath string
CloseAsync() Task<IReadOnlyList<Exception>>
  • Fixtures is the session’s TestContext. The port has no test.extend, so it has no fixtures type argument.
  • RunId and Target are the host’s run and target; AttemptId is unique to this attempt.
  • Steps() returns the steps run so far, as the report records them. Upstream’s StepRecord is ReportStep here.
  • Report() returns the session Report the trace is rendered from, after close and a successful trace write; otherwise null. Its status describes the SDK session, not your runner’s test verdict.
  • ArtifactsDir is the absolute directory for this attempt’s screenshots and recordings. The trace is not in it: read TracePath.
  • TracePath is the absolute path to trace.md; attach or read it after CloseAsync() settles.

Cancels remaining operations, ends the attempt, and writes its trace. It returns soft expectation failures (one ASSERTION_FAILED), abandoned operations (STEP_NOT_AWAITED), and cleanup failures instead of throwing them. Upstream returns SerializedError[]; the port returns the exceptions, and E2EException.Code carries the code.

var failures = await session.CloseAsync();
if (failures.Count > 0)
{
throw new AggregateException("e2e session close failed", failures);
}
  • Idempotent: later calls return the same failures.
  • The host stays open for the next test.
  • A session is IAsyncDisposable: await using var session = await host.OpenSessionAsync(...) closes it at scope exit and throws an AggregateException of the returned failures.

Soft expectation failures are collected for the session’s whole lifetime. Check CloseAsync()’s result to fail your runner’s test.

  • When the deadline passes, a poll or operation that was watching it reports its own cut. Everything else still running, including an operation an engine or a model never answered, fails with TEST_TIMEOUT.
  • Either way the session’s outcome is timed out and CloseAsync() returns the timeout. Later operations fail with TEST_TIMEOUT; close the session.

At close, e2e writes trace.md to the test’s directory (<results>/<test>, beside the attempt’s directory) through the same redacted formatting reports use. A hosted session writes its trace, not report.json. For your runner’s reporting, use OnStep, Steps(), Report(), and CloseAsync()’s failures. Your runner’s test-body failures are not passed to e2e.

Differences from upstream: the attempt’s directory is <results>/<test>/attempt-1 and trace.md sits beside it, in <results>/<test>; read TracePath for its location. Upstream’s directory is results/host/<segment>. Two sessions with one name share it; RUNNER_BUSY is not raised, since the port has no suite runner. See Compatibility.

Type Describes
OpenHostOptions, E2EHost E2EHost.OpenAsync’s options and the host
OpenSessionOptions, E2ESession host.OpenSessionAsync’s options and the session
StepProgress, ReportStep One OnStep progress event, and one step Steps() returns (upstream’s StepRecord)
E2EException One failure CloseAsync() returns; its Code is the error code (upstream’s SerializedError, and isE2EError is a type test for it)

A host for each test class and a session for each test

Section titled “A host for each test class and a session for each test”

A host for the worker and a session for each test, in NUnit:

[TestFixture]
public class SettingsTests
{
private E2EHost _host = null!;
[OneTimeSetUp]
public async Task OpenHost() =>
_host = await E2EHost.OpenAsync(new OpenHostOptions { Target = "web" });
[OneTimeTearDown]
public async Task CloseHost()
{
var failures = await _host.CloseAsync();
if (failures.Count > 0)
{
throw new AggregateException("e2e host close failed", failures);
}
}
[Test]
public async Task Turns_on_dark_mode()
{
var session = await _host.OpenSessionAsync(new OpenSessionOptions
{
Name = TestContext.CurrentContext.Test.FullName,
OnStep = step =>
{
if (step.Phase == StepProgressPhase.End)
{
Console.WriteLine($"{step.Api} {step.Label}: {step.Status}");
}
},
});
try
{
await session.Agent.ActAsync("open settings and turn on dark mode");
await session.App.ScreenshotAsync();
}
finally
{
var failures = await session.CloseAsync();
TestContext.AddTestAttachment(session.TracePath, "e2e trace");
if (failures.Count > 0)
{
throw new AggregateException("e2e session close failed", failures);
}
}
}
}