Scenario Testing

Scenario tests are colocated with routes as run.test.ts files. Each file default-exports a route-scoped suite built with scenarios("/route"), and dawn test runs every scenario through the route runtime. Plain default-exported scenario arrays are not supported.

A minimal test

A scenario file has no describe() or test() wrapper. Pass the route ID to scenarios(), add cases with .scenario(), then set the input and expected status explicitly:

import { scenarios } from "@dawn-ai/sdk/testing"
 
export default scenarios("/support/[tenant]").scenario("greets a tenant", (s) =>
  s
    .input({ tenant: "acme" })
    .expectPassed()
    .expectOutput({ tenant: "acme", greeting: "Hello, acme!" }),
)

Run all scenarios:

dawn test

Dawn discovers every run.test.ts under src/app/, verifies that the declared route matches the file's directory, invokes the route, and evaluates the expectations. dawn typegen writes the route and application-tool types used by the builder, so route names, .mockTool(), and .expectTool() are discoverable in IntelliSense.

Builder API

Every .scenario(name, configure) callback must return a builder that has called .input() exactly once and selected either .expectPassed() or .expectFailed(). The remaining methods add expectations or choose where the scenario runs:

• .input(value) sets the route input.
• .expectPassed() and .expectFailed() select the required result status.
• .expectOutput(value) matches a passed route's returned state.
• .expectError(value) matches a failed route's modeled error.
• .expectMeta(value) matches { mode, routeId, routePath, executionSource }.
• .assert(callback) runs a synchronous or asynchronous custom assertion after declarative expectations.
• .server(url) sends the scenario through a running Dawn-compatible server instead of invoking in-process.
• .mockTool(name, implementation) replaces one application tool for this in-process invocation.
• .expectTool(name, configure) asserts calls made to a tool mocked earlier in the same scenario.

Passing scenarios can use .expectOutput(); failing scenarios can use .expectError(). The builder's type states hide combinations that cannot succeed and the loader validates the same rules at runtime.

For programmatic assertions, import the existing helpers from @dawn-ai/sdk/testing:

import { expectMeta, expectOutput, scenarios } from "@dawn-ai/sdk/testing"
 
export default scenarios("/support/[tenant]").scenario("custom assert", (s) =>
  s
    .input({ tenant: "acme" })
    .expectPassed()
    .assert((result) => {
      expectOutput(result, { greeting: "Hello, acme!" })
      expectMeta(result, { mode: "agent", routeId: "/support/[tenant]" })
    }),
)

expectOutput, expectMeta, and expectError use the same result model as declarative expectations and produce focused mismatch messages.

Mocking application tools

Use .mockTool() when an in-process scenario should replace an external or nondeterministic application tool while keeping the rest of the route real. Tool names, parameters, and awaited return values come from the generated route types:

import { scenarios } from "@dawn-ai/sdk/testing"
 
export default scenarios("/research").scenario("uses a controlled search result", (s) =>
  s
    .input({ messages: [{ role: "user", content: "Research Dawn" }] })
    .mockTool("searchWeb", async ({ query }) => ({
      results: [{ title: "Dawn", url: "https://example.test/dawn", query }],
    }))
    .expectPassed()
    .expectTool("searchWeb", (call) =>
      call.calledOnce().withArgs({ query: "Dawn" }),
    ),
)

Mocks are partial: application tools not named by .mockTool() keep their real implementations. Dawn resolves shared and route-local tool precedence first, then replaces only the selected definition's implementation while preserving its schema, description, scope, and source metadata.

.expectTool() is available only for tools mocked earlier in the same scenario. Its call builder supports:

• .called() for one or more calls.
• .calledOnce() for exactly one call.
• .calledTimes(n) for exactly n calls.
• .notCalled() for zero calls.
• .withArgs(partial) for at least one call containing the supplied deep-partial object. Primitive values and arrays match exactly.

Count and argument assertions are independent when combined. Multiple .withArgs() assertions each need a matching invocation, and one compatible invocation may satisfy more than one matcher. Call ordering and return-value assertions are not part of this API.

Mocks apply only to the root route's generated application-tool set: route-local tools and shared tools. They cannot shadow built-in planning, workspace, memory, skill, or subagent capability tools, and a parent route's mocks never propagate into a child subagent route.

Each scenario invocation receives a fresh override set and call journal. Dawn never mutates cached route modules, so mocks cannot leak into another scenario, a later dawn run, a server request, or a concurrently executing invocation.

Against a live dev server

Use .server(url) to exercise a scenario through a running dawn dev, built Dawn server, or staging deployment. There is no command-level --url flag on dawn test.

import { scenarios } from "@dawn-ai/sdk/testing"
 
export default scenarios("/support/[tenant]").scenario(
  "greets a tenant via dev server",
  (s) =>
    s
      .input({ tenant: "acme" })
      .server("http://127.0.0.1:3001")
      .expectPassed()
      .expectOutput({ tenant: "acme", greeting: "Hello, acme!" }),
)

Start the dev server first, then run dawn test:

dawn dev --port 3001 &
dawn test

When to use a server-backed scenario

A server-backed scenario is useful when the claim depends on:

• JSON request and response serialization;
• request middleware and request-scoped context;
• runtime boot and route lookup wiring;
• packaged static modules in a built runtime; or
• real staging infrastructure and deployment configuration.

Server-backed scenarios execute in another process and exchange JSON. JavaScript mock functions cannot cross that boundary, so .server() and .mockTool() are mutually exclusive in the builder and in runtime validation. Dawn does not install a test backdoor or a serializable mock interpreter in the server.

Control server dependencies at their real boundary instead. Start a local fake HTTP service, point the server at a model proxy, or use a dedicated staging dependency before running dawn test.

Agents, retries, and middleware

Two cross-cutting features shape what scenarios assert:

• Agent retries — agent({ retry: { maxAttempts, baseDelay } }) retries on transient errors. To assert the exhausted-retry path, call .expectFailed().expectError(...) or use .assert() with expectError. See Retry.
• Middleware — src/middleware.ts runs before local Agent Protocol run and stream requests. Server-backed scenarios targeting a Dawn runtime exercise middleware; in-process scenarios bypass it. A scenario whose middleware calls reject(...) should use .expectFailed().expectError(...). See Middleware.

Rules

1. 1

   File location

   run.test.ts must live in the route's directory, not a sibling or nested folder. Dawn matches tests to routes by directory.

2. 2

   Route-scoped suite

   Default-export scenarios("/route").scenario(...). The declared route must match the file location, and plain scenario arrays are rejected.

3. 3

   Complete every scenario

   Call .input() once and select .expectPassed() or .expectFailed(). Add declarative expectations or .assert() for the behavior the scenario owns.

4. 4

   Use typed application-tool mocks narrowly

   Mock only the external or nondeterministic application tools needed for the claim. Leave deterministic tools real, and use a server-backed scenario when the transport boundary itself is under test.

5. 5

   Keep scenarios focused

   One scenario = one claim about the route's behavior. If you're adding branches, add more scenarios — don't inflate a single one.

CI

Use dawn verify as the integrity gate (it covers app, routes, typegen, and deps in one call), then run dawn test:

- run: pnpm exec dawn verify
- run: pnpm exec dawn test

dawn verify runs typegen and check internally, plus the deps check (missing packages, missing env vars) that bare dawn check && dawn typegen does not. Typegen also refreshes .dawn/scenarios.generated.d.ts, which powers route and application-tool completion in run.test.ts. dawn test exits non-zero on any failure and outputs a diff per mismatched scenario.

Unit-testing tools and middleware

The scenario harness drives a whole route through the runtime. Sometimes you want to test one unit in isolation — a single route tool, a FilesystemMiddleware, or ctx.fs-using code — without standing up an agent. @dawn-ai/testing ships three harnesses for this. They run against the real WorkspaceFs and filesystem backend over a temp directory, so real permission gating, realpath resolution, and parent-directory creation all apply.

All three are async create*Harness factories that return a handle with .close() and [Symbol.asyncDispose] — the same convention as createAgentHarness. Across @dawn-ai/testing, every harness/handle is created with a create* factory and torn down with close() (or await using).

A tool that uses ctx.fs

createToolHarness(tool) builds the DawnToolContext and gives you a reusable invoke(). Assert both the return value and what landed on disk via h.workspace.read(...):

import { afterEach } from "vitest"
import { createToolHarness } from "@dawn-ai/testing"
import type { DawnToolContext } from "@dawn-ai/sdk"
 
const saveNote = async (input: { name: string; body: string }, ctx: DawnToolContext) => {
  const { bytesWritten } = await ctx.fs.writeFile(`notes/${input.name}.md`, input.body)
  return { bytesWritten }
}
 
let h: Awaited<ReturnType<typeof createToolHarness>>
afterEach(() => h.close())
 
test("saveNote writes into the workspace", async () => {
  h = await createToolHarness(saveNote)
 
  const result = await h.invoke({ name: "todo", body: "ship it" })
 
  expect(result.bytesWritten).toBe(7)
  expect(await h.workspace.read("notes/todo.md")).toBe("ship it")
})

invoke() is reusable and shares one workspace across calls, so you can assert cumulative state across several invocations. Pass { workspace } to share a fixture you already own (the harness won't close it), or { permissions } to exercise allow/deny gating instead of the permissive default.

Testing ctx.fs code directly

createWorkspaceHarness() is the shared fixture the tool harness builds on. Use it directly when the code under test takes a WorkspaceFs. Seed the workspace with write, run your code against h.fs, and assert with read:

import { createWorkspaceHarness } from "@dawn-ai/testing"
import type { WorkspaceFs } from "@dawn-ai/sdk"
 
const appendLine = async (fs: WorkspaceFs, path: string, line: string) => {
  const prev = await fs.readFile(path)
  await fs.writeFile(path, `${prev}\n${line}`)
}
 
test("appendLine round-trips through the real WorkspaceFs", async () => {
  await using h = await createWorkspaceHarness()
  await h.write("log.txt", "first")
 
  await appendLine(h.fs, "log.txt", "second")
 
  expect(await h.read("log.txt")).toBe("first\nsecond")
})

The await using form auto-disposes the harness (cleaning up the temp dir) at the end of the block — the modern alternative to afterEach(() => h.close()). It works on all three harnesses.

A filesystem middleware

createMiddlewareHarness(mw) composes a FilesystemMiddleware over a temp localFilesystem and exposes the wrapped backend plus a ctx to call it with. assertForwardsAll() catches the most common middleware bug — silently dropping a backend method (required or optional) that the middleware doesn't intercept:

import { createMiddlewareHarness } from "@dawn-ai/testing"
import type { FilesystemMiddleware } from "@dawn-ai/workspace"
 
const uppercaseReads: FilesystemMiddleware = (next) => ({
  ...next,
  readFile: async (path, ctx) => (await next.readFile(path, ctx)).toUpperCase(),
})
 
test("uppercaseReads forwards every other backend method", async () => {
  await using h = await createMiddlewareHarness(uppercaseReads)
 
  await h.backend.writeFile("a.txt", "hello", h.ctx)
  expect(await h.backend.readFile("a.txt", h.ctx)).toBe("HELLO")
 
  // Fails loudly if the middleware forgot to spread a method like realPath.
  h.assertForwardsAll()
})

Related