Class ScenarioBuilder

Fluent builder for constructing ScenarioCreateParams.

Remarks

Overview

The ScenarioBuilder provides a step-by-step, chainable interface for configuring all aspects of a scenario before pushing it to the platform.

Quickstart

import { RunloopSDK } from '@runloop/api-client';

const runloop = new RunloopSDK();
const scenario = await runloop.scenario
  .builder('my-scenario')
  .fromBlueprint(blueprint)
  .withWorkingDirectory('/app')
  .withProblemStatement('Fix the bug in main.py')
  .addTestCommandScorer('tests', { test_command: 'pytest' })
  .push();

Constructors

constructor

new ScenarioBuilder(name: string, client: Runloop): ScenarioBuilder
Create a new ScenarioBuilder.
Parameters
- name: string
  Name for the scenario
- client: Runloop
  The Runloop client instance
Returns ScenarioBuilder
- Defined in src/sdk/scenario-builder.ts:67

Accessors

name

get name(): string
Get the scenario name.

Returns string
The scenario name
- Defined in src/sdk/scenario-builder.ts:76

Methods

fromBlueprint

fromBlueprint(blueprint: RunloopSDK.Blueprint): this
Set a blueprint to define the baseline environment for the scenario.
Parameters
- blueprint: RunloopSDK.Blueprint
  Blueprint to use
Returns this
Builder for chaining
- Defined in src/sdk/scenario-builder.ts:86

fromSnapshot

fromSnapshot(snapshot: RunloopSDK.Snapshot): this
Set a snapshot to define the baseline environment for the scenario.
Parameters
- snapshot: RunloopSDK.Snapshot
  Snapshot to use
Returns this
Builder for chaining
- Defined in src/sdk/scenario-builder.ts:98

withWorkingDirectory

withWorkingDirectory(directory: string): this
Set the working directory for the scenario.
Parameters
- directory: string
  Working directory path
Returns this
Builder for chaining
- Defined in src/sdk/scenario-builder.ts:110

withProblemStatement

withProblemStatement(statement: string): this
Set the problem statement for the scenario. This will be provided as input context to the agent.
Parameters
- statement: string
  Problem statement text
Returns this
Builder for chaining
- Defined in src/sdk/scenario-builder.ts:122

withAdditionalContext

withAdditionalContext(context: unknown): this
Set additional structured context for the scenario.
Parameters
- context: unknown
  Additional context (JSON-serializable)
Returns this
Builder for chaining
- Defined in src/sdk/scenario-builder.ts:133

addTestCommandScorer

addTestCommandScorer(
name: string,
opts: { test_command: string; weight?: number; test_files?: TestFile[] },
): this
Add a test-based scorer that runs a test command.
Parameters
- name: string
  Name of the scoring function
- opts: { test_command: string; weight?: number; test_files?: TestFile[] }
  Scorer options
  - test_command: string
    Command to run tests
  - Optionalweight?: number
    Weight for this scorer (default: 1.0)
  - Optionaltest_files?: TestFile[]
    Test files to create before running
Returns this
Builder for chaining
Example
```
builder.addTestCommandScorer('tests', {
  test_command: 'pytest',
  test_files: [{ file_path: 'test_main.py', file_contents: 'def test_foo(): ...' }],
});
```
- Defined in src/sdk/scenario-builder.ts:156

addShellCommandScorer

addShellCommandScorer(
name: string,
opts: { command: string; weight?: number },
): this
Add a command scorer that runs a shell command. Scoring passes if the command returns exit code 0.
Parameters
- name: string
  Name of the scoring function
- opts: { command: string; weight?: number }
  Scorer options
  - command: string
    Shell command to execute
  - Optionalweight?: number
    Weight for this scorer (default: 1.0)
Returns this
Builder for chaining
- Defined in src/sdk/scenario-builder.ts:182

addBashScriptScorer

addBashScriptScorer(
name: string,
opts: { bash_script: string; weight?: number },
): this
Add a standalone bash script scorer. The script should output "score=X.X" where X.X is a float between 0.0 and 1.0.
Parameters
- name: string
  Name of the scoring function
- opts: { bash_script: string; weight?: number }
  Scorer options
  - bash_script: string
    Bash script content
  - Optionalweight?: number
    Weight for this scorer (default: 1.0)
Returns this
Builder for chaining
- Defined in src/sdk/scenario-builder.ts:200

addPythonScriptScorer

addPythonScriptScorer(
    name: string,
    opts: {
        python_script: string;
        weight?: number;
        python_version_constraint?: string;
        requirements_contents?: string;
    },
): this
Add a standalone Python script scorer. The script should print the score in the range [0.0, 1.0] to stdout.
Parameters
- name: string
  Name of the scoring function
- opts: {
      python_script: string;
      weight?: number;
      python_version_constraint?: string;
      requirements_contents?: string;
  }
  Scorer options
  - python_script: string
    Python script content
  - Optionalweight?: number
    Weight for this scorer (default: 1.0)
  - Optionalpython_version_constraint?: string
    Python version (default "==3.12.10")
  - Optionalrequirements_contents?: string
    pip requirements.txt content
Returns this
Builder for chaining
- Defined in src/sdk/scenario-builder.ts:220

addAstGrepScorer

addAstGrepScorer(
    name: string,
    opts: {
        pattern: string;
        weight?: number;
        search_directory?: string;
        lang?: string;
    },
): this
Add an AST grep scorer that matches code patterns.
Parameters
- name: string
  Name of the scoring function
- opts: { pattern: string; weight?: number; search_directory?: string; lang?: string }
  Scorer options
  - pattern: string
    AST pattern to match
  - Optionalweight?: number
    Weight for this scorer (default: 1.0)
  - Optionalsearch_directory?: string
    Directory to search (default: ".")
  - Optionallang?: string
    Language of the pattern
Returns this
Builder for chaining
- Defined in src/sdk/scenario-builder.ts:251

addCustomScorer

addCustomScorer(
    name: string,
    opts: {
        custom_scorer_type: string;
        weight?: number;
        scorer_params?: unknown;
    },
): this
Add a custom scorer registered with Runloop.
Parameters
- name: string
  Name of the scoring function
- opts: { custom_scorer_type: string; weight?: number; scorer_params?: unknown }
  Scorer options
  - custom_scorer_type: string
    Type identifier registered with Runloop
  - Optionalweight?: number
    Weight for this scorer (default: 1.0)
  - Optionalscorer_params?: unknown
    Additional JSON parameters for the scorer
Returns this
Builder for chaining
- Defined in src/sdk/scenario-builder.ts:279

withMetadata

withMetadata(metadata: Record<string, string>): this
Set metadata for the scenario.
Parameters
- metadata: Record<string, string>
  Key-value metadata
Returns this
Builder for chaining
- Defined in src/sdk/scenario-builder.ts:301

withReferenceOutput

withReferenceOutput(output: string): this
Set the reference solution or gold patch for validation.
Parameters
- output: string
  Reference solution (e.g., git diff)
Returns this
Builder for chaining
- Defined in src/sdk/scenario-builder.ts:312

withRequiredEnvVars

withRequiredEnvVars(envVars: string[]): this
Set required environment variables.
Parameters
- envVars: string[]
  List of required environment variable names
Returns this
Builder for chaining
- Defined in src/sdk/scenario-builder.ts:323

withRequiredSecrets

withRequiredSecrets(secrets: string[]): this
Set required secrets.
Parameters
- secrets: string[]
  List of required secret names
Returns this
Builder for chaining
- Defined in src/sdk/scenario-builder.ts:334

withValidationType

withValidationType(
validationType: "UNSPECIFIED" | "FORWARD" | "REVERSE" | "EVALUATION",
): this
Set the validation strategy.
Parameters
- validationType: "UNSPECIFIED" | "FORWARD" | "REVERSE" | "EVALUATION"
  Validation type
Returns this
Builder for chaining
- Defined in src/sdk/scenario-builder.ts:345

build

build(): ScenarioCreateParams
Build the scenario creation parameters.

Validates that required fields are set and normalizes scorer weights to sum to 1.0.

Returns ScenarioCreateParams
Parameters for scenario creation
Example
```
const params = builder.build();
// Use params with the raw API client
const scenarioView = await client.scenarios.create(params);
```
Throws
If problem statement is missing or no scorers are configured
- Defined in src/sdk/scenario-builder.ts:366

push

push(options?: RequestOptions<unknown>): Promise<RunloopSDK.Scenario>
Create the scenario on the platform.

Calls build to validate and assemble parameters, then creates the scenario via the API.
Parameters
- Optionaloptions: RequestOptions<unknown>
  Request options
Returns Promise<RunloopSDK.Scenario>
Created Scenario instance
Example
```
const scenario = await runloop.scenario
  .builder('my-scenario')
  .withProblemStatement('Fix the bug')
  .addTestCommandScorer('tests', { test_command: 'pytest' })
  .push();
console.log(`Created scenario: ${scenario.id}`);
```
Throws
If required fields are missing
- Defined in src/sdk/scenario-builder.ts:431

Class ScenarioBuilder

Remarks

Overview

Quickstart

Index

Constructors

Accessors

Methods

Constructors

constructor

Parameters

Returns ScenarioBuilder

Accessors

name

Returns string

Methods

fromBlueprint

Parameters

Returns this

fromSnapshot

Parameters

Returns this

withWorkingDirectory

Parameters

Returns this

withProblemStatement

Parameters

Returns this

withAdditionalContext

Parameters

Returns this

addTestCommandScorer

Parameters

test_command: string

Optionalweight?: number

Optionaltest_files?: TestFile[]

Returns this

Example

addShellCommandScorer

Parameters

command: string

Optionalweight?: number

Returns this

addBashScriptScorer

Parameters

bash_script: string

Optionalweight?: number

Returns this

addPythonScriptScorer

Parameters

python_script: string

Optionalweight?: number

Optionalpython_version_constraint?: string

Optionalrequirements_contents?: string

Returns this

addAstGrepScorer

Parameters

pattern: string

Optionalweight?: number

Optionalsearch_directory?: string

Optionallang?: string

Returns this

addCustomScorer

Parameters

custom_scorer_type: string

Optionalweight?: number

Optionalscorer_params?: unknown

Returns this

withMetadata

Parameters

Returns this

withReferenceOutput

Parameters

Returns this

withRequiredEnvVars

Parameters

Returns this

withRequiredSecrets

Parameters

Returns this

`Optional`weight?: number

`Optional`test_files?: TestFile[]

`Optional`weight?: number

`Optional`weight?: number

`Optional`weight?: number

`Optional`python_version_constraint?: string

`Optional`requirements_contents?: string

`Optional`weight?: number

`Optional`search_directory?: string

`Optional`lang?: string

`Optional`weight?: number

`Optional`scorer_params?: unknown