@runloop/api-client - v1.4.0
    Preparing search index...

    Class ScorerOps

    Scorer SDK interface for managing custom scorers.

    Scorers are custom scoring functions used to evaluate scenario outputs. A scorer is a script that runs and prints a score in the range [0.0, 1.0], e.g. echo "0.5".

    This interface is accessed via RunloopSDK.scorer. Create scorers with ScorerOps.create or reference an existing scorer by ID with ScorerOps.fromId to obtain a Scorer instance.

    import { RunloopSDK } from '@runloop/api-client';

    const runloop = new RunloopSDK();

    // Create a scorer
    const scorer = await runloop.scorer.create({
      type: 'my_scorer',
      bash_script: 'echo "1.0"',
    });

    // Update the scorer
    await scorer.update({ bash_script: 'echo "0.5"' });

    // Validate the scorer with a scoring context
    const result = await scorer.validate({ scoring_context: { output: 'hello' } });
    console.log(result.scoring_result.score);

    Get scorer info (typical usage):

    const runloop = new RunloopSDK();
    const scorer = await runloop.scorer.create({
      type: 'my_scorer',
      bash_script: 'echo "1.0"',
    });

    const info = await scorer.getInfo();
    console.log(`Scorer ${info.id} (${info.type})`);
    Index

    Methods

    create fromId list

    Methods

    • Create a new custom scorer.

      Parameters

      • params: ScorerCreateParams

        Parameters for creating the scorer

      • Optionaloptions: RequestOptions<unknown>

        Request options

      Returns Promise<RunloopSDK.Scorer>

      A Scorer instance

      const runloop = new RunloopSDK();
      const scorer = await runloop.scorer.create({
        type: 'my_scorer',
        bash_script: 'echo "1.0"',
      });

      const info = await scorer.getInfo();
      console.log(info.id);

    • Get a scorer object by its ID.

      Parameters

      • id: string

        The ID of the scorer

      Returns RunloopSDK.Scorer

      A Scorer instance

      const runloop = new RunloopSDK();
      const scorer = runloop.scorer.fromId('scs_123');
      const info = await scorer.getInfo();
      console.log(info.type);

    • List scorers with optional filters (paginated).

      Parameters

      • Optionalparams: ScorerListParams

        Optional filter parameters

      • Optionaloptions: RequestOptions<unknown>

        Request options

      Returns Promise<RunloopSDK.Scorer[]>

      An array of Scorer instances

      const runloop = new RunloopSDK();
      const scorers = await runloop.scorer.list({ limit: 10 });
      console.log(scorers.map((s) => s.id));

    Settings

    Member Visibility

    On This Page

    OverviewUsage
    Methods