Do you already have a working eval? Copy the instructions below to have an agent create your Braintrust files for you.

Prepare the files with your LLM

Share these instructions and your finished evaluation with your LLM, then test the generated package yourself.

View the full instructions

Prepare your package

Add a braintrust/ folder at the root of your evaluation ZIP with these four files:

FileWhat to include
README.mdSetup, task and scorer behavior, dependencies or model assumptions, and your tested run instructions.
dataset.jsonA nonempty array of cases with input and, where needed, reference answers in expected.
tasks.jsonA nonempty array of complete prompt or inline-code tasks that produce answers from your inputs.
scorers.jsonA nonempty array of inline-code scorers or LLM judges that grade those answers.

Keep reference answers separate from task inputs. Leave credentials, destination IDs, and model/provider settings out of the package; the person importing selects their own project and models.

Use Braintrust’s guides for datasets, code scorers, and LLM judges. EvalMarket supports the subset below, so an arbitrary Braintrust API payload may not pass its file checks.

Package limits and accepted fields
  • Files: nonempty UTF-8, at most 5 MiB each. Existing ZIP limits apply. JSON must not contain duplicate keys, NaN, or Infinity.
  • Cases: required input accepts any JSON value. Optional fields are id (unique nonempty string), expected (any JSON value), metadata (object), and tags (string array).
  • Tasks and scorers: required name, slug, function_type (task or scorer), and function_data. Slugs must be unique across both files: 1–80 lowercase letters/digits separated by single hyphens or underscores. Optional fields are description, tags, and function_schema with parameters/returns JSON schemas using local references only.
  • Code: self-contained inline Node 22 or Python 3.12 source with a handler entry point. No external bundles, dependency installation, or TypeScript compilation. Match the included cases and scorer contract; Python object inputs are passed as named arguments, so accept the input fields and **kwargs.
  • Prompt tasks: chat messages with Mustache {{input}} or {{input.field}} bindings (triple braces allowed), and no classifier parser. The helper maps the package role task to Braintrust’s native llm function type.
  • LLM scorers: chat messages plus an llm_classifier parser with boolean use_cot and choice_scores mapping choices to finite scores from 0 to 1; optional allow_skip is boolean. Supported template formats are mustache, nunjucks, and none.

Use the nested structure in the examples. Code function_data contains only type and data; inline data contains only type, runtime_context, and code. Prompt function_data contains only type: "prompt"; prompt_data contains prompt, options, the scorer’s parser, and optional template_format.

Chat messages need a system/developer/user/assistant role, nonempty text content, and optionally a nonempty name. Options allow params and an optional nonempty position string; {} is valid. Allowed numeric params: temperature, top_p, top_k, max_tokens, max_completion_tokens, frequency_penalty, presence_penalty, n, reasoning_budget, max_tokens_to_sample, maxOutputTokens, topP, topK. Also allowed: booleans use_cache, reasoning_enabled; string arrays stop, stop_sequences; reasoning_effort (none/minimal/low/medium/high); and verbosity (low/medium/high). Test compatibility with your selected provider.

Other fields are rejected. Tools, MCP, preprocessors, remote-eval servers, attachment transfer, and categorical classifiers are unsupported. Do not substitute buyer-written task logic for a complete working task.

Example dataset, tasks, and scorers

These synthetic examples show the accepted structure. Replace them with your own tested cases, task logic, and rubric.

Dataset — save as dataset.json:

JSON
[
  {
    "id": "addition",
    "input": "What is 2 + 2? Return an integer.",
    "expected": 4,
    "metadata": {
      "topic": "arithmetic"
    },
    "tags": [
      "synthetic"
    ]
  },
  {
    "input": "What is 6 * 7? Return an integer.",
    "expected": 42
  }
]

Prompt task — save as tasks.json:

JSON
[
  {
    "name": "Answer arithmetic questions",
    "slug": "answer-arithmetic",
    "function_type": "task",
    "function_data": {
      "type": "prompt"
    },
    "prompt_data": {
      "template_format": "mustache",
      "prompt": {
        "type": "chat",
        "messages": [
          {
            "role": "system",
            "content": "Solve the arithmetic question. Return only the integer answer."
          },
          {
            "role": "user",
            "content": "{{input}}"
          }
        ]
      },
      "options": {
        "params": {
          "temperature": 0,
          "max_tokens": 1024
        }
      }
    }
  }
]

JavaScript scorer — save as scorers.json:

JSON
[
  {
    "name": "Exact integer match",
    "slug": "exact-integer",
    "function_type": "scorer",
    "function_data": {
      "type": "code",
      "data": {
        "type": "inline",
        "runtime_context": {
          "runtime": "node",
          "version": "22"
        },
        "code": "function handler({output, expected}) { const answer = typeof output === 'string' && /^[+-]?[0-9]+$/.test(output.trim()) ? Number(output.trim()) : output; return {score: Number(typeof answer === 'number' && Number.isInteger(answer) && answer === expected)}; }"
      }
    }
  }
]

Python alternative — use this code with runtime python, version 3.12:

Python
import re

def handler(output, expected, **kwargs):
    if isinstance(output, str):
        if not re.fullmatch(r"[+-]?[0-9]+", output.strip()):
            return {"score": 0}
        try:
            output = int(output.strip())
        except ValueError:
            return {"score": 0}
    return {"score": int(type(output) is int and output == expected)}

LLM judge alternative — save as scorers.json, without model or provider settings:

JSON
[
  {
    "name": "Synthetic answer judge",
    "slug": "answer-judge",
    "function_type": "scorer",
    "function_data": {
      "type": "prompt"
    },
    "prompt_data": {
      "prompt": {
        "type": "chat",
        "messages": [
          {
            "role": "system",
            "content": "Choose correct if the answer matches the reference, otherwise incorrect."
          },
          {
            "role": "user",
            "content": "Task: {{input}}\nAnswer: {{output}}\nReference: {{expected}}"
          }
        ]
      },
      "options": {
        "params": {
          "temperature": 0
        }
      },
      "parser": {
        "type": "llm_classifier",
        "use_cot": true,
        "choice_scores": {
          "correct": 1,
          "incorrect": 0
        },
        "allow_skip": false
      }
    }
  }
]

Test the import

You need Python 3.10+, a Braintrust API key, and access to a project. Configure model providers in Braintrust for prompt tasks or LLM judges. No Braintrust SDK or CLI is required for the import.

Download import_eval.py

Put the helper beside your braintrust/ folder and run it from that directory:

Linux, macOS, or WSL

Terminal
python3 import_eval.py braintrust

Windows

Terminal
python import_eval.py braintrust
  1. Enter your key at the masked prompt, or set BRAINTRUST_API_KEY locally. The helper sends it directly to Braintrust; EvalMarket does not receive it.
  2. Select your project and a model/provider for each prompt task or LLM scorer.
  3. Review and confirm. The helper validates the package, creates the dataset and functions, then checks the uploaded content by reading it back. Inspect the printed resource IDs in Braintrust.
Model selection, scripted imports, and older packages

The picker uses models discovered from your account; listing a model does not verify inference access. If a model is missing, check provider configuration and API-key access in Braintrust. Choose models compatible with your prompts and parameters.

For unattended imports, set the environment key and pass --project-id PROJECT_UUID --yes, plus --model function-slug=provider/model-name for every prompt task and LLM scorer. If ambiguous, add --provider function-slug=provider-configuration-name. Choices must match discovered account options. --yes confirms the import only; it never starts an experiment.

The helper uses https://api.braintrust.dev. Custom endpoints, EU-specific endpoints, and BYOC are not currently supported.

Older packages without tasks need an updated seller package. Legacy model/provider settings are discarded in memory so the importer can choose their own; new submissions must omit them. Source files remain unchanged.

Verify tasks and scorers

A successful import does not test behavior. Run every included task and scorer on representative cases. Check known correct and incorrect outputs against your rubric, and resolve runtime or model-access errors before treating scores as verified results.

After an interactive import, choose whether to start an experiment. Select a task if prompted; it runs with the imported dataset and scorers. Runs may incur model charges. The helper returns a link without waiting for results, so open it and check that the run finished successfully.

To run a prompt task later, follow Braintrust’s experiment setup guide. Inline-code tasks are saved functions and are not listed in the UI task picker; provide tested API/SDK run instructions for buyers. See how to interpret results when reviewing scores.

If experiment start is unconfirmed, inspect the named experiment before retrying. Your verified import remains complete.

Before you list

Record your tested procedure, runtime or model/provider assumptions, results, and verification date in braintrust/README.md. Identify anything untested so buyers know what to expect.

Submit the ZIP through EvalMarket and check the Braintrust Easy Import result. This checks required files and structure; it does not certify task behavior, code safety, or evaluation quality.