Do you already have a working eval? Copy the instructions below to have an agent create your Braintrust files for you.
Prepare the files with your LLM
Share these instructions and your finished evaluation with your LLM, then test the generated package yourself.
View the full instructions
Prepare your package
Add a braintrust/ folder at the root of your evaluation ZIP with these four files:
| File | What to include |
|---|---|
README.md | Setup, task and scorer behavior, dependencies or model assumptions, and your tested run instructions. |
dataset.json | A nonempty array of cases with input and, where needed, reference answers in expected. |
tasks.json | A nonempty array of complete prompt or inline-code tasks that produce answers from your inputs. |
scorers.json | A nonempty array of inline-code scorers or LLM judges that grade those answers. |
Keep reference answers separate from task inputs. Leave credentials, destination IDs, and model/provider settings out of the package; the person importing selects their own project and models.
Use Braintrust’s guides for datasets, code scorers, and LLM judges. EvalMarket supports the subset below, so an arbitrary Braintrust API payload may not pass its file checks.
Package limits and accepted fields
- Files: nonempty UTF-8, at most 5 MiB each. Existing ZIP limits apply. JSON must not contain duplicate keys, NaN, or Infinity.
- Cases: required
inputaccepts any JSON value. Optional fields areid(unique nonempty string),expected(any JSON value),metadata(object), andtags(string array). - Tasks and scorers: required
name,slug,function_type(taskorscorer), andfunction_data. Slugs must be unique across both files: 1–80 lowercase letters/digits separated by single hyphens or underscores. Optional fields aredescription,tags, andfunction_schemawithparameters/returnsJSON schemas using local references only. - Code: self-contained inline Node 22 or Python 3.12 source with a
handlerentry point. No external bundles, dependency installation, or TypeScript compilation. Match the included cases and scorer contract; Python object inputs are passed as named arguments, so accept the input fields and**kwargs. - Prompt tasks: chat messages with Mustache
{{input}}or{{input.field}}bindings (triple braces allowed), and no classifier parser. The helper maps the package roletaskto Braintrust’s nativellmfunction type. - LLM scorers: chat messages plus an
llm_classifierparser with booleanuse_cotandchoice_scoresmapping choices to finite scores from 0 to 1; optionalallow_skipis boolean. Supported template formats aremustache,nunjucks, andnone.
Use the nested structure in the examples. Code function_data contains only type and data; inline data contains only type, runtime_context, and code. Prompt function_data contains only type: "prompt"; prompt_data contains prompt, options, the scorer’s parser, and optional template_format.
Chat messages need a system/developer/user/assistant role, nonempty text content, and optionally a nonempty name. Options allow params and an optional nonempty position string; {} is valid. Allowed numeric params: temperature, top_p, top_k, max_tokens, max_completion_tokens, frequency_penalty, presence_penalty, n, reasoning_budget, max_tokens_to_sample, maxOutputTokens, topP, topK. Also allowed: booleans use_cache, reasoning_enabled; string arrays stop, stop_sequences; reasoning_effort (none/minimal/low/medium/high); and verbosity (low/medium/high). Test compatibility with your selected provider.
Other fields are rejected. Tools, MCP, preprocessors, remote-eval servers, attachment transfer, and categorical classifiers are unsupported. Do not substitute buyer-written task logic for a complete working task.
Example dataset, tasks, and scorers
These synthetic examples show the accepted structure. Replace them with your own tested cases, task logic, and rubric.
Dataset — save as dataset.json:
[
{
"id": "addition",
"input": "What is 2 + 2? Return an integer.",
"expected": 4,
"metadata": {
"topic": "arithmetic"
},
"tags": [
"synthetic"
]
},
{
"input": "What is 6 * 7? Return an integer.",
"expected": 42
}
]Prompt task — save as tasks.json:
[
{
"name": "Answer arithmetic questions",
"slug": "answer-arithmetic",
"function_type": "task",
"function_data": {
"type": "prompt"
},
"prompt_data": {
"template_format": "mustache",
"prompt": {
"type": "chat",
"messages": [
{
"role": "system",
"content": "Solve the arithmetic question. Return only the integer answer."
},
{
"role": "user",
"content": "{{input}}"
}
]
},
"options": {
"params": {
"temperature": 0,
"max_tokens": 1024
}
}
}
}
]JavaScript scorer — save as scorers.json:
[
{
"name": "Exact integer match",
"slug": "exact-integer",
"function_type": "scorer",
"function_data": {
"type": "code",
"data": {
"type": "inline",
"runtime_context": {
"runtime": "node",
"version": "22"
},
"code": "function handler({output, expected}) { const answer = typeof output === 'string' && /^[+-]?[0-9]+$/.test(output.trim()) ? Number(output.trim()) : output; return {score: Number(typeof answer === 'number' && Number.isInteger(answer) && answer === expected)}; }"
}
}
}
]Python alternative — use this code with runtime python, version 3.12:
import re
def handler(output, expected, **kwargs):
if isinstance(output, str):
if not re.fullmatch(r"[+-]?[0-9]+", output.strip()):
return {"score": 0}
try:
output = int(output.strip())
except ValueError:
return {"score": 0}
return {"score": int(type(output) is int and output == expected)}
LLM judge alternative — save as scorers.json, without model or provider settings:
[
{
"name": "Synthetic answer judge",
"slug": "answer-judge",
"function_type": "scorer",
"function_data": {
"type": "prompt"
},
"prompt_data": {
"prompt": {
"type": "chat",
"messages": [
{
"role": "system",
"content": "Choose correct if the answer matches the reference, otherwise incorrect."
},
{
"role": "user",
"content": "Task: {{input}}\nAnswer: {{output}}\nReference: {{expected}}"
}
]
},
"options": {
"params": {
"temperature": 0
}
},
"parser": {
"type": "llm_classifier",
"use_cot": true,
"choice_scores": {
"correct": 1,
"incorrect": 0
},
"allow_skip": false
}
}
}
]Test the import
You need Python 3.10+, a Braintrust API key, and access to a project. Configure model providers in Braintrust for prompt tasks or LLM judges. No Braintrust SDK or CLI is required for the import.
Put the helper beside your braintrust/ folder and run it from that directory:
Linux, macOS, or WSL
python3 import_eval.py braintrustWindows
python import_eval.py braintrust- Enter your key at the masked prompt, or set
BRAINTRUST_API_KEYlocally. The helper sends it directly to Braintrust; EvalMarket does not receive it. - Select your project and a model/provider for each prompt task or LLM scorer.
- Review and confirm. The helper validates the package, creates the dataset and functions, then checks the uploaded content by reading it back. Inspect the printed resource IDs in Braintrust.
Model selection, scripted imports, and older packages
The picker uses models discovered from your account; listing a model does not verify inference access. If a model is missing, check provider configuration and API-key access in Braintrust. Choose models compatible with your prompts and parameters.
For unattended imports, set the environment key and pass --project-id PROJECT_UUID --yes, plus --model function-slug=provider/model-name for every prompt task and LLM scorer. If ambiguous, add --provider function-slug=provider-configuration-name. Choices must match discovered account options. --yes confirms the import only; it never starts an experiment.
The helper uses https://api.braintrust.dev. Custom endpoints, EU-specific endpoints, and BYOC are not currently supported.
Older packages without tasks need an updated seller package. Legacy model/provider settings are discarded in memory so the importer can choose their own; new submissions must omit them. Source files remain unchanged.
Verify tasks and scorers
A successful import does not test behavior. Run every included task and scorer on representative cases. Check known correct and incorrect outputs against your rubric, and resolve runtime or model-access errors before treating scores as verified results.
After an interactive import, choose whether to start an experiment. Select a task if prompted; it runs with the imported dataset and scorers. Runs may incur model charges. The helper returns a link without waiting for results, so open it and check that the run finished successfully.
To run a prompt task later, follow Braintrust’s experiment setup guide. Inline-code tasks are saved functions and are not listed in the UI task picker; provide tested API/SDK run instructions for buyers. See how to interpret results when reviewing scores.
If experiment start is unconfirmed, inspect the named experiment before retrying. Your verified import remains complete.
Before you list
Record your tested procedure, runtime or model/provider assumptions, results, and verification date in braintrust/README.md. Identify anything untested so buyers know what to expect.
Submit the ZIP through EvalMarket and check the Braintrust Easy Import result. This checks required files and structure; it does not certify task behavior, code safety, or evaluation quality.