# Braintrust for sellers

Prepare your Braintrust package, test it in your own project, and verify the results before listing on EvalMarket.

Source: https://docs.evalmarket.ai/platform-integrations/braintrust/

Do you already have a working eval? Copy the instructions below to have an agent create your Braintrust files for you.

## Prepare the files with your LLM

Share these instructions and your finished evaluation with your LLM, then test the generated package yourself.

### View the full instructions

```text
Prepare a seller-owned Braintrust edition of my existing evaluation including its complete task setup, without inventing or changing task logic, cases, expected answers, grading logic, or rubric. Ask me about missing requirements. Add braintrust/README.md, braintrust/dataset.json, braintrust/tasks.json, and braintrust/scorers.json at the ZIP root; preserve my other files. Follow https://www.evalmarket.ai/seller/import-guide#braintrust and https://docs.braintrust.dev/openapi.json.

README.md must be nonempty UTF-8 and explain the seller-tested import, included task setup, task/scorer behavior, shared input/output contract, runtimes/dependencies or model assumptions, and buyer setup. Each required file must be at most 5 MiB. Existing ZIP limits apply. There is no extra total case cap.

dataset.json: nonempty array of native objects. Required input preserves any JSON type. Only optional id (unique nonempty string), expected (any JSON value), metadata (object), tags (string array) are allowed. Keep references in expected or scorer metadata, separate from task inputs. Generate no placeholder references. Reject duplicate keys/IDs, NaN/Infinity, and unknown/control fields.

tasks.json: required nonempty array of complete task definitions with function_type:"task" as the package role. The helper saves answering prompts as native function_type:"llm" so Braintrust lists them under + Task > Prompt; code tasks remain native task functions for API/SDK execution by ID and require seller-tested run instructions. Use the same name/slug/function_data and optional fields as scorers; slugs must be unique across tasks AND scorers. Code tasks receive the case input: Node uses handler(input); Python receives object input fields as keyword arguments, or scalar/array input as the input keyword, plus hooks. Use a matching handler signature with **kwargs in Python and return the task output. Prompt tasks supply the answering messages and options (without model or endpoint_name), and template_format:"mustache", without a scoring parser. Template bindings may reference only input or input.field (double or triple braces); no expected/output/metadata bindings, blocks, tools, or preprocessors. All tasks run against the packaged dataset and scorer contract. Buyers configure required access; they do not author or connect a task. Reject missing task setup; do not fabricate a generic task from dataset inputs.

scorers.json: nonempty array. Every scorer requires name (nonempty), slug (unique, 1–80 lowercase letters/digits separated by single hyphens/underscores), function_type:"scorer", and function_data. Optional description (string), tags (string array), function_schema (parameters/returns as JSON schema objects or booleans, local references only). Do not include project IDs, remote object IDs, credentials, origins, or remote references.

Code scorer: function_data = {"type":"code","data":{"type":"inline","runtime_context":{"runtime":"node","version":"22"},"code":"..."}}; python/3.12 is also supported. Use Braintrust's handler entry point: function handler({output, expected}) in JavaScript or def handler(output, expected, **kwargs) in Python. Preserve type-sensitive scoring. Include self-contained source as a string using only hosted runtime dependencies. No bundles, TypeScript compilation, dependency installation, or prompt_data on code scorers.

LLM scorer: function_data:{"type":"prompt"}; prompt_data requires prompt:{type:"chat",messages:[{role:"system"|"developer"|"user"|"assistant",content:"nonempty text",name?:"nonempty"}]}, options:{params?:{...},position?:"nonempty"}, parser:{type:"llm_classifier",use_cot:boolean,choice_scores:{"choice":0.0},allow_skip?:boolean}. Scores must be finite numbers in [0,1]. Optional template_format is mustache, nunjucks, or none. Do not include model or endpoint_name in task/scorer options: the importer generates those fields from buyer account selections. Empty options:{} is valid. Preserve messages, variables, choice names and scores, model parameters, and optional skip behavior. No tools, MCP, preprocessors, categorical classifiers, or attachments. Supported numeric params: temperature, top_p, top_k, max_tokens, max_completion_tokens, frequency_penalty, presence_penalty, n, reasoning_budget, max_tokens_to_sample, maxOutputTokens, topP, topK. Boolean params: use_cache, reasoning_enabled. String arrays: stop, stop_sequences. reasoning_effort: none/minimal/low/medium/high; verbosity: low/medium/high. Other fields are unsupported.

Download the shared import_eval.py from Docs. With Python 3.10+, run python import_eval.py braintrust (python3 on Linux/macOS/WSL) beside the extracted folder. Set BRAINTRUST_API_KEY locally or enter it at the masked prompt. Never put secrets in files or command arguments. Select an accessible project. For every prompt task and LLM scorer, choose a numbered model/provider pair discovered from your account; q cancels and n/p page long lists. There are no seller defaults or manual model entries. Discovery may omit project-only providers/overrides and does not test inference access or chat support. If no choices are discovered, the helper stops before writes; check existing Braintrust provider configuration and API-key access. Prompts, grading and model parameters stay unchanged. Review and confirm the displayed dataset, tasks, and scorers. Invalid project/model choices repeat the numbered list. Optional buyer --model function-slug=model-name and --provider function-slug=provider-configuration-name choices must match discovery. A model with exactly one matching provider selects that pair; otherwise choose a matching provider. Provider-only flags still require a model choice. Unattended imports require --project-id PROJECT_UUID --yes and --model for every prompt task/LLM scorer, plus --provider when ambiguous. Code-only packages need no models. Older packages containing model/provider fields remain importable: the helper discards those fields in memory, preserves seller files, and requires fresh buyer account selections. New submissions must omit them. No Braintrust SDK/CLI is needed. Only https://api.braintrust.dev is supported.

The helper creates fresh resources, checks for collisions, uploads and reads back the native files, and reports confirmed versus uncertain partial results. Inspect the named project/resources before retrying a failure; there is no automatic retry, resume, or cleanup. Upload/verification does not run tasks, models, or scorers, and seller code never executes locally. After successful interactive import, the helper asks "Would you like to start an experiment? [y/N]". Explicit acceptance runs one imported task on the full dataset with the imported scorers, at concurrency 1, and returns the experiment link without waiting for results. Multiple tasks prompt a choice. Enter/no skips execution; --yes confirms import only, and unattended imports never launch. Configure model access first; execution may incur charges. An unconfirmed start can already be running: inspect Braintrust before retrying; do not re-import. Record untested behavior honestly; a structural pass does not certify code safety or model access.

To evaluate later, configure provider/model access in Braintrust, for a prompt task open Experiments > + Experiment > + Task > Prompt, select the imported prompt, dataset and scorers, and deliberately click Create to run. Refresh Braintrust after import. Inline code tasks are not listed in the UI picker; provide tested API/SDK run instructions for their saved IDs. The task setup is already included. Only case input goes to the supplied answering task; expected answers and grading material stay with scorers. Seller code runs with Braintrust hosted-function permissions. Missing model access is a setup error, not a zero score. Include the complete braintrust/ folder in the ZIP and use EvalMarket's optional Easy Import checks. LangSmith can coexist independently in langchain/.
```

## Prepare your package

Add a `braintrust/` folder at the root of your evaluation ZIP with these four files:

| File | What to include |
| --- | --- |
| `README.md` | Setup, task and scorer behavior, dependencies or model assumptions, and your tested run instructions. |
| `dataset.json` | A nonempty array of cases with `input` and, where needed, reference answers in `expected`. |
| `tasks.json` | A nonempty array of complete prompt or inline-code tasks that produce answers from your inputs. |
| `scorers.json` | A nonempty array of inline-code scorers or LLM judges that grade those answers. |

Keep reference answers separate from task inputs. Leave credentials, destination IDs, and model/provider settings out of the package; the person importing selects their own project and models.

Use Braintrust’s guides for [datasets](https://docs.braintrust.dev/annotate/datasets), [code scorers](https://docs.braintrust.dev/evaluate/custom-code), and [LLM judges](https://docs.braintrust.dev/evaluate/llm-as-a-judge). EvalMarket supports the subset below, so an arbitrary Braintrust API payload may not pass its file checks.

### Package limits and accepted fields

- **Files:** nonempty UTF-8, at most 5 MiB each. Existing ZIP limits apply. JSON must not contain duplicate keys, NaN, or Infinity.
- **Cases:** required `input` accepts any JSON value. Optional fields are `id` (unique nonempty string), `expected` (any JSON value), `metadata` (object), and `tags` (string array).
- **Tasks and scorers:** required `name`, `slug`, `function_type` (`task` or `scorer`), and `function_data`. Slugs must be unique across both files: 1–80 lowercase letters/digits separated by single hyphens or underscores. Optional fields are `description`, `tags`, and `function_schema` with `parameters`/`returns` JSON schemas using local references only.
- **Code:** self-contained inline Node 22 or Python 3.12 source with a `handler` entry point. No external bundles, dependency installation, or TypeScript compilation. Match the included cases and scorer contract; Python object inputs are passed as named arguments, so accept the input fields and `**kwargs`.
- **Prompt tasks:** chat messages with Mustache `{{input}}` or `{{input.field}}` bindings (triple braces allowed), and no classifier parser. The helper maps the package role `task` to Braintrust’s native `llm` function type.
- **LLM scorers:** chat messages plus an `llm_classifier` parser with boolean `use_cot` and `choice_scores` mapping choices to finite scores from 0 to 1; optional `allow_skip` is boolean. Supported template formats are `mustache`, `nunjucks`, and `none`.

Use the nested structure in the examples. Code `function_data` contains only `type` and `data`; inline data contains only `type`, `runtime_context`, and `code`. Prompt `function_data` contains only `type: "prompt"`; `prompt_data` contains `prompt`, `options`, the scorer’s `parser`, and optional `template_format`.

Chat messages need a system/developer/user/assistant role, nonempty text content, and optionally a nonempty name. Options allow `params` and an optional nonempty `position` string; `{}` is valid. Allowed numeric params: `temperature, top_p, top_k, max_tokens, max_completion_tokens, frequency_penalty, presence_penalty, n, reasoning_budget, max_tokens_to_sample, maxOutputTokens, topP, topK`. Also allowed: booleans `use_cache, reasoning_enabled`; string arrays `stop, stop_sequences`; `reasoning_effort` (none/minimal/low/medium/high); and `verbosity` (low/medium/high). Test compatibility with your selected provider.

Other fields are rejected. Tools, MCP, preprocessors, remote-eval servers, attachment transfer, and categorical classifiers are unsupported. Do not substitute buyer-written task logic for a complete working task.

### Example dataset, tasks, and scorers

These synthetic examples show the accepted structure. Replace them with your own tested cases, task logic, and rubric.

**Dataset** — save as `dataset.json`:

```json
[
  {
    "id": "addition",
    "input": "What is 2 + 2? Return an integer.",
    "expected": 4,
    "metadata": {
      "topic": "arithmetic"
    },
    "tags": [
      "synthetic"
    ]
  },
  {
    "input": "What is 6 * 7? Return an integer.",
    "expected": 42
  }
]
```

**Prompt task** — save as `tasks.json`:

```json
[
  {
    "name": "Answer arithmetic questions",
    "slug": "answer-arithmetic",
    "function_type": "task",
    "function_data": {
      "type": "prompt"
    },
    "prompt_data": {
      "template_format": "mustache",
      "prompt": {
        "type": "chat",
        "messages": [
          {
            "role": "system",
            "content": "Solve the arithmetic question. Return only the integer answer."
          },
          {
            "role": "user",
            "content": "{{input}}"
          }
        ]
      },
      "options": {
        "params": {
          "temperature": 0,
          "max_tokens": 1024
        }
      }
    }
  }
]
```

**JavaScript scorer** — save as `scorers.json`:

```json
[
  {
    "name": "Exact integer match",
    "slug": "exact-integer",
    "function_type": "scorer",
    "function_data": {
      "type": "code",
      "data": {
        "type": "inline",
        "runtime_context": {
          "runtime": "node",
          "version": "22"
        },
        "code": "function handler({output, expected}) { const answer = typeof output === 'string' && /^[+-]?[0-9]+$/.test(output.trim()) ? Number(output.trim()) : output; return {score: Number(typeof answer === 'number' && Number.isInteger(answer) && answer === expected)}; }"
      }
    }
  }
]
```

**Python alternative** — use this code with runtime `python`, version `3.12`:

```python
import re

def handler(output, expected, **kwargs):
    if isinstance(output, str):
        if not re.fullmatch(r"[+-]?[0-9]+", output.strip()):
            return {"score": 0}
        try:
            output = int(output.strip())
        except ValueError:
            return {"score": 0}
    return {"score": int(type(output) is int and output == expected)}
```

**LLM judge alternative** — save as `scorers.json`, without model or provider settings:

```json
[
  {
    "name": "Synthetic answer judge",
    "slug": "answer-judge",
    "function_type": "scorer",
    "function_data": {
      "type": "prompt"
    },
    "prompt_data": {
      "prompt": {
        "type": "chat",
        "messages": [
          {
            "role": "system",
            "content": "Choose correct if the answer matches the reference, otherwise incorrect."
          },
          {
            "role": "user",
            "content": "Task: {{input}}\nAnswer: {{output}}\nReference: {{expected}}"
          }
        ]
      },
      "options": {
        "params": {
          "temperature": 0
        }
      },
      "parser": {
        "type": "llm_classifier",
        "use_cot": true,
        "choice_scores": {
          "correct": 1,
          "incorrect": 0
        },
        "allow_skip": false
      }
    }
  }
]
```

## Test the import

You need Python 3.10+, a Braintrust API key, and access to a project. Configure [model providers in Braintrust](https://docs.braintrust.dev/admin/ai-providers) for prompt tasks or LLM judges. No Braintrust SDK or CLI is required for the import.

[Download import\_eval.py](https://www.evalmarket.ai/import-guide/import_eval.py)

Put the helper beside your `braintrust/` folder and run it from that directory:

**Linux, macOS, or WSL**

```sh
python3 import_eval.py braintrust
```

**Windows**

```sh
python import_eval.py braintrust
```

1. Enter your key at the masked prompt, or set `BRAINTRUST_API_KEY` locally. The helper sends it directly to Braintrust; EvalMarket does not receive it.
2. Select your project and a model/provider for each prompt task or LLM scorer.
3. Review and confirm. The helper validates the package, creates the dataset and functions, then checks the uploaded content by reading it back. Inspect the printed resource IDs in Braintrust.

> **If an import fails:** inspect the reported project, resource names, and IDs before retrying. A partial upload may have succeeded. Each invocation creates new resources; the helper does not resume or clean up a previous attempt.

### Model selection, scripted imports, and older packages

The picker uses models discovered from your account; listing a model does not verify inference access. If a model is missing, check provider configuration and API-key access in Braintrust. Choose models compatible with your prompts and parameters.

For unattended imports, set the environment key and pass `--project-id PROJECT_UUID --yes`, plus `--model function-slug=provider/model-name` for every prompt task and LLM scorer. If ambiguous, add `--provider function-slug=provider-configuration-name`. Choices must match discovered account options. `--yes` confirms the import only; it never starts an experiment.

The helper uses `https://api.braintrust.dev`. Custom endpoints, EU-specific endpoints, and BYOC are not currently supported.

Older packages without tasks need an updated seller package. Legacy model/provider settings are discarded in memory so the importer can choose their own; new submissions must omit them. Source files remain unchanged.

## Verify tasks and scorers

**A successful import does not test behavior.** Run every included task and scorer on representative cases. Check known correct and incorrect outputs against your rubric, and resolve runtime or model-access errors before treating scores as verified results.

After an interactive import, choose whether to start an experiment. Select a task if prompted; it runs with the imported dataset and scorers. Runs may incur model charges. The helper returns a link without waiting for results, so open it and check that the run finished successfully.

To run a prompt task later, follow Braintrust’s [experiment setup guide](https://docs.braintrust.dev/evaluate/run-in-ui). Inline-code tasks are saved functions and are not listed in the UI task picker; provide tested [API/SDK run instructions](https://docs.braintrust.dev/evaluate/run-in-code) for buyers. See [how to interpret results](https://docs.braintrust.dev/evaluate/interpret-results) when reviewing scores.

If experiment start is unconfirmed, inspect the named experiment before retrying. Your verified import remains complete.

## Before you list

Record your tested procedure, runtime or model/provider assumptions, results, and verification date in `braintrust/README.md`. Identify anything untested so buyers know what to expect.

Submit the ZIP through [EvalMarket](https://www.evalmarket.ai/seller) and check the Braintrust Easy Import result. This checks required files and structure; it does not certify task behavior, code safety, or evaluation quality.
