Do you already have a working eval? Copy the instructions below to have an agent create your LangSmith files for you.

Prepare the files with your LLM

Share these instructions and your finished evaluation with your LLM, then test the generated package yourself.

View the full instructions

Prepare your package

Add a lowercase langchain/ folder at the root of your evaluation ZIP with these files:

FileWhat to include
README.mdSetup, dependencies, expected application outputs, metric meanings, and your tested import and run instructions.
dataset.jsonA nonempty array of examples with an inputs object and optional reference answers in an outputs object.
evaluator.jsonOne evaluator’s name and judge_type: deterministic, code, or llm. Code judges also declare file and function.

Add the files for your judge type:

Keep your complete task implementation and setup alongside these files. The helper imports the dataset and one evaluator; buyers still need your instructions to run the task. Keep credentials out of the ZIP.

Package limits and dataset example

The langchain/ folder and evaluator.json are EvalMarket’s packaging convention; the dataset and judge files use LangSmith’s native formats. All checked files must be nonempty UTF-8 and at most 5 MiB each. Existing ZIP limits apply. JSON comments, duplicate keys, and non-finite numbers are rejected.

Package structure
your-eval.zip
├── your existing files and instructions
└── langchain/
    ├── README.md
    ├── dataset.json
    ├── evaluator.json
    └── judge files listed below

Dataset — save as dataset.json. Keep reference answers in outputs, separate from application inputs:

JSON
[
  {
    "inputs": {
      "question": "What is 2 + 2?"
    },
    "outputs": {
      "answer": "4"
    }
  }
]

Additional files stay in your ZIP but are not interpreted by the Easy Import check. Include any dependencies and task setup buyers need.

Deterministic checks or custom code

Save the evaluator metadata as evaluator.json. Use deterministic for fixed checks or code for other custom code; both upload your native evaluator file.

JSON
{
  "name": "Answer correctness",
  "judge_type": "deterministic",
  "file": "evaluator.py",
  "function": "perform_eval"
}

For this CLI upload path, use two positional arguments, run and example, and return a dictionary mapping metric names to scores. Read application data from run["inputs"] and run["outputs"], and references from example["outputs"].

Python example — save as evaluator.py, replacing the example logic with your tested grading rules:

Python
def perform_eval(run, example):
    actual = run["outputs"].get("answer")
    reference = example["outputs"].get("answer")
    return {"correct": actual == reference}

Declare the exact filename and function name in evaluator.json. Check supported runtimes and dependencies in the code evaluator documentation; arbitrary local scripts may need changes. The file check does not execute code or verify its behavior.

LLM-as-judge files and examples

Save this metadata as evaluator.json:

JSON
{
  "name": "Answer correctness",
  "judge_type": "llm"
}
  • prompt.json: your full grading instructions as a nonempty array of [role, content] string pairs or objects with nonempty string role and content fields. The helper uses Mustache templates.
  • rubric.json: a JSON feedback schema with type: "object" and nonempty properties containing schema objects. A native wrapper with the schema under parameters is also accepted.
  • model.json: a nonempty serialized configuration from your working LangSmith setup, without credentials. This is your tested reference; the default import uses the buyer’s saved model configuration instead.
  • variables.json: prompt variable names mapped to nonempty LangSmith field-path strings. Use {} only if you have verified the default mappings.

Prompt — save as prompt.json, replacing this example with your rubric:

JSON
[
  [
    "system",
    "Judge whether the answer correctly answers the question using the reference."
  ],
  [
    "user",
    "Question: {{question}}\nAnswer: {{actual}}\nReference: {{expected}}"
  ]
]

Feedback schema — save as rubric.json:

JSON
{
  "type": "object",
  "properties": {
    "correctness": {
      "type": "boolean",
      "description": "True when the answer is correct according to the grading instructions."
    }
  },
  "required": [
    "correctness"
  ]
}

Variable mappings — save as variables.json. Here, output.answer is the application’s answer and reference.answer is the dataset’s reference answer:

JSON
{
  "question": "input.question",
  "actual": "output.answer",
  "expected": "reference.answer"
}

EvalMarket checks these JSON shapes, not full schema validity, variable resolution, model access, or grading quality. Test your prompt, mappings, and scores using LangSmith’s LLM-as-judge configuration guide.

Test the import

Install Python 3.10+ and the official LangSmith CLI. You need access to a LangSmith workspace and, for LLM judges, a saved model enabled for evaluators.

Download import_eval.py

Put the helper beside your langchain/ folder and run it from that directory:

Linux, macOS, or WSL

Terminal
python3 import_eval.py langchain

Windows

Terminal
python import_eval.py langchain
  1. Use your current CLI login, or follow the browser sign-in and workspace selection when prompted.
  2. For an LLM judge, select a saved model. If none are available, configure one in Settings → Model configurations and rerun. Selection happens before any upload.
  3. Open the new dataset in LangSmith and inspect its examples and evaluator. The helper names the dataset using the evaluator’s name and a timestamp.
Model selection and command options

For an LLM judge, enter the number beside a saved model when prompted:

Example
Which model would you like to configure as your judge?

The helper uses that saved configuration and its provider access. Buyers do not need to create or edit a model file. Keep authentication in the CLI, local environment, or LangSmith.

Use --profile work to choose an existing CLI profile, or --dataset "My eval" to set the dataset name. To test your packaged model configuration or script an import, pass --model-config ./langchain/model.json to skip the picker. Neither profile nor dataset overrides are required for the normal flow.

The integration’s commands were checked against LangSmith CLI 0.4.0. Record the CLI and SDK versions you actually test in your README.

Verify your evaluator

The import runs no evaluation and sets evaluator sampling to zero. Follow your packaged task instructions to generate application outputs, then enable grading when ready in LangSmith. See evaluators on experiments and running an evaluation for the platform steps.

Test representative correct and incorrect outputs. Confirm that input, output, and reference mappings resolve correctly and that the resulting metrics match your rubric. Resolve runtime or model-access errors before treating scores as verified. Model calls may incur charges.

Inspect the completed run using LangSmith’s experiment results guide. Import success alone does not verify grading behavior.

Before you list

Record the tested procedure, CLI/SDK versions, dependencies, model assumptions, results, and verification date in langchain/README.md. Identify anything incomplete or untested.

Check your ZIP on EvalMarket and fix any missing or malformed files. Easy Import checks required files and basic formatting; it does not run code, call models, or certify compatibility.