Do you already have a working eval? Copy the instructions below to have an agent create your LangSmith files for you.
Prepare the files with your LLM
Share these instructions and your finished evaluation with your LLM, then test the generated package yourself.
View the full instructions
Prepare your package
Add a lowercase langchain/ folder at the root of your evaluation ZIP with these files:
| File | What to include |
|---|---|
README.md | Setup, dependencies, expected application outputs, metric meanings, and your tested import and run instructions. |
dataset.json | A nonempty array of examples with an inputs object and optional reference answers in an outputs object. |
evaluator.json | One evaluator’s name and judge_type: deterministic, code, or llm. Code judges also declare file and function. |
Add the files for your judge type:
- Deterministic or custom code:
evaluator.py,evaluator.js, orevaluator.ts, using LangSmith’s native code evaluator format. - LLM-as-judge:
prompt.json,rubric.json,model.json, andvariables.json. See LangSmith’s prompt and feedback guide.
Keep your complete task implementation and setup alongside these files. The helper imports the dataset and one evaluator; buyers still need your instructions to run the task. Keep credentials out of the ZIP.
Package limits and dataset example
The langchain/ folder and evaluator.json are EvalMarket’s packaging convention; the dataset and judge files use LangSmith’s native formats. All checked files must be nonempty UTF-8 and at most 5 MiB each. Existing ZIP limits apply. JSON comments, duplicate keys, and non-finite numbers are rejected.
your-eval.zip
├── your existing files and instructions
└── langchain/
├── README.md
├── dataset.json
├── evaluator.json
└── judge files listed belowDataset — save as dataset.json. Keep reference answers in outputs, separate from application inputs:
[
{
"inputs": {
"question": "What is 2 + 2?"
},
"outputs": {
"answer": "4"
}
}
]Additional files stay in your ZIP but are not interpreted by the Easy Import check. Include any dependencies and task setup buyers need.
Deterministic checks or custom code
Save the evaluator metadata as evaluator.json. Use deterministic for fixed checks or code for other custom code; both upload your native evaluator file.
{
"name": "Answer correctness",
"judge_type": "deterministic",
"file": "evaluator.py",
"function": "perform_eval"
}For this CLI upload path, use two positional arguments, run and example, and return a dictionary mapping metric names to scores. Read application data from run["inputs"] and run["outputs"], and references from example["outputs"].
Python example — save as evaluator.py, replacing the example logic with your tested grading rules:
def perform_eval(run, example):
actual = run["outputs"].get("answer")
reference = example["outputs"].get("answer")
return {"correct": actual == reference}Declare the exact filename and function name in evaluator.json. Check supported runtimes and dependencies in the code evaluator documentation; arbitrary local scripts may need changes. The file check does not execute code or verify its behavior.
LLM-as-judge files and examples
Save this metadata as evaluator.json:
{
"name": "Answer correctness",
"judge_type": "llm"
}prompt.json: your full grading instructions as a nonempty array of[role, content]string pairs or objects with nonempty stringroleandcontentfields. The helper uses Mustache templates.rubric.json: a JSON feedback schema withtype: "object"and nonemptypropertiescontaining schema objects. A native wrapper with the schema underparametersis also accepted.model.json: a nonempty serialized configuration from your working LangSmith setup, without credentials. This is your tested reference; the default import uses the buyer’s saved model configuration instead.variables.json: prompt variable names mapped to nonempty LangSmith field-path strings. Use{}only if you have verified the default mappings.
Prompt — save as prompt.json, replacing this example with your rubric:
[
[
"system",
"Judge whether the answer correctly answers the question using the reference."
],
[
"user",
"Question: {{question}}\nAnswer: {{actual}}\nReference: {{expected}}"
]
]Feedback schema — save as rubric.json:
{
"type": "object",
"properties": {
"correctness": {
"type": "boolean",
"description": "True when the answer is correct according to the grading instructions."
}
},
"required": [
"correctness"
]
}Variable mappings — save as variables.json. Here, output.answer is the application’s answer and reference.answer is the dataset’s reference answer:
{
"question": "input.question",
"actual": "output.answer",
"expected": "reference.answer"
}EvalMarket checks these JSON shapes, not full schema validity, variable resolution, model access, or grading quality. Test your prompt, mappings, and scores using LangSmith’s LLM-as-judge configuration guide.
Test the import
Install Python 3.10+ and the official LangSmith CLI. You need access to a LangSmith workspace and, for LLM judges, a saved model enabled for evaluators.
Put the helper beside your langchain/ folder and run it from that directory:
Linux, macOS, or WSL
python3 import_eval.py langchainWindows
python import_eval.py langchain- Use your current CLI login, or follow the browser sign-in and workspace selection when prompted.
- For an LLM judge, select a saved model. If none are available, configure one in Settings → Model configurations and rerun. Selection happens before any upload.
- Open the new dataset in LangSmith and inspect its examples and evaluator. The helper names the dataset using the evaluator’s name and a timestamp.
Model selection and command options
For an LLM judge, enter the number beside a saved model when prompted:
Which model would you like to configure as your judge?The helper uses that saved configuration and its provider access. Buyers do not need to create or edit a model file. Keep authentication in the CLI, local environment, or LangSmith.
Use --profile work to choose an existing CLI profile, or --dataset "My eval" to set the dataset name. To test your packaged model configuration or script an import, pass --model-config ./langchain/model.json to skip the picker. Neither profile nor dataset overrides are required for the normal flow.
The integration’s commands were checked against LangSmith CLI 0.4.0. Record the CLI and SDK versions you actually test in your README.
Verify your evaluator
The import runs no evaluation and sets evaluator sampling to zero. Follow your packaged task instructions to generate application outputs, then enable grading when ready in LangSmith. See evaluators on experiments and running an evaluation for the platform steps.
Test representative correct and incorrect outputs. Confirm that input, output, and reference mappings resolve correctly and that the resulting metrics match your rubric. Resolve runtime or model-access errors before treating scores as verified. Model calls may incur charges.
Inspect the completed run using LangSmith’s experiment results guide. Import success alone does not verify grading behavior.
Before you list
Record the tested procedure, CLI/SDK versions, dependencies, model assumptions, results, and verification date in langchain/README.md. Identify anything incomplete or untested.
Check your ZIP on EvalMarket and fix any missing or malformed files. Easy Import checks required files and basic formatting; it does not run code, call models, or certify compatibility.