# LangSmith for sellers

Prepare your LangSmith package, test it in your own workspace, and verify your evaluator before listing on EvalMarket.

Source: https://docs.evalmarket.ai/platform-integrations/langsmith/

Do you already have a working eval? Copy the instructions below to have an agent create your LangSmith files for you.

## Prepare the files with your LLM

Share these instructions and your finished evaluation with your LLM, then test the generated package yourself.

### View the full instructions

```text
Help me prepare a LangChain-compatible version of my finished evaluation for EvalMarket, using native LangSmith dataset and evaluator files.

1. Inspect my existing eval.
Ask me to provide the finished eval files and grading instructions if you cannot access them. Identify the packaged task implementation and setup, inputs, expected application outputs, reference answers, grading rules, dependencies, and current judge. Preserve the original task setup, files and grading meaning. Every eval must include its tasks and setup; buyers configure required access rather than authoring or connecting a replacement task. Do not invent task logic, test cases, reference answers, thresholds, or verification results. Ask about missing information or changes that would alter what the eval measures.

2. Choose the judge type with me.
Use "deterministic" for fixed checks, "llm" for LLM-as-judge, or "code" for other custom code. Both deterministic and custom-code judges use seller-supplied native code evaluators. The package describes one evaluator; explain any limitations if my eval needs multiple judges or unsupported features instead of silently dropping them.

3. Add a lowercase langchain/ folder at the root of my existing ZIP.
Keep the original eval alongside it. Every package needs these nonempty UTF-8 files, each no larger than 5 MiB:
- README.md: setup and import instructions, prerequisites, tested CLI/SDK versions, expected application output fields, metric meanings, dependencies, buyer model/credential setup, and actual verification status/date.
- dataset.json: a nonempty JSON array of examples. Each example has an "inputs" object and optionally an "outputs" object containing reference answers. Preserve the existing dataset and map its fields explicitly.
- evaluator.json: an object with a nonempty "name" and a "judge_type" of "deterministic", "llm", or "code". Code judges also declare "file" and "function".
Use valid JSON without comments, duplicate keys, or non-finite numbers. Do not put credentials or API keys in the package.

4. For a deterministic or custom-code judge:
Create evaluator.py, evaluator.js, or evaluator.ts inside langchain/ using the native LangSmith code evaluator format. Put the existing assertions and grading logic in that file, preserving their behavior. Declare the exact filename and entrypoint in evaluator.json, for example:
{"name":"Answer correctness","judge_type":"deterministic","file":"evaluator.py","function":"perform_eval"}
For the CLI's native code evaluator upload, use exactly two positional arguments: perform_eval(run, example). Read inputs and application outputs from run["inputs"] and run["outputs"], and reference answers from example["outputs"]. Return a dictionary mapping metric names to scores, such as {"correct": true}; do not use the SDK evaluator signature with separate inputs, outputs, and reference_outputs arguments or the SDK's key/score return wrapper. Check the supported runtime, dependencies, and feedback format in the official documentation: https://docs.langchain.com/langsmith/code-evaluator-ui . Do not assume arbitrary local scripts or dependencies work in LangSmith.

5. For an LLM-as-judge evaluator:
Use evaluator.json such as {"name":"Answer correctness","judge_type":"llm"} and include all four files:
- prompt.json: a nonempty array of [role, content] string pairs, or objects with string "role" and "content" fields. Preserve my full grading instructions and intended use of references.
- rubric.json: the native JSON output schema for feedback, with "type":"object" and nonempty "properties" whose values are schema objects. A native LangSmith wrapper with that schema under "parameters" is also supported. Describe the existing metrics and score meanings faithfully; do not substitute a prose rubric or invent new aggregation or thresholds.
- model.json: a nonempty native serialized model configuration from my working LangSmith setup, without credential values, included as the seller's tested reference. Ask me for my provider/model and a suitable exported configuration if needed; do not invent a configuration or require OpenAI. Buyers normally choose a saved model through the helper and do not create or edit this file.
- variables.json: an object mapping prompt variable names to LangSmith field-path strings. For inputs.question, an application output answer, and reference outputs.answer, an example is {"question":"input.question","actual":"output.answer","expected":"reference.answer"}. Match the actual fields and prompt variables; use {} only if defaults are appropriate and verified.
Consult the native prompt and feedback documentation: https://docs.langchain.com/langsmith/llm-as-judge .

6. Deliver the files and explain the mapping.
Create the actual langchain/ files when you have filesystem access, or provide each complete file labeled with its path. Explain how the original cases, rules, and scores map to them. Identify unresolved decisions and any missing model configuration. Clearly mark incomplete or untested work; do not present placeholders as a verified package.

7. Help me verify with the same helper buyers use.
Use the standard import_eval.py helper available from EvalMarket's Docs page: https://www.evalmarket.ai/seller/import-guide . It uses the official LangSmith CLI (commands checked against version 0.4.0). With Python 3.10+ and the CLI installed, place the helper beside the extracted langchain/ folder and run:
Linux/macOS/WSL: python3 import_eval.py langchain
Windows: python import_eval.py langchain
The helper uses the current LangSmith login or starts the CLI's browser login and workspace selection when needed. It automatically names the new dataset using the eval's name and a timestamp. Buyers do not need to supply --profile or --dataset; those are optional overrides for people who already manage multiple logins or want a specific dataset name.
For an LLM judge, the helper lists saved LangSmith models enabled for evaluators and asks "Which model would you like to configure as your judge?" Select a number; the helper uses that saved configuration and provider access without asking buyers to create any files. If no models are available, add or enable one in LangSmith Settings > Model configurations and rerun. Selection happens before any upload. An optional --model-config ./langchain/model.json flag uses an existing native file instead, for seller verification or scripted imports. Keep authentication in the CLI/local environment or LangSmith. Each completed import creates a new dataset and installs the evaluator with sampling zero; it does not run an evaluation. Inspect partial resources before retrying a failed import.
Guide me through checking the imported dataset/evaluator and separately testing representative application outputs in LangSmith. Record what was actually verified in README.md. Do not claim a successful import or grading test unless it was observed.

8. Package and explain the final check.
Include langchain/ in the original ZIP and have me select it on EvalMarket's seller ZIP submission form. Easy Import checks verifies required files and basic formatting only. It does not convert files, execute code, call models, or certify importability. I am responsible for verifying that my evaluator imports and works in LangSmith before listing it.
```

## Prepare your package

Add a lowercase `langchain/` folder at the root of your evaluation ZIP with these files:

| File | What to include |
| --- | --- |
| `README.md` | Setup, dependencies, expected application outputs, metric meanings, and your tested import and run instructions. |
| `dataset.json` | A nonempty array of examples with an `inputs` object and optional reference answers in an `outputs` object. |
| `evaluator.json` | One evaluator’s `name` and `judge_type`: `deterministic`, `code`, or `llm`. Code judges also declare `file` and `function`. |

Add the files for your judge type:

- **Deterministic or custom code:** `evaluator.py`, `evaluator.js`, or `evaluator.ts`, using LangSmith’s [native code evaluator format](https://docs.langchain.com/langsmith/code-evaluator-ui).
- **LLM-as-judge:** `prompt.json`, `rubric.json`, `model.json`, and `variables.json`. See LangSmith’s [prompt and feedback guide](https://docs.langchain.com/langsmith/llm-as-judge).

Keep your complete task implementation and setup alongside these files. The helper imports the dataset and one evaluator; buyers still need your instructions to run the task. Keep credentials out of the ZIP.

### Package limits and dataset example

The `langchain/` folder and `evaluator.json` are EvalMarket’s packaging convention; the dataset and judge files use LangSmith’s native formats. All checked files must be nonempty UTF-8 and at most 5 MiB each. Existing ZIP limits apply. JSON comments, duplicate keys, and non-finite numbers are rejected.

```text
your-eval.zip
├── your existing files and instructions
└── langchain/
    ├── README.md
    ├── dataset.json
    ├── evaluator.json
    └── judge files listed below
```

**Dataset** — save as `dataset.json`. Keep reference answers in `outputs`, separate from application inputs:

```json
[
  {
    "inputs": {
      "question": "What is 2 + 2?"
    },
    "outputs": {
      "answer": "4"
    }
  }
]
```

Additional files stay in your ZIP but are not interpreted by the Easy Import check. Include any dependencies and task setup buyers need.

### Deterministic checks or custom code

Save the evaluator metadata as `evaluator.json`. Use `deterministic` for fixed checks or `code` for other custom code; both upload your native evaluator file.

```json
{
  "name": "Answer correctness",
  "judge_type": "deterministic",
  "file": "evaluator.py",
  "function": "perform_eval"
}
```

For this CLI upload path, use two positional arguments, `run` and `example`, and return a dictionary mapping metric names to scores. Read application data from `run["inputs"]` and `run["outputs"]`, and references from `example["outputs"]`.

**Python example** — save as `evaluator.py`, replacing the example logic with your tested grading rules:

```python
def perform_eval(run, example):
    actual = run["outputs"].get("answer")
    reference = example["outputs"].get("answer")
    return {"correct": actual == reference}
```

Declare the exact filename and function name in `evaluator.json`. Check supported runtimes and dependencies in the [code evaluator documentation](https://docs.langchain.com/langsmith/code-evaluator-ui); arbitrary local scripts may need changes. The file check does not execute code or verify its behavior.

### LLM-as-judge files and examples

Save this metadata as `evaluator.json`:

```json
{
  "name": "Answer correctness",
  "judge_type": "llm"
}
```

- **`prompt.json`:** your full grading instructions as a nonempty array of `[role, content]` string pairs or objects with nonempty string `role` and `content` fields. The helper uses Mustache templates.
- **`rubric.json`:** a JSON feedback schema with `type: "object"` and nonempty `properties` containing schema objects. A native wrapper with the schema under `parameters` is also accepted.
- **`model.json`:** a nonempty serialized configuration from your working LangSmith setup, without credentials. This is your tested reference; the default import uses the buyer’s saved model configuration instead.
- **`variables.json`:** prompt variable names mapped to nonempty LangSmith field-path strings. Use `{}` only if you have verified the default mappings.

**Prompt** — save as `prompt.json`, replacing this example with your rubric:

```json
[
  [
    "system",
    "Judge whether the answer correctly answers the question using the reference."
  ],
  [
    "user",
    "Question: {{question}}\nAnswer: {{actual}}\nReference: {{expected}}"
  ]
]
```

**Feedback schema** — save as `rubric.json`:

```json
{
  "type": "object",
  "properties": {
    "correctness": {
      "type": "boolean",
      "description": "True when the answer is correct according to the grading instructions."
    }
  },
  "required": [
    "correctness"
  ]
}
```

**Variable mappings** — save as `variables.json`. Here, `output.answer` is the application’s answer and `reference.answer` is the dataset’s reference answer:

```json
{
  "question": "input.question",
  "actual": "output.answer",
  "expected": "reference.answer"
}
```

EvalMarket checks these JSON shapes, not full schema validity, variable resolution, model access, or grading quality. Test your prompt, mappings, and scores using LangSmith’s [LLM-as-judge configuration guide](https://docs.langchain.com/langsmith/llm-as-judge).

## Test the import

Install Python 3.10+ and the [official LangSmith CLI](https://docs.langchain.com/langsmith/langsmith-cli). You need access to a LangSmith workspace and, for LLM judges, a saved model enabled for evaluators.

[Download import\_eval.py](https://www.evalmarket.ai/import-guide/import_eval.py)

Put the helper beside your `langchain/` folder and run it from that directory:

**Linux, macOS, or WSL**

```sh
python3 import_eval.py langchain
```

**Windows**

```sh
python import_eval.py langchain
```

1. Use your current CLI login, or follow the browser sign-in and workspace selection when prompted.
2. For an LLM judge, select a saved model. If none are available, configure one in [Settings → Model configurations](https://docs.langchain.com/langsmith/model-configurations) and rerun. Selection happens before any upload.
3. Open the new dataset in LangSmith and inspect its examples and evaluator. The helper names the dataset using the evaluator’s name and a timestamp.

> **If an import fails:** inspect the workspace before retrying. A partial upload may have succeeded. Each invocation creates a new dataset; the helper does not retry, replace, or clean up resources.

### Model selection and command options

For an LLM judge, enter the number beside a saved model when prompted:

```text
Which model would you like to configure as your judge?
```

The helper uses that saved configuration and its provider access. Buyers do not need to create or edit a model file. Keep authentication in the CLI, local environment, or LangSmith.

Use `--profile work` to choose an existing CLI profile, or `--dataset "My eval"` to set the dataset name. To test your packaged model configuration or script an import, pass `--model-config ./langchain/model.json` to skip the picker. Neither profile nor dataset overrides are required for the normal flow.

The integration’s commands were checked against LangSmith CLI 0.4.0. Record the CLI and SDK versions you actually test in your README.

## Verify your evaluator

**The import runs no evaluation and sets evaluator sampling to zero.** Follow your packaged task instructions to generate application outputs, then enable grading when ready in LangSmith. See [evaluators on experiments](https://docs.langchain.com/langsmith/bind-evaluator-to-dataset) and [running an evaluation](https://docs.langchain.com/langsmith/evaluate-llm-application) for the platform steps.

Test representative correct and incorrect outputs. Confirm that input, output, and reference mappings resolve correctly and that the resulting metrics match your rubric. Resolve runtime or model-access errors before treating scores as verified. Model calls may incur charges.

Inspect the completed run using LangSmith’s [experiment results guide](https://docs.langchain.com/langsmith/analyze-an-experiment). Import success alone does not verify grading behavior.

## Before you list

Record the tested procedure, CLI/SDK versions, dependencies, model assumptions, results, and verification date in `langchain/README.md`. Identify anything incomplete or untested.

[Check your ZIP on EvalMarket](https://www.evalmarket.ai/seller/submit-zip) and fix any missing or malformed files. Easy Import checks required files and basic formatting; it does not run code, call models, or certify compatibility.
