DEV Community

Cover image for How to fine-tune a small model on your Langfuse or LangSmith traces
Tyler Edwards
Tyler Edwards

Posted on Originally published at overmindlab.ai

How to fine-tune a small model on your Langfuse or LangSmith traces

If your agent logs to Langfuse or LangSmith, you already have a record of it doing its job thousands of times. Each record is called a trace, and it holds the input, every model and tool call along the way, and the output. Those traces can train a small model you own to do one of the agent's jobs, and a small model is usually cheaper and faster to run than the large one the agent calls today. Training a model further on examples like these is called fine-tuning.

Originally published at overmindlab.ai.

It works like training a new hire from a file of past cases. You pick the cases that were handled well, black out the customer names, and test the new hire on cases they haven't seen. Exporting the traces takes a short script, and most of the time goes into the same three jobs. You decide which traces to keep, clean them, and test the trained model in a way that can't flatter it.

All of this assumes the traces already exist. If you're still working out what to capture or how to label it, read how to turn traces into a training dataset first.

Pick one step of the agent to train

A single agent run usually contains several model calls that do different jobs. One call might classify the request and a later one write the reply. Train a model for one of those calls at a time. A single step has a defined input and output, and you can tell whether it succeeded, so the dataset stays small and the trained model is easy to measure and swap in.

Start with the step that runs most often, or the one where a mistake costs the most. In your tracer it appears as an observation or run with a stable name, something like classify_ticket or extract_invoice, and the export scripts below filter on that name.

Which traces are worth training on?

Supervised fine-tuning teaches a model to copy its examples, and it copies the mistakes as faithfully as everything else. So for each trace you're asking whether you'd want that output repeated. Your tracer usually holds four signals that help, from weakest to strongest.

  1. The call returned valid output without an error or a retry. You need this, but a confident wrong answer passes it too.
  2. An automated check (an eval) scored the output, whether that was another model acting as judge or a piece of code. This is useful at volume once you have compared a sample of its verdicts with a person's.
  3. The user gave a thumbs up, or the ticket closed without escalation.
  4. Someone corrected the output. The corrected version is the best training target you can get, because it is exactly what you wanted.

Keep the traces that pass the signals you trust. Put failures in your test set unless a person corrected them, in which case train on the corrected version.

Before you train on any output, check who wrote it. If a GPT or Claude model wrote it, the vendor's terms apply. OpenAI's usage policies, effective November 20, 2026, bar using its outputs "to generate, prepare, select, or supply training material or signals for models that are intended as alternatives to or substitutes for OpenAI models or services." Anthropic's usage policy, effective November 12, 2026, prohibits using outputs to train an AI model without Anthropic's prior authorization. OpenAI's wording includes "select", so a GPT model that judges which traces to keep may fall under it too. Outputs your team wrote are yours. A GPT or Claude output that someone edited may still count as the vendor's. If you're unsure, ask your legal team before you train. Teams leaving OpenAI fine-tuning face the same question, and the migration guide covers what they can take with them.

Export the traces

Each model call you export holds the messages your agent sent and the reply it got back. Each one becomes a chat row whose final assistant message is the accepted or corrected output.

Both tools changed their read APIs in 2026, so any tutorial that calls langfuse.api.trace.list or client.list_runs is out of date. Check your retention window before you do anything else. LangSmith's base tier keeps traces for 14 days and Langfuse's Hobby plan shows 30 days of history, so older traces drop out while you wait.

From Langfuse

Langfuse stores each trace as a tree of observations, and its Observations API returns them a page at a time. It can't filter observations by score, so you fetch the scores you trust from the Scores API first and then keep the model calls that belong to those traces. Human corrections are stored as scores with the data type CORRECTION.

export_langfuse.py

from datetime import datetime, timedelta, timezone

from langfuse import get_client  # langfuse>=5, Python 3.11+

# Reads LANGFUSE_PUBLIC_KEY and LANGFUSE_SECRET_KEY.
langfuse = get_client()
end = datetime.now(timezone.utc)
start = end - timedelta(days=30)

# 1. Traces that got positive user feedback (100 scores a page at most).
good_traces, cursor = set(), None
while True:
    page = langfuse.api.scores.get_many(
        name="user-feedback", data_type="BOOLEAN",
        value="true",  # thumbs up; NUMERIC and "1" if you log numbers
        fields="subject", from_timestamp=start, to_timestamp=end,
        limit=100, cursor=cursor,
    )
    for score in page.data:
        if score.subject and score.subject.kind == "trace":
            good_traces.add(score.subject.id)
        elif score.subject and score.subject.kind == "observation":
            good_traces.add(score.subject.trace_id)
    cursor = page.meta.cursor
    if not cursor:
        break

# 2. The model calls for one step, kept only from those traces.
calls, cursor = [], None
while True:
    page = langfuse.api.observations.get_many(
        type="GENERATION", name="classify_ticket",
        from_start_time=start, to_start_time=end,
        fields="core,basic,io", limit=500, cursor=cursor,
    )
    calls += [obs for obs in page.data if obs.trace_id in good_traces]
    cursor = page.meta.cursor
    if not cursor:
        break

print(len(calls), "calls from", len(good_traces), "traces")
Enter fullscreen mode Exit fullscreen mode

input and output come back as raw strings, so run json.loads on them before you build chat rows. If you'd rather not write code at all, you can filter a Langfuse table by score in the UI and export it as CSV or JSON, and Enterprise, self-hosted and Pro (with the Teams add-on) plans can schedule exports to S3, Google Cloud Storage or Azure. Langfuse doesn't train models itself. Its Fireworks guide curates a dataset by score and converts it to chat JSONL so you can train on Fireworks.

From LangSmith

In LangSmith, runs.query replaces list_runs, which LangSmith Cloud removes on January 31, 2027. Its filter language can match on feedback directly, and the trace_filter argument keeps only the model calls from traces whose root run got a thumbs up.

export_langsmith.py

import asyncio
from datetime import datetime, timedelta, timezone

from langsmith import Client  # langsmith>=0.10.15


async def export_calls():
    client = Client()  # reads LANGSMITH_API_KEY
    project = client.read_project(project_name="support-agent")
    calls = []
    async for run in client.runs.query(
        project_ids=[str(project.id)],
        # Without this, runs.query only looks back one day.
        min_start_time=datetime.now(timezone.utc) - timedelta(days=14),
        run_type="LLM",
        filter='eq(name, "classify_ticket")',
        trace_filter=(
            'and(eq(feedback_key, "user_score"), '
            "eq(feedback_score, 1))"
        ),
        selects=["ID", "TRACE_ID", "START_TIME", "INPUTS", "OUTPUTS"],
    ):
        calls.append(run)
    return calls


calls = asyncio.run(export_calls())
print(len(calls), "calls")
Enter fullscreen mode Exit fullscreen mode

If you've already curated a LangSmith dataset, you can download it in OpenAI's fine-tuning format from the UI or with client.read_dataset_openai_finetuning(dataset_name=...). Bulk export of whole projects to S3 as Parquet needs the Enterprise plan for accounts created after August 3, 2026. Unlike Langfuse, LangSmith does offer fine-tuning. Its smithtune command-line tool, in beta since September 2026, trains a LoRA model on Fireworks or Baseten with your own provider key and compares the base and tuned models in a LangSmith experiment.

Clean the data before you train

Duplicates are the first problem. Production traffic repeats itself, and one template that runs thousands of times a day can swamp everything else in the dataset. Drop exact duplicates by hashing the normalized input, then go after near-duplicates, such as the same request with a different order ID. In a study of language model training data, Lee et al. found that after deduplication, models repeated memorized text ten times less often and reached the same or better accuracy in fewer training steps.

A language model can repeat text from its training data word for word (Carlini et al., 2021), so names, emails and account numbers in your traces should never reach the training set. Both tracers can mask data before it leaves your app, which means new traces arrive clean. Langfuse's Python SDK accepts a mask_otel_spans function, and LangSmith's client accepts an anonymizer or can hide inputs and outputs entirely. That doesn't help with the traces you already have. For those, run a scrubber over the inputs and outputs before the data leaves your systems. The open-source Presidio library finds personal data such as names and email addresses and replaces it with placeholders.

scrub.py

from presidio_analyzer import AnalyzerEngine
from presidio_anonymizer import AnonymizerEngine

analyzer, anonymizer = AnalyzerEngine(), AnonymizerEngine()

def scrub(text: str) -> str:
    found = analyzer.analyze(text=text, language="en")
    return anonymizer.anonymize(text=text, analyzer_results=found).text

print(scrub("Refund order 4417 for Jane Smith, [email protected]"))
# Refund order <DATE_TIME> for <PERSON>, <EMAIL_ADDRESS>
Enter fullscreen mode Exit fullscreen mode

With placeholders like these, the model still learns the shape of a request but never sees the real values. Check a sample by hand, though, because detectors get it wrong in both directions. In the output above, Presidio took the order number for a date, and it can just as easily miss an internal account number.

The labels need a check as well. If your targets come from outcomes and nobody edited them by hand, pull 50 at random and read them against a written definition of a correct output. If more than a few are wrong, fix the rule that produced them before you train.

Which training method do your traces support?

Supervised fine-tuning is the place to begin. It needs the simplest data, and it is the baseline any other method has to beat. Whether the other two methods are open to you depends on what your traces contain.

Method What each example needs Where it comes from in traces Use it when
Supervised fine-tuning (SFT) An input and one good output Accepted or corrected outputs You can say what the right output is
Preference training (DPO) An input, a better output and a worse one Edits (edited version vs original), regenerations, A/B picks You can tell better from worse but rarely have a perfect answer
Reinforcement learning (RL) An input and a scoring function Inputs from traces; the score comes from code you write Correctness can be checked by code and SFT has stalled

Choosing a training method from what your traces hold

DPO trains on pairs of answers to the same prompt, one preferred and one not. GRPO, introduced in the DeepSeekMath paper, samples several answers to each prompt, scores each one and rewards those that beat the group's average, so it needs a scoring function and no written-out target for each prompt. The guide to types of fine-tuning covers supervised and reinforcement fine-tuning in more depth.

How many examples do you need?

No single number holds for every task. OpenAI's fine-tuning guide sets a minimum of 10 examples and recommends starting with 50 well-crafted ones. The research favors quality over volume. LIMA tuned a 65B model's chat style with 1,000 curated examples, and AlpaGasus beat a model trained on 52,000 instructions with one trained on 9,000 filtered ones. Across the 31 tasks in LoRA Land, the size of the training set did not predict how much fine-tuning helped.

You can measure this on your own data. Train on a quarter of your examples, then half, then all of them, and score each model on the same test set. If the score is still climbing at full size, more data will help. If it flattened early, more data won't fix it, and the place to look is the quality of the data or the definition of the task.

Keep the test set out of the training data

A model that has already seen the test questions scores better than it deserves. With traces that happens easily, because the same user, session or template turns up again and again.

  • Split by session or user. If you split by row, steps from one conversation can land on both sides.
  • Split by time when you can. Train on older traces and test on the newest week, which matches real use, where the model only sees new traffic.
  • Remove near-duplicates across the split as well as inside the training set.
  • Freeze the test set and tune settings on a separate validation slice. Each look at the test set while you iterate leaks a little of it into your choices.

Exact-match checks are not enough on their own. Yang et al. showed that paraphrased test questions slip past n-gram decontamination, and that a 13B model trained on them can match GPT-4 on the benchmark it has effectively seen.

This script splits by time, then drops any training row whose session also appears in the test set.

split.py

import json

rows = [json.loads(line) for line in open("clean.jsonl")]
# Assumes ISO 8601 timestamps in UTC, which sort correctly as text.
cutoff = "2026-11-01"  # the newest traffic becomes the test set

train = [r for r in rows if r["timestamp"] < cutoff]
test = [r for r in rows if r["timestamp"] >= cutoff]

# Drop any training row from a session that also appears in the test set.
test_sessions = {r["session_id"] for r in test}
train = [r for r in train if r["session_id"] not in test_sessions]

print(len(train), "train,", len(test), "test")
Enter fullscreen mode Exit fullscreen mode

Measure the result against what you run today

Score three models on the same frozen test set, with the same checks. The first is the base model before training, which shows you what training added. The second is the model your agent calls today, which sets the bar. Your trained model has to beat the base and at least match the model you run today.

Which check to use depends on the step. Exact match or F1 works for labels and extracted fields. JSON output needs schema validation, and for free text you can use an LLM judge with a short yes-or-no rubric. Break the score down by segment too, because an average can hide a rare case that got worse. Before you switch, run the trained model on live traffic in shadow mode, where it answers copies of real requests while users still see the current model.

The same work in Overmind

Overmind runs this work in one place, and your coding agent can drive each step.

  1. Connect Langfuse, LangSmith, Braintrust or Galileo on the Integrations page with the provider's API keys, and choose how far back to import. The traces land in Observability, and Overmind can keep polling for new ones. The import brings the traces but leaves behind the scores and feedback attached to them (Braintrust scores arrive only as raw metadata), so filter on those in your tracer or recreate the checks as Overmind evaluators. Python agents can also send traces with the Overmind SDK, and any other stack can send them over OpenTelemetry.
  2. In Observability, select traces and click Add to dataset. Each trace becomes one row. When you split the rows into training and evaluation sets, Overmind removes exact duplicates and keeps rows with the same input, trace or conversation on one side. If training rows still overlap your evaluation set, you get a warning when you train.
  3. Clean the data in the Data Workshop. Your coding agent writes each cleaning step as a small Python script, which Overmind runs in an isolated sandbox and versions. Overmind does not detect personal data for you, so scrub it before import or in one of these steps.
  4. Write evaluators as code checks or model judges. The ones you enable score imported traces as they arrive.
  5. Train the model. Overmind trains with supervised fine-tuning, either LoRA or, on selected models up to 14B, full fine-tuning. It can score the trained model against its untouched base on your evaluation set, and against the model you use today if OpenRouter serves it.
  6. Call the model with the OpenAI SDK at https://api.overmindlab.ai/api/v1, or download the weights and run them yourself.

If your traces are already in Langfuse or LangSmith, your coding agent can set this up for you. Open your agent's repo in Claude Code, Cursor, Codex or OpenCode and paste Set me up on Overmind https://console.overmindlab.ai/setup.md into the chat. You can also start for free and connect your tracer from the console.

FAQ

Can I fine-tune a model directly on my Langfuse traces?

Langfuse records and evaluates traces but doesn't train models, so the training has to happen somewhere else. Export the traces, filter and clean them, then train on them with a fine-tuning tool. Overmind can import them straight from Langfuse.

Does LangSmith fine-tune models?

Yes. LangSmith Fine-Tuning, in public beta since September 2026, uses the smithtune command-line tool to run supervised LoRA fine-tuning on Fireworks or Baseten with your own provider key. It then compares the base and tuned models in a LangSmith experiment, and can deploy the tuned model.

How many traces do I need to fine-tune a model?

Fewer than you might expect, if they're clean. OpenAI's guide recommends starting with 50 well-crafted examples. In research, LIMA tuned a model's chat style with 1,000 curated examples, and AlpaGasus beat a 52,000-example dataset with 9,000 filtered ones. To find where your own curve flattens, train on growing slices of your data.

Should I train on traces where the agent failed?

Only once someone has corrected them. Supervised fine-tuning copies its examples, so a failed run teaches the model to fail the same way. Put uncorrected failures in your test set.

Can I train on traces where GPT or Claude wrote the output?

Check the vendor's terms first. OpenAI's usage policies from November 20, 2026 and Anthropic's from November 12, 2026 restrict using their outputs to train other models. Outputs your team wrote are yours, and model outputs that someone edited are a grey area. Ask your legal team if you're unsure.

Overmind is the model training platform for AI teams. Start for free.

Top comments (0)