LMnetwork: automated prompt optimization for LangGraph agents

Table of Contents

An internal tool that treats prompts like weights. It runs a LangGraph agent over a golden dataset, scores every output with an LLM judge, then evolves the prompt text until the scores climb.

Highlights
#

Built LMnetwork, a prompt-optimization framework that tunes LangGraph agent prompts against golden datasets with an LLM-as-judge, lifting a production procedure-extraction agent from 0.51 to 0.87 and a classification agent from 0.60 to 0.90.
Designed an RL-style optimization loop that pits several competing edit strategies against each other every iteration and keeps the highest-scoring one, feeding past strategies and their scores back in so the search never settles on a single idea.
Solved the failure mode that breaks LLM-driven text edits (search strings that don’t match verbatim) with a fuzzy search/replace engine: exact single-match by default, SequenceMatcher sliding-window fallback when the model’s whitespace drifts.
Ran evaluation, strategy generation, and candidate scoring as LangGraph Send fan-outs, so a run scores all N strategies across M samples concurrently instead of walking them one at a time.

Architecture / How it works
#

LMnetwork: how prompt optimization works — The optimization loop end to end: test prompts, score with an LLM judge, try many strategies, keep the best, repeat until the target score.

flowchart TD
    A[Load JSONL dataset] --> B[Baseline eval: fan out every sample,
run target graph, LLM judge scores it]
    B --> C[Reduce: mean score, track best]
    C --> D{score ≥ target
or max iters?}
    D -- yes --> Z[Restore best prompts as active, complete run]
    D -- no --> E[Strategy generator LLM:
N diverse strategies, aware of past strategies + scores]
    E --> F[Fan out one optimizer worker per strategy]
    F --> G[Each worker proposes search/replace edits.
Fuzzy edit engine applies them into a candidate prompt set]
    G --> H[Fan out: score every candidate on the full dataset]
    H --> I[Tournament: pick highest-mean candidate as winner]
    I --> J[Persist winning prompt version + strategy history to SQLite]
    J --> B

The loop is a compiled LangGraph StateGraph. Every expensive step (evaluating samples, running optimizer workers, scoring candidates) is a Send fan-out, so a run with N strategies and M samples dispatches its N×M candidate evaluations in parallel instead of in sequence.

One iteration as a parallel tournament: strategies fan out, every candidate is scored, only the winner survives — One iteration from the inside: the generator proposes N competing strategies with past strategies and their scores in view, every candidate prompt set is scored on the full dataset in parallel, and only the highest-mean winner survives into the next round.

Portfolio Case Study
#

Problem. This came out of a larger production project that ran on a pile of hand-written prompts. We already had golden result sets we trusted, and at some point the obvious question surfaced: instead of hand-tuning prompts against those golden sets, could we optimize them automatically? The first test wasn’t the production system at all. I pointed it at generating hip-hop verses in the style of a specific artist, purely to see whether “make a prompt better on its own” was even a real thing. Once the lyric scores started climbing, we turned it on our actual production prompts.

Approach. The core is a training loop, deliberately shaped to feel like reinforcement learning rather than a single greedy edit. Each iteration doesn’t just ask the model for one improvement. It generates several competing strategies, and it shows the strategy generator the previous strategies together with the scores they earned, so the next round of ideas builds on what worked and drops what didn’t. Every strategy becomes a candidate prompt set, every candidate is scored on the whole dataset, and only the winner survives to the next iteration. Spawning many strategies per iteration is what keeps the search from getting stuck on one idea.

The optimizer also adapts how aggressive it is. When the current score sits well below target it makes substantial rewrites; as it closes in, it switches to surgical, minimal edits.

What was hard. The genuinely frustrating part is that models are bad at editing text through a search/replace tool. The search string the model emits rarely matches the prompt verbatim (a stray space, a changed line break), and a strict tool just drops the edit on the floor. This is a known-enough problem that Cursor built a separately fine-tuned model to sit between the main model and the file and apply edits for it (instant-apply). We didn’t want a second model in the loop, so we took the cheap and fast route: a more forgiving edit tool. It still prefers a clean single exact match, but when that fails it falls back to a fuzzy match over sliding windows, applies the edit only when there’s a unique best candidate above a threshold, and otherwise records exactly why it skipped. That one change is what made automated edits reliable enough to trust in a loop.

Getting the LLM-as-judge scoring stable enough to optimize against was the other quiet difficulty. The whole loop is only as trustworthy as the number the judge hands back.

Outcome. On real production workloads the climbs were clear. A procedure-finding agent went from 0.51 to 0.87, and an administrative-regime classifier from 0.60 to around 0.90. It moved from a for-fun experiment to a tool we ran many times against real prompts, with every run’s iterations, candidates, and prompt versions persisted so we could go back and see what changed.

Non-goals. It’s an internal tool, and it’s Gemini-focused on purpose. Prompting technique is model-specific, and at the time Gemini was the one model with a context window large enough for the long agentic flows we were running, so tuning against a single target model was the right scope rather than a shortcut. It optimizes prompt text only and never touches model weights. The dashboard is a local, read-only review surface with no auth, there to inspect training rather than to be a product.