MODEL EVALUATION / LangChain
Eval Engineering
Turns an agent repository and traces into reviewed task specs, verifiers and Harbor runs, with a cost confirmation before scored runs.
“Inspect an agent repository and optional traces, interview the user, write reviewed Task Specs, build and audit Harbor tasks, and bootstrap reusable project World Knowledge Skills. Use for agent evals, benchmark design, Task generation, controlled Environments, synthetic data, Verifiers, Harbor runs, calibration, or continuous benchmark maintenance.”
01 / THE REASONING
Why this made the selection.
- Before a scored model run, the file requires confirming the model, trial count, judge, timeout and maximum expected cost with the user, unless that run plan was already authorized.
- It states that pass rates and model ordering do not prove task quality, requires reading at least one full run, and sorts failures into agent, harness, environment, verifier, leakage and infrastructure causes.
02 / THE REVIEW RECORD
What we actually inspected.
Source review has boundaries.
A clear record is more useful than a “safe” badge.
Material inspected
- config/skills/eval-engineering/SKILL.md (flow, terms, reference routing, Spec2Task pre-run checks, run audit, safety, completion criteria)
- README.md (installation)
- Repository tree listing the skill's references and scripts
- LICENSE MIT, copyright LangChain, Inc.
Our findings
- Task.md stays beside the Harbor task and must never be copied or mounted into the evaluated agent's workspace or image.
- Access, startup, reset, timeout, judge and verifier failures count as invalid runs, not failed agent work.
Not established by this review
- The installer was not run, and the skill was not executed against a project or service. No Harbor task was built or run.
- The reference files and the compare_tool_schemas.py and snapshot_sqlite_state.py scripts were not opened.
The review applies to the material and revision named here. A newer upstream release can change its behavior.
03 / PUT IT TO WORK
Add a skill to your workflow.
DOCUMENTED COMMAND
/plugin marketplace add langchain-ai/langchain-skills Copying does not execute this command. It may retrieve a newer version than the reviewed source.
- The README then runs /plugin install langchain-skills@langchain-skills. Its npx examples use --skill '*', which selects every skill in the repository.
- Neither command pins revision 16a992f09ab3ccfb642dd4f08d350d833d1990bf. Compare the installed eval-engineering directory with that revision.
- Keep secrets out of the conversation; the file says never to read, print, copy, store or ask the user to paste secret values.
Before you start
- Claude Code for the plugin path
- Harbor and a Docker-compatible runtime only if you build and run tasks
04 / FOLLOW THE EVIDENCE
The source trail.
Our notes are separate from the original resource.
Check upstream before adopting a new version.
- Skill definition ↗Checked
https://github.com/langchain-ai/langchain-skills/blob/16a992f09ab3ccfb642dd4f08d350d833d1990bf/config/skills/eval-engineering/SKILL.md
Supports: summary, upstreamDescription, whySelected, limitations, access, review
https://github.com/langchain-ai/langchain-skills/blob/16a992f09ab3ccfb642dd4f08d350d833d1990bf/README.md
Supports: install, compatibility
- Repository licence ↗Checked
https://github.com/langchain-ai/langchain-skills/blob/16a992f09ab3ccfb642dd4f08d350d833d1990bf/LICENSE
Supports: license