How to Use claude plugin eval in Claude Code

The new claude plugin eval command in Claude Code (Anthropic's terminal-based AI coding tool) lets you score whether a plugin (a bundle of add-on functionality for Claude Code) you built actually helps — or still needs work. It runs the same test cases with your plugin on and off and compares the results, so you get a number instead of a guess about whether things "feel" better.

What is claude plugin eval?

claude plugin eval is a command that measures the real-world effect of a plugin or skill (a pre-written set of instructions for a specific task) using test cases. According to the official announcement (@ClaudeDevs, September 11, 2026), it creates test cases, runs them with your plugin applied and scores the results, then reruns the same cases without the plugin so you can compare the two. For example, if you built a plugin meant to make code review more thorough, this command lets you check whether review quality actually improved.

Before you start

• Claude Code installed and up to date — run claude update in your terminal so the new command is available.
• A plugin folder to test — you need an existing plugin or skill project folder. This feature has nothing to work with if you haven't built a plugin yet.
• A few real prompts — having some actual requests you use with the plugin ready in advance makes the next step much faster.

Step-by-step instructions

The overall flow is: initialize, pilot run, full run, then check results.

1. Go to your plugin's folder and initialize. In your terminal, move into the plugin's folder and run:

   claude plugin eval init

   Success looks like this: Claude asks what good and bad output looks like, then drafts test cases for you.
2. Describe good/bad output and give real prompts. Follow the on-screen prompts to explain what counts as a success versus a failure, and enter a few of the real prompts you prepared. Claude drafts test cases and checks, pilots the suite, and tells you what a full run will cost.
3. Do a pilot run first. Evals call the model, so they use tokens (Token — the unit AI uses to process text, also the basis for API billing) and results can vary between runs. It's safer to run a single pilot before committing to a full run:

   claude plugin eval --runs 1

4. Run the full evaluation once you're satisfied.

   claude plugin eval

   Success looks like this: your terminal shows each test case's score with and without the plugin side by side, plus an HTML report with full detail.

Checking your results

Results show up in two places: terminal scores and an HTML report. The terminal displays a "with plugin" score and a "without plugin" score for each test case, and the gap between them is the actual value your plugin is adding. If your account supports it, the same report is also published as a private artifact (Artifact — a space where Claude stores and shares generated output).

Common pitfalls and things to watch out for

• Cost and variance — evals actually call the model, so they cost tokens and scores can shift slightly between runs. Always try --runs 1 first before a full run.
• Only evaluate plugins you trust — a plugin's hooks (scripts that run automatically at certain points) and MCP (Model Context Protocol — a standard for connecting external tools) servers run with your own account permissions during the eval. Don't evaluate plugins from sources you don't trust.
• Run claude update first — if the command isn't available, your Claude Code version is likely outdated. Run claude update and try again.

Frequently asked questions

Q. What do I need before using claude plugin eval?
You need a plugin folder to test, a few real prompts you actually use, and an up-to-date version of Claude Code. Run claude plugin eval init inside the folder and Claude guides you through the rest.

Q. Do I have to run a pilot before the full run?
It's not strictly required, but it's recommended. Since evals call the model, cost tokens, and results can vary between runs, checking cost and output first with --runs 1 is the safer approach before running the full evaluation.

Q. Where can I see the results?
By default, case-by-case scores appear in your terminal, and an HTML report with full detail is generated alongside it. If your account supports it, the same report is also saved as a private artifact.

Q. Is it safe to evaluate any plugin?
Caution is needed. During the eval, the plugin's hooks and MCP servers run with your own account permissions, so it's safest to only evaluate plugins from sources you know and trust.

Related articles

• Connect Blender to Claude Code with an MCP Server: Step-by-Step Guide
• Claude Code Installation Guide: native vs npm, Windows Included
• Claude Code Skills (Agent Skills) Guide: Extend Claude with SKILL.md
• Claude Code settings.json Guide — Permissions, Env Vars & Tools
• Claude Code Pricing Explained (2026): Plans, API Costs & Limits