How to Measure MCP Retrieval Quality

To measure MCP retrieval quality, write a fixed list of questions with the fact each answer must contain, ask each one, and mark yes only when the passage the server returned contains that fact. Count the yeses, name each miss, and rerun the same list after every edit. Request volume shows the server was used. This shows whether it handed the agent the right paragraph.

By the end you will have a short scoring sheet, a result you can write in one honest sentence, and a way to tell whether your next docs edit helped.

What retrieval quality means

It is about the passage, not the prose around it. When an agent calls your MCP server, the server returns a few passages from your content and the agent writes a reply from them. A polished reply can restate the wrong passage in clean sentences. Retrieval quality asks a narrower question: did the passage handed to the agent contain the fact the person needed?

You answer it by reading. No chart can tell you whether a paragraph about key rotation mentions the 24 hour overlap. You open the passage and look.

Why request volume is not quality

The free Context MCP Studio dashboard shows request volume, success rate, and which tool and source handled each call. A busy server with a high success rate is being used, and success there means the request completed. It does not mean the passage held the fact.

A server can be busy all week and still hand the agent a neighboring section on every refund question. Volume tells you where to look, and reading tells you what you found.

How to write the question list

The running example here is a Billing API server that covers authentication, refunds, and webhooks.

1. Pick five to ten questions from real tasks. "How long do the old and new webhook keys overlap during rotation?" "What is the refund window for annual plans?" "Which header authenticates a request?"
2. Write the expected fact in a few words next to each one. "Overlap is 24 hours." "Refund window is 30 days." Keep it short enough to spot in a passage.
3. Add one question the sources do not cover, such as "What is the office wifi password?"
4. Freeze the list. The same questions and the same expected facts, rerun after every edit. A list that changes every week cannot show whether an edit helped.

For a simpler pass or fail version of this sheet, see How to test an MCP server for accuracy. For a more detailed way to mark an answer, see the MCP server rubric.

How to score each passage

Yes when the passage contains the fact.

No when the passage is from a different section, a different source, or the right page without the fact. If the constraint the person needs is missing, that is a no, even when the topic is right.

The out-of-scope question is a yes only when the server says its content does not cover it. MCP Studio does this by design: when nothing matches, the server tells the agent so instead of offering the nearest paragraph as an answer. A reply that invents a wifi password is a no.

How to run it in MCP Studio

MCP Studio is the no-code builder on Appa Tools. Open the server on the dashboard and confirm indexing is complete first, since a run during indexing measures a half-built server.

1. Ask each question once, in a fresh chat with the server enabled.
2. Record the cited page and copy the passage. On Core plus Action analytics, $99 a month, the Action tab shows the page, section, and excerpt for each request, so you are not copying passages out of the chat by hand. Core alone, at $49 a month, shows the question text in the request log. New accounts get a 30-day trial with every tier unlocked.
3. Mark yes or no against the fact, not against how smooth the reply sounds.
4. For each no, make one edit. A heading, a source, or a retrieval rule. Then rerun the whole list. Do not mix a partial rerun into the old count.

How to report the result

Write the result as a record of one run, with its limits attached. Here is an example of that sentence, not a product score: "4 of 5 answerable questions returned a passage that contained the fact, on the Billing API server, in Claude, on September 22, 2026, after indexing completed. The fifth cited the overview. The list is ours."

That sentence says what was measured, on which server, in which client, and when. It is not a universal score, and it is not comparable to a three-question run on a different server.

What to do with a no

Move the missing constraint next to the steps it qualifies, or prefer the right source. Then rerun the full list. The edit path for each kind of miss is in How to fix an MCP server returning the wrong information. To see which pages your agent relies on day to day, rather than in one test, read Measure which documentation an AI agent actually uses.

Frequently asked questions

What is MCP retrieval quality?

MCP retrieval quality is whether the passages an MCP server returns contain the fact the question needed. It is judged on the passage itself, not on how fluent the agent's reply sounds. You measure it by asking fixed questions and checking each returned passage against an expected fact.

How many questions do I need to measure retrieval quality?

Five to ten questions drawn from real tasks is enough to show where a server misses, plus one question the sources do not cover. Keep the list fixed between runs so the results stay comparable.

Is a high request count a sign of good retrieval?

No. A high request count means agents are calling the server, and a high success rate means the requests completed. Neither tells you the returned passage held the right fact, so read the passages for the questions that matter.

Do I need paid analytics to measure retrieval quality?

No. You can ask each question in your client and open the cited page to check the passage. Core plus Action analytics makes it faster by showing the page, section, and excerpt for every request in one place.

Can I compare my retrieval score with another server?

Only if both runs used the same questions, the same expected facts, the same client, and fully indexed sources. A score from your own list describes your server on that day, so treat it as a record of progress rather than a ranking.

How should I score a question my docs do not cover?

Score it as a pass when the server says its content does not cover the question. Score it as a fail when the reply offers an answer anyway, because that answer did not come from your content.

Score your server with MCP Studio

Open your server, ask the five to ten questions you wrote down, and mark each passage yes or no against its fact. Keep the count next to the server name, the client, and the date, then rerun it after your next edit. Open the MCP Studio dashboard.

Where to go next

• Test an MCP server for accuracy
• The MCP server rubric
• What the questions people ask your docs can tell you
• Action and Predictive analytics