2026-09-22 • Updated 2026-09-26 • 7 min read • Measure
How to Test an MCP Server for Accuracy
Learn to test an MCP server with a written list of questions, the page each answer must cite, the fact it must contain, and one question it should decline.
To test an MCP server for accuracy, write down five to ten real questions, the page each answer should cite, and the fact that page must contain, plus one question your sources do not cover. Run them after indexing finishes, in a fresh chat each time. Score a pass only when the citation and the fact both match, and count the uncovered question as a pass when the server says it is not covered.
By the end you will have a test sheet you can rerun after every docs edit, and a result that tells you exactly which question failed and why.
What an accuracy test sheet contains
Questions from real tasks, written the way people ask them. Five to ten is enough. Three will show a broken server. Ten will show which task fails. Word them the way a customer or teammate would ask, not the way your headings are phrased, because matching your own headings tests nothing.
The page each answer should cite, and the fact it must contain. A few words is enough, such as the header name or the size limit exactly as the page states it. This is what turns "it seemed fine" into a result the next person can rerun.
One question your sources do not cover. This checks the behavior that makes a server trustworthy. When nothing in your content matches, a Context MCP Studio server says so, and the agent is asked to pass that on rather than guess. Your test should confirm that happens.
The sheet is yours. It describes your server on your content, so it is not a score to paste onto someone else's server. For a more detailed way to mark an answer, with a worked comparison, see the MCP server rubric. Use the sheet every time you change a page, and the rubric when pass or fail is too coarse.
Why the link and the fact both have to match
A fluent paragraph with the wrong link fails. The agent can restate a neighboring page in clean sentences, and the reply will read well. The link tells you where it came from.
The right link without the fact fails too. A citation to the upload page that never states the header name leaves the person unable to act.
A nearby paragraph fails the out-of-scope question. The wifi password question is a pass only when the server says its content does not cover it.
A reworded question is not a rerun. Changing the wording until one row passes teaches you a prompt. It does not tell you whether the page is fixed.
How to run the test in MCP Studio
MCP Studio is the no-code builder on Appa Tools. The server has to exist and show indexing complete before the sheet means anything. Open it on the dashboard. If you still need a server, create one in the wizard.
The running example is a server built from a file upload API's documentation.
| Question | Expected | Pass if |
|---|---|---|
| Which header authenticates an upload? | Authentication page | The citation is that page and the header name matches |
| What is the maximum upload size? | Limits page | The number matches the page |
| What is the office wifi password? | Not in the docs | The server says it is not covered |
- Ask each row in a fresh chat with the server enabled. Do not hint at the page title.
- Use the client people actually use. A second client is a second run, labeled separately.
- Open each citation and check the fact against the page.
- Save the date, the client, and the fails next to the sheet.
The free MCP Studio dashboard shows that each call happened and which tool and source handled it. It does not compare a fact to a page, because that judgment is yours. On Core plus Action analytics, $99 a month, the Action tab shows the page, section, and excerpt for every request, which makes step 3 much faster. New accounts get a 30-day trial with every tier unlocked. The dashboard guide describes each view.
How to read the results
A pass on the upload header row is the right page and the right header name. A pass on the wifi row is a clear "not covered". Anything else is a fail, even when the reply sounds confident.
Write the result as a record of one run: the server, the client, the date, and how many rows passed. A run started while indexing was still in progress is not comparable to a run on a finished server. If you keep the early numbers, say so next to them.
If a row fails
Fix one fail at a time using How to fix an MCP server returning the wrong information, then rerun the whole list, not only the row you fixed. An edit that fixes one answer can move another.
If two pages could both be correct, pick the canonical one in the sheet. If the other keeps winning, prefer the canonical source with a retrieval rule. Headings that use the reader's words are covered in Make your documentation AI-ready. To score passages in more detail over time, read Measure MCP retrieval quality.
Frequently asked questions
How do I know if my MCP server is accurate?
Ask it questions you already know the answers to and check two things on each reply: that it cites the page you expected, and that the cited passage contains the fact. Include one question your content does not cover and confirm the server says so.
How many questions should an MCP accuracy test have?
Five to ten questions drawn from real tasks is enough for most servers. Three will reveal a server that is badly broken, and ten will show which specific task fails. Keep the list fixed so each run is comparable with the last.
Should I test with a question my docs cannot answer?
Yes. A question outside your content checks that the server says it is not covered instead of producing an answer anyway. It is one of the most useful rows on the sheet, because it confirms you can trust the server's other answers.
Can I compare accuracy scores between two MCP servers?
Only when both runs use the same questions, the same expected facts, the same client, and fully indexed sources. A score from your own sheet describes your server on your content on one day, so treat it as a record of progress.
Does the MCP Studio dashboard measure accuracy for me?
The dashboard shows that requests happened and which tool and source handled them, and Action analytics shows the exact passage returned for each one. Deciding whether that passage holds the right fact is a judgment you make with your test sheet.
Test your server with MCP Studio
Open your server after indexing shows complete and ask the three questions in the table, or your own five to ten. Keep a row as a pass only when the link and the fact both match, or when the out-of-scope question is declined. Open the MCP Studio dashboard.