How to Know If Your Docs MCP Is Actually Accurate: Introducing the MCP Server Rubric

Building an MCP server is the easy part. Knowing if it's working is harder.

Once an MCP server is connected, there's an obvious question to ask: how do you know the answers it gives are more accurate?

That's a fair question, and "it seems better" isn't a satisfying answer. So we built a rubric, a five-dimension scoring method for evaluating AI answer quality with and without an MCP connection, and then we used it on a real test.

This post shares the method, the results from that test, and a free download of the rubric so you can run this experiment on your own documentation.

The short version: On a 250-point scale, an unconnected Claude session scored 139 answering five Metabase questions. The same session, with an MCP server connected to Metabase's documentation through MCP Studio, scored 245. That's a 76.3% improvement.

Here's exactly how those numbers were calculated.

Why Metabase?

Metabase made an ideal test subject for a specific reason: it has three distinct public-facing resources that can all be indexed into a single MCP server. A technical documentation site, a tutorials and use-case library, and a public GitHub repository for its docs. That's a realistic, multi-source MCP server setup, and it means ground truth is easy to verify independently by anyone reading this post.

Five questions, one rule

Each question had to pass a single test: does it have one unambiguous answer explicitly stated in the current Metabase docs?

That ruled out best practices and architectural opinions. It kept questions about specific commands, environment variable names, minimum version requirements, and exact configuration steps. These are the kinds of facts that AI models often get confident about but that change between software releases.

1. What Java version is required to run Metabase?
2. What environment variable changes the port Metabase runs on from its default?
3. What are the exact environment variables needed to connect Metabase to a Postgres application database?
4. What command migrates Metabase's data from H2 to a production database, and what's the critical gotcha about the file path?
5. What is the minimum supported MySQL version for Metabase's application database?

Real questions, documented precisely, easy to get wrong without the right context.

The MCP Server Accuracy Rubric

Scoring an AI answer as "good" or "bad" doesn't generate a number you can compare. So we broke quality down into five independent dimensions, each scored 0–10, for a maximum of 50 points per question and 250 points total across five questions.

Here's each dimension and what the scores mean.

1. Factual Accuracy

Is every verifiable claim correct per the current documentation?

Score	What it means
10	Every fact matches the current docs exactly
7–9	One minor detail is slightly off or paraphrased
4–6	Core answer is right but at least one specific fact (version number, flag, variable name) is wrong
1–3	The direction is right but the specific values are wrong
0	The answer is confidently and substantively incorrect

2. Specificity

Are exact values given, like commands, variable names, and version numbers, rather than vague guidance?

Score	What it means
10	Concrete, usable values given for every part of the answer
7–9	Most values are given; one part is hedged or paraphrased
4–6	The right topic is covered but not with enough detail to act on
1–3	Vague throughout
0	No useful specifics at all

3. Completeness

Does it cover all parts of the question, including the caveats needed to act safely?

Score	What it means
10	All sub-questions answered; all important prerequisites mentioned
7–9	One minor sub-point or safety caveat is missing
4–6	The main answer is there but a meaningful piece is left out
1–3	Significant gaps; a reader would need to look elsewhere
0	Answer is essentially incomplete

4. Currency

Does the answer reflect the current release, not a superseded one?

Score	What it means
10	All version-specific details match the current release
7–9	Mostly current with one minor dated reference
4–6	Some details reflect an older release
1–3	The answer primarily describes outdated behavior
0	The information is from an old version and substantively wrong today

5. Source Groundability

Can every claim be traced to a specific statement in the documentation, not just something that sounds like it could be there?

This is the most important dimension, and the one you can't evaluate by re-reading the answer. It asks: if someone opened the documentation right now, would they find this exact claim? A fabricated flag name, an invented version number, a setting that has the shape of real documentation but no actual origin. These all score low here.

Score	What it means
10	Every claim traceable to a specific page and section
7–9	Most claims traceable; one slightly paraphrased
4–6	Core facts are real but at least one specific claim has no origin in the docs
1–3	Multiple claims exist nowhere in the documentation
0	Answer contains invented specifics throughout

---

Ground truth for each question was established before either answer was scored, by reading the current Metabase docs directly:

• Java version → running the Metabase JAR
• Port variable → environment variables
• Postgres and MySQL app DB → configuring the application database
• Migration command → migrating from H2
• Migration error message → troubleshooting load-from-h2

The results

Without MCP

Question	Factual	Specificity	Completeness	Currency	Groundability	Total
Q1 — Java version	2	6	6	1	3	18
Q2 — Port variable	10	8	7	10	7	42
Q3 — Postgres vars	9	9	6	9	6	39
Q4 — H2 migration	4	5	5	6	3	23
Q5 — MySQL minimum	2	6	4	2	3	17
Total	27	34	28	28	22	139 / 250 — 55.6%

With MCP

Question	Factual	Specificity	Completeness	Currency	Groundability	Total
Q1 — Java version	10	10	9	10	9	48
Q2 — Port variable	10	10	9	10	9	48
Q3 — Postgres vars	10	10	10	10	9	49
Q4 — H2 migration	10	10	10	10	10	50
Q5 — MySQL minimum	10	10	10	10	10	50
Total	50	50	48	50	47	245 / 250 — 98.0%

Overall

	Score	Percentage
Without MCP	139 / 250	55.6%
With MCP	245 / 250	98.0%
Improvement	+106 points	+76.3%

What changed and where

Q1 (Java version): +166.7%. Without MCP, Claude answered Java 21, a real, recent LTS release that sounds right but is explicitly unsupported. Java 25 is the current requirement. The MCP server retrieved this from the troubleshooting: running Metabase page: "Metabase should be run on Java version 25 (older versions are unsupported)." An archived v0.43 page also appeared in the retrieval results. It ranked last, below all current content.

Q2 (Port variable): +14.3%, the smallest gain. MB_JETTY_PORT is stable and widely documented, so the unconnected model already knew it. Retrieved sources: environment variables and customizing the Jetty webserver. This is what the rubric looks like when grounding adds precision rather than correctness.

Q4 (H2 migration): +117.4%. The unconnected answer invented a --h2-jar-path flag that does not exist in Metabase's CLI. With MCP, the answer scored a perfect 50: correct full command, correct path semantics (omit .mv.db), the exact error string from the troubleshooting page, same-version requirement, and empty-target-database requirement. These were pulled from two different documentation pages and stitched into one complete answer.

Q5 (MySQL minimum): +194.1%. The largest gain, and the most interesting result. Five questions produced only four retrieval calls. There was never a dedicated search for the MySQL minimum. Every fact in that answer (MySQL 8.4.0, MariaDB 10.6.0, utf8mb4 character set, innodb_large_prefix=ON) came from a passage already retrieved for Q4. Lines 33–72 of migrating-from-h2.md contain Metabase's full list of supported application databases. One retrieval call answered two questions. The unconnected model had answered MySQL 8.0.17 and MariaDB 10.4+, both wrong.

The pattern

Question	Gain	Type
Q5 — MySQL minimum	+194.1%	Version-pinned, changes between releases
Q1 — Java version	+166.7%	Version-pinned, changes between releases
Q4 — H2 migration	+117.4%	Syntax-exact, spans multiple pages
Q3 — Postgres vars	+25.6%	Stable, widely documented
Q2 — Port variable	+14.3%	Stable, widely documented

The gains cluster around version-pinned facts and syntax-exact procedures. These are the parts of documentation that change between releases and live on pages a model's training data may not have seen, or may have seen in an old form. For stable trivia that appears in a thousand blog posts, grounding adds polish. For the facts that change, it's the difference between an answer you can run and one that fails in production.

The useful question for your own docs: how much of what your users ask about is version-pinned or syntax-exact? For most infrastructure, API, and onboarding documentation, the honest answer is most of it.

Honest caveats

Five questions is a small sample showing a large effect on this class of question, not a universal constant. Every score above is included so anyone can disagree with a specific number rather than the overall conclusion. All five questions were asked in a single prompt, which is why Q5 was answered by a retrieval that ran for Q4. Asking them separately might score differently.

Run this on your own documentation

That's the whole reason for sharing this method. The rubric is free to use. Pick five questions your team keeps getting wrong, find the documented answers, run both conditions, and score them.

Building the MCP server side of this takes about two minutes. Paste your documentation URL, a GitHub repo, or any website, and MCP Studio handles the rest. The free tier covers everything this test required.

Once it's live, the MCP Studio Analytics dashboard shows exactly which pages were retrieved for every answer, so you can trace each score back to a real source, the same way every question above was verified.

Download the MCP Server Accuracy Rubric (PDF) →

Build your MCP server for free →

Keep going

• Create your first MCP server — start here
• Connect to Claude, Cursor, or VS Code — required for the with-MCP condition
• How MCP Studio works, end to end — from source to answer

Building grounded AI into your own product? The MCP Studio SDK embeds the full wizard inside your app. Start at appatools.com.