Skills Written from Your Docs Answer the Questions in Them

A skill is a set of instructions your coding assistant can follow. The useful question is a small one: if the assistant follows the skill, does it get the documentation right, and does it miss those same facts when the skill is not there?

We ran that check on skills written by docs-to-self-evaluating-skills. The project reads documentation you have indexed in MCP Studio and writes one skill per passage, with checks taken from that same passage.

How this was measured

The documentation is public, and so is every skill, question, and chunk identity from the run. They are on the measurement page.

The docs. Two sites: Tally's developer documentation and Tally's webhook help.

The MCP server. It was made in the MCP Studio wizard, with no code. Add each of those URLs as a website source, deploy, and wait until both sources show as indexed. The server from this measurement is tally-api-docs-xilrlv, at https://appatools.com/mcp-studio/api/mcp/tally-api-docs-xilrlv. The developer docs source id is cmvlqi9iu0008j1lprcag0hag. The webhook help source id is cmvlqi9iu0009j1lp35xzvnq3.

The chunks endpoint. List recently indexed chunks returns the passages MCP Studio stored for one source, newest first. The project reads a page of 100, then sends nextCursor back as cursor until nextCursor is null.

curl "https://appatools.com/mcp-studio/api/mcp/tally-api-docs-xilrlv/sources/cmvlqi9iu0008j1lprcag0hag/chunks?limit=100" \
  -H "Authorization: Bearer $MCP_STUDIO_API_KEY"

The same call with source id cmvlqi9iu0009j1lp35xzvnq3 reads the webhook help. You need your own API key. None is included here.

docs-to-self-evaluating-skills was pointed at those two sources with one skill per chunk. A chunk whose text was an exact copy of another chunk was skipped. That left 269 skills from chunks, plus two skills that map the pages.

Grading. The skill grades used gpt-4o-mini, which is the model named in the 24 October 2026 report. The ten questions below used gpt-4o at temperature 0.2, five answers each. Both setups are written out on the measurement page, next to the files they graded.

What we asked

The documentation was Tally's public developer docs, the pages a person reads to connect an app to Tally, plus Tally's webhook help page. Each passage became a skill. Each skill came with checks written from that passage: a field name, a limit, a header, the kind of fact a correct answer has to contain.

Every check was asked twice. Once, the answer had to follow the skill. Once, it had the question and nothing else. A check passes only when the grading can quote the part of the answer that satisfies it. Nothing was run. A passing answer has the right facts on the page. It is not code we executed.

What came back

The report is from 24 October 2026. It covers 267 skills and 3,158 checks.

Following the skill, 1,891 checks passed. That is 59.9%. The same questions without the skill passed 19.6%. The report stores the without-skill score as a percent per skill, so that 19.6% is those percents added up, and the last digit is rounded.

243 of the 267 skills scored higher with the skill than without it. Ten scored the same. Fourteen scored lower on that day. A skill's score moves by a check or two from one run to the next even when its text did not change, which is why the project keeps a revision only when it passes at least two more checks.

A second list of questions, written first

The checks above are written from the same passage as the skill. That is the right test of whether the skill teaches its passage. We also kept a separate list of ten questions a developer would ask while connecting an app to Tally. The list was written down before these skills existed, and each question was asked five times.

Without a skill, 10 of 50 answers had the facts, which is 20%. With the skills, and with the assistant allowed to open only the few whose names fit the question, 36 of 50 did on the latest stored run, which is 72%. Across the stored runs of that setup, the score with skills stayed between 66% and 72%, and the score without them stayed between 20% and 22%. We had written down that a gap of 15 points would count. The gap was about 50 points.

A question whose answer is not on the page stays unanswered either way. The skill can only carry what the documentation says.

What you can do with this

Index the documentation in MCP Studio. Point the project at that source and run it. You get a folder of skills, and a report with each skill's score alongside the score for the same questions without it. Copy the skills you want into the project where you are building.

The numbers above are this project, on Tally's public docs, for one day. Your own documentation produces your own report, and that report is the one that tells you whether the skills are worth keeping.

The project and the setup steps are in the repository. A walkthrough of the same flow is in the docs.