How the Tally skills measurement was done

This page is the record for the measurement behind Skills written from your docs answer the questions in them. It lists the documentation, the MCP server, the chunks endpoint calls, every chunk that became a skill, the skills, and the questions.

The passage text of each chunk stays on Tally's site. Copying those pages here would republish their documentation. Each row links to the page the chunk came from, with the chunk's index on that page. The skills and the questions are ours, and they are on this page.

The documentation

The MCP server

The server was made in MCP Studio, with no code:

  1. Sign in and open the wizard.
  2. Add https://developers.tally.so/ as a website source.
  3. Add https://tally.so/help/webhooks as a second website source.
  4. Deploy, and wait until both sources show as indexed.

The deployed server slug is tally-api-docs-xilrlv. Reading it requires an MCP Studio API key. The key is not on this page.

Assets

The MCP server from this measurement. Copy the address into an MCP client that has an API key.

https://appatools.com/mcp-studio/api/mcp/tally-api-docs-xilrlv

The chunks endpoint

List recently indexed chunks returns the passages MCP Studio stored for one source, newest first, 100 at a time. The project calls it with the server slug and each source id, then passes nextCursor back as cursor until nextCursor is null.

curl "https://appatools.com/mcp-studio/api/mcp/tally-api-docs-xilrlv/sources/cmvlqi9iu0008j1lprcag0hag/chunks?limit=100" \
  -H "Authorization: Bearer $MCP_STUDIO_API_KEY"

curl "https://appatools.com/mcp-studio/api/mcp/tally-api-docs-xilrlv/sources/cmvlqi9iu0009j1lp35xzvnq3/chunks?limit=100" \
  -H "Authorization: Bearer $MCP_STUDIO_API_KEY"

The run used docs-to-self-evaluating-skills with granularity set to chunk. The config is skills.config.json. One skill was written per chunk. Chunks whose text matched another chunk exactly were skipped. That left 269 skills from chunks, plus 2 map skills that list the pages, 271 in total.

How a skill was graded

The model for this part was gpt-4o-mini, named in the 24 October 2026 report. For each skill the project wrote a few questions from that skill's passage, and for each question a few checks that name a fact a correct answer must contain. Every question was answered twice: once following the skill, once without it. A check passes only when the grader quotes the part of the answer that satisfies it. Nothing was executed.

On that report, 1,891 of 3,158 checks passed with the skill (59.9%). Without the skill, about 19.6% passed. The report stores the without-skill side as a percent per skill, so that figure is those percents added up. 243 of 267 scored skills were higher with the skill, 10 were level, and 14 were lower on that day.

Open a skill below to read it. If it has checks, the questions and the expected facts are in the same view. The full score table is the report.

The ten questions written beforehand

These were written before the chunk skills existed. Each was asked five times. The model was gpt-4o at temperature 0.2. One arm had the question only. The chunk-skills arm saw each skill's name and description, opened up to six, and answered from those. A text answer passes when it contains every fact listed for that question. Question 4 is different: the JSON it returns is sent to Tally's API, and it passes only if Tally creates the form. The rule written down beforehand was that skills count if they beat the question-only arm by at least 15 points out of 50 answers.

The latest stored run was 10 of 50 without a skill (20%) and 36 of 50 with the chunk skills (72%). Across the stored runs of that setup, the with-skill score stayed between 66% and 72%, and the without-skill score stayed between 20% and 22%.

Every chunk that became a skill

269 chunks, 10 per page. Search, or narrow by source and page.

PageIndexSkill

Every skill

271 skills. Search, or narrow by the source the skill was written from.

SkillDescription