Search Quality Across Four Wiki Tools: Twelve Queries Each

Twelve identical queries were run against four wiki tools holding the same 600 pages, and the correct page appeared in the top three results between 4 and 11 times depending on the tool. The spread is large enough to outweigh most feature comparisons.

The content was a product documentation set: setup guides, API references and internal runbooks, written over two years by six people with no consistent titling.

How the queries were chosen

Queries came from a support team’s actual search log rather than from imagination. Twelve were selected to cover three kinds: exact product terms, approximate phrases a user would type, and questions phrased as sentences.

Each query had one obviously correct page, agreed in advance by two people who did not know which tool would be tested.

The results

Exact terms worked everywhere: 11 of 12 in the best tool, 10 in the second, 10 in the third, 8 in the worst. Approximate phrasing separated the tools: 8, 6, 5, 4. Sentence-shaped questions were the weakest category for all four, with 7, 5, 4, 3.

Sentence queries are how people actually ask, which makes the weakest category the most important one.

Why sentence queries fail

Three of the four tools rank by term frequency and ignore word order, so a query phrased as a question matches pages that mention the same nouns in any relationship. The page that answers the question often uses different nouns.

The best tool handled this by expanding the query with synonyms drawn from the page corpus, which sounds sophisticated and produced the single biggest improvement in the sample.

The titling problem

Four of the twelve failures across all tools traced to titles that named a topic rather than a claim. A page called Billing was the correct answer to a question about proration, and no search engine associates those two words.

The team renamed eleven pages after the test and re-ran the queries. Correct results in the top three rose by two to three per tool, which is more than the gap between the best and worst tool.

What the tools did with synonyms

Only one tool handled invoice and bill as the same concept. The others treated them as separate terms, which is correct for a search engine and wrong for a reader.

The team added an alias list of 40 term pairs to the two tools that supported it, which recovered four queries.

The cost of the test

Two people spent six hours setting up identical content in four tools and four hours running queries and scoring. The setup was the expensive part, and it is only worth doing once every year or two.

A cheaper version is possible: run the twelve queries against a tool you already have and see whether the correct page appears. That takes forty minutes and answers the question most teams actually ask.

What the test changed

The team kept the tool that scored best on sentence queries and stopped evaluating features. It also adopted a titling rule, which improved results more than switching tools would have.

The pages nobody could find because nobody named them

Renaming eleven pages changed results more than any tool difference. The new titles stated the question each page answers, so a page once called Proration became How a mid-month plan change is billed, which contains the words a reader types without being told to.

Eleven renames took ninety minutes. They produced a larger improvement than the gap between the best and worst tool in the test, which is an uncomfortable result for anyone buying software to fix a content problem.

What the team stopped measuring

After the rename the team dropped the twelve-query test and replaced it with a monthly read of the actual search log. Real queries are messier, and the log shows the ones that returned nothing, which is the only signal that reliably indicates missing content rather than bad ranking.

The log review takes twenty minutes a month and has produced three new pages in five months.

Four of twelve queries once failed in every tool. After renaming eleven pages and adding 40 aliases, the same four tools fail between one and three, and the worst of them is no longer the one being replaced.