CodeAtlas

• 36questions with labelled answer files
• 0.8195% interval 0.67 to 0.92hit rate@5, shipped configuration
• 0.6595% interval 0.52 to 0.78recall@5, shipped configuration
• 0.57795% interval 0.444 to 0.702mean reciprocal rank
• 18configurations scored on the same questions

What is measured

The corpus is four repositories of mine, checked out at fixed commits and listed file by file, with line counts and hashes, in the results directory. The evaluator refuses to run on a corpus that differs from that list, so the numbers here can be regenerated by anyone.

The question set is 36 questions I wrote against these projects, phrased the way someone new to a repository would ask rather than by pasting identifiers, each with the files that answer it. Relevance is judged at file level: a retrieved chunk counts when its source file is one of the labelled answers. I wrote both the questions and the code, and there has been one labeller so far; the labelling protocol for a second round with two independent labellers is in the repository.

The questions are split 50/50 into a dev half and a test half. The only setting chosen on the data, the per-file cap value, is chosen on the dev half; every table reports all 36 questions with a bootstrap interval, because eighteen questions per half say very little on their own.

Metrics

• hit rate@k: 1 when any labelled file appears in the top k chunks.
• recall@k: the share of the labelled files that appear in the top k chunks. For a question with four answer files, one of them in the top 5 is a hit rate of 1 and a recall of 0.25.
• MRR: one over the rank of the first relevant chunk, averaged over questions.
• nDCG@10: discounted gain with one binary gain per relevant file at its first chunk, normalised by the ideal ordering of the labelled set.

Every cell carries a 95% percentile bootstrap interval over questions (10,000 resamples). Every claim that one configuration beats another rests on a paired bootstrap of the per-question differences and an exact sign test on the questions where the two differ. When the interval includes zero the page says so.

Results

The shipped configuration is merged structural chunking, hybrid retrieval, and a per-file cap of 1.

Every configuration

• BM25structural, merged0.670.342
• densestructural, merged0.830.532
• hybrid (RRF)structural, merged0.720.547
• hybrid + per-file capstructural, merged0.810.577
• hybrid + reranker (text only)structural, merged0.720.423
• hybrid + reranker (prefixed)structural, merged0.780.455
• BM25structural, unmerged0.560.382
• densestructural, unmerged0.690.474
• hybrid (RRF)structural, unmerged0.640.521
• hybrid + per-file capstructural, unmerged0.750.568
• hybrid + reranker (text only)structural, unmerged0.670.436
• hybrid + reranker (prefixed)structural, unmerged0.780.461
• BM25window0.640.385
• densewindow0.750.514
• hybrid (RRF)window0.690.515
• hybrid + per-file capwindow0.810.567
• hybrid + reranker (text only)window0.670.449
• hybrid + reranker (prefixed)window0.670.442

What beat what, and what was within noise

Structural boundaries against windows

Merged structural chunking with hybrid retrieval scores hit rate@1 0.42 against 0.39 for windows (difference +0.03, 95% interval -0.06 to +0.11; better on 2 questions, worse on 1, same on 33). The interval includes zero, so this is within noise at this sample size. The same chunking scores MRR 0.547 against 0.515 for windows (difference +0.032, 95% interval -0.027 to +0.089; better on 11 questions, worse on 4, same on 21). The interval includes zero, so this is within noise at this sample size. Both chunkers are packed to the same token budget, so the comparison is about where boundaries fall, not about chunk size.

Hybrid against dense

Hybrid retrieval scores MRR 0.547 against 0.532 for dense retrieval alone (difference +0.015, 95% interval -0.126 to +0.160; better on 14 questions, worse on 11, same on 11). The interval includes zero, so this is within noise at this sample size. Hybrid retrieval scores hit rate@5 0.72 against 0.83 for dense retrieval (difference -0.11, 95% interval -0.25 to +0.00; better on 1 question, worse on 5, same on 30). The interval includes zero, so this is within noise at this sample size. Fusing in the BM25 ranking pushes some files that dense retrieval had in the top 5 below the cut.

Hybrid against BM25

Hybrid retrieval scores MRR 0.547 against 0.342 for BM25 alone (difference +0.205, 95% interval +0.095 to +0.318; better on 20 questions, worse on 3, same on 13). The interval excludes zero.

The per-file cap

With the cap, hybrid retrieval scores recall@5 0.65 against 0.57 for the same ranking without it (difference +0.08, 95% interval +0.01 to +0.17; better on 4 questions, worse on 0, same on 32). The interval excludes zero. The cap scores MRR 0.577 against 0.547 for no cap (difference +0.029, 95% interval +0.011 to +0.052; better on 9 questions, worse on 0, same on 27). The interval excludes zero. Under file-level relevance the cap cannot lower hit rate or MRR, because the first chunk of every file is always kept and only moves up; the metric it can lower is recall, and it did not.

The shipped configuration against the next best cell

The shipped configuration scores MRR 0.577 against 0.568 for structural, unmerged / hybrid + per-file cap (difference +0.009, 95% interval -0.039 to +0.059; better on 7 questions, worse on 4, same on 25). The interval includes zero, so this is within noise at this sample size.

All comparisons

The cross-encoder reranker

Reranking the top 30 with cross-encoder/ms-marco-MiniLM-L-6-v2 was run two ways: on the chunk text alone, and on the same path :: name payload the embedder sees, so that the reranker is not judged on less information than the first stage had. Every chunk fits the reranker's window, so nothing is truncated in either variant.

Reranking the chunk text alone scores MRR 0.423 against 0.547 for hybrid retrieval without it (difference -0.124, 95% interval -0.220 to -0.042; better on 5 questions, worse on 12, same on 19). The interval excludes zero. Reranking the same path-and-name payload the embedder sees scores MRR 0.455 against 0.547 for hybrid retrieval (difference -0.092, 95% interval -0.194 to +0.008; better on 10 questions, worse on 13, same on 13). The interval includes zero, so this is within noise at this sample size.

The prefixed variant scores MRR 0.455 against 0.423 for the text-only variant (difference +0.032, 95% interval -0.024 to +0.082; better on 16 questions, worse on 3, same on 17). The interval includes zero, so this is within noise at this sample size. On window chunks, the text-only reranker scores MRR 0.449 against 0.515 for hybrid retrieval (difference -0.066, 95% interval -0.183 to +0.045; better on 10 questions, worse on 10, same on 16). The interval includes zero, so this is within noise at this sample size.

What the data supports: in this setup the reranker does not improve ranking, and on merged structural chunks the text-only variant is worse by more than the noise. It does not support a cause. Whether the model's training domain, the small candidate set, or the file-level judgement explains it is not something this experiment tested.

Where it fails

The shipped configuration misses 7 of 36 questions at rank 5: tf03, tf08, tf09, tf12, dl06, dl13, gc05. Two things were measured about what fills the top 5 on those questions.

The documentation share differs by +0.04 between misses and hits, and its interval includes zero, so it is within noise. The wrong-project share differs by +0.24, and its interval excludes zero. The corpus holds four separate projects and nothing in the retriever scopes a query to one of them; a project filter is the obvious next fix, and it would be tested the same way.

How it works

files ──► chunker ──┬─► BM25 index          ┐
                    └─► MiniLM embeddings   ├─► RRF fusion ─► per-file cap ─► top k
                                            ┘

Tree-sitter parses Python and Java. A class becomes a header chunk (its signature, fields and docstring) plus one chunk per method, so no line is indexed twice; imports and module-level code become their own chunks; Markdown splits on headings and carries the heading path into the text. Files without a grammar (JavaScript, YAML) use windows in every chunking. Every chunk is sized to fit the embedder's window: sentence-transformers/all-MiniLM-L6-v2 reads at most 256 word pieces, so the path :: name prefix plus the text is kept within 254 tokens, and a declaration longer than that is split into consecutive parts. The window baseline packs lines to the same budget with a quarter of each window repeated in the next.

BM25 runs on a tokeniser that splits identifiers (RateLimitFilter is reachable from "rate limit"), drops English stopwords and stems with Snowball. Fusion is Reciprocal Rank Fusion with k = 60 over the top 60 of each list. The per-file cap keeps the first chunk of every file at the head of the list and pushes the rest below every survivor, so nothing is discarded. Both models run on CPU at pinned revisions.

Limitations

• The set is 36 questions. The intervals are the honest width: a hit rate of 0.81 at rank 5 sits in 0.67 to 0.92. Most differences between configurations are inside that width.
• One labeller, who also wrote the code. A second round with two independent labellers, an agreement statistic and an adjudication log is specified but not done.
• File-level relevance is generous: a chunk from the right file counts even when it is the wrong function in that file.
• The dev/test halves are small. The cap value was chosen on eighteen questions; the test half agrees on direction, and that is all it can say.
• Retrieval only: no generation, so no measurement of whether an answer follows from the retrieved context.