OpenWiki Technical Review: Agent Documentation and Retrieval Readiness

This review asks how OpenWiki can be evaluated as a context-generation system for coding agents: how quickly it produces documentation, how faithfully it represents a repository, and whether agents retrieve useful guidance from the result.

The public project is broader than a conventional vector index. It combines repository analysis, documentation generation, agent instruction updates, connectors, local persistence, and evaluation tooling. A fair review must test those layers separately.

OpenWiki writes structured repository documentation and points coding agents toward that material through managed instruction snippets. Before measuring retrieval precision, an evaluator must determine whether the generated pages contain correct, current, and navigable facts.

Create a gold set from maintainer-verified architecture, commands, ownership boundaries, and operational constraints. Score factual precision, factual recall, unsupported statements, broken links, duplication, and update freshness. Retrieval cannot recover facts that generation omitted or distorted.

The reviewed connector documentation describes ingestion from Git repositories and several external sources into a local raw cache. Agent-facing tools can list, ingest, and read connector material while constraining raw-file access to connector directories.

This creates useful context but also makes provenance important. Generated claims should remain traceable to their source type and collection time. Benchmark connector-enabled and repository-only runs separately so external context does not obscure code-documentation quality.

OpenWiki documents a DeepSWE harness that compares a baseline coding agent with an OpenWiki-augmented condition. The design uses paired conditions and separates experiments, which is the right shape for measuring whether generated documentation changes downstream task performance.

A complete report should add direct overhead: wiki-generation time, generation tokens and cost, documentation size, agent tokens consumed, retrieval calls, and net task latency. Better task outcomes may justify overhead, but that trade-off must be visible.

Select repositories across languages and size bands, pin each commit, and generate OpenWiki output from a clean environment. Measure cold and incremental generation time, tokens, cost, output size, changed-page accuracy, and invalid references.

Then run a fixed set of repository questions and coding tasks in baseline and OpenWiki-assisted conditions. Score source-grounded answer precision, task success, time to first relevant fact, total agent tokens, and stale-answer rate after controlled repository changes.

OpenWiki has an evaluation-friendly shape: generated artifacts are inspectable, repository refs can be pinned, instruction integration is visible, connector boundaries are documented, and paired downstream evaluation already has project scaffolding.

Our source-only conclusion is that “retrieval precision” alone is too narrow. The decisive metric is net agent utility after generation cost, factual quality, retrieval behavior, and maintenance freshness are combined. Runtime results remain necessary before making performance claims.