--- type: output topic: agent-wiki status: active created: 2026-04-12 updated: 2026-04-12 source_notes: - "[[research/agent-wiki/health-check]]" - "[[research/agent-wiki/outputs/wiki-vs-rag-comparison]]" derived_from: - "web research pass, 2026-04-12" confidence: medium output_kind: gap-scan --- # Technical Gap Scan ## Question What technical gaps remain in `research/agent-wiki` after comparing the current compiled-wiki thesis against recent work on RAG, agent memory, attribution, and poisoning? ## Short Answer The workspace has a good conceptual thesis, but it is still missing implementation evidence. The strongest next move is to ingest one or two technical sources that convert "governed interpretation layer" from a principle into concrete controls: memory lifecycle rules, provenance enforcement, adversarial ingest testing, and RAG-style evaluation. ## Missing Gaps ### 1. Memory lifecycle controls The workspace says persistent synthesis can compound value, but it does not yet define how memories or concept updates should be added, revised, deleted, or quarantined. Useful source: [How Memory Management Impacts LLM Agents](https://arxiv.org/abs/2505.16067) studies memory addition and deletion. Its most relevant finding is that agents can follow similar retrieved experiences too strongly, which creates error propagation and misleading experience replay. Implication for this workspace: add a lightweight memory lifecycle page or rules section covering: - when a source digest can update a concept - when an old synthesis should be deprecated instead of merged - when a concept needs a retrieval check before reuse - whether failed future answers should become negative examples for memory hygiene ### 2. Adversarial ingest and knowledge-base poisoning The workspace already names memory poisoning as a risk, but it has not grounded that risk in concrete attack models. Useful sources: - [PoisonedRAG](https://www.usenix.org/conference/usenixsecurity25/presentation/zou-poisonedrag) treats the RAG knowledge database as an attack surface and shows that a few injected malicious texts can steer answers. - [CPA-RAG](https://arxiv.org/abs/2505.19864) extends the concern to black-box attacks that generate query-relevant adversarial texts. - [A Few Words Can Distort Graphs](https://arxiv.org/abs/2508.04276) shows that GraphRAG-style structure is also vulnerable when upstream text is manipulated before graph construction. Implication for this workspace: a markdown wiki should not ingest every source directly into concepts. It needs an ingest quarantine: - source digest first - contradiction scan second - concept update third - high-confidence promotion only after source provenance and conflict checks ### 3. Citation and attribution faithfulness The current notes treat source links as useful, but they do not distinguish between "the cited source supports the claim" and "the model actually relied on the cited source." Useful sources: - [Correctness is not Faithfulness in RAG Attributions](https://arxiv.org/abs/2412.18004) argues that citation correctness alone is insufficient and reports attribution faithfulness failures. - [LLatrieval](https://www.cs.jhu.edu/~kevinduh/t/naacl24/final_pdf/paper595.pdf) uses LLM-driven retrieval verification and update loops to improve verifiable generation. Implication for this workspace: concept pages should not merely list supporting sources. For stronger claims, they should map claims to source spans or source digest bullets, and the workflow should include a "retrieve then verify" mode before high-stakes reuse. ### 4. Evaluation harness for the hybrid model The workspace currently has a thesis that the compiled wiki should orient and canonical retrieval should verify. It lacks an evaluation harness to test whether this actually works. Useful sources: - [ARES](https://aclanthology.org/2024.naacl-long.20/) evaluates RAG systems along context relevance, answer faithfulness, and answer relevance, using tailored judges and a small validation set. - [RAGChecker](https://arxiv.org/abs/2408.08067) proposes fine-grained diagnostic metrics for retrieval and generation modules. Implication for this workspace: create a small local eval set: - 10 recurring questions over `agent-wiki` - expected source notes for each question - expected concept pages for orientation - failure labels for stale citation, unsupported claim, missed contradiction, and answer drift ### 5. Source reliability and recency policy The workspace says direct retrieval is the backstop for freshness, but it does not say how to rank source reliability or what happens when a trusted source changes. Useful source: RA-RAG estimates source reliability and incorporates it into retrieval and answer generation, but also notes the need for periodic or online updates when source reliability changes. Implication for this workspace: add source trust tiers: - canonical source - current technical article - speculative essay - dissenting view - unverified clipping Then make recency and reliability visible in source digests. ### 6. Markdown-first scaling evidence The existing corpus still lacks a concrete implementation case study for a maintained markdown-first wiki. MemGPT and Mem0 are relevant memory-system references, but they are not the same as a local Obsidian-style compiled wiki. Useful sources: - [MemGPT](https://arxiv.org/abs/2310.08560) frames agent memory as virtual context management across memory tiers. - [Mem0](https://arxiv.org/abs/2504.19413) argues for structured, persistent memory for long-running conversational agents and reports latency/token-cost benefits versus full-context approaches. Implication for this workspace: do not assume memory-system results transfer cleanly to markdown compilation. The missing evidence is a small local prototype: compare answer quality, source traceability, and maintenance time for compiled-wiki answers versus direct retrieval answers over the same vault subset. ### 7. Promotion criteria as a first-class governance object Promotion is currently embedded across `governed-interpretation-layer`, `compiled-knowledge-layer`, and the health check. It probably deserves its own concept page if this workspace continues. Potential page: `concepts/promotion-as-trust-decision.md` Core rule: promotion is not deduplication. It is a claim that a concept has become reusable, source-backed, and stable enough to shape work outside its originating research folder. ## Best Next Ingest Ingest these in order: 1. [How Memory Management Impacts LLM Agents](https://arxiv.org/abs/2505.16067) for memory lifecycle controls. 2. [PoisonedRAG](https://www.usenix.org/conference/usenixsecurity25/presentation/zou-poisonedrag) for concrete adversarial ingest risks. 3. [ARES](https://aclanthology.org/2024.naacl-long.20/) or [RAGChecker](https://arxiv.org/abs/2408.08067) for an evaluation harness. 4. [Correctness is not Faithfulness in RAG Attributions](https://arxiv.org/abs/2412.18004) for claim-to-citation governance. ## Practical Next Move Create one new source digest for the memory-management paper, then update `governed-interpretation-layer` with explicit lifecycle controls. After that, add a tiny eval file with recurring questions and failure labels. That would move the workspace from "good thesis" to "operational research system."