--- type: idea title: Dissenting views on persistent agent wikis created: 2026-04-07 status: active origin_sources: - "[[clippings/26-04-06 - Github Repo - karpathyllm-wiki]]" - "GPT Researcher contrarian pass (2026-04-07)" related_auto: [] used_in: [] --- ## Claim The strongest dissent against a Karpathy-style persistent wiki is not "RAG wins." It is that a compiled knowledge layer can turn ordinary model errors into durable system state. Once bad summaries, poisoned memories, or stale interpretations are written back into the wiki, they stop being transient output and start behaving like infrastructure. ## Best contrarian case ### 1. Persistent memory creates a delayed attack surface The sharpest technical objection is memory poisoning. In a normal chat failure, the bad output dies with the session. In a persistent wiki or long-term memory system, the bad output can be stored, retrieved later, and silently influence future reasoning. This is not hypothetical. Recent work and security analysis explicitly describe: - poisoning long-term memory or knowledge bases so malicious content is retrieved later as if it were trusted context - indirect prompt injection that turns future sessions into a delayed compromise - cross-temporal attacks where the damage appears long after the original injection Strong sources: - Memory Poisoning Attack and Defense on Memory Based LLM-Agents (arXiv, 2026): https://arxiv.org/abs/2601.05504 - AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Bases: https://www.proceedings.com/079017-4136.html - Taming OpenClaw: Security Analysis and Mitigation of Autonomous LLM Agent Threats: https://arxiv.org/html/2603.11619v1 ### 2. A maintained wiki weakens provenance unless evidence stays attached A persistent wiki is basically a recursive summarization system. That gives it power, but it also creates compression risk. Summaries can be fluent, useful, and wrong at the same time. Once those summaries are edited into higher-level concept pages, the provenance chain gets thinner unless every durable claim still points back to supporting source digests. This is why the real danger is not only hallucination. It is hallucination lock-in. Older summarization work still matters here: - On Faithfulness and Factuality in Abstractive Summarization: https://aclanthology.org/2020.acl-main.173/ - Asking and Answering Questions to Evaluate the Factual Consistency of Summaries: https://aclanthology.org/2020.acl-main.450/ - Evaluating the Tradeoff Between Abstractiveness and Factuality in Abstractive Summarization: https://aclanthology.org/2023.findings-eacl.156/ ### 3. Drift is a state-management problem, not just a prompting problem The contrarian view says long-running memory systems fail because they accumulate state without enough governance. A wiki that keeps rewriting itself can drift semantically even when each individual edit looks locally reasonable. The GPT Researcher pass kept surfacing the same pattern: - memory integrity is now treated as a first-class security problem - point defenses are weak against multi-stage, cross-lifecycle failures - autonomous workflow drift is increasingly recognized as a distinct risk class Strong sources: - Taming OpenClaw: Security Analysis and Mitigation of Autonomous LLM Agent Threats: https://arxiv.org/html/2603.11619v1 - Results from the 2026 OWASP Top 10 for LLM Applications Survey: https://www.linkedin.com/pulse/results-from-2026-owasp-top-10-llm-applications-survey-steve-wilson-qorxc - GenAI security: How to protect LLMs from AI-powered attacks: https://www.vectra.ai/topics/genai-security ### 4. Evaluation gets harder when the system owns history Stateless retrieval systems are not easy to evaluate, but persistent wikis add a new problem: you are no longer evaluating a prompt-plus-retrieval flow, you are evaluating an evolving interpretation layer with hidden history. That creates several difficulties: - a bad concept page may stay plausible for weeks - the system can degrade gradually without a clean failure boundary - evaluation has to cover trajectories, revisions, and memory consistency, not just final answers - deployment behavior can diverge from evaluation behavior once the memory layer starts changing This is one reason the security literature is shifting toward lifecycle and governance language instead of single-turn prompt language. ### 5. Direct retrieval can be safer when freshness matters more than synthesis The best anti-wiki argument is often operational, not ideological. Query-time retrieval over canonical notes is easier to refresh, easier to inspect, and easier to verify against source material. A compiled wiki may be better for continuity and evolving theses, but it is usually worse as an unquestioned truth store. That does not mean "RAG is safe." Poisoned retrieval corpora are also vulnerable. It means direct retrieval over canonical documents is often the cleaner backstop when you care about recency, source fidelity, and claim verification. Useful framing sources: - Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks: https://arxiv.org/abs/2005.11401 - Rowen: Adaptive Retrieval-Augmented Generation for Hallucination Mitigation in LLMs: https://arxiv.gg/abs/2402.10612 ## Working synthesis The strongest dissenting view is: Persistent agent wikis are valuable as interpretation layers, but dangerous as silent authority layers. If the wiki becomes the thing you trust instead of the thing you inspect, it can: - launder stale summaries into canon - preserve poisoned or weakly grounded claims across sessions - make provenance harder to recover - hide gradual drift behind coherent prose ## What this means for this vault This does not kill the `research//` design. It sharpens the rules for it. The contrarian case implies that our system should: 1. Keep canonical source notes read-only. 2. Treat compiled concepts as interpretations, not ground truth. 3. Preserve contradictions instead of cosmetically resolving them. 4. Require source links on durable synthesized claims. 5. Keep direct retrieval over canonical notes available for verification and recency checks. 6. Promote concepts slowly, because promotion is a trust decision. That is why the right system is hybrid: - compiled knowledge for continuity, synthesis, and evolving theses - canonical notes plus retrieval for verification, provenance, and freshness ## Strongest sources from this pass - Memory Poisoning Attack and Defense on Memory Based LLM-Agents: https://arxiv.org/abs/2601.05504 - AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Bases: https://www.proceedings.com/079017-4136.html - Taming OpenClaw: Security Analysis and Mitigation of Autonomous LLM Agent Threats: https://arxiv.org/html/2603.11619v1 - Results from the 2026 OWASP Top 10 for LLM Applications Survey: https://www.linkedin.com/pulse/results-from-2026-owasp-top-10-llm-applications-survey-steve-wilson-qorxc - GenAI security: How to protect LLMs from AI-powered attacks: https://www.vectra.ai/topics/genai-security ## Related - [[research/agent-wiki/concepts/compiled-knowledge-layer]] - [[research/agent-wiki/concepts/simplicity-before-autonomy]] - [[research/agent-wiki/outputs/first-pass-thesis]] ## Draft hooks - Possible headline: Persistent wikis are powerful, but they can turn model errors into infrastructure - Possible angle: The real enemy of trustworthy memory is not forgetfulness, but ungoverned state - What this is really about: how to compound insight without compounding corrupted history