--- type: output topic: agent-wiki status: active created: 2026-04-12 updated: 2026-04-12 source_notes: - "[[research/agent-wiki/health-check]]" - "[[research/agent-wiki/outputs/technical-gap-scan-2026-04-12]]" derived_from: - "GPT Researcher session research-c60cdbc01708" confidence: medium output_kind: research-rerun --- # GPT Researcher Rerun ## Question What implementation-heavy technical sources should this workspace use to close the remaining gaps around persistent agent memory, provenance, poisoning, citation faithfulness, and RAG evaluation? ## Run Summary GPT Researcher completed a full deep-research run after switching the skill to the current Anthropic key, Tavily search, and local HuggingFace embeddings. - Research ID: `research-c60cdbc01708` - Report: `C:\Users\darsh.shah\.codex\skills\gpt-researcher\sessions\research-c60cdbc01708\report.md` - Sources: `C:\Users\darsh.shah\.codex\skills\gpt-researcher\sessions\research-c60cdbc01708\sources.json` - Source count: 16 - Source URL count: 19 Quality note: the run found useful technical material, but it also scraped some lower-quality or inaccessible sources. Treat the report as a scouting pass, not as a promoted source digest. ## High-Signal Sources - [AgentPoison](https://arxiv.org/abs/2407.12784) - Strong source for knowledge-base and long-term-memory poisoning. It reports high attack success against RAG-based agents with very small poison rates, so it is directly relevant to the workspace's governance concern. - [MemoryGraft](https://arxiv.org/html/2512.16962v1) - Strong source for persistent experience-memory poisoning and the risk that agents imitate retrieved "successful" procedures. It is useful for designing memory lifecycle and quarantine rules. - [Hugging Face RAG Evaluation Cookbook](https://huggingface.co/learn/cookbook/rag_evaluation) - Practical source for a local evaluation harness: generate QA pairs, filter them with critique agents, run RAG variants, and score outputs. - [Confident AI RAG evaluation metrics](https://www.confident-ai.com/blog/rag-evaluation-metrics-answer-relevancy-faithfulness-and-more) - Useful practitioner source for distinguishing faithfulness, answer relevance, contextual relevance, recall, and precision. - [Snowflake EvaluateRagFaithfulness](https://docs.snowflake.com/en/user-guide/data-integration/openflow/processors/evaluateragfaithfulness) - Useful implementation reference for treating faithfulness evaluation as a processor/pipeline step. - [Evaluating Faithfulness in Agentic RAG Systems for e-Governance](https://github.com/gpapageorgiouedu/Evaluating-Faithfulness-in-Agentic-RAG-Systems-for-e-Governance-applications-LLM-Based) - Useful implementation reference for statement-level faithfulness checks across agentic RAG pipelines. ## Missing Gaps Confirmed ### 1. Memory lifecycle is now the highest-priority gap The existing workspace says the compiled wiki should be governed, but it does not yet say what happens to a memory or concept once it enters the system. Needed controls: - source digest before concept update - quarantine for untrusted or adversarially interesting sources - deprecation path for stale claims - contradiction register before merge - retrieval check before high-stakes reuse ### 2. Poisoning should be modeled at the retrieval layer, not only the prompt layer AgentPoison and MemoryGraft both push against a prompt-only security model. The risk is not just "the user prompt is malicious"; the risk is that retrieved memory can become the attack delivery mechanism. Implication: `governed-interpretation-layer` should explicitly include retrieval-time checks, such as provenance confidence, source trust tier, recency anomaly, and "new source cannot directly overwrite promoted concept" rules. ### 3. Citation faithfulness needs claim-level checks The current concept pages list supporting sources, but they do not map individual claims to support. The research pass reinforces that this is too weak for promoted or high-stakes concepts. Practical rule: local concept pages can keep lightweight source lists, but promotion candidates should include a claim-support map or a short "evidence table" before being moved outside the workspace. ### 4. The evaluation harness should be tiny and local first The best next implementation is not a full RAG benchmark. It is a small `evals/` file inside `research/agent-wiki` with recurring questions and expected evidence. Suggested starter eval set: - "What is a compiled knowledge layer?" - "When should compiled wiki answers fall back to retrieval?" - "What can poison a persistent agent wiki?" - "What is the minimum provenance rule for concept promotion?" - "Which claims in this workspace are weakly supported?" Each eval should record: - expected concept pages - expected source digests - forbidden unsupported claims - failure labels: stale citation, missing contradiction, unsupported claim, overconfident promotion ### 5. Promotion deserves a dedicated governance concept The rerun strengthens the case for `concepts/promotion-as-trust-decision.md`. Promotion is the point where a local interpretation starts shaping wider work, so it should require higher evidence and lifecycle discipline than ordinary workspace notes. ## Updated Recommendation Do not ingest the full GPT Researcher report as a source digest. Instead: 1. Ingest `AgentPoison` as a focused source digest on memory/RAG poisoning. 2. Ingest `MemoryGraft` as a focused source digest on persistent experience-memory poisoning. 3. Add `concepts/promotion-as-trust-decision.md`. 4. Add `evals/agent-wiki-hybrid-eval.md` with 5-10 recurring questions. 5. Update `governed-interpretation-layer` with explicit lifecycle controls: source quarantine, contradiction scan, retrieval check, and promotion gate. ## Residual Tool Notes The GPT Researcher skill is now usable with: - Tavily retrieval - Anthropic LLM config - local HuggingFace embeddings Remaining noise: - HuggingFace warns about unauthenticated downloads because no `HF_TOKEN` is configured. This is harmless for the current local embedding model, but an HF token would improve rate limits. - The underlying GPT Researcher package emits a nonfatal coroutine warning during cost logging.