Two weeks ago I wrote about how I built a personal knowledge base to stop losing AI knowledge between sessions. The idea: instead of letting every conversation disappear, you distill what you learn into a structured Obsidian vault. Over time, Claude reads those notes before answering.
After publishing I received this message:
I realized that the post explained the *what* and *how*. But.. it really works better than just asking Claude without context?
So I built a small eval to find out.
But.. what is an eval?
Basically, it’s a way to stop guessing. Instead of saying “it seems to work fine,” you give the AI a set of questions and score the answers. It’s the reality check your system needs.
The setup
Two systems, same ten questions:
- System A (with vault): Claude gets the question + the relevant notes from my Obsidian.
- System B (baseline): Just Claude, no context.
The questions were auto-generated from the vault itself, so they are about things I actually care about.
The architecture
Three steps. (Full code on GitHub)
- Generation: A script reads my notes and creates a test set of questions.
- Inference: For every question, I get two answers: one with context, one without.
- Judgment: A third LLM (the “Judge”) looks at both answers without knowing which is which. It gives a score from 1 to 5 on three things: technical precision, personal grounding (how much it knows about me), and practical utility.
What I found
- Technical precision: Tied. The vault doesn’t make Claude “smarter” at general stuff. It already knows the docs.
- Personal grounding: System A wins by far. Without context, the answer is for anyone. With the vault, the answer is for me. It knows my stack, my tools, and my past decisions.
- Practical utility: System A wins. A generic answer is correct but doesn’t help you decide. An answer that knows your specific constraints is actually useful.
Is it worth it?
Yes, if your questions are specific to your work.
A second brain doesn’t make you smarter at general concepts. For that, the model is already good enough. What it does is eliminate the gap between what you know and what the model knows about your specific situation. It turns a “consultant” into a “partner.”
Beyond a personal second brain
The same idea could be applied at a company level as a shared brain. Imagine a project where the AI doesn’t just know the code, but the decisions behind it. When people leave or new members join, the context isn’t lost. You don’t just have documentation; you have the project’s history and rationale.