I'm building an AI agent that manages my Obsidian vault through natural language. It can search notes, read content, create files, move things around - all from a chat interface inside Obsidian. The whole point was to keep my notes private.
Then I switched from a local LLM to Anthropic's API because the local model was painfully slow. That forced a question I'd been avoiding: what data is actually leaving my machine?
Here's what I found.
The Architecture
My setup has three layers:
- Obsidian with the Copilot plugin as the chat UI
- A local FastAPI server that runs a PydanticAI agent with three vault tools
- An LLM provider - either Ollama running locally or Anthropic's cloud API
When I type a message in Obsidian, the Copilot plugin sends it to my local server. The server runs the agent, which can search, read, and modify vault files. The LLM generates the responses.
The privacy question comes down to: at each layer, what gets sent where?
Layer 1: Obsidian Itself
Obsidian is fully private. Their privacy policy is unusually clear:
- No telemetry
- No data collection
- No vault content ever sent to their servers
- Internet connections only for update checks (which you can disable)
The only exception is Obsidian Sync, which is opt-in, paid, and end-to-end encrypted. I'm not using it.
Your .md files sit on your local filesystem. Obsidian reads them directly. Nothing phones home.
Layer 2: The Copilot Plugin
This was my main concern. The Obsidian Copilot plugin is the chat interface that talks to my agent server. Does it scan my vault in the background? Does it index notes and send them somewhere?
No. The plugin only sends data when you actively send a chat message. What gets included in each request:
- Your typed message (always)
- The currently open note (only if you toggle "Include Active Note" - off by default)
- Specific notes you reference with
@notementions - Chat history from the current conversation
That's the complete list. Notes you don't mention, don't reference, and don't have open are never touched by the plugin. There's no background indexing, no silent vault scanning, no embeddings being generated and shipped off.
The plugin does have a Vault QA feature that can search your notes, but even that uses local lexical search (BM25) by default. The search happens on your machine. Only the results you interact with get sent to the LLM.
There was a previous concern about the plugin sending queries to a third-party server called Brevilabs for intent analysis. That code has since been fully removed from the codebase. Current versions don't contact Brevilabs unless you're using their paid Plus tier features (PDF parsing, web search).
Layer 3: The LLM Provider
This is where data leaves my machine - and where the privacy tradeoff lives.
When I was running Ollama locally, the entire pipeline stayed on my laptop. Messages went from Obsidian to my server to the local model and back. Nothing left the machine. Fully private.
When I switched to Anthropic's API, the conversation content now transits their servers. What Anthropic receives:
- The system prompt (my agent's instructions)
- The chat messages (my questions and the agent's responses)
- Tool call results - if the agent searches my vault or reads a note, that content gets sent to Anthropic so it can formulate a response
Per Anthropic's data policy, API inputs are not used for model training. But the data does pass through their infrastructure.
The Privacy Boundary
Here's what this means in practice:
Notes I never discuss with the agent are fully private. They're markdown files on my disk. No plugin, no server, no API touches them. They don't exist to the system unless I bring them into a conversation.
Notes I do discuss get sent to whatever LLM provider I'm using. If I ask "summarize my project notes," the agent reads those files from disk and sends the content to Anthropic as part of the conversation context.
The boundary is simple: the chat is the gate. Nothing crosses it unless I initiate the conversation.
The macOS Permissions Side Quest
While setting this up, I ran into a macOS issue that's worth mentioning. My vault lived in ~/Documents, which is a TCC-protected directory on modern macOS. My IDE couldn't access it without Full Disk Access - a permission that felt way too broad.
I tried a symlink from an unprotected directory into ~/Documents. Didn't work. macOS resolves symlinks to their real target path before checking permissions. Apple has been actively patching symlink-based TCC bypasses.
The fix was simple: move the vault out of ~/Documents to a user-created directory that isn't TCC-protected. No permission prompts, no Full Disk Access grants. The vault works the same - Obsidian doesn't care where the folder lives.
If your vault is in ~/Documents, ~/Desktop, or ~/Downloads and you're running into permission issues with development tools, just move it. Open Obsidian, use the vault switcher, point it at the new location. Everything (notes, plugins, .obsidian config) travels with the folder.
What I'd Tell Someone Building This
If you're connecting an AI agent to your personal notes:
-
Obsidian is not the privacy risk. It's one of the most privacy-respecting apps I've used. No telemetry, no cloud dependency, local-first by design.
-
The Copilot plugin is not the privacy risk. It doesn't scan or index your vault behind your back. It sends what you tell it to send.
-
Your LLM provider is the privacy boundary. Local model = fully private but slow. Cloud API = fast but your conversation data transits external servers. Pick based on the sensitivity of what you're discussing.
-
You can have both. My agent supports switching between Ollama and Anthropic with a single environment variable. Sensitive notes get the local model. Everything else gets the fast one. The architecture doesn't force a choice.
The whole point of building your own agent instead of using a SaaS product is that you control the pipeline. You decide what crosses the boundary and what stays local.
This is the kind of problem I enjoy solving - understanding systems deeply before building on top of them. I take the same approach when building websites for contractors. Want to know more? Read about how I work.
