# Why a Coding Agent Needs Three Different Ways to "Read" Code

If you're building something like Devin, an agent that goes end-to-end from a task description to a working code change. The very first problem you hit isn't code generation. It's *comprehension*. Before an agent can safely touch a codebase, it has to be able to answer basic questions: Where is this function defined? What calls it? What does it return? Is this file even valid right now?

The instinct is to just hand the model `read_file` and let it grep around. That works for a five-file toy project. It falls over immediately on anything real the model burns its context window reading whole files to find one function, and it has no way to know that the `parse_config` it just read is a *different* `parse_config` than the one being called three directories away.

This experiment tackles that problem with two tools that sound similar but do fundamentally different jobs: **tree-sitter** and the **Language Server Protocol (LSP)**. Here's the reasoning behind using both, not just one.

## Tree-sitter: fast, local, and doesn't need to know your project

Tree-sitter's job in this script is narrow: given a file and a symbol name, parse just that file into a syntax tree and find the node whose name matches. That's `find_symbol`. Under the hood it's simple parse the file's bytes into an AST with the right grammar (`tree_sitter_python`, `_go`, `_javascript`, `_typescript`), walk the tree looking for definition-shaped nodes (`function_definition`, `class_declaration`, `method_definition`, etc.), and compare each candidate's name field against the string the agent asked for.

The reason this tool exists at all, alongside a full language server, comes down to **cost and availability**:

*   **It's instant.** No subprocess, no protocol handshake, no project-wide indexing. Parsing one file with tree-sitter takes milliseconds. Spinning up `gopls` or `pyright` and waiting for it to build a full workspace model does not.
    
*   **It doesn't need the rest of the project to be coherent.** A tree-sitter parse only needs valid-ish syntax in *this file*. It works before dependencies are installed, before `gopls` has finished indexing, even inside a codebase with broken imports elsewhere.
    
*   **It gives the agent a foothold for everything else.** This is the more interesting design choice: `find_symbol` doesn't just return source code. It returns the **1-based line/character position of the symbol's name**, specifically so that position can be fed straight into `goto_definition`, `find_references`, or `hover` without a separate lookup step. That's a small detail, but it matters LSP calls are positional (a line/column in a file), and models are bad at counting characters by eye from a text dump. Tree-sitter becomes the thing that hands the model *correct coordinates* to hand back to the LSP.
    

Notice also what tree-sitter deliberately *doesn't* try to do: cross-file resolution. It can't tell you that the `Client` class used in `handlers.py` is actually imported from `client/base.py`. That's not a limitation to work around — it's the boundary that defines why the LSP is there too.

## LSP: slow to start, but it actually understands the project

The four LSP-backed tools — `goto_definition`, `find_references`, `hover`, `get_diagnostics` — exist for exactly the question tree-sitter can't answer: *how does this symbol relate to the rest of the codebase?*

This is where most of the script's complexity lives, and it's worth explaining why, because "just call the language server" hides a lot of real engineering:

**LSP servers are long-running, stateful processes, not stateless functions.** `LSPClient` spawns something like `pyright-langserver --stdio` or `gopls serve` and talks to it over JSON-RPC framed with `Content-Length` headers on stdin/stdout the same protocol VS Code uses. That process has to be initialized once (`initialize` → `initialized` handshake), told about the workspace root, and then kept alive for the life of the session because it's continuously building an in-memory model of the project (symbol tables, import graphs, type inference). Treating it like a one-shot subprocess call would mean re-indexing the whole project on every single query, which is untenable.

**Servers talk back, not just respond.** This is why there's a background reader thread and a notification queue instead of a simple request/response wrapper. Diagnostics (`textDocument/publishDiagnostics`) arrive *unprompted*, whenever the server feels like re-analyzing a file — not as a reply to a specific request. Some servers (pyright, notably) also make *requests back to the client*, like `workspace/configuration`, and will stall indefinitely if nobody answers. The `_notification_listener` thread and `_handle_server_request` method exist specifically to keep that conversation alive in both directions while the main thread is free to do other things.

**Warm starting is a deliberate latency hedge.** As soon as the sandbox detects a project's primary language (via manifest files like `pyproject.toml`/`go.mod`, falling back to extension counts), it kicks off `warm_start` in a background thread because spawning the server and `didOpen`\-ing up to 500 source files *before the agent has asked a single question*. The comment in the code makes the reasoning explicit: `gopls` and `pyright` both do most of their expensive work (project graph loading, type inference) as a side effect of being told what's open, so the win isn't a special "index everything now" API — there isn't one for `typescript-language-server` — it's just front-loading that same cost so it overlaps with the agent's early exploration turns instead of blocking the first real query.

**Every server is configured slightly differently, and the code doesn't pretend otherwise.** `pyright` needs an explicit `workspace/didChangeConfiguration` push to turn on whole-workspace diagnostics instead of open-files-only. `gopls` doesn't need that because it indexes the whole module by design. `typescript-language-server` doesn't have a workspace diagnostic mode at all, warm-starting it only buys you a pre-loaded `tsserver` process. The `ServerConfig` dataclass and per-language `settings` dict exist to hold these differences explicitly rather than hard-coding one server's assumptions everywhere.

**Coordinates need translating both ways.** Tools are exposed to the model as 1-based line/character, because that's what a human (and apparently a model) naturally reasons about from reading numbered source. LSP itself is 0-based. So every LSP-backed method in `Sandbox` does the `-1` on the way in and `+1` on the way back out — a small but easy-to-get-wrong detail that's worth calling out because it's exactly the kind of bug that would silently point the agent at the wrong line.

## Why not just use one of them?

Because they answer different questions at different costs, and an agent loop is sensitive to both:

|  | Tree-sitter (`find_symbol`) | LSP (`goto_definition`, `find_references`, `hover`, `get_diagnostics`) |
| --- | --- | --- |
| Scope | Single file | Whole indexed workspace |
| Startup cost | ~0 | Seconds to tens of seconds (process spawn + indexing) |
| Understands imports/types | No | Yes |
| Needs project to be "healthy" | No | Mostly no, but benefits from it |
| Best for | "I know the name, give me its code and a position" | "Where does this actually come from / who uses this / is this broken" |

The system prompt actually encodes this tradeoff directly for the model: *prefer the cheapest tool that answers your question*, and reach for LSP specifically when the question is relational rather than local. That's the real design insight here — it's not "LSP is better than tree-sitter" or vice versa, it's that a code-understanding agent needs a cheap *local* lookup and an expensive *global* one, and conflating them either wastes tokens re-reading whole files or wastes time and process overhead asking a language server questions a regex could've answered.

## The shape this sets up for the bigger project

For an end-to-end coding agent, this comprehension layer is the foundation everything else stacks on. You can't safely rename a function, fix a call site, or reason about blast radius without `find_references`. You can't verify a generated edit didn't break type-checking without `get_diagnostics`. And you can't afford to have the model read entire files every time it needs one function's body — that's context budget you'll want later for the actual editing and reasoning steps.

Worth flagging as open questions for where this goes next: the diagnostics cache and `_opened_files` set live in memory for the session, so nothing currently invalidates or re-fetches a file's diagnostics after the agent (eventually) edits it — that'll matter as soon as writing code enters the loop. There's also no `textDocument/didChange` handling yet, only `didOpen`, which is fine for a read-only exploration tool but will need to change the moment the agent starts making edits and wants fresh diagnostics without a full re-open.