Atlas vs Graphify: does a code graph actually help?
In my post about systems thinking, I wrote about using AI to map a system and build an initial mental model of it. That sounds great, but there is another question behind it: what does the agent actually need to create a good map? Is reading the code enough, or does giving it a graph of the code make a real difference?
I ran an experiment comparing Atlas with Graphify. And I added a third option: an agent with plain filesystem search and file reading, no navigation skill. Because if I want to know whether adding a tool helps, I need to compare it with not having the tool at all.
The result wasn’t a simple win for Atlas. Plain navigation was the cheapest and fastest overall. Atlas used fewer harness calls. Graphify covered one extra behavior. Which sounds a little bit messy, but that’s actually the interesting part.
What I tested
The idea was to test navigation across three programming languages: Python, TypeScript, and Elixir. I wanted the agents to explain how things actually work, not just find a function or list dependencies. For Python, they followed indexing and persistence; for TypeScript, registration and graph rendering; and for Elixir, orders and LiveView updates.
For each language I ran three agents in fresh contexts, using the same model, GPT-6.1 Sol, with medium reasoning. One used Atlas and its latest navigation skill, one used Graphify 0.9.90 and its Pi skill, and one used plain search/read. They worked from identical clean source snapshots, with matched scope, maps of no more than 700 words, and a suggested budget of 25 investigation calls. All of them could read the source to check what they found.
An independent auditor checked the sessions and the answers against the source. There were twelve source-derived checkpoints per language, things like whether a write is atomic, what happens after a commit, or how the UI gets updated. I wanted to know whether the agent understood the behavior, not whether it produced a nice-looking explanation.
There is an important detail about Graphify here. This run used its directed AST code graph, with querying, paths and explanations available. It did not run the full semantic-document extraction, community naming and visualization pipeline. The agents also only partially followed its skill. So this is a comparison of these navigation setups, not a verdict on everything Graphify can do.
Fewer calls, but not fewer tokens
These are the totals across the three languages for each strategy:
| Strategy | Uncached input + output tokens | Harness calls | Summed agent time | Reported cost |
|---|---|---|---|---|
| Plain search/read | 127,850 | 75 | 530s | $0.435 |
| Atlas + skill | 136,198 | 59 | 591s | $0.475 |
| Graphify + skill | 208,324 | 69 | 686s | $0.692 |
The time is the sum of the three agent durations, not how long the concurrent batch took. Costs are Pi-reported estimates, not invoices. The token column excludes cached input; including it gives approximately 1.206 million tokens for plain, 1.273 million for Atlas and 2.195 million for Graphify. Plain was lowest with either definition.
Atlas reduced the harness calls by about 21% compared with plain navigation, but consumed about 7% more uncached input/output tokens. Graphify consumed about 63% more of those tokens and took about 29% more summed time than plain.
At first, fewer calls sounds like less work. But one harness call can contain multiple commands. Atlas’s 59 calls included 58 graph CLI invocations, while Graphify’s 69 included 24, counting help and save commands. So I can’t say Atlas performed fewer underlying operations just because the harness counter is smaller.
This made me think about how I measure work done by agents: a metric can be correct and still not mean what I initially think it means. Fewer exchanges with the harness is useful to know. It isn’t automatically less computation, less context, or a cheaper answer.
The aggregate also hides differences between the language-specific tasks. Graphify was fastest on the Python task, Atlas on TypeScript, and plain on Elixir. Atlas used the fewest uncached input/output tokens on Elixir, although plain still had the lower estimated cost there. There wasn’t one strategy that won across all three languages.
Did they understand the systems?
On checkpoint coverage, Graphify got 36 out of 36. Atlas and plain got 35 out of 36. All three covered every checkpoint in Python and TypeScript.
The difference was in Elixir. Graphify explicitly traced how session notifications reload the incoming order histories. Atlas and plain mentioned loading histories, but didn’t explicitly follow that notification-driven path. It’s a small difference in the score, but it is a real part of how the system stays synchronized.
Nevertheless, coverage is not precision. Getting twelve checkpoints right doesn’t tell me what percentage of every claim in the answer is true. The auditor checked errors separately, and that’s where the diagrams became interesting.
Both Atlas and plain produced an Elixir diagram with a misleading edge. The Atlas diagram implied factory stock production emits an order_created notification; the plain diagram grouped it with order_updated. In the source, factory production emits only player_updated. Their prose mostly got this distinction right, but the arrows didn’t.
Graphify’s diagrams had no major errors in the audit, although its answer had a minor TypeScript mistake about which library owns layout computation and some source references pointed to the wrong lines. Atlas’s source references were more accurate in the checked samples. None of this gives me a defensible overall precision winner.
I like diagrams because they help me think about a system. But an arrow is also a claim. If I combine two paths to make the picture simpler, I might accidentally invent a relationship that doesn’t exist. A nice diagram can give me confidence in the wrong mental model.
A graph is a map, not the behavior
The most useful learning for me is that finding relationships and understanding behavior are different things.
In the TypeScript code, registering a repository in the browser does not clone or index it. It stores metadata. Importing the architecture is a separate terminal operation, and the UI reads the stored snapshot. A graph showing the form, repository service and graph UI might help the agent find those pieces, but it doesn’t establish that separation by itself.
The same happens with transaction boundaries. In the Python code, analysis happens before storage is opened, and graph replacement is atomic. Yet writing the JSON response to stdout can fail after the database commit. In the Elixir code, order creation and weekly/session updates are separate commits, while broadcasts happen after the transaction. A relationship between functions doesn’t tell me which failure rolls back which change.
For those details, the agents still had to read the source. The graph could help with navigation, but it wasn’t a replacement for checking what the code actually does.
This connects with what I learned building agents: adding more tools isn’t the same as improving the system. What matters is how the harness uses them, what they put into the context, and whether they help with the particular task.
What I can take from this
I need to be honest about the limits. This was one run per combination, using one selected codebase for each language, with warm source and index caches. It doesn’t tell me how these tools perform on Python, TypeScript or Elixir in general. All nine agents ran concurrently on a shared machine. Provider errors affected some runs, especially Graphify on Elixir, so the timings include retries. I didn’t run repeated trials, cold-start comparisons or runtime verification of the applications.
Graphify’s pilot memory wasn’t reset before measurement, although the auditor saw no use of that memory in the measured runs. Its Python corpus also exceeded the skill’s suggested 500-file narrowing threshold. The source snapshots matched, but Atlas and Graphify didn’t index identical sets of languages and files. Setup timings excluded installation and semantic work. Those are reasons to keep the conclusion small, not hide them behind a ranking.
For these tasks, plain search/read was a strong baseline. Atlas reduced harness calls without reducing aggregate tokens or cost. Graphify found one additional synchronization behavior, at a higher aggregate cost. That’s what I can say from the evidence.
For Atlas, the next question I want to explore isn’t just whether it can expose more relationships. It’s whether it can help the agent get to the right source with less context, and whether that holds across repeated runs. I still see value in having a map of a system. But I want to measure when that map helps, rather than assume it must help because it’s there.
The full report has the setup, pinned source revisions and results for each language. The metrics and independent evaluation are there too.