Matato Agent
Over the last few months I have been experimenting with building my own AI agent, one that can do things for me in the background 24/7. What I have as a result is a small, easy-to-upgrade, self-improving, self-organizing agent: a blog writer and coder that runs on a computer isolated from my main workspace, and that I can chat with over Telegram.
The application is a simple TypeScript project with a lot of markdown files that uses the PI-SDK as its main harness. No database, no crazy access, but easy to expand and to grant more permissions or access to other tools.
This started as an experiment to find out the best way to build such an agentic system: from programming language to harnesses, architecture, patterns and more. The journey has been great, and in this post I will walk you through the learnings.
How it started
The first step was reading about the different harnesses I could use. I investigated for a while, until I decided to go with LangChain and simple agents. It seemed nice and easy, and the mental model of agents in LangChain matched my previous experience working with the OpenAI Agents SDK and Convex agents, so I knew something like that could work.
It started capturing messages from Telegram and answering them. I gave it access to a couple of tools like web-search, and the defaults from the harness like bash.
Then I added a few more features. I wanted to migrate my quick-note app, which is a note-taking web application for random small ideas (not a Notion or Obsidian replacement), and to build a tasks section. I added tools and started a programmatic adventure to give access to a storage layer, using SQLite.
At this point everything was mainly done via code and a bunch of sub-agents. First there was a sub-agent that would classify the intent of the message: is it a query? is it a command? is it about creating a task or saving a note? This would produce a structured output with a predefined schema, and pass it to a part of the project that would decide which underlying agent to call. The underlying agents would then call other agents, like in the case of notes, or call tools such as CRUD operations on entities in the database.
The UI
Since I wanted to migrate an existing web application, I created a UI that would work like the one in my original quick-note. I used TanStack, running locally with Prisma pointing to my SQLite database. Basically I copied a lot of code from my original quick-note application, plus some other things. And that comes with authentication too. If I’m going to have a user interface, I’ll need authentication as well. And that’s where things actually started getting a little bit messy.
I still wanted to have access to the agent in a binary way, not only through a UI but also with the option to run it locally on any computer without having to start the UI. For this I started with two different modules: a front-end module and an API module that would expose access to the agent.
And at this point I thought, how could I run the agent? Where would I deploy it? I decided to look for something where the background job of the agent could live nicely alongside the UI, and I found Railway: nice tool, great user experience.
A mid journey of learnings
The Elixir
But as I needed to deploy multiple applications and move back and forth, I asked myself: why? That’s when I thought about Elixir. I love it and I had always wanted to build something useful with it, apart from the simple applications I build in my free time. So I thought maybe this could be a great opportunity to try it out, and I gave it a shot. I migrated everything to Elixir because why not, the AI does the typing. I found it interesting and I developed a new application, now written in Elixir. I used Jido, which is an agent framework that also has an AI module on top. You have tools, skills, heartbeat, and multiple kinds of agents for multiple kinds of workflows.
After weeks of working with the Elixir version and having it running for a while, I started noticing the agent wasn’t always capturing the right intent. It would do things incorrectly, use the wrong tool or the wrong skill. I need to be honest: I believe it was a skill issue (a human one) and not a tool issue. Nevertheless, when working with Jido I saw the power of a good harness, and I ended up removing a lot of code like the intent classifier and a bunch of sub-agents.
The problem I had was that the coding harnesses I was using weren’t doing things correctly. There were a lot of strange patterns, mistakes, etc. I’m not as good in Elixir as in other languages I use every day, and I ended up fighting the AI to take the direction I wanted more than actually making progress.
Since this was an AI experiment — first to use more AI for coding, and second to build my AI assistant — I wanted to use a programming language where the coding agent would perform better and that had a rich ecosystem for the problem space. So I moved back to TypeScript.
Deep Agents
Another reason to migrate everything back to TypeScript was to see if, with DeepAgents from LangChain, I would be able to get more interesting, more deterministic results by removing a lot of code and giving more freedom to the tool. Which sounds counterproductive of course: you’re looking for some predictability, some determinism in the process, and what you do is give more freedom to the non-deterministic machine…
DeepAgents wasn’t quite right. It’s an interesting tool but the agent wasn’t exactly what I was looking for. By design, DeepAgents are slow and they do a lot of thinking. That’s not bad in itself, but for what I wanted it wasn’t optimal. At the same time, I started removing more code and focusing on sub-agents and skills. Yes, one big learning is that most of your code can be replaced by a markdown file.
PI Agent
This is when I first heard about PI Agent and its SDK. At first I said, hey, this looks like a nice harness, and I started playing with it on the side. I found it so powerful that I dropped DeepAgents and the LangChain ecosystem for it. Also, at this point I already had a bunch of UI, DB access, background jobs, etc., and everything was nicely modularized, so it was easy to drop and replace.
PI Agent is just simple, easy to use, and works great with minimal configuration. It comes with basic tooling in place, it supports skills, and it has a large range of providers. It’s designed to be expandable: start small and adapt as you need. But the most important lesson from it was that simplicity matters when working with AI harnesses: the less, the better.
Arriving back at square one
As soon as I saw how good the results were, and how good the harness was at just using any tool, something clicked in my brain, and that’s when I understood agentic systems. It’s not about multi-agents, or big fat agents that do everything and can orchestrate a swarm of micro-agents. It’s about having a single good harness that can do the heavy lifting, and finding ways to reuse it as much as you can. So I dropped everything and kept a simple background app that can do everything.
Having a single harness that works great, is strong, and can do multiple kinds of tasks is way simpler than trying to orchestrate a bunch of small agents that specialize in one thing.
I deleted all the UIs. I decided to keep working with my quick-note app, since due to its nature and simplicity it’s way easier to use AI there as a simple amplifier, instead of trying to solve the full note-taking space in one prompt. I have been adding features to quick-note and that will be in a future post. Now Matato Agent uses its HTTP API to call quick-note.
I removed the database and I’m working with simple JSON files that the project (or a bit of code) creates at runtime. By deleting the database I also deleted my Railway deployment, and now I deploy on a computer I have at home. It has full access to the OS, it has access to other projects as well as itself, running 24/7. It generates posts from my notes, it tracks my tasks, it creates GitHub issues to fix itself or improve other systems. All this using just simple tools like bash, or custom tools I create.