Before moving to agentic engineering, I was a traditional enterprise software engineer. Right out of college, I started in the best place possible – building a hypervisor in a small startup. Then on to adventures in startups and large companies as a technical lead and people manager. Over the last 16 years I’ve worn most hats in an Engineering org and a few Product ones, too. Now, as the CTO of Gas City, Inc., my focus is split. I use my experience to plan and architect the next release of our OSS projects as well as our upcoming commercial products. But I don’t write or read any of the code anymore. I’m no longer a “developer” in that sense. Instead, I’ve become the chief operator of the software factory that writes the code, runs the tests and ships the releases, as well as handling the incoming issues and PRs.
At any given moment, hundreds of agents are designing, implementing and reviewing inside the software factory that runs our open-source Beads and Gas City projects. We burn through billions of tokens a day, and that turns into hundreds of pull requests that need to be merged. That’s just me, running one factory.
Without agents, solo developers and small projects never ran into this problem. It was something only big companies had to deal with. Google built TAP for it. Uber built SubmitQueue.
You used to need a building full of engineers to generate hundreds of PRs a day. Now you need a laptop and a token budget.
I am one of those developers running into large company problems and I love it. I instrument, analyze, find the bottleneck, and use my experience to guide my agents to remove it. Lately, the bottleneck standing in my way has been CI. The crush of incoming PRs completely ground our CI to a halt: constant merges, constant rebases, a red main, and far too much human involvement.
The Number We Optimize
“What can be measured can be managed”, as the saying goes. I’ve come to believe that the number that matters most is the time from the first line of code written to that code landing in main. That window is your conflict exposure. It’s how long one agent’s work can break another’s, and how long a broken test can hide. With hundreds of agents working at once, the window is never empty: something is always in flight next to you. But you can still make that window as small as possible. Steve Yegge saw the same thing building Gas Town: “As soon as you start swarming workers, you run into the Merge Queue (MQ) problem.” Uber measured it at scale:
Agents squash the front of that window. Ideas become code in minutes. Our factory and agentic workflows handle all code reviews. But we still have to run the tests. CI sits right on the critical path.
And every time an agent has to stop and ask a human, “there’s an incompatibility here, what do you want to do?”, the whole change waits on the most expensive resource in the system: human attention.
Following the Data, One Bottleneck at a Time
Everything Everywhere All At Once
I got myself into this mess because I didn’t spend time thinking about CI design. I let the agents decide for themselves, and agents test everything. Every PR ran the unit tests, the integration tests, a full end-to-end suite and even expensive Docker-based tests. Then it ran all of it again after merge, and again any time there was a merge conflict.
The data was ugly. A single Beads CI run could burn over 100 minutes of compute. When agents merged in waves, PRs sat in GitHub’s runner queue for an hour and a half or more before a single test started.
Agents are happy to pile on more tests without stepping back to simplify or refactor the ones they have. Without guidance, they don’t reach for the abstractions that make code cheap to test: pure functions, and interfaces that can use test doubles for speed. They write another integration test, which isn’t a problem until each one takes minutes and runs on hardware that can’t handle them all in parallel.
Since April, the number of test functions in Gas City has grown by 600%, and more than half of our two million lines are now tests.
None of this is the agents’ fault. Testing strategy is a judgment call: what each phase is for, how much risk you’ll carry to main, what a failure costs you. That judgment wasn’t written down anywhere an agent could read it. So my first fix wasn’t technical at all. It was a conversation.
Try it. “I’d like to codify our testing strategy so we test at the right level: faster iteration, faster PRs, more maintainable software. Inventory our tests by type with counts and typical run times. Then interview me, one question at a time, about how I think about test quality, risk, and what belongs in PR checks versus post-merge. Write the result into our agent instructions so future agents follow it.”
Bottleneck moved to: deciding what runs where.
The Standard Playbook
My first real fixes were standard industry practice from the days before agents. Build a real test pyramid: mostly fast unit tests, fewer integration tests, end-to-end tests only where nothing else will do.
Then split the phases. On each PR, run the subset most likely to catch a bug. After merge, on main, run the rest, and accept that some failures will land there. There will be fewer of them, and at the end of the day the whole system moves faster.
What made this hard to get right was the lack of data. If you were already set up to capture every test and its timing, you could query it to find the hotspots. If you weren’t, the problem was too big, so you relied on intuition. With agents, exhaustive analysis is cheap. They are happy to read through months of old CI runs and logs, correlate failures with test types, and help you find the efficient split between PR and main.
It got me about half.
Try it. “Using our CI history (the last few hundred runs, their logs and their failures), work out which tests actually catch regressions and when. For each test group, report how often it fails, whether those failures were real bugs or flakes, and how long it takes. Propose which groups gate every PR and which run after merge, with the time saved and the risk we take on. Show me the data before changing any workflow.”
Bottleneck moved to: the machines.
More Runners
The next fix was the bluntest one: run more of the test suite in parallel trading compute and cost for latency. I split the slow jobs into shards and spread them across more runners. GitHub gives open-source projects runners for free, so parallelism costs nothing but a config change (there are hidden priority queues and parallelism constraints, but you get the idea). Then the free runners ran out. In our busiest merge waves, jobs waited in line for a runner to exist at all. Moving our CI to Blacksmith runners took the slowest tenth of our job starts from 28 minutes of waiting to about 10 seconds. Shout out to Blacksmith for sponsoring our Beads and Gas City open source projects! We couldn’t have made it through this phase without you.
Try it. “Review our CI workflows and recent run timings. Find where jobs or tests run serially that could run in parallel, where runners sit idle or queue, and which job sets the critical path. Plan changes that cut the time from push to all checks green, with the expected saving for each, then make them one at a time and measure each one.”
Bottleneck moved to: re-running work that didn’t change.
Let the Build Graph Decide
Unfortunately, half wasn’t enough. Our PRs were on treadmills. Two PRs touch nearby files, one merges, and the other has to rebase and rerun everything. Agents often handle the rebase, which means inference costs rise with conflicts. We were burning compute on tests that had already passed and tokens on merges, and every so often an agent gave up and had to ask me.
What I really wanted to know was which subset of tests to rerun for a given change to verify we didn’t break anything. Unfortunately, that’s not an easy answer. There’s a lot of recent excitement about classifiers choosing tests and about inference deciding what to test (Peter Steinberger’s plan is to let Codex decide which tests actually need to run), and there are old techniques like deterministic path filtering. I took a slightly different route. I used the build graph.
Bazel (Buck2 works the same way) knows the dependency graph. Every compile, link and test is an action, keyed by a hash of everything that goes into it. Change a file and only the actions downstream of it change: those rebuild and those tests rerun. Everything else comes from the cache. By moving to Bazel, we moved from every PR running 100% of tests to a typical PR only running 2–6% of our tests.
It isn’t perfect, but it’s manageable. It over-tests anything that links half the codebase and under-tests anything that’s incorrectly configured. However, I’ve found that agents are extremely effective at auditing and fixing those kind of build configuration errors as they come up. It only works if your tests are hermetic, but that’s a good thing: hermetic tests are higher quality and more reproducible anyway.
Try it. “Assess what it would take to move our build and tests to Bazel (or Buck2): our languages and toolchains, what our tests read from the environment, and anything that would block hermetic, cacheable tests. Then interview me about the setup I want: local caching, a shared remote cache, remote execution, and who gets read or write access. Run it side by side with our current CI and prove each suite runs the same tests with the same results before we cut over.”
Bottleneck moved to: slow tests.
Fast Tests, Fewer Tests
All the work until now simplified the problem into three parallel paths to continue the cost reduction:
- Run fewer tests, based solely on the blast radius of a change.
- Make each individual test as fast as possible while keeping quality.
- Make sure every test is worth having.
Before Bazel, it was more efficient for us to batch tests around shared resources, which left long poles of ten minutes or more. That efficiency inverts when only one of those batched tests needs to run for a given change. Now it’s more effective to split each test out on its own and run a subset of them all in parallel.
Also, we know that agents write too many useless tests. Luckily they are also good at finding those tests and deleting them. All you have to do is ask.
Try it. “Set a budget of 5 minutes for every integration test. For each test over budget, explain why it’s slow and propose a way to keep the same coverage in less time: a lower-level test, an in-memory fake, sharding, or deleting a duplicate. Fix them one at a time and show that coverage didn’t drop.”
Bottleneck moved to: everyone doing the same work twice.
Share the Work
Now we’re running the minimum set of tests for the exact change we’re making, but we can still get more efficient with shared compute and caching.
Think about what happens every time an agent pulls the latest main: if one agent already built that code and ran those tests, why should every other agent do the same work again? With Bazel, it doesn’t have to. Every action’s key is a hash of its inputs, and its result goes into an action cache under that key. When anyone, human or agent, builds the same inputs, Bazel finds the key and downloads the result instead of running it. Put that cache on a shared server, and one agent’s work becomes everyone’s head start.
Remote execution is the other half. A build farm takes each action, runs it on big machines, writes the result to the shared cache and hands it back. That flips how you size CI. Once the tests run on the farm, in parallel, the GitHub Actions steps are just thin shims that ask the farm for work, so they can run on the smallest runners. A 2-vCPU runner now keeps 64 remote actions in flight instead of 2, and the remote test farm allowed us to drop our long pole from 841 seconds to 167. Keep the beefy machines as workers and the steps tiny.
This technology isn’t new. But, the maintenance overhead and domain knowledge requirement didn’t make sense for smaller projects. For the projects where it would help, the cost to change was too great. A migration like this used to take quarters or years.
Airbnb’s move of its JVM monorepo to Bazel took 4.5 years. My agents had a working Bazel setup running beside the old CI, with full test parity, in about two days.
After the migration, we proved equivalence suite by suite before anything was cut over. The Gas City migration wasn’t the size of the Airbnb migration, but it wasn’t small either.
Our Topology
So, what does our stack look like now?
Our community is a mix: humans and agents, full-time employees and open-source volunteers. This is the setup we found works best for us.
Every client whether it is a human or an agent talks to the same endpoint. Behind it is a bare-metal NVMe cache with Cloudflare R2 as the slow tier in the cloud. Bare metal remote workers scale up and down to meet demand controlling costs while keeping our job queue short.
The important part is that code on main is cached. Every actor in our system, from my laptop to the agent fleet to a first-time contributor, gets the benefit immediately and never has to redo that work.
Pick your tier. You don’t need a farm to get most of this. A local cache is free and makes your own edit-test loop fast. A shared cache lets your agents reuse each other’s work. A read-only community cache gives your contributors the same. Remote execution is for when your machines are the bottleneck.
One of the experiments we’re trying is opening up the read-only cache to our community, as a kind of thank-you. People spend their attention, and their agents’ time, on our projects. The least we can do is make sure they never rebuild what has already been built.
What It Bought Us
The cutover finished a few days ago. We went from independent, long-running tests directly on GitHub runners to thin wrappers that lean on a cache and remote execution, and that shrinks the conflict window we started with. The cost drop is just as dramatic.
That’s a lot of tokens. The agents feel it too. Steve Yegge’s crew agents used to “work for 10-15 minutes, and then idle-wait on monitors for 45-60 minutes”. Ours were no different. The agent-hours spent each day waiting on checks fall from about 52 to about 13, and far less inference gets wasted on timeouts and failures because a test ran too long. For the humans, there are fewer red builds on main to triage, fewer escalations, and a shorter loop from handing off a design to auditing the result. We can spend that attention on more important work.
Everyone at the Edge Is Working on This
This is the problem everyone pushing the limits is facing. Anthropic saw a 25x increase in CI jobs over six months and built test impact analysis. Geoffrey Huntley would hand agents trusted remote builders and leave traditional CI for the complex integration work. David Crawshaw argues for replacing CI with a merge queue that runs every test. Peter Steinberger runs OpenClaw’s CI across up to 10k vCPUs.
Different answers, same diagnosis. The tools we have were built for commits at human speed. A software factory needs new technology and new techniques. Until then, it’s now cheap and easy to use the same tools that already solved these scale problems at large human companies.
Where Are We?
This journey has reinforced my conviction about data and iterative improvement: make the data available, and let the agents optimize, so we’re instrumenting everything. All build and test data now goes into our analytics, and our factory can quickly identify bottlenecks, propose solutions and roll out changes without a human involved. These same analytics are going into the hosted Gas City product as well, so our customers can track and optimize in the same way.
For now, our test suite is averaging about 7 minutes end to end. Ideally we push it under 5, as close to zero as possible. The factory has already identified the next few items in our way. When that work is done, we’ll have the data to show us what’s next. That’s the job now.
Have thoughts about this post? Join the Gas Town Hall Discord!