Building a Local LLM Coding Environment: The Why

Summary (human): Building a software development AI suite on local LLMs and mid-cost, off-the-shelf hardware is still a technical challenge. But it is possible. A setup like this does two things. It keeps me from adding to data center demand that consumes huge amounts of resources. And it keeps my work tools out from under the whims of frontier corporations and erratic politics.

Level set: The environmental damage from unbridled, cloud-based AI disturbs me. Today's data centers already cause a lot of it, and our current path of AI use will cause far more. At the same time, I am neck deep in building practical AI tools and improvements for business. I work on tech stacks for Sales, Marketing, Customer Success, and Operations, and I have done this work for a couple of years now.

I use AI heavily as a coding assistant. I lean on it as an agentic decision maker too. I brainstorm architecture with it and hand off tasks for it to finish on its own.

The speed of the jumps in frontier AI, and the chaos of our societal response, overwhelmed me. Eventually I paused. I started looking at the bigger systems at play and asking a few questions:

Do we really need frontier-level AI for everything?
Is this huge investment in data centers really the only path forward as AI changes how we build technology?
How do we handle the risks of constant, huge updates to software with little real-world testing? (Yes, AI is just software.)
How do we handle the risks of corporations and governments fighting for the money and control these technologies bring?

Obviously, plenty of other people are asking these questions. But I'm a practical kinda guy. So I decided to test one answer to all four questions, for one heavy, professional AI user: me.

Here is the experiment. I am a senior systems engineer. I produce a lot of code, and my environment needs real speed and reliability. I set out to build a local LLM coding environment with two requirements.

It matches the frontier-model functionality I use today in four areas:
- Planning the architecture
- Writing an engineering plan an AI coding agent can iterate on
- Producing the code
- Reviewing and testing the code
It runs only open-weight models and open-source (or at least free) servers and harnesses. I store the weights and pin the versions, with no auto-updates. The goal is a frozen, working system. Updates, access changes, cloud outages, corporate acquisitions, and network failures can't put my work at risk.

The experiment took some fascinating deep dives into the real-world practicalities of a system like this.* The necessities turned out to be:

Hardware. I researched GPU, CPU, unified RAM, chip architecture, and cost, and then bought it. Then I learned which limits, software, and settings get the most out of that hardware for local models.
A local model server. I had to understand how settings and limits interact across the model, server, harness, and hardware.
Local models. Size, capabilities, inference, quantization, temperature, thinking modes, and context size.
A harness. This is the agentic work environment. It supports role profiles, gives access to skills, and enforces behavior and access rules as far as it can.

Only then could I start testing. I tested models in three roles: architect, coder, and reviewer. I put a different model in each role to reduce bias.

After testing, I set up the harness with a clean, working first workflow:

The architect designs the system and writes the plan.
The plan breaks into coder-sized slices.
The coder builds one slice.
The reviewer passes or fails the code and explains why.
On a fail, the coder tries again. On a pass, the coder moves to the next slice.

Upcoming articles will break down each of the four deep dives, the test results, and the workflow details. For now, here's a taste of the result: “Hooray, it is possible!” <grin>

* I could not have done this kind of deep dive in any reasonable time without AI, if at all. AI is an amazing research and development tool.