Rationale
-“I tried asking an LLM to help me with this database task and all I got back was outdated and wrong answers”
-“The LLM said it had written the code but when I tried it it had just skipped parts of it and completely misunderstood others”
-“I’m not saving any time when I have to sit and copy paste results and new queries back and forth between terminals”
This you?
Back in March I posted the results of my own very first “vibe code using an LLM” results. Before then the above was me - but since other people whom I trust said it worked for them I realized I was probably not using LLMs in the best way, and my opinion on their usefulness was highly skewed from those experiences.
I’m now at ~6 months of usage and it pains me whenever I’m working with a customer where LLM usage is not part of the toolset. See, I’m using LLMs to enhance my way of working in all three areas that are part of what I do - Cybersecurity (red and blue), System adm and Software Development. Everyone hates when people say they’re “10x more productive” so I won’t state a number, but it’s a lot.
Anyway, this post (which will be updated regularly as new tools get released) is my little try to help others get over the initial friction of getting LLMs to work for them, by drawing upon what I’ve found works for me.
Lesson 1 - What
What LLMs are not
One of the models I will recommend takes up 14GB of disk space. It’s quite obvious that all the knowledge in the world cannot fit inside a 14GB file, so the first revelation is that LLMs are not databases. Sure, depending on the focus of their training they might have a lot of relevant innate knowledge, but you cannot trust it.
What LLMs are
Eager and fast. The best description I’ve seen, and I’m sad that I can’t attribute it properly, is; “Think of an LLM as a very junior, but very eager, employee”. That’s precisely the way of looking at it that works. Regardless of how much an LLM can impress you, a lot of that comes from their ability to work through large datasets much more quickly than we can ourselves. We need to supply them with what to work with, and how, though.
Meaning …
Use LLMs to “do the work”, but just as a junior human developer they need access to API documentation, development methodology guidelines and quality assurance.
Lesson 2 - How
Local or cloud
I’m almost exclusively using local LLMs myself. There are many reasons why (political, environmental, security) but the biggest is that I believe the current cloud AI economic model in the US is a bubble that will pop. I want my own way of working to survive that, and so I make do with the hardware I managed to get before prices shot through the roof. However, not everyone has the same limitations or abilities, so I’ll describe both.
Local
If you have at least 32GB memory and a graphics card with 16GB VRAM - or a system with 48GB+ unified memory - you should be able to run a local LLM with capable results. The things you need are:
- llama.cpp - the “engine” that runs the model and has an API it can be accessed through
- the LLM model file itself - basically a bunch of numbers. The trained neurons.
Cloud
No need to shell out huge sums of money to one of the names you might have heard of when your’re just about to start. Due to the harness (“where you interact with the LLM”) I will recommend in the next section, just go with Opencode . They have free models for testing, but I recommend a $10/month “Go” plan. You’ll get a lot of usage out of that one, especially choosing the cheaper models.
Common
Here’s the meat of this post I guess. The whole reason why I’m writing this is because “just use an LLM” isn’t very helpful. We need to help this eager and junior thing to perform at acceptable levels, and that’s done through a lot of steering.
I recommend Opencode as a harness. You’ll do the interactions with the LLM(s) through it, and you’ll add small little helpers that enforce a capable way of working around it.
Install Opencode.
Install two plugins:
- DCP to better handle the LLM’s internal memory during large sessions
- Superpowers to enforce good analysis, design and reviewed test driven development
Install MCPs relevant to what you will be working with:
- tinysearch so that your LLMs can research how to do things
- pyghidra if you’re doing reverse engineering
- jcodemunch if you’re doing a lot of software development
- kubernetes if you’re sysadming K8s clusters
- browser if you’re doing a lot of frontend work
… and there are of course many more, but these are a good start.
They also all work the same regardless if you’re using cloud or local models, and you can of course also switch models depending on the needs of your current task.
Lesson 3 - Why
Why spend the time on setting all this up, learning how to do things in a different way when we already know how to do it? I’ve been a programmer for 40 years this year. A degreed Software Engineer since 28. I’ve run public services for other people since 35. I know this shit.
… but the amount of work I’m able to do, and the quality of it, is vastly enhanced from these LLM helpers. See, it’s a question of time vs effort. I know that I could use my powerful workstation as a build server so that when the computers that host external services need to build from source their CPUs won’t get bogged down increasing lag on what they serve. But, that’s not something that happens very often - I can’t prioritize setting that up instead of other things to spend my time on.
Prompting an LLM a few times though, now that’s different. I have such a build server setup now (a rather impressive one if I might say so myself).
The same is then true for everything else too. My K3s cluster has better administration. My personal code projects have better CI/CD pipelines. My reverse engineering is done at 1/10 the time it takes for me to do all of it manually.
I’m fine not writing all the lines of the code myself, just as I’m fine not writing all the assembler instructions myself when using a higher level language. Sure, LLMs are not deterministic in the same way and you need to use these capable harnesses I’ve described to avoid pitfalls, but that’s just the same as if you were working with other humans if I am to be perfectly honest.
Very junior, but very eager, other humans.
I idle in #LocalLLaMa and myself and others there would be happy to help out with more details, suggested parameters for running LLMs etc. Just come say hi.