The Hand of the King

I am not a skeptic on how useful LLMs are, but I feel that we are making mistakes we've already learned lessons from. I don't know if we've never learned them or we are just missing opportunities with LLM agents.

Or I am just plain wrong.

Prompts as Changes

Producing vibe-coded commits and not keeping the prompts that generated them feels like manual testing and debugging software from 20y ago: throwing away effort that is useful for other developers.

SDD addresses that partially: Storing the initial prompts as documentation and the resulting code as commit. It throws away a lot of intermediary steps though:

• Corrections and customizations on the proposed prompts.
• Changes to the code prior the end result.
• Disposable attempts used to refine the prompt that “survived” user iterations.

To solved this with current tooling and practices we could (1) commit often to a branch and (2) keep it after merging it back to the trunk.

Why we wouldn't do that for non-LLM generated code?

Developers usually just write code. They document it at a later stage. Having just intermediary code that doesn't work without the developers objective is not useful.

The “document-first” approach is an emergent behavior made possible by LLMs, with that we have the original intention of the developer before any code was produced. Testing that intention against the end result (code) might yield some insights.

What value can we extract from these Intermediaty Steps?

I see a lot of parallel to the opportunities of keeping an automated test suite:

• Interpret the developer's original intention on producing the result based on how iterations progressed. we might answer the “Is it a feature or a bug” question.
• Refining the project objectives and development practices to adopt what has not yet been stated on root project instruction-set.
• Try only the instructions against a new LLM model.

I feel that we are learning, just like with TDD, what is useful and what is not regarding what artifacts to keep and how to store and navigate them.

Software Architecture

Is there an architecture that better relates to how LLM works with code?

I can see micro-services being easier to deal with than big monoliths because of context-size limit. Even though LLMs interactions with workspaces (multiple, isolated, repositories under the same editor instance) remains janky at best that seems to be the best kind of brownfield to work with.

I am betting that we won't solve the context-size issue with LLMs in the long (5y) run and that an architecture that aids in that regard is going to get blindly promoted (e.g: micro-services). Maybe this is the year of Clean Architecture.

Maybe Unix philosophy is yet to make a huge comeback.

Strong Typed Languages

Assuming that LLMs can't produce reliable automated test suites, languages with great compilers and tooling will become more relevant as they provide the essential feedback required to steer LLM agents into producing working code.

How long until we see a scoring card on how good a language is with LLMs? How will this impact JS vs TS growth?

Conclusions

Of course I don't have any.

I fear, though, we are going to survive the FOMO of LLM Agents just to fallback to promoting techniques or tooling that are just good enough in producing marginally better results than their counterparts.

I'd hope that with LLMs, which makes easier to navigate massive contents, we would dive deeper into modeling and problem exploration rather than rapidly producing results. We'd be more open to share knowledge than earn money. I wish we would not fall for the “walled garden” of information again as we did with internet content.

Maybe I am wrong.

— The Hand of the King