AI just scored 0% on the hardest coding test. The reason should change your next year of work.
This should give you an insight.
Meta, Stanford, and Harvard just released the hardest AI coding test yet. They call it ProgramBench.
The setup: give the AI a compiled program plus its documentation. Tell it to write source code that matches the original behavior.
Nine top models tried. Claude Opus 4.7, Opus 4.6, Sonnet 4.6, Gemini 3.1 Pro, GPT 5.4, and four smaller models. (well I wish they did this test when 5.5 came out lol)
Two hundred software projects total, including FFmpeg, the PHP interpreter, and SQLite.
Pass rate across every model: 0%.
Zero across the board. Even Opus 4.7, the best performer, nearly passed only 3% of the 200 tasks. Every other model couldn’t nearly pass a single one.
AI writes your blog posts and builds landing pages without issue. So why can’t it do this?
The difference between writing code and designing software
The same Meta team built the older benchmark, SWE-Bench. It gave AI a codebase with bugs and asked it to fix them. AI did fine on that one, sometimes outperforming human engineers.
But that test was reading comprehension plus local surgery: find the broken thing and fix it.
ProgramBench is different. Nothing exists at the start. AI has to decide:
What programming language to use
How to organize the files
Which data structures matter
What gets split into modules
What gets bundled into one big function
Every architectural decision, from scratch.
That’s where it falls apart.
The “single-file monsters” reveal what’s missing
The researchers compared AI’s output to human-written code on the same projects.
Median number of files for human code: 15. Median for AI: 3. Sixty percent of AI solutions used 1 to 3 files for the entire project.
The functions AI writes are 1.5x longer than human ones, and there are 70 to 90% fewer of them. Translation: AI crams everything into one giant chunk because that’s what “works.” It doesn’t understand why we split things up.
Human engineers spend time defining clean interfaces. They separate database logic from API logic for a reason: when one piece needs to change in month 6, the others survive.
AI doesn’t know any of that.
The 36Kr coverage was direct about the takeaway. AI is now competent at code production. It still can’t do software design.
Why this is worth paying attention even if you don’t write code
Zoom out. The skill AI lacks, holistic system design, applies straight to your business. Same muscle:
Design an offer that fits a specific market
Build a customer acquisition system that survives changes in ad costs
Architect an outreach campaign that hits the right people with a hook that lands
Run a business with separated, swappable parts instead of one giant tangled mess
This is architecture. It’s the most expensive part of building anything. And it just became the rarest skill on the market.
Implementation is abundant. AI does the typing. The deciding is still yours.
The 1930 test that explains why
A separate research project just dropped: a 13-billion parameter LLM trained only on text from before 1931. They call it Talkie-1930.
Then they ran an experiment. Could a model with 1930 knowledge learn Python (a language that didn’t exist until 1991) from a few in-context examples?
It could pick up some patterns from examples in the prompt, but the results were weak. The model’s worldview ends in 1930. It has never seen any technology invented since.
Same principle applies to modern AI. The model can only recombine what got encoded into it during training. Anything that doesn’t exist in the corpus, AI can’t dream up.
If your job is to build something that already exists, AI can do it, sometimes better than you.
If your job is to imagine something that doesn’t exist yet, AI can’t help. You’re on your own. Which means your work just got harder to replace than it was six months ago.
How to actually develop this skill
You might think “architecting systems” requires a CS degree or 10 years of engineering experience. It doesn’t. Because now we have AI to help with that.
What it requires is one specific muscle: decomposition.
Decomposition is taking a complicated thing and breaking it into its actual physical pieces.
I’ve watched this be the dividing line in business after business. Some people, when they hear about a new tool, react with “that sounds complicated, I’ll learn it later.” Others react with “wait, what’s actually inside that thing?”
The second group eventually becomes the architects. They’re not smarter than the first. They just kept asking.
The 4-step procedure that does the work for you
The procedure I run on anything:
Pick the thing that looks intimidating, whether that’s an open-source repo, a competitor’s funnel, or the latest “AI stack” everyone’s posting about.
Ask “what are the actual physical pieces here?” Focus only on the parts for now, not how they connect.
For each piece, ask “what does it actually do?” Answer in one plain sentence without jargon.
For each piece, ask “what’s the simplest version of this?”
Run it once and the thing breaks into 4 to 6 pieces, usually in under an hour.
Why don’t we do it with RAG (retrieval-augmented generation). The jargon sounds scary: embeddings, vector databases, cosine similarity, semantic search etc.
Decomposed:
Big text source
Cut into smaller chunks
Turn each chunk into a number array (an embedding)
When someone asks a question, find chunks whose number arrays are closest to the question’s number array
Feed those chunks to the LLM
Five steps total. The marketing made it sound complicated. The system itself is straightforward.
This works on anything intimidating in your business. Pick whatever’s been bugging you (a competitor setup, a funnel you keep avoiding) and decompose it tonight.
Why most people won’t do this
A few days ago my partner and I were chatting about something I forget. She said “I wasn’t curious to that degree.”
That sentence stuck because she’s not unusual. Most people stop at “good enough” instead of asking the next question.
Most people frame this as a personality issue. It’s actually a procedural one. Run the four-step decomposition once a week and “naturally curious” stops being relevant.
You don’t have to want to know. You just have to run the procedure on one thing per week.
Two years of that and you can decompose almost anything. People will start calling you “smart” or “a fast learner.” You’ll know it’s just the procedure compounding.
Edge cases (where this advice might fail)
A few honest counter-points:
“But AI keeps getting better. Won’t it eventually architect too?”
Maybe. ProgramBench data suggests something different. Task difficulty is “model-agnostic,” meaning even the strongest models fail on the same tasks the weakest do. That’s a structural limit. More training data won’t fix it. Betting your career on AI catching up before you adapt is betting against structure.
“I’m not technical, so this doesn’t apply to me.”
Replace “architect software” with “architect offers” and you have the same muscle. A local lawn care guy I know designed a service specifically for heritage tree owners in his town. He worked out who actually pays for it and what AI and competitors can’t deliver. He’s running the same architecture procedure I run, just in a different vocabulary.
“Vibe coding works fine for some people. Why’s that?”
It works fine for demos. The moment something needs to change, the single-file monster collapses. Same logic for vibe-building a business: it survives until your first hard pivot.
Action checklist
This week. Pick one tool or system you’ve been avoiding. Run the 4-step decomposition and write down the pieces.
This month. Audit your offer with one question per line item: is this retrievable (anyone can buy this from anyone) or constructible (only you can build this for this customer)? Retrievable means AI is coming for it this year. Constructible means AI is making you harder to replace.
Ongoing. Stop optimizing your AI tool stack. The marginal return is tiny compared to practicing decomposition on one new system per week. That return compounds for years.
Right now. Pick the thing you’d most like to understand and have most been avoiding. Decompose it tonight instead of reading about it.
Author’s note: An LLM was used only for light copy editing, including spelling, grammar, and clarity. The ideas, meaning, tone, and structure are entirely my own.