Hey HN, one of the side projects my startup maintains is this arena of 52 apps implemented by different models. Inkling is the most exciting model launch we've seen from an American lab in a while, so I spun up some NVidia B200's and got to work.

Hope you find it helpful or interesting.

Casually spin up some b200's who are you sir..

Great demo/comparision and definitely exiting.

I ran them for a very short time!

Similarly, we updated our model arena (52 apps each built by 26 models) to have GPT 5.6 Sol, Terra, and Luna today:

https://arena.logic.inc/

It's really interesting to see the Sol/Terra/Luna apps side-by-side.

I need to add these stats somewhere in the UI, but one interesting take away: Terra took 1/2 as much wall-clock time as Sol, but Luna took more wall-clock time than Sol (by about 23%). It's still much much cheaper, but it seems like Terra is likely a more optimal time/cost balance for most use cases.

The Terra quality is usually nearly as good as Sol, but much faster and cheaper. I do appreciate Sol's design sensibilities (see, for example, the audio sequencer). It's the first model in a while that is clearly distinct on that front. They'd all converged to very similar visuals for a while.

What caught my eye was:

            Model  Lines of Code  File Size  Gzip Size 
      GPT-5.6 Sol          1,264    35.5 KB    10.0 KB 
    GPT-5.6 Terra            827    20.0 KB     6.7 KB

Yea, that's an interesting result as well. The Terra apps don't feel 35% less feature-rich. So it seems quite token efficient.

If you like this kind of comparison, we have an arena of 52 apps one-shotted across 21 models here: https://arena.logic.inc/

I keep it pretty up to date (tomorrow Grok 4.5 and Sonnet 5 should be pushed).

I want this with smaller models as well like Gemma 4 or Qwen 3.6

Awesome - will work on getting those in.

It would be nice to see how "long" -- e.g. how many self-turns/tool calls/etc the prompt takes to resolve as well.

I know that models like Gemma4-e4b will take longer self-turns but IBM's Granite models will take shorter self-turns in exchange for more tool calls.

Having 'local runable' to compare would be awesome. For example I have a 48G MacBook.

It'd be interesting to add Nemotron, which is quite popular on Spark alongside Qwen 3.6.

Really nice site! From your experience, what’s your go-to model for nice storefronts?

This is impressive and, I think, complements well whatever benchmark is the hottest right now.

Impressive, specially the amount of models used for the comparison.

Yep, 100% correct. We're still reviewing and advising on test cases. We also write a PRD beforehand (with the LLM interviewing us!) so the scope and expectations tend to be fairly well-defined.

It doesn't require removing them if you think you'll need them. It just requires writing tests for those edge cases so you have confidence that the code will work correctly if/when those branches do eventually run.

I don't think anyone wants production code paths that have never been tried, right?

FWIW all of the content on our eng blog is good ol' cage-free grass-fed human-written content.

(If the analogy, in the first paragraph, of a Roomba dragging poop around the house didn't convince you)

I suspect it will still fall on humans (with machine assistance?) to move the field forward and innovate, but in terms of training an LLM on genuinely new concepts, they tend to be pretty nimble on that front (in my experience).

Especially with the massive context windows modern LLMs have. The core idea that the GPT-3 paper introduced was (summarizing):

  A sufficiently large language model can perform new tasks it has never seen using only a few examples provided at inference time, without any gradient updates or fine-tuning.

I never claim that 100% coverage has anything to do with code breaking. The only claim made is that anything less than 100% does guarantee that some piece of code is not automatically exercised, which we don't allow.

It's a footnote on the post, but I expand on this with:

  100% coverage is actually the minimum bar we set. We encourage writing tests for as many scenarios as is possible, even if it means the same lines getting exercised multiple times. It gets us closer to 100% path coverage as well, though we don’t enforce (or measure) that

> I never claim that 100% coverage has anything to do with code breaking.

But what I care about is code breaking (or rather, it not breaking). I'd rather put effort into ensuring my test suite does provide a useful benefit in that regard, rather than measure an arbitrary target which is not a good measure of that.

I feel this comment is lost on those who have never achieved it and gave up along the journey.

SimpleCov in ruby has 2 metrics, line coverage and branch coverage. If you really want to be strict, get to 100% branch coverage. This really helps you flesh out all the various scenarios

Brakes in cars here in Germany are integrated with less than 50 % coverage in the final model testing that goes to production.

Seems like even if people could potentially die, industry standards are not really 100% realistic. (Also, redundancy in production is more of a solution than having some failures and recalls, which are solved with money.)

Can you say more? I see a lot of teams struggling with getting AI to work for them. A lot of folks expect it to be a little more magical and "free" than it actually is. So this post is just me sharing what works well for us on a very seasoned eng team.

As someone who struggles to realise productivity gains with AI (see recent comment history) I appreciate the article.

100% coverage for AI generated code is a very different value proposition than 100% coverage for human generated code (for the reasons outlined in the article).

it is MUCH easier for solo devs to get agents to work for them than it is for teams to get agents to work for them.

that's interesting, whats the reason for that?

Hi, the reason I have this expectation is that on a (cognitively) diverse team there will be a range of reactions that all need to be accommodated.

some (many?) devs don't want agents. Either because the agent takes away the 'fun' part of their work, or because they don't trust the agent, or because they truly do not find a use for it in their process.

I remember being on teams which only remained functional because two devs tried very hard to stay out of one another's way. Nothing wrong with either of them, their approach to the work was just not very compatible.

In the same way, I expect diverse teams to struggle with finding a mode of adoption that does not negatively impact on the existing styles of some members.

thanks for the reply, thats interesting

i was thinking it was more like llms when used personally can make huge refactorings and code changes that you review yourself and just check it in, but with a team its harder to make sweeping changes that an llm might make more possible cause now everyone's changes start to conflict... but i guess thats not much of an issue in practice?

oh yeah well that's an extreme example of how one dev's use could overwhelm a team's capacity.

Reranking is definitely the way to go. We personally found common reranker models to be a little too opaque (can't explain to the user why this result was picked) and not quite steerable enough, so we just use another LLM for reranking.

We open-sourced our impl just this week: https://github.com/with-logic/intent

We use Groq with gpt-oss-20b, which gives great results and only adds ~250ms to the processing pipeline.

If you use mini / flash models from OpenAI / Gemini, expect it to be 2.5s-3s of overhead.