No, because the Chinese have nothing to gain by prompting their models to organise into "swarms" and "go rogue". BS like this is PR moves if z, company that tries to convince investors they have "the best AI in the world".

I'd rather buy two used rtx3090 than a single r9700 AI pro. More VRAM (some wasted due to it being non continuous), more RAM bandwidth, more aggregate compute.

Only if AMD made a card like this with 48G+ I'd consider it.

Also these 20-30t/s jumping to 150-200... Watch out for the massaged numbers coming from vendors.

I believe Intel has claimed something like 1400tok/s (generation! Not prefill) of Qwen3.6-moe on Arc b70.

I was actually very interested in this so I checked the details. Turns out it was 200 simultaneous users running the same 1024 token prompt :D so all the experts got maximum parallelism.

How often are you going to run 200 parallel sessions with a tiny context and same prompt running at 7tok/s.

Based on how much my rtx3090 is getting on a single user (150tok/s) I'm estimating b70 to probably get less than that.

Sadly nvidia is king now.

Also, most of us already have nvidia cards and no inference software supports mixing let's say nvidia, Intel and amd cards in inference of one model.

2x RTX 3090 is not enough for proper use. ideal is 64GB+ so that you get proper cache and 256k context with good enough models. (e.g. Qwen 3.8 27B with MXFP4). And that is tight already.

You would need 3x RTX 3090 - but then the PCIe bandwidth comes an issue if you really want tensor parallelism with three cards. 3x PCIe5 x16 is not cheap with direct CPU access.

So surprisingly, 2x r9700 starts be a nice deal.

> Also these 20-30t/s jumping to 150-200... Watch out for the massaged numbers coming from vendors.

Well, luckily these are not vendor numbers. Prefill also scales almost linearly with the amount of GPUs.

> 2x RTX 3090 is not enough for proper use

I don't see how. Even on 32GB you can run Q6_K_XL quant with MTP at 200k context k=q8_0, v=q5_1. So 48GB VRAM is good enough to run Q8 at long context. Also with tings like ninfer and it's various forks I'm seeing people get very good performance out of Qwen3.8 models on all sorts of NVIDIA cards.

You don't need PCIe 5.0 x16 since RTX 30 are not PCIe 5.0 to begin with.

Well, that makes them just even slower then

The whole point of dual 3090 over other cards is the nvlink support. At that point the pcie doesn't really matter.

I'm questioning their results. It doesn't take much to beat the frontier in single benchmarks if one puts extra software between the model and the harness.

This is also a reason why comparing "naked models" for which weights are available and frontier where providers can do whatever they want behind the scenes is unfair.

Specifically, all it took to boost Qwen3.8-27B to get 10% more points on SWEbench Pro and Terminal Bench 2.0 with a proxy that has just these basics: - tweaks few decode settings like slightly higher temperature - detects when model gets stuck and tells it to "go on" - detects responses cut in the middle, empty responses that contain only reasoning, formats not passing verification etc and tells the model to "try better"

And that is it. 10% more. I admit on a subset of tasks, but results are results, even on a subset.

Yeah but what is the point of "naked model" evaluations? It seems to me that the only interesting question is capability per time and cost (and conditioned on other important things, like privacy). It doesn't matter to me which part of that equation is being implemented by model weights and which part by other supporting software.

It is useful to compare like for like.

Currently if my hypothesis about frontier labs doing creative tricks between the model and the client is true (and the results seem to favour it so far) the benchmarks are giving us an artificially lowered results for open weights models.

I have yet to test opus/sonet via my proxy. If Qwen gets 10% better and Opus stays the same that suggests one if two things: - either opus doesn't need it - or it's already done behind the scenes.

In my view, the useful like for like comparison is to the entire system that people actually use. Nobody uses a "naked model", so what is the point of these benchmarks that use them in that way?

I think the benchmarks should be trying to use realistic harnesses for both proprietary and open weight models.

Harnesses can fairly easily be adapted to other models. Moving capabilities from one model to another is a more involved process requiring distillation, training, etc.

I don't quite understand your point. Why are you moving those capabilities from one model to another, or improving the built-in capabilities of a model, what is the goal? If having the capabilities in the model itself improves the overall capability, then using that better model with the same harness should achieve better results. Or if the capabilities are the same, but they've been moved from the harness into the model, then there should be some other detectable improvement, like to speed or cost.

I don't know about others, I can only speak for myself. But I do appreciate numbers for bare models, numbers for model + harness, numbers comparing different models in the same harness, and numbers comparing several models across several harnesses.

It's a lot of information to ingest, but it gives me some idea of which part of the system is doing which part of the work, how well different harnesses and models interoperate, and more insight into the part of the equation under my direct control as a software developer.

I agree that all else equal, the more comparisons the merrier. But IMO, there has been way too much focus on benchmarks for the bare models, when it seems to me that what actually matters is what you can do with the technology, and that is never limited to what a bare model can do.

> Specifically, all it took to boost Qwen3.8-27B to get 10% more points on SWEbench Pro and Terminal Bench 2.0 with a proxy that has just these basics

Do you have a link/paper for this you could share?

I haven't written one. You can easily replicate it if you wish just based on my comment and a day spent with Claude Code.

In fact that is how I got the idea. There is a 4 month old post on SWEbench github that claimed 20 point boost (beating opus as well) for another small Qwen model when it's traffic was sent via a similar proxy and claude code was used as the harness.

The author didn't post their full code, but he did post all sorts of logs showing which features of his proxy were activated during which run.

I wanted to exclude the harness and validate his claims regarding only the proxy's impact so I reimplemplemented a subset of his features that seemed to give me the biggest "wins" with the smallest amount of work. These are the ones I list in the parent.

It took one afternoon last week plus the runs that took few days.

I saw about 10% better results.

As I mentioned, it is a subset of tasks, not the entire set, but results so far are very encouraging.

Since then I implemented many more features in the proxy (best of N for example), but i haven't had time to measure the impact yet. His proxy also had best of N, but that feature wasn't activated in on the runs he reports.

I find it very disingenuous when tjose companies talk about models "going rogue" or "escaping their sandboxes".

All those activities take place during so called "security testing" when the model is prompted to use "any means necessary" to achieve a, certain goal.

Is it surprising turn the model trained on exploits and vulnerabilities does exactly that?

We could talk about "models going rogue" only if did anything AGAINST it's prompt.

Don't they have very low limits?

What do people use these tiny limits for?

I started measuring my Claude Max x5 use and last week (they gave me 50% more) I used 1.3B input tokens. Some 130M were cache writes, rest was cached. And 5M output.

This puts things in perspective. We're taking thousands of bucks weekly even if I managed to switch to Kimi K3.

What is the majority of this use? Infrastructure upgrades, troubleshooting and so on. Ingesting quite a bit of documentation at beginning of each session.

Sessions run from few hours to a month long and 1M context usually hovers near 30-60%.

When the weights are closed I don't believe any benchmark.

I just got Qwen3.8-27B to score extra 10% on SWE Pro by adding a proxy in front of it that has few simple "harness like features": - when the model gets stuck it tells it to "go on" - when it sends no output, malformed json, slips to wrong tool use format, etc it asks it to "try again better" - detects repetition and tells the model. - injects a prompt about "planning tool use" when it seems to be using same tools repeatedly. - injects a reminder it can use tools if there are no tool uses for over X messages.

10% - with just that.

I have more to test. My point is, open weights models get tested on naked model quality. "Frontier" models get tested as a model + whatever secret sauce they choose to put in front.

Have you posted the code to this proxy somewhere? I'm dabbling in custom harnesses and would love to learn from it.

Seconding this, would also like to hear more about the proxy.

Are the weights public? I'm not seeing them

I recently heard a local EU politician on the radio trying to convince the listeners "things made with AI should be inherently non copyrightable, because they lack human creative labour".

I wish I could tell him there is this thing called photography where all a human does is presses a button and copyrightable artefacts are created by capturing how the world looks. For example a landscape that human had no influence over creating!

Surely this "non copyrightability" should apply to photography too.

/sarcasm

AI is just a tool. You can make slop or you can make beautiful art. Is it fair to Picasso's copyright holder that all his paintings were used for training and you can prompt "this in style of Picasso"?

No, especially if the company that trained the LLM makes money on it.

But there is a technical solution. Simply during training record who owns what training data and record how much influence on each weight this training run had.

Then during inference it could be recorded if the weights that trained on Picasso's stuff activated or not. And if someone pays for inference the original owner of tgat data should get a cut.

Maybe even blockchain tech could be used for this... Imagine the hype! AI and crypto together!

I'm with you on this. I remember when the Internet became a thing, how revolutionary it was to all areas of my life.

This is comparable.

While I dislike the bonkers valuations, and "were building Agi so it tells us how to be profitable" is onion worthy statement LLMs are an absolutely revolutionary technology.

Even with all the hallucinations.

Imagine telling someone 15 years ago Internet/Google is useless because people sometimes do not tell the truth online... It's like that.

I also hate the fact AI is being used as justification to grab all compute in the world and lock it up in one country's datacenters.

I hope this endeavour fails, but sadly being realistic I have to admit I think the so called "AI bubble" will not pop. Instead the companies will be bailed out with printed money.

I recently looked up there are approximately $20Trillion in circulation. Even if they print extra $2T it will dilute existing money supply by 10%. So everyone that uses USD as a currency will pay for the datacenter build out wether we want it or not.

All the compute will be slurped from the market for this and companies like nvidia that get used to 50bln deals will never go back to making consumer stuff.

I never thought I'll live to see the PC revolution reverse, but that is what seems to be happening.

>> I've never been the kind of coder >Looks like you were never really a coder, to be honest.

I do not understand the bitterness here at all. AI is just a tool you can use for better or worse.

I learnt basic programming at an old age of 10 and 6502 assembler 2 years later from (paper)books.

There was no Internet and I dreamed of one day owning a magical software program called Macro Assembler so I could use labels and advanced loops in assembler programs instead of tediously translating examples from the book or magazines using a pencil and paper into assembly without such features.

The AI today is like that Macro Assembler for me back then. You, a human, are simply moved one layer of abstraction higher.

Did I enjoy writing that assembler back then when the goal was to complete a calculation in the time it took the crt tube's electron gun to draw one line on the screen? Sure. Would I want to write accounting software in it? Hell no.

There have been very crappy coders and great coders before and after AI. Just like almost no one writes assembler anymore, almost no one will write normal code in the age of AI. But knowledge of it, how it should be written is still going to be important.

Your value as a human is in the architecture of the software and choices that influence it's entire functioning. In maintainability, scalability and resilience present in your design from day 1 not added a year later.

You know how much slop and crappy work I saw before AI? A lot.