Nice concept but It would've been better if you don't force people to sign in with the auth box always shown I had to delete it from the node tree using chrome devtools. also why not enable WASD keys or even arrow keys to move?
It was only when native English speakers—or those I presumed were—started calling out how bad "GPT/Claude speak" has become that I realized I wasn't actually losing my grip on English as a second language. For a second, I thought, Oh, I learned this language on my own, but it seems I've hit a wall and need to study further. It didn't help that I've also been trying to acquire Swedish as a third language for a while now.
One thing that I find is that it doesn't seem to grasp levels of jargon-use. Like I ask a basic question, okay, a few questions later, suddenly there's abbreviations and weird formulations everywhere.
Not the original commenter, but I did this in all of them.
Their skills formats are basically identical, so I setup simlinks from their own skills directories into a shared one so Claude, Codex, Cursor, and anything else that comes out will all read and write to the same shared skills.
It's great having access to the same skills no matter the harness being used
Can you please share, if you are comfortable of course, what did the model(s) mess up? what were you using codex/cc/pi? did the project have a solid agent.md/claude.md? I am genuinely curious whenever someone have such a low success rate with models what is happening because it could be fixed maybe?
From my own experience using agents for the past year or so. The rate if I have to guess, is well above 70%. I mainly use claude (opus) on typescript react projects that are well setup with minimal plugins/MCPs!
> For instance the task naming in the task file starts with an optimistic 1, 2, 3, 5, 5a but then eventually gets to 8a, 8a1, and then ends up with 8b2c2b3 and “8b2c2b2b checkpoint1”. The code that it produced got ever more wild. I don’t want to bore you with what it tried to build, but here are some example pieces of the interpreter changes:
Hardcoded constants everywhere
Multiple same-line macro invocations in C
Random indexes in production code
Hideous tokenizer code in C
Just earlier Opus 5 was supposed to migrate a Python codebase from kafka-python client-per-send to a client singleton. It decided it should add a partition key "that nobody asked for" which it admitted to at the end of the implementation.
It randomly decided going from no explicit partition key which would randomly add to a Kafka partition to partioning key company was the right solution.
In fact, that was fairly counterproductive since this codebase dumps events to the topic per company so you'd get a lumpy distribution where only one partition is ever active at once (starving other consumers)
The fact it decided it should add this without prompting was a bit shocking and I'll probably try to make it explicitly clear in the context not to "do random shit that wasn't asked for"
One thing I've always been curious about is, often times it seems that models don't seem to have these addendums like "dont do excessive/random shit" by default? Or I suppose, if it did have something like that, and still screws up like in your example, it clearly isn't working, so assume that it doesn't.
But the phrase "don't do random shit", semantically, from a monkey's paw perspective, could imply curtailing the model's creativity and 'thinking out of the box' capacity, that might have existed in its 'reasoning' process. So I'm always concerned if it's possible that, adding these phrases might be part of the reason why a model performs dumber than it should.
I don't have empirical evidence to support that supposition though.
I think of it as a tradeoff between creativity and specificity. Every instruction you give reduces creativity, and at best, increases specificity (I imagine a lot of prompts like "make no mistakes" do literally nothing but pollute context, but I haven't evaled them)
So if you're doing something very ordinary, fewer instructions result in better results. If you're doing something fairly off-piste, you have to give instructions to that effect and accept less creativity. For situations where you want it to do something extremely specific, tons of instructions and accept that you're going to get much closer output but much worse "intelligence"
Another way to think about it is Type 1 and Type 2 errors or sensitivity and specificity from statistical testing - do you want an agent that solves any problem but goes off the rails 10% of the time, or do you want an agent that can only solve 10% of the problems but nails them 100% of the time (sensitivity and specificity, respectively)
This is why I like to really interrogate Claude on what it’s planning to do, what the implementation will look like etc before writing code. Without enough direction and time thinking about it, it can definitely go in an unintended direction
I usually use Fable with superpowers to generate plans and do subagent driven development but unfortunately I was at my Fable weekly limit with some other usage left.
I thought this was written a year ago or so, but it seems to be from July—or at least this is what the Wayback Machine can show. These points at the bottom about "looking under the hood" are either of someone who has not kept up with the technology or does not really grasp what it is capable of: an oversimplification that can be excused by neither.
What really is annoying in all these "stances" is how it boils to a pursuit of quality and idealistic views that are —while appreciated— not really going to necessarily help the product. If someone is going to choose you over X or Y just because you don't use AI, then by all means let them do so; but most likely someone will use AI to replace you, even if you were a programming language or an OS!
If your problem is with environment, sovereignty, and tech feudalism, then by all means I totally agree with you; it sucks, and we should find solutions, not abandon the tech just because it enables greedy psychos to gain more billions. If anything, it's now more important than ever to have a voice, as this technology can lead the way to meaningful automations to help small business owners get on even ground with bigger players by providing them necessary tools to focus on what they actually do, rather than chasing tech and SaaS solutions that dry their wallets. Or can enable individuals who can't afford education/research in a certain area to at least get started by asking questions and finding resources without the stigma or lostness of doing so that many face.
Finally. I am sorry but the "AI will replace you" argument is getting old and silly! If you are to be replaced by AI or X, then neither AI nor X are the problem: it's the people who chose to replace you in the first place. You were replaceable to them; they just lacked the opportunity.
At the end, I think all opinions matter, but sometimes opinions become a jail.
As I was reading and seeing the video, which is easier to demonstrate as you guessed, I was asking my self the same questions you answered in the "Caveats" section.
I love that you are trying something and you shared it, I relate very much to some parts of the post minus the fatigue part maybe I am it's because I am a very chatty person in general so prompting is not a problem who knows!
I had my own attempts to improve working with AI but to my fault I rarely commit to a project no matter if I wrote it or AI wrote it for me!
I tried this over a year ago https://github.com/ramigb/promachos (before I found out about spec kit and similar solutions)
Then I tried this https://github.com/ramigb/groundcheck recently which is to actually help me in the review process specially if there is intent documentation like ADRs or similar.
Your post inspired me to try from a different angle. Thank you.
Same happened to me. Very disappointing, this should be mentioned before the user even authenticates in the onboarding phase of the product asking the user if they have acquired the whitelisting from OpenAI and want to proceed or not! But running for 35+ minutes plus to give such a response is very disappointing and very awkward! Not to mention the lost weekly tokens!
That's beautiful! I see a .claude folder in your code, I am curious if you've "vibecoded" the whole project or just had claude there for some tasks! not that it matters or takes away from your work but just pure curiosity as someone who enjoys betting on the LLM output XD
I found that out while working with music models like Suno! I love creating music for my own listening experience as a hobbyist and when I give suno a prompt no matter how well crafted it is the outcome varies from "meh" to "that's good" ... while when I upload semi finished beat I made and prompt it to cover it the results consistently leave me speechless! Could be a bias since the music has a lot of elements I created but this workflow is similar across other generative models for me.
reply