All of these "hacks" are snakeoil and I think deep down we all know. Whether it's caveman, RTK, or whatever other vibe-coded productivity/token cost saving hacks/skills/claude.md.
What I had success with (although benchmarks are older) is to index the codebase with a dedicated local code embedding model. It's a bit expensive on the CPU side but in my benchmarks it reduced token use and wall clock time significantly. Of course, it's always dependent on statistical noise + host system load, and running sufficiently large benchmarks is simply too expensive, so take em with a grain of salt.
Why does it work you may ask? Well, LLMs basically brute force words/phrases and pipe that into find/grep/pgrep/whatever (or as recently discussed here write a python script for it - https://news.ycombinator.com/item?id=49654229). Semantic search looks for similarities so you have to do less brute forcing. Comes of course at the cost of indexing everything first.
I should not trust their "vibe-coded productivity/token cost saving hacks" but I should trust yours?
Save 30% token costs when using Claude Code, Codex, OpenCode for free - with open source, local semantic search. Works for small and large codebases and monorepos! Enterprise-ready and fully compliant via Ollama and SQLite-vec.
Releases v0.0.42 Latest last month
Why should I trust that what you're peddling isn't snakeoil?
I literally say you should take benchmarks with a grain of salt :)
> Of course, it's always dependent on statistical noise + host system load, and running sufficiently large benchmarks is simply too expensive, so take em with a grain of salt.
And the savings listed are coming from a benchmark harness that implements different OSS bugs one time with and one without lumen - in those cases the % saved are reproducible (caveat: it was on older models, Opus 4.6 I believe).
Also I explain WHY it saves tokens - because the model doesn’t have to brute force different terms until it finds the match it needs, but uses semantic „distance“ so the embedding does it for the model.
I just went through a lot of benchmarking and the only thing that seemed better than rg was chunkhound, which sounds similar to this project. Actually a small Jina embedding model actually did better than voyage AI, but took a long time to index. Also chunkhound doesn’t work well with worktrees. In the end, I decided to stick with rg.
I spent way too long trying to reproduce the results in Pi and failing before I decided that I shouldn't trust author benchmarks for any of these tools. Then I found that I couldn't even close to reproduce their benchmark results using the exact model and their harness.
If any person other than the author has time to verify these Lumen benchmark results I'd be curious to hear it. I don't have the time to do it myself at the moment.
I'm in the process of evals for these tools after my org adopted them. My RTK findings are the same. It worsens task performance and overall you don't save money. I wanted to give the same treatment to other tools like ponytail and caveman (especially caveman, I mean there's no way that telling a computer to talk like a caveman is a valid engineering technique right?). To my horror, caveman is looking to be the only tool that actually doesn't regress on reasoning while taking costs down. But I still have a lot more evals to write, so this isn't conclusive or anything. (Also I haven't tried Lumen yet)
I am actually rather fond of caveman. I haven't evaluated it for token cost, in part because frankly I think that part of the pitch is a load of malarkey. Output that's shown to the user is such a small percentage of overall tokens these days.
But anecdotally I do think it saves me quite a lot of time on reading LLM outputs. And that, if nothing else, is good for my sanity.
The caveman gimmick makes sense to me as a clever hack. Caveman talk is a longstanding meme that's presumably well-represented in the models' training data. So just asking it to do that is just an ultra-concise way to tell the LLM to be ultra-concise. Which, in turn, is theoretically good for accuracy because putting too many instructions in the prompt is bad for task performance.
Similar for ponytail, I don’t know if it saves tokens, but there is less output to read (and usually less over engineering). Occasionally I have to push for more complex code, but that is much nicer than constantly asking for simpler code.
That's also a good point. When I'm using caveman (and especially cavekit), I don't have to spend quite so much energy on dealing with it building features I didn't ask for and don't want.
It combines search with tree sitter grammars so the results can annotate usage vs definitions, cite line number ranges, inline the actual definition if it's short, etc. Not as precise as LSP but simple (no daemon), human readable, and in many cases works without configuration.
A bit different but in the same domain GitNexus has been really good for me. Symbol based search and impact analysis. Once I added git hooks to auto index and added some explicit instructions it saved quite a bit for me. I have no affiliation just wanted to add that as some thing to try out.
You use caveman to save on tokens, I use it to make the agent shut up. We are not the same =)
When caveman was released, token efficiency was a lot more relevant. But in the time I've actually come to like its matter-of-fact and short way of responding. There are other similar ones, but caveman is familiar.
Jetbrains IDEs are a perfect solution for this. They expose IDE actions (e.g, search, see occurrences, go to implementation) in their MCP server, which the harnesses can then call directly instead of figuring out the code themselves.
I also found LSP like skills to usually have no advantage over rg. The agent needs to read the code to understand it; navigation is a small portion of that.
This is the way. Take it with a grain of salt (as the author mentions) but we have gotten much better results and less token usage after we hooked up our codebase to an indexing service’s MCP (in our case sourcegraph). But any indexing service will do.
skills have a pretty straightforward token win - you remove the decision making from the agent so it does specific things in a row. at some point youre better off inverting control to a script that only occasionally calls the agent for a small decision
Probably the first time reading that the Ory stack is unfinished! Sorry you had a frustrating time, but there's 10 years of development and many happy customers + adopters who see it differently! Polis / Boxy still works as before, we didn't gut or take away anything.
Open source development needs to be paid by someone - most of the time people complaining about paying for software are working themselves (for money!) in some company making huge bucks, or looking up to "successful (as in money) tech leaders".
For Ory, B2B login is a good value differentiator, because it's required by companies selling to other companies meaning they can spend some money on licenses to further develop software.
Ory powers the largest technology providers, and super small solo projects. It's robust, stable, Apache2 licensed. It's the best CIAM tech out there that's free (!!).
In the end, everyone is entitled to their opinion but the "open source can't make money" train is honestly a bottom tier opinion and I'm tired of reading it on HN, probably written by people making $100K+ a year for writing software and using open source daily (without paying a dime).
It's like the people complaining that Wikipedia is collecting too many donations, while they cheer on Apple or Anthropic or whoever raking in billions of dollars.
Somehow, only if it's open source / non profit it's bad to make money. If it's proprietary nobody gives a damn. Says a lot about society.
>Open source development needs to be paid by someone - most of the time people complaining about paying for software are working themselves (for money!) in some company making huge bucks, or looking up to "successful (as in money) tech leaders".
There is software that is cutting edge and always changing, and those types of products need to be paid for much more than software that is stable.
With a stable product like Auth (which requires only security fixes and minor features), the 'pay per MAU' model employed across Auth companies is unreasonable
A combination of the people and companies using the product for free or selling its support (like RedHat and IBM for KeyCloak) along with an open license allowing it to be offered as a cloud service should be sufficient?
If you want to pay per monthly active user for the rest of your life, up to you.
Vercel have raised multiple rounds, last one was in 2025 and $300m. So we don't know what the VC's are going to demand for revenue targets. https://en.wikipedia.org/wiki/Vercel
Auth is not stable, it‘s constantly changing and evolving and also a lot of work to keep secure, and scalable. Auth is critical infrastructure and certainly not free. Most companies with homegrown at some point go to a vendor because it is so much work to DIY.
I can’t speak for Vercel‘s goals or pricing - but Ory is evidently still open source while many others went other routes!
i'm curious, outside of new models of authn (such as passwordless, totp, hash algos etc) and new models of authz (rbac, zero trust, etc), what else is "constantly changing"?
even the items i mentioned only change every 5 years or so, in my experience. i accept that there will be a lot of work preventing attackers from gaining unauthorized access, but again this feels partially solved by just rejigging the authn flow (rather than username -> password -> totp (leads to password sprays), just do username -> totp -> password)
KeyCloak is great if you want a full stack Java server to run internal workforce for example, but Ory is much better at running high scale (eg at OpenAI https://www.ory.com/case-studies/openai) and in a composable fashion.
Yes we have an commercial version because how else can one finance world class open source powering the biggest software names on the planet? It‘s a good thing that Ory has a business model that works, not a bad thing. And by the way, IBM finds ways to charge you for KeyCloak too ;)
Valid points (although Keycloak was Redhat not IBM and then donated by them to CNCF), but should "security SLAs for CVEs" be listed as a premium feature?
Looked at the case study, uses Cockroach which is now commercial, so potentially with the dual costs of Ory and Cockroach licenses, unless you need massive scale, would be too expensive for small/medium and also startups? Unless your sole focus is on enterprises?
My mistake - I thought it‘s now just under the IBM corp but it is indeed in CNCF. Still, IBM offers a commercial product around KeyCloak.
If you serve 900m weekly active users, you need this type of distributed database architecture that is expensive to run. But at that point the cost of running it is a fraction of overall infra spend. No start up really needs this level of scale, only Enterprises (hence it‘s gated). Making Cockroach work is more work than just wiring up the SQL, you actually need to deal with it like dynamodb under the hood and use primary keys efficiently, avoid hotspots, and all that jazz.
Most companies (like Cloudflare!) do just fine with Postgres and one of our services. Ory Hydra is written in Go, doesn’t need JVM, very little RAM, doesn’t need caches or start up time due to cold starts. The architecture is different and that makes it cheap and fast to run. From the blog post - they run Hydra on 0.6 vCPU and 200MB of RAM. That’s probably as cheap as it gets!
It‘s a different tool for a different problem than KeyCloak - both have their place.
For API Keys we just launched Ory Talos (https://github.com/ory/talos) - a perfect alternative for when OAuth2 is too much for the use case.
There are use cases and security concerns that legitimize using OAuth2 - with specs like DPoP you can make these flows more secure. In my view the use cases presented here is a good one for OAuth2, but it certainly doesn’t make sense everywhere - complexity makes system harder to secure.
Ory Hydra was one of the few tools I remember being actually good and lightweight and useable. Tried setting up and using KeyCloak for a while, absolute nightmare
Author of Ory Hydra here! Very cool to see this blog post and technical description! I never would have thought this piece of software would secure the internet companies in the world :) Also great to see that the 2.x version performs so well for you! The CPU use is ridiculously small for that scale! We have a commercial variant that‘s even faster, if you ever run into trouble.
If anyone here is interested in providing their own oauth, IAM, rebac permissions, API keys, agent security - check out our open source & commercial products at https://github.com/ory and https://www.ory.com/
Just a passerby, but wanted to say thanks for your work. Ory services are a delight and I was excited to see them spring up years ago and even more excited to see them continue to be developed and put to good use!
For AI Agents we have added token derivation to Ory Talos which allows you to exchange a static API key for a ephemeral, short lived, and restricted token. It can be both a JWT and a Macaroon (super interesting for caveats)!
However this would require GitHub to use Ory Talos and it‘s not a solution for third party credentials really.
So your project solves that need quite nicely, and I‘ll check it out in more detail today :)
We built Ory Talos (not to confuse with Talos Linux) to solve API keys (think OpenAI and Anthropic API keys) at scale and with the best practices around capabilities and securities.
What are parallel agents worth to professional engineers if reviewing the code is a pain (aka non existent) in Zed? Please add proper code review tools (compare with branch|file|revision) and GH pr review tools like IntelliJ!
Besides some of the obvious hacks to reduce token usage, properly indexed code bases (think IntelliJ) reduce token usage significantly (30%-50%, while keeping or exceeding result quality compared with baseline) as shown with https://github.com/ory/lumen
Anthropic is not incentivized to reduce token use, only to increase it, which is what we are seeing with Opus 4.6 and now they are putting the screws on
reply