Yes, those ones would be at least a somewhat-plausible simulation of a real scenario people care about: a once-clean codebase that was allowed to become messy by a succession of insufficiently-careful vibeslop PRs.
I'm not a huge fan of their methodology for the AI-degraded cases either (ideally one would set up the mirror pairs by taking some real repositories and rewinding history a month or so and then having a succession of independent agents reimplement each bit of feature work and bugfixes over that period of time), but it's at least a coarse approximation whereas I just don't trust the cleanup methodology to resemble anything real in the first place.
Our initial suggestion was to do something along the lines of your proposal. But we found that when we ask an agent to implement features for `t-n`^th PR, even when we are overly specific makes the code rather divergent, to the point that sometimes the `t-n+1`th task description doesn't really make a lot of sense.
In fact, there are some papers (that we cited) which create a set of tasks doing exactly this, and it is non-trivial [1].
I was reading your comment, agreeing with it but still feeling why this is a bad comment. It just occurred to me that an anecdotal statement like this is the antithesis of scientific discourse. We have a paper here, trying to answer a question, and anecdotal testimonials can only harm the discussion by biasing readers without adding anything of value to let anyone objectively conclude anything on the problem.
The most useful discussion would be if we all read the paper and critique its methodology or results.
I was reading your comment, disagreeing with it but still feeling why this is a good comment. It just occurred to me that this is not science: science must be reproducible and this is just an historical report on artifact that will be unavailable soon.
There was never a time when a book gave the public an overview of the universe. ABHOT was so popular for being a book no one actually read, theres even an index named after Hawking due to it: https://en.wikipedia.org/wiki/Hawking_Index
Did _you_ read that book?
There however definitely was a piece of media that captured public minds and educated them about the cosmos. And that was the show Cosmos. The original of course. Not the NDT drivel.
I am pretty certain this "hawking index" meme must be a new-ish thing. I read the book as teenager and know also others who did. It is a fairly easily readable book imo, so I don't think this characterisation is warranted by the qualities of the book itself.
I suspect that a popsci book becoming a bestseller creates a larger-than-the-usual-nerds audience, a big part of which lacks the motivation to actually finish it. I expect that in places like this you will find higher frequency of people who have actually read it.
Moreover, when i read the book i did not have easy access to pop-sci sources as a (practically pre-internet) teenager in a small town of a small country, like i would have had today. I got upon a booklet of a small publishing house with the titles of translated pop-sci books and would order them from a local bookstore. Maybe if I was already familiar enough with the topic through youtube videos etc I would not have finished either.
> this "hawking index" meme must be a new-ish thing
TBF the methodology and hypotheses that it's based on aren't that bad. I'm sure Amazon has better data, but for a "publicly accessible" data (at that time) I can see it. The problem is that while lots of people might abandon the book, that doesn't mean that still loads of people don't read it fully. They are, after all, extremely popular books. Obviously some people will have received / impulse bought / FOMO / new year resolution / etc the books, but from the sales numbers that's still a lot of people that did enjoy them. Marketing aside, the book is pretty approachable, like were Sagan's books and so on.
Interesting. I've been gifted that book at just the right time in my life as a teenager. It captured me. I read it the to end many times. Not understanding everything or course. But it created a spark which laid the groundwork for my entire career.
From that list I've read three books: "A Brief History of Time", "Thinking, Fast and Slow", and "Capital in XXI Century". First two I've read from start to end. The last I didn't read to the end, I think I've read ~50% of it. My numbers correlate with that list, the least "readable" book in the list is the book I didn't read through.
However I still doubt the methodology. It is not obvious for me that if a book was read in full, then highlights from it would be distributed uniformly all over the book.
I did read it and I'm sure a whole generation of people also did, it is a very clear and readable book. Don't underestimate or minimize the impact of Hawking's book when it was released.
As an early teen I begged my parents for both A Brief History of Time and The Grand Design. Read both several times. ~15 years later, my parents are still holding on to them and my Dad has read them a couple of times too. It was great reading and played a significant role in my choice of research as a career.
Never heard of the "people don't actually read it" meme.
I read it in high school in 82. I still have the book.
Fun fact: The very first book ever sold on Amazon was "Fluid Concepts And Creative Analogies: Computer Models Of The Fundamental Mechanisms Of Thought" also by Douglas Hofstadter
I sold cars for a decade and wouldn't even take the insanely cheap priced company demo car, just driving my 10 year old car instead.
On that note though, most car salesman are somewhat stupid, although the best ones are atleast normal. (I was a cal mechE grad and top 0.2% nationwide car salesman). They also slowly brainwash themselves into believing most of what they say.
You may have been lucky or privileged in working in high competency environments where the essence of containerization is truly captured in the dev practices. The vast majority of mediocre eng shops do "dockerize everything" but would still end up with leaks everywhere, artifacts in s3, endpoint dependencies, mounts that depend on the helm chart etc.
IMO containerization as a principle has failed the test of making mediocre teams productive. Such teams cargo cult ape on this term and think because they've done it theyre now webscale.
I personally like to still start with "this code should run anywhere" principle. On python especially with uv now this seems like a safer bet. At the least this forces the engineers to explicitly list out stupid dependencies in the reader.
I mean, to be honest, I'm the one who sets the precedent for most of the teams that I work with; that's one of my roles.
And personally, I don't necessarily follow a hard line with this. My Python API services run foregrounded in a terminal with uv run, but that doesn't mean that I'm using SQLite instead of Postgres. Any of the backend services I need that don't make sense to be foregrounded still run the same versions as prod containerized. And front end teams will tend to run a containerized API service since they're not modifying it as often.
Point being there is some conscious leeway here, but it's bounded and well-defined.
The important point is that your benchmark is pretty much irrelevant for the actual usage. Thus whatever conclusion you draw is not just irrelevant but misleading.
You're correct, but there's a good reason: they need to draw over other apps to do what they do. So it's not necessarily nefarious. But it is an excellent reason to build the functionality into the OS.
(The reason the permission is so dangerous is they can trick you into pressing the wrong button by relabeling dangerous text with innocuous text.)
The presence of a good reason is exactly why you have to be so careful. Creating an app with a legitimate reason to request permission, only to also abuse it, is a great strategy for an attacker.
Absolutely, which is why I really appreciate the network permission on GrapheneOS. It makes me more comfortable to allow other permissions knowing no data can be exfiltrated.
It's wild to me that "internet access" is not revokable or even displayed in the Play Store in stock Android. It's such a huge security and privacy concern, even if most apps semi-legitimately need it.
Or, it would be wild, if it weren't fairly obvious that this is just Google protecting their mobile ad revenue.
More than half of the ones I have installed have no internet access. Most because they don't have the permission (thanks, F-Droid!) and the rest because I've rejected that permission (thanks, GrapheneOS!)
Not in all app repositories. This isn't so common among open source software as it is in the commercial/adware ones you find very prominently in Google's curated collection
Well, that depends on the other apps you have installed. Unless things have changed in newer versions, apps with no networking can still do IPC, so any app can for example use Cronet to make network requests via Google Play Services, regardless of the toggle, as long as sandboxed Google Play Services has network permission.
Mostly asking it as a question, given that graphene runs Google play services (optionally) as a normal, sandboxed service with no special permissions might help a bit, but I guess unless you disable networking for every other service installed, this is sort of impossible to plug 100%? IPC can be quite the security hole.
Only if the other services provide a network proxy right? You'd need to find an exploit in the app otherwise.
Edit: although, I just remembered that it's actually as simple as sending "open this URL" intents to the Android equivalent of sensible-browser, which everyone will have installed. That does rely on users not understanding or caring about what's happening or it only works for the first user
Yes IPC is definitely a security hole, but because the two apps communicating need to both explicitly support it (I really doubt there'd be an exploitable vulnerability here of all places), it's a much smaller concern. Here I'd mainly worry about apps like Google Photos talking to Google Play Services. GrapheneOS has mentioned they'd like to implement IPC scopes to isolate apps, just like contact scopes and storage scopes.
And it even fails in the way that apps will see no-wifi and believe the entire device is offline. That way they can't detect it and mess around without harming regular offline users.
Depends. One of the reasons I stopped using Spotify was because its offline mode refused to work when my device had all radios off (who even thought of this?). Once I turned on Wi-Fi, even without connecting to a network, it would load immediately.
Funnily enough, the Huawei P30 Pro I was using previously (back when these things could run Android with GMS) had network toggles for all apps. They weren't in the permissions menu, as they were meant for data saving purposes but you had the additional granularity of choosing if an app could access mobile data or Wi-Fi separately.
That solved the problem of ads in games much better than DNS :D
Any self-respecting OS has packet filtering, this isn't unique to or surprising from GrapheneOS. On my Samsung/OneUI I use AFWall+ which sets iptables rules iirc
