From further down in the comment thread, Linus says that speeding this up should help other workloads too, and he's sick of using make replacements:
---
+Peter oh, it's absolutely true that 'make' is a pig, and does too much, and we don't exactly help the situation by using tons of GNU make features and complex variables and various random shell escapes etc etc.
So there's no question that some "makefile compiler" could optimize this all. But quite frankly, I've had my fill of random make replacements. imake, cmake, qmake, they all solve some problem, and they all have their own quirks and idiocies.
So while I'd love for 'make' to be super-efficient, at the same time I'd much rather optimize the kernel to do what make needs really well, and have CPU's that don't take too long either.
Because let's face it, even if the kernel build process was some super-efficient thing, real life isn't that anyway. I guarantee you that the "tons of small scripts etc" that the kernel build does is a real load somewhere totally unrelated. Optimizing page faults will help other loads.
> Because let's face it, even if the kernel build process was some super-efficient thing, real life isn't that anyway. I guarantee you that the "tons of small scripts etc" that the kernel build does is a real load somewhere totally unrelated. Optimizing page faults will help other loads.
Suppose the kernel build were as efficient as Chrome's is claimed to be on this page (<5s) and wasn't stressing his system. Would Linus then approve of anyone submitting patches to deliberately slow down the Linux kernel build just to show up slownesses in page fault and encourage kernel devs to spend time on optimizing that?
No, of course not! That would be idiocy and the person submitting the patches would probably be banned by Linus in his titanic rage. However, since the slow build & page faults is the status quo, Linus is making lemonade of it...
I think you missed the point. He's not denying that make is slow and could be faster or that most of it doesn't have to do with page-faults. All he's saying is that page-faults in general are slower and speeding them up would improve lots of various different kinds of loads besides just the kernel build system (Which would probably not improve very much from just that alone).
I'm sure he'd be happy to take patches that make 'make' faster, but that's simply not what he's trying to address here.
So, that is actually an interesting thought experiment, thanks for that.
However I'm not sure it is directly applicable here. There are two courses of action that could solve this problem:
A, an action that improves kernel builds
B, an action that improves several workloads
For A and B of similar cost, it makes sense to do action B in preference to action A.
Your argument speaks to A being of positive utility, but given a finite knapsack of effort smaller than the set of possible actions that fit in the knapsack, a greedy algorithm that places any positive item into the knapsack is not optimal.
I don't follow your reasoning here, but let me expand my observation further: Linus can A. improve his build system (low-hanging fruit which the Chrome numbers suggest could yield an order of magnitude better performance), or B. he can search among a variety of difficult unlikely-to-yield-major improvements (like yelling at Intel engineers 'make it go faster!') which will improve his build system and also other hypothetical loads (which are unlikely to be large gains if any at all; what, is Intel too ignorant to try to make the TLB fast?). He is claiming B is better in part because of the hypothetical loads makes it better in total.
Most people would consider A a more reasonable reaction, especially after hearing that Linus's best idea for doing B is apparently going all the way down to the hardware level in search of some improvement. We can see this by intuitively asking what people's reactions would be to a proposal to induce B if the equilibrium were already at A.
On the other hand, Linus is one of a handful of people in the world who may be in a position to get results by yelling at Intel engineers to 'make it go faster!'. This isn't because Intel is too ignorant to do things on their own, but because practically everything can be optimized further, and Linus may have enough sway to focus the engineers' attention on the problem that he wants solved.
Personally, I don't care much about the speed of the Linux kernel build system, but I do care about the speed with which page faults are handled by the CPU. Even if the chances of success are lower, if he is able to succeed in speeding up every page fault on future Intel processors, I would consider that a much greater good.
The real problem (as I see it) is that I think he's trying to optimize the wrong thing. His worst-case test is based on trying to repeatedly fault in an uncacheable page: every lookup TLB lookup fails at every level of the cache. Likely, Intel has chosen to optimize the real situation where page translations are cached when they are repeatedly accessed.
Intuition is probably not a good guide here. An improvement to the kernel build process is relevant to the thousands of machines that are used for kernel development; an improvement to the page fault speed is relevant to the over 1 billion devices that run the kernel. That's a pretty big multiplier.
(I believe the GPs reasoning is that "improve the build system" and "retard the build system" are not symmetrical, because both directions require positive effort to be expended).
On the other hand, if build system improvements truly are low-hanging fruit, then there are a lot of people out there who are capable of helping. A much smaller number of people are capable of productively working on page fault optimizations, so having someone like Linus address low hanging build system fruit instead would be a waste of talent.
We have a different prior expectation of the utility and feasibility of action B. In my view, Linus is uniquely suited to do something useful here, and his chance of success is high. In your view, Intel is already doing the best it can, and his likelihood of success is low. Our positions follow.
A isn't really very low-hanging though, rewriting the build-system for something as complex as the kernel would be quite the undertaking even just from a technical standpoint, let alone that you now have to change thousands of peoples workflow by having them install and use a new build-system (some of those people are probably dedicated to just maintaining the build-system too, so their job would radically change) as well as figure out a suitable build-system in the first place (which one would work well for the kernel? which one works on all the architectures people want to compile linux on? etc)
B from linus' perspective just means "wait a year for things to automatically get better (after throwing some money at it)" which seems like the low-effort solution.
Not sure why, but your URL has a bunch of junk on the end of it.
I don't get what you're arguing against. Even if the kernel were superfast, there's probably another "real load" out there somewhere that legitimately runs into lots of page faults.
It's a question of what should be addressed first.
Linus is actively arguing against a faster kernel build. It sounds like it makes sense, because it gives him leverage with CPU vendors.
But it doesn't, really. If kernel build were already fast, he would never in a million years slow it down just to get that leverage.
Now maybe he's perfectly aware of this status quo bias, and he's taking advantage of it to meliorate something he would otherwise have no power over. Sneaky.
Still, what's the priority? He's made his point now, hasn't he? He could work on making a faster build process, now.
Actually, his main point is "here is a workload that is kernel-bound that ordinary users deal with, and here is where the time goes." He has explained in the past that he works on whatever interests him, period. So profiling page faults must be more interesting to him than speeding up the build per se. That's all.
"When a proposal to change a certain parameter is thought to have bad overall consequences, consider a change to the same parameter in the opposite direction. If this is also thought to have bad overall consequences..."
How is speeding up kernel build times a "bad overall consequence"? Thats exactly what Linus is trying to do.
Except Linus is not addressing that problem directly. If he was, he would probably be writing a make he'd be satisfied with. Instead, he's working on page faults.
Speeding up page faults does _directly_ improve build times. I don't understand why you think writing yet another make is the obvious direct fix, but fixing page faults is not. Or why you think Linus shouldn't fix things he is clearly interested in.
Oh no. Not on your life. It would be better for the entire world if Linus had a heart attack rather than rewriting Make. Make is already unusable; git-style make would be an abomination.
Can you even imagine reading the man pages of all the different git-make commands? The syntax?
I hope Linus keeps contributing to humanity and doing what he's doing. He's obviously a great engineer, and if he works hard, he might undo the wrong of releasing git into the world.
I can't tell if you're joking or not. The popularity of GitHub and its position as the defacto modern source control management tool demonstrate that very few people hold your point of view. Sure, Git has a lot of options, but what do you need to actually be productive with Git? Probably only a handful of commands: git clone, git pull, git push, git commit, git add, git checkout, git branch, git log. And not too much more than that. At my university Git is used in every single lab based CS course and I don't know of a single person who has had trouble learning to use it.
One of the main criticisms of git has been its user interface (which to be fair, has improved considerably over time). People often compare it to Mercurial's, unfavourably.
No, it demonstrates that things that the Linux kernel adopts tend to get adopted. That isn't very surprising. Git being faster than the dVCS's of the time probably helped. Popularity doesn't equate to quality. QWERTY is still the default keyboard layout.
I am willing to bet money (let's say 0.1 bitcoins) that there are plenty of people at your university who have trouble with Git. We can haggle on terms, but I'd bet that you'd see 10 people at least struggling with git in an intro course.
Once you learn to give it the right incantations, and learn never to deviate from the path you know, any tool can become usable. But it will never be a part of you and will never make you stronger.
This has crippled a generation of developers and I'm afraid it will be a massive barrier to entry, stunting humanity's growth probably by 10-20 years.
QWERTY exists because alternatives were not objectively better. Everytime anyone proposed an alternative, they had to not just be a little better, they had to be so much better that a lifetime of QWERTY muscle-memory being undone was justified.
It's the same reason so many new display technologies have failed: its not good enough to be better then an LCD eventually. You need to be better now, and better then the LCD which will come about as evolutionary improvements in manufacturing too. Being a little cheaper in the initial plant cost is meaningless if the plant has been built and we understand its processes.
Which of course is why the commentary on Git is absurd: people struggle with Git? People struggled with CVS. People struggle with the notion of "files" and programming in general. The alternative has to both exist, and be easier to use from the get-go. Not just a different set of traps.
At some level, command-line conventions are arbitrary. "ls" could just as easily be "dir" or "list." "revert" could just as easily be "checkout" in some contexts.
UNIX has never been a zero-learning-curve OS. That's OK. There are other operating systems that fill that role. git is a UNIX tool which is intuitive once you learn it.
The world doesn't owe you anything, and if you don't want to learn git, or any aspect of programming, nobody is forcing you to. There are lots of other things to do out there. Some of them even pay more.
The parent quote doesn't deserve to be downvoted but it seems to have been massively downvoted for the same reason a massive number of people use GIT who shouldn't - the bandwagon effect.
Git may well be a perfect tool for the Linux kernel but it's still an opaque nightmare for the many average non-kernel-developers who use it as "the new good version control system" (which it isn't, it's only a tool for a specific purpose made by a famous person).
The poster's point is valid - if Linus created a make replacement for his purposes, the effects of everyone else inappropriately adopting it would be horrific (even if it was indeed, a great make for the kernel and just for the kernel).
---
+Peter oh, it's absolutely true that 'make' is a pig, and does too much, and we don't exactly help the situation by using tons of GNU make features and complex variables and various random shell escapes etc etc.
So there's no question that some "makefile compiler" could optimize this all. But quite frankly, I've had my fill of random make replacements. imake, cmake, qmake, they all solve some problem, and they all have their own quirks and idiocies.
So while I'd love for 'make' to be super-efficient, at the same time I'd much rather optimize the kernel to do what make needs really well, and have CPU's that don't take too long either.
Because let's face it, even if the kernel build process was some super-efficient thing, real life isn't that anyway. I guarantee you that the "tons of small scripts etc" that the kernel build does is a real load somewhere totally unrelated. Optimizing page faults will help other loads.