I got an Xbox Ally X (with bazzite as Gabe intended) after enjoying the steam deck so much. I get it's not for everyone, but I wasn't playing games on my desktop due to life/kids/etc. The handheld is awesome for my personal case, I love casual games and use it on the couch/plane/bed. If you want a PC with a keyboard and mouse that works with everything, get a gaming laptop. If you want to pull out a game for 15 minutes, play, then hit sleep and come back later, a handheld is absolutely the best way to go.
It is entirely possible to recognize the things that are good about the US while also seeing the very terrible things. Nothing is perfect and if we tell ourselves it is by criticizing anyone who says otherwise, we are robbing not only the world but ourselves of any opportunity to make it better.
I have been bothering the VM team for years for VM GPU pass through. I worked on the Apple Silicon Mac Pro and it would have made way more sense if you could run a linux VM and pass through the GPU that goes inside the case!
Sadly, as you can tell, they have not taken me up on my requests. Awesome that other people got it working!
It looks like the pass through part here was implemented using standard DriverKit interfaces, if I'm not mistaken. That is, the PCIe BAR can already be mapped from the user-space, without any extra modifications to macOS. It's just a matter of VMMs, such as QEMU, adopting this interface in addition to Linux VFIO and the like (unless you're talking about Virtualization.framework, which is kind of a VMM of its own).
Isn't driverkit essentially a separate user space stack compared to regular code? I remember seeing the driverkit specific dyld caches in macos root partition images that included their own copies of everything down to libsystem. Getting driverkit code to run in the same process as normal user code seems like it'd be quite an uphill battle.
Presumably with the right entitlements you can just hit the same (presumably IOKit) syscalls that driverkit does. But that's an extra layer of reverse engineering, and you're not really using driverkit anymore.
it is a separate stack, but that probably doesn't matter much. a user process (in my case, qemu) can communicate with a driverkit driver. the user process can also map memory through the driver, which is how this pci passthrough system works.
i don't think the issues with the project really are specific to driverkit.
>> This project requires a special entitlement from Apple. I’ve requested it, and heard they may be open to granting it, but I have not yet heard back, and I’m told that the wait time could be months.
> I have been bothering the VM team for years for VM GPU pass through.
Good luck. I'm sure they're keen on giving people access to this so that people can spend their money on NVIDIA GPUs instead of buying more expensive Macs. :)
Would of course be awesome, but I'd be very surprised if it happened.
There isn't a more expensive Mac option to buy if what you're after is a gaming GPU. It's more likely that the VM team sees this as a very low benefit ticket to pursue given the tiny segment of Mac gamers hoping to improve their options with a Linux VM for gaming.
(Meanwhile, I'm recompiling Wine to see if I can patch it to address an issue that was hotfixed in Proton two weeks ago but isn't in a CrossOver build yet, so yeah, there's maybe some arguments to be made here that I'd be a potential beneficiary. If I weren't too cheap to spring for an eGPU in today's market, anyway.)
The entitlement in question is the standard `com.apple.developer.driverkit.transport.pci` [0], required for anything that touches the PCIe bus [1]. Apple is generally restrictive with how much third-party applications can do on machines with SIP/"full security", so I'm not exactly surprised. It's not an Apple-private entitlement, however.
The VFIO-style driver made by the author of this also appears generic enough to support all kinds of PCIe, not just GPUs. Apple might find a way to weasel out of this ("hey, this is for hardware companies and you don't seem to be affiliated with one", "your driver requests too broad access", etc.) if there really is a conflict of interest, but so far, there's a chance it will just get rubber-stamped.
I can see them rejecting it for legitimate reasons, though, at least as far as "legitimate" with Apple goes. This driver is essentially a thin layer over PCIDriverKit, exposing all functionality that's supposed to be behind the entitlement to arbitrary applications, in similar fashion to WinRing0. They probably didn't come up with all this bureaucracy only to sign something like that in the end. We'll see what happens.
1. Virtualization.framework seems to support some form of GPU passthrough from the host (granted, not eGPU - it's for the integrated GPU). I think the primary use case is having macOS guests get acceleration, while still sharing GPU time with the host. There is also a patch that recently hit QEMU mainline that supports using the "venus server" with virtio-gpu to support a similar functionality for Linux guests under Hypervisor.framework.
2. Apple internally has some kind of PCI Passthrough support available in Virtualization.framework. It seems like the code is shipped to customers in the framework, but it relies on some kind of kext or kernel component that isn't shipped in retail macOS. I can't say if that's intended to ever be released to customers, but clearly someone at Apple has thought about this the feature.
I experimented with booting Arm macOS 14-26 in QEMU a while back, building on the work of Alexander Graf for macOS 12-13, and reverse-engineered substantial parts of Hypervisor.framework, the in-kernel hypervisor, and a bit of Virtualization.framework. Got newer versions of Sequoia to boot past the log in screen, with GPU acceleration too.
Unless there's another method I missed, the internal GPU "pass through" of Virtualization.framework you're thinking of might actually just be paravirualization, at least that's what the name suggests. It's implemented in the public ParavirtualizedGraphics framework [0], albeit for PG on Arm macOS, the relevant interfaces are private [1]. I haven't looked that deep into it per se, but, fixing the bugs around it, I've run into a few clues suggesting that it's just a command stream + shared memory being passed around. It also uses its own generic driver on the guest side.
Great job, by the way! Love how authors of pieces like this casually come here to comment :)
There's some randomness around Tahoe for FileVault and it crashing because Data is detected as not encrypted (and that's not OK on bare metal). If hitting that case you might need to enable FileVault inside the VM (and remember to sync aux storage afterwards if not done)
I see the author of this patch set has run into a few similar issues as me. Strangely, not all of them: I don't see patches for the new PCI MSI-X device introduced somewhere in Sequoia (IIRC), a source of kernel panics for me; and there's still a bug in the PG MMIO path that casts all writes to 32-bit, this one caused me a lot of headaches (no errors but no video adapter detected). I'm somewhat surprised to hear that this works!
There's two significant problems that we both have came across:
- LLVM now favoring pre-indexed load/stores which trap with ISV=0 for MMIO accesses, and those ending up in the GIC initialization path of the newer macOS kernels (looks like there's a separate patch set for this [0]),
- Hypervisor.framework trapping PAC HVC calls.
It seems like the latter has been worked around here by signing QEMU with an Apple-private entitlement and running with SIP off, but there's actually a different way! While some HVCs are trapped right in the host kernel, the PAC trapping happens within Hypervisor.framework itself (at least in the host OSes I tested). It's possible to patch out this functionality without special privileges or talk to the in-kernel hypervisor directly. I originally tested with the former, and chose to implement the latter as a separate accel in the code I was planning to submit upstream, but the complexity of it exploded and, besides confirming that it would have worked, I haven't managed to finish my implementation.
Does the Tahoe crash you mentioned manifest itself in stage 2 iBoot panics? I must admit 26 was never quite my priority so I haven't looked into it, but if so, it might have been closer to booting than I thought :)
It was a kernel panic for Tahoe. Anything between macOS 12 and 26 wasn't tested so releases in-between might have more issues.
The userspace reboot after FileVault password entry acts a bit oddly with QEMU input devices so you might need to attach a new USB tablet or kbd from the monitor.
> looks like there's a separate patch set for this
Yup and it's a bit of a problem to figure out the right thing to do for it on the upstreaming side as normal guests aren't supposed to do that.
> It's possible to patch out this functionality without special privileges or talk to the in-kernel hypervisor directly
Or pre-patch them all to HVC #1 works too. Patching the host Hypervisor.framework sounds quite brittle especially after they moved to a pile of C++
Oh how I wish Arm didn't let anything like this slip into the architecture spec to begin with! Massive source of pain, especially with protected memory/CCA guests. It's not macOS triggering this in isolation either. Most start up binaries for QNX do this too, somehow also in the GIC init path.
I've looked at how different hypervisors/VMMs handle this and, if this makes that patch set any less hacky, Virtualization.framework, QNX Hypervisor, and (I think) VMware all decode and emulate those instructions in software. Virtualization.framework is a remarkable spaghetti in this regard :)
> Or pre-patch them all to HVC #1 works too. Patching the host Hypervisor.framework sounds quite brittle especially after they moved to a pile of C++
Possibly! IIRC, if HCR_EL2.HCD==1, HVC should trap as undefined instruction. Not sure how much of HCR_EL2 can be set from the user-space, but perhaps this could be the least invasive way.
Simply ignoring the instruction, though, is not enough. I remember in my setup, with HVC handling stubbed out, secondary cores would always fail to start. I suspect this to be the culprit.
The SMP bring-up code would fail to pass pointer authentication on the first indirect branch. It would then immediately pivot into FLEH->SLEH->panic(). panic() shortly would attempt to make an indirect jump itself, hoping to crash the other processors, but instead, getting stuck in a loop of calling itself. This would eventually get caught by a stack overflow guard somewhere in FLEH/SLEH, which would place the core in an infinite loop, and... the system would continue to run with just the boot core. Yo dawg, I heard you like panics :)
> Any resulting exception is taken to the Exception level at which the HVC instruction is executed.
instead of trapping to the hypervisor
> I've looked at how different hypervisors/VMMs handle this and, if this makes that patch set any less hacky, Virtualization.framework, QNX Hypervisor, and (I think) VMware all decode and emulate those instructions in software. Virtualization.framework is a remarkable spaghetti in this regard :)
And so does Hyper-V.
> It's not macOS triggering this in isolation either
There are some nightmare cases that SEPOS specifically triggers, such as doing isv=0 accesses to GICR... when using the Apple vGIC handling _that_ becomes truly bizarre.
> Simply ignoring the instruction, though, is not enough
My bad! I mean, ehh, I guess you could maintain a breakpoint in the guest kernel's exception vector table or have QEMU inject its own "zero-level exception handler" whose only purpose would be to capture those HVCs, but that's not as straightforward as I originally thought. And since those PAC calls are expected to set a few Apple-specific registers anyway, using the entitlement or skipping Hypervisor.framework and talking straight to the kernel seem like the only viable options when macOS is the guest.
> There are some nightmare cases that SEPOS specifically triggers, such as doing isv=0 accesses to GICR... when using the Apple vGIC handling _that_ becomes truly bizarre.
Interesting! Are there any resources out there about virtualizing sepOS?
Not much public yet about VRE virtualisation (which includes SEP) at this point.
> whose only purpose would be to capture those HVCs
quite expensive because you get to trap ~ all EL0 -> EL1 priv transitions through the virtualisation infrastructure as the sync handler has a lot going through it
Only if you used a breakpoint or something similar. I believe a "shadow" exception vector like that can run entirely in the guest context with the guest not even being aware of this (MRS is generally always trapped so you can return the address of the real one while still taking exceptions to the injected one).
Figuring out where to put it and how to keep it mapped is another problem, though!
> I imagine the C-suite might get over it at some point.
Not at Apple. They have a looooooong institutional memory that's passed down. They're still pissed at Gizmodo. I'm shocked they made a deal with Intel. I think if it hadn't been for the global political uncertainty right now they would not have signed that fab deal with Intel.
Maybe with Tim and Jensen going on holiday together in China, the relationship might be healed somewhat.
Things have moved on since the days where GPUs in Macs were a priority.
But then the AI race has changed things. So who knows - maybe we will one day see official eGPU support from Apple and new drivers from nVidia. Wouldn't put on money on it though....
> I still believe the lack of NVIDIA GPU support in the Mac Pro will go down as one of the greatest missed opportunities in tech.
I don’t know about that. Apple supported some full size GPUs in past product lines and the number of users was very small. Granted, LLMs change that demand but the audience for Mac Pro buyers who would use a full-size GPU that is impossible to obtain is almost nothing compared to their laptop sales.
Part of the reason the new Mac Pro failed to find an audience can definitely be blamed on macOS' hostility to third party hardware. Who knows what Apple would be worth if they beat Nvidia's Grace CPU to the datacenter market. It was certainly their opportunity.
True, they could do any number of things. But a datacenter play would appear quite random to investors and their core audience. Broadcom + Nvidia however...
The Mac Pro isn't a Macbook Pro. It has socketed PCI slots and should be able to support the user's hardware in macOS' software, regardless of how Apple feels.
Seriously, the decades-long grudge against Nvidia that we always hear about seems like the most ridiculous and immature business move. I expect that kind of thing from an individual, you know, “I NEVER fly American Airlines!!!” but in business, such a permanent ban on one of the two players in a market, the leader no less. I don’t get it.
Maybe it doesn’t matter that much now because they’ve literally exited all the businesses where an external GPU is going to matter. But sticking with AMD all that time out of spite is just wild.
If you want a workstation, you are probably better off building it yourself, or having your local computer store do it. The primary exceptions are AMD strix halos or the nvidia dgx spark.
I haven’t seen a non-laughable workstation config from the big vendors since the dot com bubble. Presumably they exist, I guess?
DISCLAIMER: Only speaking for myself, not employers or affiliates.
I've been pretty darn happy with the Puget Systems custom workstation I ordered last year before the memory craze started (especially since it has 192GiB of DDR5).
I also ordered another family member a custom "Tiki" system from Falcon Northwest and that has also been quite excellent from what I've seen and they've told me.
Now is obviously not the most economical time to order a new system, but when it is appropriate (and for what it's worth) I think those are two great system builders.
I wouldn’t count them as a big vendor, but I’ve only heard good things. Local shops around here charge like $99 to put a machine together, install an OS and run burn in testing. You get more choice than an outfit like puget, but less carefully tested part / cooling selection, etc.
The last I checked, the really big players tended to add value add gimmicks (water cooling is a common one, custom psu form factors are another) with reliability / compatibility issues. That’s the tier to avoid, not the Puget systems of the world.
I picked both Puget Systems and Falcon Northwest because for the most part, both focus on pre-tested off-the-shelf parts with good reliability data from their own servicing.
My Puget Systems workstation for example has a simple AIO for cooling with some Noctua fans and a Fractal Design 7 XL full tower case.
The Tiki system I ordered for a family member from Falcon Northwest does have a custom case, but almost everything else is fairly standard inside. The super small form factor was important to them.
Could I have built either of these systems myself? Absolutely -- I've done that for at least prior 20 years or so, and I've built dozens for employers, but it sure was nice to buy one that just worked this time instead of having to having to fiddle with memory sticks or find exactly the right bios settings for stability, etc.
I'm well aware of the premium I paid but I can honestly say it has been incredibly nice to have a workstation that just works without having to fiddle with bios updates or hardware. I also don't really have the time to spare so I was entirely willing to trade funds for time.
Non-standard parts are not about value-adding, they're about cost-cutting if you're feeling charitable, and about forcing vendor lock-in if you're not.
It is too inefficient to design a machine which _might_ have two GPU and a flock of additional drives installed into it. It just makes sense to instead design around having independent hardware in its own case, which can meet its own power/cooling needs. This has been a design goal since the trashcan Mac.
Having a PCIe bus increases bandwidth and reduces latency, but once you account for eGPU and for people who would be happy building custom solutions on platforms other than macOS, there's likely not enough identified market for a modular design.
It feels like half the problem in this blog post is dealing with memory access issues induced by QEMU and the VM boundary... it's probably something dumb I'm missing, but if you boot up Ubuntu in Docker, wouldn't the NVIDIA drivers still load? And then you wouldn't have to fight Apple about the memory management because OSX would still own the memory?
> but if you boot up Ubuntu in Docker, wouldn't the NVIDIA drivers still load?
Even if the drivers loaded, they can't talk to the GPU from within docker (unless one implements PCI passthrough). MacOS owns the PCI bus in this scenario.
Literally my only complaint about Sebastian Lague is that he doesn't have enough videos. I usually put them on to fall asleep. I've thought about making my own just to include more content like that out there. The fairly deep technical topics explored in a calm but thorough way.
I don't think there should be an investigation. Data got leaked? That's a fine. Consequences happened? The people who stole it are accountable but so are the people who had the data in the first place. Just don't have the data. There are plenty of companies out there who don't have cyber security incidents despite being huge targets, what are they doing? Insurance is also a thing if companies are that worried about fines or getting sued.
I remember circa 2010 a friend of mine at college was like “blackboard sucks, let’s build something new”. At the time I poo pood the idea and lo and behold canvas came out a year later. Outside looking in, they been crushing it.
One of my mentors created Blackboard. It used to be very very good, but he sold it to private equity, and they immediately fired all of the customer support and developers, 3xd prices overnight leading to the 'blackboard sucks' problem. This gave the opening for Canvas to eventually come on to the scene and dominate.
My wife and I each have to use it as we're both following an online master's at the same university... it's definitely gone downhill (compared to the days where I originally used it ~20 yrs ago in college; tracker-riddled, slow); surprisingly, a recent change made it so that you can only attend online lessons in Chrome (haven't had time to see if this is just a user-agent thing).
I worked in a college IT department around that time and the common belief was that all LMSes suck. There are just too many different ways that too many different people want to do things that it's just bound to be hated. Kind of like Jira / Asana for software dev project management.
As someone who has used both as a student and a TA I find blackboard miles better, much easier to find what i'm looking for and my professors seem to have better luck laying out their course on blackboard than canvas.
I actually disagree, based on my time using Blackboard as an admin, student, and teacher. Although my experience is a few years out of date, I found the interface cumbersome and the performance slow.
It depends on what vintage of Blackboard your IT team has installed. We moved from a circa 2011 BB instance to Canvas in 2022, and it was hands down superior. A different university is running the most recent BB and it’s similar to Canvas.
I used both and could not tell you the major differences. I feel like they are equivalent in the bread and butter features. Most people don't use 99% of the functions they bake into these. Just use it to hold the syllabus, maybe hold the slides, submit assignments, and spreadsheet for grades. All stuff you can do with email + spreadsheet already. Maybe throw in a shared drive for larger files, which every university in the country already pays for.
They are definitely crushing it on sales. The actual product is a radioactive dumpster fire that is simultaneously hostile to students, teachers, and parents.
Cruft can be removed yes but the telemetry is literally how they measure quality. Their metrics of whether it’s good or not is the status of telemetry on a given device set and figuring out root cause. So they can’t easily roll back the telemetry without a massive rethink in their quality strategy
Along with this, which is how nearly every major piece of software works today, removing telemetry likely wouldn't have a measurable impact on performance or memory usage. Those aren't heavy subsystems.
I was at Microsoft for that particular transition but it is still wild to me that a post about improving quality starts with details about windows insiders.
reply