Notice how the active OS X builds reaches the limit of 216 jobs and the backlog of OS X jobs increases without decreasing. This is a capacity issue that is likely related to high OS X build costs.
Another thing to notice is that the backlog correlates strongly with time of day in the EST timezone, where the backlog starts to take off after 9am EST.
Their slowest build jobs are all on Darwin/OSX-based systems and (notably) this is a problem numerous other projects have -- slow jobs on OSX build systems. There is a large backlog of OSX build jobs that starts around 9am EST.
Is that true for the paid version of Travis as well? If not, might be worth paying for it, to support a community as large as Rust's. (If the paid version is that backlogged as well, I'm surprised they don't put more resources into accelerating it; perhaps not enough of their customers care about macOS?)
Alternatively, given that Mozilla has their own extensive CI infrastructure (for Firefox), perhaps it'd make sense to migrate to that instead of Travis?
> Given that Mozilla has their own extensive CI infrastructure
While Mozilla does sponsor a lot of Rust stuff, it's fundamentally a community project, and so many of us (including myself) would prefer to not to move closer to something that's Mozilla-specific.
FWIW Mac jobs are also often capacity-limited on Mozilla infrastructure. It's not unusual to have thousands of jobs waiting on hundreds of machines, which leads to multi-hour queues at peak times.
Fundamentally the problem with CI testing on Mac is that Apple don't provide the components you want to build a large pool of machines at reasonable cost, so everyone ends up doing slightly mad things involving hundreds of mac minis (obviously the Windows situation also isn't great; this kind of infrastructure loves free software on commodity hardware).
Have you heard anything from them about improving the issues with macOS? You've certainly got a case study in it being a problem.
> While Mozilla does sponsor a lot of Rust stuff, it's fundamentally a community project, and so many of us (including myself) would prefer to not to move closer to something that's Mozilla-specific.
Fair enough; I only suggested it because I had the impression that Mozilla's bits were all based on Open Source systems, and it would just be a matter of sharing hosting infrastructure rather than duplicating it.
I'm not on the infra team so I'm not sure; I do know they pay attention to this stuff though.
Yeah, it's not really so much "it's not open source" as it is "Mozilla is the only organization who's really using this stuff". I used to worry about this a lot, but I think we've mostly gotten away from it now that a lot of organizations are using Rust. That is, if Rust was a Mozilla-only technology, I think Rust would have failed its mission. As such, I'm wary of things that seemingly tie Rust back into Mozilla too specifically.
> The Mac queues for travis-ci.org and travis-ci.com are separate in terms of provisioned capacity, but use the same underlying infrastructure. We make sure we have enough provisioned capacity for our customers on travis-ci.com who are testing private Repositories on our Mac infrastructure. For travis-ci.org we provide a greater capacity, in terms of concurrent VMs, but have had to make some hard decisions on how and when to increase this capacity as the cost for helping the open source community is fairly high.
I'm curious which configuration that is, and how it could be accelerated.