I would definitely argue against that given the perf that you can get out of CSPRNGs. AES/CTR could give you over 100 million random bytes per second back in 2009 (source: https://www.cryptopp.com/benchmarks.html).
How much faster do you need your RNG to be in any non-niche situation?
The niche situations aren't going to use the built-in rand() anyway because they can write their own or use someone else's that's faster. Even if they end up using the same algorithm (e.g. a Mersenne Twister) they can still often go faster.
Interestingly according to https://bocoup.com/weblog/random-numbers Firefox is just going to use its CSPRNG once the crypto api is ready. (Bug 322529 is still in NEW state, though.)
How much faster do you need your RNG to be in any non-niche situation?