Perfect pitch is not correlated with the ability to see written notes and immediately know how to play them on any instrument. It is only barely correlated with the same but for singing the note.
I do not have perfect pitch, but I could always do what GP's kid does. I am now a pretty good amateur singer.
What matters for this is learning to read the music notation and developing a sense (mostly subconscious) of scales and harmony. Perfect pitch does not help either of those things.
There is a paper by Vincent Lefèvre that indeed proves that floating point division and a floor implement the euclidean division, with a careful analysis of when. [1] A corollary of the main theorem in the paper is that, with round to nearest, if x and y fit in 53 bits unsigned then it works out.
Agreed, correctly rounded libm functions are great, as long as they don't have miserable worse case behavior (as was famously the case with glibc's pow at one point).
One thing I was thinking of doing is manually SLP-vectorizing the high-precision fallbacks that they use when they're close to a rounding boundary, so that you can get better worst-case behavior – but obviously it's good enough already for most purposes.
I'm honestly surprised though that JS engines don't just keep using fdlibm though. The ECMAScript spec explicitly encourages it iirc. And if Math.tanh is on your hot path in JavaScript then you're doing something quite bizarre...
> as was famously the case with glibc's pow at one point
Pow is famously hard anyway because it's bivariate and there is no currently known way to work around the table-maker's dilemma (TMD). CORE-MATH even crashes upon a new required precision record, because it intentionally avoids Ziv's rounding.
I was a bit puzzled by your second sentence, so I searched around a bit and... do I have this right?
- There’s a well-known way (“Ziv’s rounding”) to get (among other things) a correctly rounded double-precision pow(), but in bad cases it can get slow, meaning really quite slow in practice and we’ve got no idea how slow in the worst case (nobody knows what the worst case is).
- There’s a recent, guaranteed-correct way[1] to get (specifically) a correctly rounded double-precision pow(), but the last step requires “enough” precision and we’ve got no idea how much that actually is, so CORE-MATH uses 256 bits of mantissa and crashes if that turns out not to be enough (no such cases are currently known).
- (Bonus) For most special functions [not just the bivariate ugly duckling of pow()], there’s essentially no hope of getting a correctly rounded quad-precision version anytime soon.
You are right. (Ziv's rounding is essentially a method that keeps error bounds and increases the precision on the inconclusive result.) I think atan2 suffers from the same problem as well, but haven't actually looked up.
At what point are we going to realize that computers will never work for floating point math. If your numbers are not exact there will always be a better solution for what you are doing that does not involve a computer.
I never understood why fixed precision, and integer math isn't more popular. In engineering, we used fixed point all the time, it ran on much simpler hardware and the error is mathematically easy to model. IEEE 754 floats are not only suspect when it comes to theory, but are often outperformed with integers smaller than the mantissa (so less than 24 bits of int can beat a 32 bit float), when it comes to things like loss of precision.
I recommend pretty much everyone avoid fixed point and other float alternatives, barring exceptional cases after you've done your own numerical analysis, or you lack floating point hardware (rare these days).
Yes, fixed point can use simpler hardware. That's also a completely irrelevant consideration for software. The vast majority of processors are optimized for floats now and some operations (e.g. division) are actually faster.
The precision argument also falls apart. Any float with mantissa >= X+Y can get exactly the same results as a QX.Y fixed point. The float will actually perform better across the same range because you have to round it to perform like the fixed point. That means more precision, lower error, automatic normalization, better overflow behavior, a larger working range, etc. And it'll probably be just as fast, unless you're bottlenecked on memory bandwidth of inputs (unlikely). When you inevitably want an exp() or another special function, it's a heck of a lot easier to call libm than implement your own and it will perform better.
Floats are also much easier to get right for your coworkers that aren't numerical analysts.
I didn't recommend fixed point for simpler HW - I recommended it for better precision (if you know what you are doing). First, a point I didn't make, is that if you have 32 bits of fixed, you get way more precision than with a 32 bit float. But I can think of a pretty common case where a 24 bit int would win against a 32bit float: convolution filters.
If you have a filter whose inputs are supposed to sum up to 1 (which is the most common case), integer computations mean that, even with internal overflows, the end result will be correct. In contrast, with floats, you can lose precision. If you apply said operation 10000x recursively (say, you are 'stepping' a simulation), those errors can add up bigtime.
> Floats are also much easier to get right for your coworkers that aren't numerical analysts.
That one is true, however, when you have people, such as EEs who really care about precision, and know the theory behind it, then floats are often not the obvious choice. It has other advantages, like your calculation running the exact same regardless of CPU and/or compiler, which I'm sure a lot of analysts care about. Afaik finance people don't even use floats for things like account balances, because you can't represent something like 0.1$ exactly.
Fixed point has basically no language support, and is very hard to get right, but sometimes you need to do that.
Do you have any subject matter expertise in quantization errors? Like doing simulations or DSP work? Not trying to be antagonistic, just figure out where you're coming form.
> if you have 32 bits of fixed, you get way more precision than with a 32 bit float
You get 7 more bits for the most extreme numbers. Which is a good portion of 32, but not crazy. By the time you hit double precision you're only sacrificing 10 of your 64 bits to make your range considerations a hundred times simpler.
First, a point I didn't make, is that if you have 32 bits of fixed, you get way more precision than with a 32 bit float.
That's true, but I already responded to it. If you step up to the next size of float (e.g. f64), you have more precision than the fixed32. You can do exactly the same computation in f64 with equivalent inputs, and you'll get better precision than doing it in fixed32. Or you can round at every step like fixed does and get a bit-equivalent value if you don't want the precision. It's less memory efficient, but my point is that the remaining use cases for fixed point are exceptional/situational and getting increasingly niche.
Maybe using a bigger float type is cheating, but it's basically free because of the ubiquitous support for floats.
In contrast, with floats, you can lose precision.
This only happens with values outside the range of your fixed point type if you use larger floats as mentioned above. I consider that a different argument. You can alternatively view this as the float handling a situation more gracefully than fixed point would have.
Afaik finance people don't even use floats for things like account balances, because you can't represent something like 0.1$ exactly.
Finance types typically use decimal types from what I understand. This is really just the result of using a decimal syntax to initialize/output a binary representation. Fixed point has exactly the same problem. Decimals have an analogous issue with the value 1/3.
It has other advantages, like your calculation running the exact same regardless of CPU and/or compiler, which I'm sure a lot of analysts care about.
I wrote a library that makes floats more practically deterministic across platforms for very little cost (linked at [0] so you can see the limitations and numbers), and the underlying problem is [maybe] getting a standardized solution in C++29. You can get the same thing today just by changing compiler flags. If you need the special, non-reproducible float functions, your options are mainly to import a library or implement it yourself, same as fixed point.
Not trying to be antagonistic, just figure out where you're coming form.
I work in safety-critical automotive/robotics, used to do audio DSP, contributed a bit to the aforementioned standardization, etc. I also have a talk on this topic I've been working on for the last few weeks. It's a bit of a pet subject.
Fixed point has basically no language support, and is very hard to get right, but sometimes you need to do that.
There are absolutely situations for it, but that's exactly it: it's situational. And those situations are increasingly uncommon these days, now that hardware with good IEEE support is essentially ubiquitous and compilers/standard libraries are improving their implementations.
Thanks for answering my points in detail! I think this is one of those domains where there's enough theory and practical considerations, that basically given the same set of constraints, there's generally one set of correct conclusions.
To be clear I'm not anti-float, as they have less surprising behavior than fixed point, and are much less fiddly, but I do have to note that f32 sits at that awkward spot where it's not accurate enough for a lot of numerical work, but stepping up to f64 carries a significant performance penalty on things like GPUs, and most DSPs don't really give you f64 hardware at full rate, if at all.
In contrast, 32 bit fixed point gives you 6 extra bits of precision, (IEEE754 mantissa should count as 26 bit), which can often be the saving grace.
For example, in video games, if you mandate a 0.01mm precision, with int32, you get a 40kmX40km area, which is plenty, and with float32, you have to divide every dimension by 64, which is not enough for even mid-sized maps, and you have to employ tricks or go straight to f64.
Totally agreed that f32s can be awkward. I'm really just arguing that most of the time, you're better off starting with floats and seeing if there are issues. If there are, you can often take advantage of float tooling (e.g. fpchecker [0]) to make better choices about how to proceed, since virtually no one does numerical analysis on commercial codebases in my experience. Sometimes you can skip that if there's an obvious reason (e.g. no float HW, FPGAs), but the general direction of software seems to be towards universal availability.
You're right about that particular constraint, though I'd question why the achievable 2-4mm precision at 32 or 64km are meaningfully different. Covering up unstable collision code and adaptive methods would be a rewrite?
> The vast majority of processors are optimized for floats now and some operations (e.g. division) are actually faster.
This seems backwards. Hardware is optimized for floats because people use floats. If people used fixed point, hardware would become optimized for that instead.
Given an equal number of transistors, I'm pretty sure fixed point would be a lot faster on equally optimized hardware for almost all operations.
fixed-point provides uniform precision, exact integer-scaled arithmetic, is deterministic whereas floating point is more convenient but its not a panacea
As I said, floats can provide results that are no worse than a specified fixed point type. So if you want uniform absolute precision, just round down to the required precision.
Floating point is generally deterministic in practice with a fairly minor amount of effort, the major remaining issue being library rounding. I actually wrote a library that guarantees this for arbitrary code, with some small, obvious caveats like standard library precision. And the conference talks linked above note, the standard library issues are an increasingly solved problem for modern toolchains. The remaining cases are mostly things you won't do in fixed point. Let me know if you're aware of anyone computing erfc in fixed point for determinism though.
I'm not saying there aren't any situations where other systems are justified, but you probably won't know if you fall into any of them without the kind of numerical analysis that most codebases will never receive.
Totally fair if you have full control, my experience is often with databases where you can get warnings that float implementations can even change per-operating system (thanks MySQL) or per query plan (based on plan ordering) which is ... pretty bad!
I don't know much about real query planners, but if I understand what you're saying there may be ways to improve the situation.
If you stick to the safe bits I've been discussing elsewhere in this thread and your platforms implement IEEE floats, float math will also be commutative + associative and you won't have to deal with precision loss. That means your usable range will be narrower than the same size fixed type (because it's limited by the mantissa) but it's large enough to still be useful.
Fixed-point arithmetic provides uniform absolute precision; floating-point arithmetic provides almost-uniform relative precision, as used by scientists and engineers for literally centuries (“significant digits” except the binary version is more uniform than the decimal one customary with pencil and paper).
Hardware floating point on CPUs (including SIMD units) is almost always IEEE 754 compliant these days (excepting only IBM’s weird fantasy land), and there the rules for the non-YOLO operations (+, -, *, /, sqrt, fma) are completely unambiguous and deterministic: treating the inputs as exact, compute the mathematically exact result, then either return it as is if it is exactly representable as a floating-point number, or if it’s not then round it to one that is according to the current rounding mode.
Things that can mess this up:
- GPUs just do whatever they feel like will make them look faster on benchmarks, don’t count on anything.
- Transcendental functions (exp, sin, etc.) are really hard (multiple literal PhDs) to implement according to the rules I’ve just described (“correct rounding”), so you’ve only been able to get such implementations in the last few years, and I believe no stock libm has completely switched so bring your own if you need them.
- Decimal-to-binary and binary-to-decimal conversions, by contrast, are not that hard to implement according to the rules in principle (it’s making them fast that’s difficult), yet Microsoft couldn’t get it right for literal decades, so if you need Windows then double-check CRT versions and bring in well-known open-source conversion code as necessary.
- Denormal inputs or outputs are very slow in some implementations, leading to a hardware option to flush them to zero. Either make sure to not produce them or keep an eye on the option.
- The precise bit pattern of the NaNs you get for invalid inputs may differ across platforms. Either make sure to not produce them (you really shouldn’t) or canonicalize upon de/serialization.
- Sometimes compilers will try to HALP by performing e.g. single-precision math in double-precision accumulators and only rerounding upon store to memory; by fusing * followed by + (two roundings) to hardware fma (only one); by reassociating; etc. Take care to prohibit your compiler from doing these shenanigans (no -ffast-math or -funsafe-math-optimizations ever, in your code or in any dependencies, and God help you if you’re on MSVC).
- Most shamefully, the 8087 (despite spawning the entire IEEE standard in the first place) tried to HALP by using 80-bit registers, so if you need x86-32 then be especially careful with compiler settings (I seem to remember the HALP mode might even be ABI-mandated on some 32-bit platforms so you’ll need to violate that).
The concept of floating point is solid, the IEEE standard is stellar, but the superstructure around it is just—not, requiring an unnecessary amount of vigilance to just make it work as designed.
- Transcendental functions (exp, sin, etc.) are really hard (multiple literal PhDs) to implement according to the rules I’ve just described (“correct rounding”), so you’ve only been able to get such implementations in the last few years, and I believe no stock libm has completely switched so bring your own if you need them.
This is right, but it's useful to point out how important the "correct rounding" qualification is to the difficulty of the problem. Writing a "good enough" function with floats is easy even for non-experts. exp can be efficiently implemented in hardware with a 4 element lookup table and polynomial interpolation. Sin/Cos are range reduction + a minimax polynomial from sollya. But the standard [rightly] doesn't prescribe specific implementations, so you need fully correct rounding to have cross-platform determinism.
Correct transcendentals are also difficult in fixed point. When pigweed was implementing them for their fixed impl, they got help from from an ex. Mathworks floating point expert on the LLVM team with a relevant PhD to do it correctly [0].
I go to the conference frequently and know the organizing committee. It depends on the year (2025 and 2026 did it, but not 2024). And if someone decides to register arith20xx.org, I doubt the conference would buy it.
Oh waw, I had totally forgotten about the handkerchiefs. But this is indeed how I was first thaught juggling when I was a kid. Thanks for the trip down the memory lane!
It seems to me that there's a certain "blindness" between two compiler worlds.
Compiler engineers for mostly linear-memory languages tend to only think in terms of SSA, and assume it's the only reasonable way to perform optimizations. That transpires in this particular article: the only difference between an AST and what they call IR is that the latter is SSA-based. So it's like for them something that's not SSA is not a "serious" data structure in which you can perform optimizations, i.e., it can't be an IR.
On the other side, you actually have a bunch of languages, typically GC-based for some reason, whose compilers use expression-based structures. Either in the form of an AST or stack-based IR. These compilers don't lack any optimization opportunities compared to SSA-based ones. However it often happens that compiler authors for those (I am one of them) don't always realize all the optimization set that SSA compilers do, although they could very well be applied in their AST/stack-based IR as well.
I think the WASM world is a clear example that bridges the gap you're describing.
You usually compile from SSA to WASM bytecode, and then immediately JIT (Cranelift) by reconstructing an SSA-like graph IR. If you look at the flow, it's basically:
Graph IR -> WASM (stack-based bytecode) -> Graph IR
So the stack-based IR is used as a kind of IR serialization layer. Then I realized that this works well because a stack-based IR is just a linearized encoding of a dataflow graph. The data dependencies are implicit in the stack discipline, but they can be recovered mechanically.
Once you see that, the blindness mostly disappears, since the difference between SSA/graph IRs and expression/stack-based IRs is about how the dataflow (mostly around def-use chains) is represented rather than about what optimizations are possible.
Fom there it becomes fairly obvious that graph IR techniques can be applied to expression-based structures as well, since the underlying information is the same, just represented differently.
Didn't look close enough to JSIR, but from looking around (and from building a restricted Source <-> Graph IR on JS for some code transforms), it basically shows you have at least a homomorphic mapping between expression-oriented JS and graph IR, if not even a proper isomorphism (at least in a structured and side-effect-constrained subsets).
Only compilers that already had an SSA-based pipeline transform SSA to stack-based for Wasm. And several don't like that they have to comply with Wasm structured control flow (which, granted, is independent from SSA). Compilers that have been using an expression-based IR directly compile to Wasm without using an SSA intermediary.
I was imprecise, I was specifically thinking of already SSA-based tech.
My broader point is that for SSA-based pipelines targeting Wasm, translation between SSA/graph IR and stack-based IR is largely mechanical and efficient. Whether a compiler uses SSA as an intermediary or goes straight from an AST to Wasm, the fact remains that you can round-trip between a SSA-like IR and a stack-based IR without losing the underlying dataflow information.
Yeah, mapping is not canonical and some non-semantic structure is not preserved (evaluation order, materialization points, join encoding, CFG reshaping for structured control and probably some more structure I'm not familiar with), but optimization power is unaffected.
And JSIR seems to be based on an even stronger assumption.
Would appreciate corrections if you see things differently.
Hum, IIRC, using your definition of an AOT compiler, then V8 is an AOT compiler. V8 never interprets code. It immediately compiles it to machine code. It improves it later, but it's never slow.
V8 is a JIT compiler that uses the Ignition [1] interpreter and only compiles sections of code down to machine instructions once they've been marked as hot via TurboFan [2].
V8 can also go back and forth from machine instructions back to bytecode if it identifies that certain optimization assumptions no longer hold.
I do not have perfect pitch, but I could always do what GP's kid does. I am now a pretty good amateur singer.
What matters for this is learning to read the music notation and developing a sense (mostly subconscious) of scales and harmony. Perfect pitch does not help either of those things.
reply