
Re: numerical precision of sqrt(a^2+b^2) with 16 bit floating point numbers
Posted:
Feb 22, 2014 6:08 PM


On Friday, February 21, 2014 9:22:58 AM UTC, Thomas Plehn wrote: > Excuse my short description, but what about the precision here? > > I am noot good in numerics, excuse me...
You have exactly the same problems with a naive calculation as with 32 bit, 64 bit or 80 bit precision.
1. If either a or b or both are large enough, then calculating a^2 + b^2 will overflow and you won't get a reasonable result, even though sqrt (a^2 + b^2) is nowhere near large enough to overflow.
2. If a and b are both quite small, then calculating a^2 + b^2 will underflow with massive loss of precision, or even give a result of zero, so you won't get a reasonable result, even though sqrt (a^2 + b^2) is nowhere near small enough to underflow.
Two solutions: a. You prove that a and b are not large enough that a^2 + b^2 could overflow. You also prove that either a or b is large enough so that the square doesn't underflow, and that an underflow in the other square won't afffect the result much. b. You check the magnitude of a and b. If the numbers are too big or too small you multiply both with some power of two so that the calculation will be fine, then divide the result by the same power of two.

