Search All of the Math Forum:

Views expressed in these public forums are not endorsed by NCTM or The Math Forum.

Notice: We are no longer accepting new posts, but the forums will continue to be readable.

Topic: numerical precision of sqrt(a^2+b^2) with 16 bit floating point numbers
Replies: 16   Last Post: Feb 27, 2014 6:52 PM

 Messages: [ Previous | Next ]
 gnasher729 Posts: 419 Registered: 10/7/06
Re: numerical precision of sqrt(a^2+b^2) with 16 bit floating point numbers
Posted: Feb 22, 2014 6:08 PM

On Friday, February 21, 2014 9:22:58 AM UTC, Thomas Plehn wrote:
> Excuse my short description, but what about the precision here?
>
> I am noot good in numerics, excuse me...

You have exactly the same problems with a naive calculation as with 32 bit, 64 bit or 80 bit precision.

1. If either a or b or both are large enough, then calculating a^2 + b^2 will overflow and you won't get a reasonable result, even though sqrt (a^2 + b^2) is nowhere near large enough to overflow.

2. If a and b are both quite small, then calculating a^2 + b^2 will underflow with massive loss of precision, or even give a result of zero, so you won't get a reasonable result, even though sqrt (a^2 + b^2) is nowhere near small enough to underflow.

Two solutions: a. You prove that a and b are not large enough that a^2 + b^2 could overflow. You also prove that either a or b is large enough so that the square doesn't underflow, and that an underflow in the other square won't afffect the result much. b. You check the magnitude of a and b. If the numbers are too big or too small you multiply both with some power of two so that the calculation will be fine, then divide the result by the same power of two.

Date Subject Author
2/21/14 Thomas Plehn
2/21/14 Thomas Plehn
2/21/14 Peter Percival
2/21/14 Wizard-Of-Oz
2/21/14 David C. Ullrich
2/22/14 Wizard-Of-Oz
2/22/14 gnasher729
2/22/14 Wizard-Of-Oz
2/21/14 Roland Franzius
2/22/14 gnasher729
2/23/14 Roland Franzius
2/27/14 Phil Carmody
2/21/14 Roland Franzius
2/21/14 James Waldby
2/27/14 Phil Carmody
2/22/14 gnasher729