Search All of the Math Forum:

Views expressed in these public forums are not endorsed by NCTM or The Math Forum.

Notice: We are no longer accepting new posts, but the forums will continue to be readable.

Topic: numerical precision of sqrt(a^2+b^2) with 16 bit floating point numbers
Replies: 16   Last Post: Feb 27, 2014 6:52 PM

 Messages: [ Previous | Next ]
 gnasher729 Posts: 419 Registered: 10/7/06
Re: numerical precision of sqrt(a^2+b^2) with 16 bit floating point numbers
Posted: Feb 22, 2014 6:20 PM

On Friday, February 21, 2014 9:34:42 PM UTC, Roland Franzius wrote:

> Yo can have it exactly to one bit. Suppose a>b and set a=2^m A, b=2^m B
> with 1/sqrt(2) < A < sqrt(2). Then ...

You can very easily have a result that is almost always correctly rounded by doing the calculation with 32 bit. Now using 32 bit instead of 16 would presumably cost extra time but is very easy to do. Therefore to be useful, a solution using only 16 bit arithmetic must not be more costly than just switching to 32 bit.

Date Subject Author
2/21/14 Thomas Plehn
2/21/14 Thomas Plehn
2/21/14 Peter Percival
2/21/14 Wizard-Of-Oz
2/21/14 David C. Ullrich
2/22/14 Wizard-Of-Oz
2/22/14 gnasher729
2/22/14 Wizard-Of-Oz
2/21/14 Roland Franzius
2/22/14 gnasher729
2/23/14 Roland Franzius
2/27/14 Phil Carmody
2/21/14 Roland Franzius
2/21/14 James Waldby
2/27/14 Phil Carmody
2/22/14 gnasher729