Search All of the Math Forum:

Views expressed in these public forums are not endorsed by NCTM or The Math Forum.

Notice: We are no longer accepting new posts, but the forums will continue to be readable.

Replies: 0

 dpb Posts: 9,850 Registered: 6/7/07
Posted: Jan 11, 2013 12:55 AM

Hi, I have a problem which is bothering me all the time.

I have a float value, say AAA, which is equal to 0.004

now in a very simple judgement function:

if AAA == 0.004
...

it will never enter into the condition section.
Every time, I have to do this:

if round(AAA*1000)/1000 == 0.004
...

<http://matlab.wikia.com/wiki/FAQ#Why_is_0.3_-_0.2_-_0.1_.28or_similar.29_not_equal_to_zero.3F>

Also DAGS for 'Goldberg-What every computer scientist should know about
floating point' -- I seem to have mislaid my convenient link at the
moment, sorry.

It's inherent in floating point as the above show in lesser/greater detail.

if (abs(AAA)-setValue) > tolVal

is alternative way to make a comparison where tolVal is something like
1E-7*setValue or other suitable tolerance value.

The point is that _exact_ comparisons to floating point values is often
going to cause grief owing to a bit or so in the representation that can
be off owing to whether the value is computed or entered at the command
line or otherwise obtained in slightly different manner. Even order of
computation can cause a rounding difference as can whether an
intermediate value is held at a higher precision in a floating point
register during computations rather than stored between operations.

--