The Math Forum

Search All of the Math Forum:

Views expressed in these public forums are not endorsed by NCTM or The Math Forum.

Math Forum » Discussions » Software » comp.soft-sys.matlab

Notice: We are no longer accepting new posts, but the forums will continue to be readable.

Topic: about float value
Replies: 0  

Advanced Search

Back to Topic List Back to Topic List  

Posts: 9,850
Registered: 6/7/07
about float value
Posted: Jan 11, 2013 12:55 AM
  Click to see the message monospaced in plain text Plain Text   Click to reply to this topic Reply

Hi, I have a problem which is bothering me all the time.

I have a float value, say AAA, which is equal to 0.004

now in a very simple judgement function:

if AAA == 0.004

it will never enter into the condition section.
Every time, I have to do this:

if round(AAA*1000)/1000 == 0.004


Also DAGS for 'Goldberg-What every computer scientist should know about
floating point' -- I seem to have mislaid my convenient link at the
moment, sorry.

It's inherent in floating point as the above show in lesser/greater detail.

if (abs(AAA)-setValue) > tolVal

is alternative way to make a comparison where tolVal is something like
1E-7*setValue or other suitable tolerance value.

The point is that _exact_ comparisons to floating point values is often
going to cause grief owing to a bit or so in the representation that can
be off owing to whether the value is computed or entered at the command
line or otherwise obtained in slightly different manner. Even order of
computation can cause a rounding difference as can whether an
intermediate value is held at a higher precision in a floating point
register during computations rather than stored between operations.


Point your RSS reader here for a feed of the latest messages in this topic.

[Privacy Policy] [Terms of Use]

© The Math Forum at NCTM 1994-2018. All Rights Reserved.