The Math Forum

Search All of the Math Forum:

Views expressed in these public forums are not endorsed by NCTM or The Math Forum.

Math Forum » Discussions » Software » comp.soft-sys.matlab

Notice: We are no longer accepting new posts, but the forums will continue to be readable.

Topic: precision of fractional numbers
Replies: 7   Last Post: Oct 8, 2013 10:38 AM

Advanced Search

Back to Topic List Back to Topic List Jump to Tree View Jump to Tree View   Messages: [ Previous | Next ]

Posts: 9,850
Registered: 6/7/07
Re: precision of fractional numbers
Posted: Oct 8, 2013 10:38 AM
  Click to see the message monospaced in plain text Plain Text   Click to reply to this topic Reply

On 10/8/2013 9:16 AM, someone wrote:
> "Bruno Luong" <b.luong@fogale.findmycountry> wrote in message
> <l30tbi$jau$>...

>> Sorry should read
>> > > round(x*100)/100
>> >
> Consider the OPs original request:
> More specifically, I would like Matlab to consider 3.257 as 3.26. It is
> feasible for a single number, but I am looking for a command to use in
> the first line of the script and that applies same format to all the
> numbers along the calculations.
> First, the above would apply "for a single number" but not for every
> number in the entire script.
> Second, although 3.257 would APPEAR to be 3.26 (to two decimal digits),
> I don't believe it would necessarrily be EXACTLY 3.26 due to the way
> MATLAB (or a computer, in general) stores floating point numbers. (I
> don't have MATLAB installed on this computer, so I can't give an example
> using the short and long formats.)

The question is one of interpretation of what OP means and has,
indeed...Bruno and I both basically assumed an array to be used
subsequently in which case the rounding will occur to the nearest as can
be held in dp in a float, yes.

If there are other values beyond that array on which our solutions
operate then they'll have to be treated separately.

OTOH, if it's simply a case of output format and OP doesn't _really_
care about internal representation but only the final results, then
using a format string is generally the simpler way to get the appearance
of a given precision since the Matlab workspace FORMAT display has a
limited amount of flexibility in that regard.

OTOH, if OP really means to use only N decimal digits in calculations,
that's another completely separate question that needs something like
the Fixed Point Toolbox or other implementation techniques than simply
either rounding or output formatting. (I've not used FP TB so can't
address whether it has decimal-based precision or only bit-width


Point your RSS reader here for a feed of the latest messages in this topic.

[Privacy Policy] [Terms of Use]

© The Math Forum at NCTM 1994-2018. All Rights Reserved.