Search All of the Math Forum:

Views expressed in these public forums are not endorsed by NCTM or The Math Forum.

Notice: We are no longer accepting new posts, but the forums will continue to be readable.

Topic: precision of fractional numbers
Replies: 7   Last Post: Oct 8, 2013 10:38 AM

 Messages: [ Previous | Next ]
 dpb Posts: 9,850 Registered: 6/7/07
Re: precision of fractional numbers
Posted: Oct 8, 2013 10:38 AM

On 10/8/2013 9:16 AM, someone wrote:
> "Bruno Luong" <b.luong@fogale.findmycountry> wrote in message
> <l30tbi\$jau\$1@newscl01ah.mathworks.com>...

>> > > round(x*100)/100
>> >
>
> Consider the OPs original request:
> More specifically, I would like Matlab to consider 3.257 as 3.26. It is
> feasible for a single number, but I am looking for a command to use in
> the first line of the script and that applies same format to all the
> numbers along the calculations.
>
> First, the above would apply "for a single number" but not for every
> number in the entire script.
>
> Second, although 3.257 would APPEAR to be 3.26 (to two decimal digits),
> I don't believe it would necessarrily be EXACTLY 3.26 due to the way
> MATLAB (or a computer, in general) stores floating point numbers. (I
> don't have MATLAB installed on this computer, so I can't give an example
> using the short and long formats.)

The question is one of interpretation of what OP means and has,
indeed...Bruno and I both basically assumed an array to be used
subsequently in which case the rounding will occur to the nearest as can
be held in dp in a float, yes.

If there are other values beyond that array on which our solutions
operate then they'll have to be treated separately.

OTOH, if it's simply a case of output format and OP doesn't _really_
care about internal representation but only the final results, then
using a format string is generally the simpler way to get the appearance
of a given precision since the Matlab workspace FORMAT display has a
limited amount of flexibility in that regard.

OTOH, if OP really means to use only N decimal digits in calculations,
that's another completely separate question that needs something like
the Fixed Point Toolbox or other implementation techniques than simply
either rounding or output formatting. (I've not used FP TB so can't
address whether it has decimal-based precision or only bit-width
implementations).

--

Date Subject Author