Search All of the Math Forum:
Views expressed in these public forums are not endorsed by
NCTM or The Math Forum.


dpb
Posts:
9,259
Registered:
6/7/07


Re: precision of fractional numbers
Posted:
Oct 8, 2013 10:38 AM


On 10/8/2013 9:16 AM, someone wrote: > "Bruno Luong" <b.luong@fogale.findmycountry> wrote in message > <l30tbi$jau$1@newscl01ah.mathworks.com>... >> Sorry should read >> > > round(x*100)/100 >> > > > Consider the OPs original request: > More specifically, I would like Matlab to consider 3.257 as 3.26. It is > feasible for a single number, but I am looking for a command to use in > the first line of the script and that applies same format to all the > numbers along the calculations. > > First, the above would apply "for a single number" but not for every > number in the entire script. > > Second, although 3.257 would APPEAR to be 3.26 (to two decimal digits), > I don't believe it would necessarrily be EXACTLY 3.26 due to the way > MATLAB (or a computer, in general) stores floating point numbers. (I > don't have MATLAB installed on this computer, so I can't give an example > using the short and long formats.)
The question is one of interpretation of what OP means and has, indeed...Bruno and I both basically assumed an array to be used subsequently in which case the rounding will occur to the nearest as can be held in dp in a float, yes.
If there are other values beyond that array on which our solutions operate then they'll have to be treated separately.
OTOH, if it's simply a case of output format and OP doesn't _really_ care about internal representation but only the final results, then using a format string is generally the simpler way to get the appearance of a given precision since the Matlab workspace FORMAT display has a limited amount of flexibility in that regard.
OTOH, if OP really means to use only N decimal digits in calculations, that's another completely separate question that needs something like the Fixed Point Toolbox or other implementation techniques than simply either rounding or output formatting. (I've not used FP TB so can't address whether it has decimalbased precision or only bitwidth implementations).




