Forums > Javascript

flonum accuracy problem

October 10, 2010 | 8:26 pm

Hi,

I am having a problem with getting accurate representations of floats in flonum.

I am using a JavaScript to send a float to a flonum. Just for testing I did myPI = Math.PI; then I send myPI.toFixed(10); to the Max Window, and send parseFloat(myPI); to another outlet patched to a flonum.
The Max Window shows 3.1415926536;
The flonum shows 3.1415927410 (while in the flonum inspector decimals are set to 10);

I can’t get the flonum to show anything better than 6 decimals.
In the flonum help window it says "Clicking in the Float Display Correction checkbox in the Interface pane of the Max Preferences Window will make the Max application round numbers and truncates them at what it guesses is the least significant digit."

But whether I check or uncheck that checkbox has no influence on the behavior of the flonum object. It only influences the behavior of message boxes (on my system anyhow):

Live 8.2
M4L 5.1.2
Mac OS X 10.6.4
MacBook Pro Intel

Is it supposed to be like that, or am I doing something wrong? I mean in the flonum inspector I can set the decimals to more than six. Is it really only supposed to work correctly for up to six decimals?

/ekke


October 11, 2010 | 12:50 am

yes.


Viewing 2 posts - 1 through 2 (of 2 total)