## Floating-point rounding error?

Feb 17 2010 | 10:32 am
Hi guys,
I'm working on set of boxes for controlling floating-point number entry into Max. I've come across what seems to be a rounding error in floating-point number boxes, and I was wondering if this was a mistake on my part or whether it's a bug.
I've attached the files below. To make the patch work you type an integer into the left hand side of the number and scroll through the decimal portion using the + and - signs.
If the integer value is '0' it works fine, but as the value increases an error is introduced into the calculation when the decimal portion is changed. The javascript is outputing its values to the MAX window for comparison and the numbers seem to be fine, so the error seems to be happening in Max.
The error gets larger the higher a value you use, so it doesn't happen at 1 - 10, starts at around 20 and by the time you get to 100,000 is around 0.002.
I was wondering if anyone else has come across this and if there's a fix.
I've attached the patch along with the relevant javascript files.
Thanks!
Tim

• Feb 17 2010 | 10:39 am
Hmm ..... I don't think the attachment worked so I've pasted the files below. The names of the three javascript files are at the top of the code for each:
//##### plusone.js ##########
// plusone.js
//Increments or decrements a number box by one //Rolls over between 9 and 0 and outputs a bang
inlets = 1; outlets = 3; autowatch = 1;
var x = 0;
function add() { if (x == 9) { x = 0; outlet(1, "bang"); } else { x = x+1; } outlet(0, x); }
function sub() { if (x == 0) { x = 9; outlet(2, "bang"); } else { x = x-1; } outlet(0, x); }
function zero() { x = 0; outlet (0, x); }
//###### isint.js ######
// isint.js
//tests input to see if it is an int //if so outputs int otherwise outputs zero
inlets = 1; outlets = 1; autowatch = 1;
function text(t) { obj = this.patcher.getnamed("tbx_Number"); t = parseInt(t, 10);
if (isNaN(t)) { t=0; }
outlet(0, t); obj.set(t); }
//###### buildnum.js ######
//buildnum.js //Joins elements from number boxers to make float
inlets = 1; outlets = 1; autowatch = 1;
var whole = 0; var ten = 0;
function wholenum(a) { whole = parseInt(a, 10); result(); }
function tennum(b) { ten = b; result(); }
function result() { var numout = 0;
numout = whole + (ten/10) outlet (0, numout); post(numout); post(); }
• Feb 17 2010 | 9:03 pm
I can't see any rounding error other than the basic inaccuracy of the standard 32-bit floating point number. For more info see here: http://www.lahey.com/float.htm
The javascript output in the max window doesn't show this error for small values as it is rounded to two decimal digits. If you set the integer part to 1,000,000, you'll notice that the javascript output features the same rounding error. If you set the integer part to 10,000,000, you won't get any decimal part at all in this patch (at least I don't).