## Floating-point rounding error?

Feb 17, 2010 at 10:32am

# Floating-point rounding error?

Hi guys,

I’m working on set of boxes for controlling floating-point number entry into Max. I’ve come across what seems to be a rounding error in floating-point number boxes, and I was wondering if this was a mistake on my part or whether it’s a bug.

I’ve attached the files below. To make the patch work you type an integer into the left hand side of the number and scroll through the decimal portion using the + and – signs.

If the integer value is ’0′ it works fine, but as the value increases an error is introduced into the calculation when the decimal portion is changed. The javascript is outputing its values to the MAX window for comparison and the numbers seem to be fine, so the error seems to be happening in Max.

The error gets larger the higher a value you use, so it doesn’t happen at 1 – 10, starts at around 20 and by the time you get to 100,000 is around 0.002.

I was wondering if anyone else has come across this and if there’s a fix.

I’ve attached the patch along with the relevant javascript files.

Thanks!

Tim

###### Attachments:
1. Number_boxes.zip
#48543
Feb 17, 2010 at 10:39am

Hmm ….. I don’t think the attachment worked so I’ve pasted the files below. The names of the three javascript files are at the top of the code for each:

– Pasted Max Patch, click to expand. –

//##### plusone.js ##########

// plusone.js

//Increments or decrements a number box by one
//Rolls over between 9 and 0 and outputs a bang

inlets = 1;
outlets = 3;
autowatch = 1;

var x = 0;

{
if (x == 9) {
x = 0;
outlet(1, “bang”);
}
else {
x = x+1;
}
outlet(0, x);
}

function sub()
{
if (x == 0) {
x = 9;
outlet(2, “bang”);
}
else {
x = x-1;
}
outlet(0, x);
}

function zero()
{
x = 0;
outlet (0, x);
}

//###### isint.js ######

// isint.js

//tests input to see if it is an int
//if so outputs int otherwise outputs zero

inlets = 1;
outlets = 1;
autowatch = 1;

function text(t)
{
obj = this.patcher.getnamed(“tbx_Number”);
t = parseInt(t, 10);

if (isNaN(t)) {
t=0;
}

outlet(0, t);
obj.set(t);
}

//###### buildnum.js ######

//buildnum.js
//Joins elements from number boxers to make float

inlets = 1;
outlets = 1;
autowatch = 1;

var whole = 0;
var ten = 0;

function wholenum(a)
{
whole = parseInt(a, 10);
result();
}

function tennum(b)
{
ten = b;
result();
}

function result()
{
var numout = 0;

numout = whole + (ten/10)
outlet (0, numout);
post(numout);
post();
}

#174580
Feb 17, 2010 at 9:03pm

I can’t see any rounding error other than the basic inaccuracy of the standard 32-bit floating point number. For more info see here: http://www.lahey.com/float.htm

The javascript output in the max window doesn’t show this error for small values as it is rounded to two decimal digits. If you set the integer part to 1,000,000, you’ll notice that the javascript output features the same rounding error. If you set the integer part to 10,000,000, you won’t get any decimal part at all in this patch (at least I don’t).

#174581

You must be logged in to reply to this topic.