float precision problem

Dec 18, 2011 at 10:59am

float precision problem

I want to calculate distance with gps. I moved code from processing (java) to max patches.
Problem is that in processing i have those values for example:
dLat: -1.3315805E-7
dLon: 1.7476995E-7

which is like 0.00000017476995 if i’m correct.

In max the value is zero, and so after a few calculations i get nan (not a number).
What can i do?


– Pasted Max Patch, click to expand. –

In case someone know’s processing, here is the proceesing code.

testP: gps 5.910325 30. 51.988632 57. 28.372467

testP: gps 5.91006 10. 51.988777 19. 29.984245

void setup() {
// println(getDistance(51.988632, 5.910325, 51.988777, 5.91006)); // pc > keuken
// println(getDistance(51.988777, 5.910006, 51.988739, 5.910056));
String[] lines = loadStrings(“geo.tsv”);

String[] tokens = split(lines[0], “t”);
float lon1 = float(tokens[2]);
float lat1 = float(tokens[4]);

for (int i = 1; i < 2; i++) { //lines.length-1

tokens = split(lines[i+1], “t”);
float lon2 = float(tokens[2]);
float lat2 = float(tokens[4]);

float d = getDistance(lon1, lat1, lon2, lat2);
float d = getDistance(5.910161, 51.988705, 5.910171, 51.988697);


float getDistance(float lon1, float lat1, float lon2, float lat2) {
int R = 6371; // Radius of the earth in km
float dLat = radians(lat2-lat1);
float dLon = radians(lon2-lon1);

println(“dLat: “+dLat+”ndLon: “+dLon);

float a = sin(dLat/2) * sin(dLat/2) + cos(radians(lat1)) * cos(radians(lat2)) * sin(dLon/2) * sin(dLon/2);
float c = 2 * atan2(sqrt(a), sqrt(1-a));
float d = R * c; // Distance in km

return d;

Dec 18, 2011 at 11:35am

I can’t find the corresponding expression in your Processing code (maybe the last 5 lines?); but inside your getDistance subpatch, you are trying to find the sqrt of a negative number (on the right, above [atan2]); this generates a complex number, unrepresentable in Max. That’s where the “nan” is generated. Sorry if you already knew this.



Dec 18, 2011 at 11:50am

…..to get it to “work”, I inverted the result AFTER the sqrt calculation; probably messes up the result you’re expecting but here it is anyway:

– Pasted Max Patch, click to expand. –


Dec 18, 2011 at 5:23pm

Max works with single-precision floating point values, which gives you approximately 6-digit precision in the mantissa and can handle exponents (base 10) of about +/-38.

This is all considerably less powerful than the double-precision floats used in Java, but should not be a limitation with the values you say you are using.

Square-roots of negative numbers will NaN no matter what representation you have. From a quick look, it seems that what you have implemented in your max patch is float c = 2 * atan2(sqrt(a), sqrt(0-a)); rather than float c = 2 * atan2(sqrt(a), sqrt(1-a));.

The problem is further exacerbated by your using send/receive pairs and, apparently, relying on left-to-right order. When you s/r, all bets on execution order are off.

I’ve also loadbanged the radius of the earth into your patch.

The following is only the [p GetDistance] part of your patch, you’ll have to manually copy/paste it into your top-level patch.

– Pasted Max Patch, click to expand. –
Dec 20, 2011 at 12:32pm

Thanks peter,

i allready had it fixed with javascript which is much easier to read for such things (didn’t know before that javascript was possible).

But good to know it is possible.


You must be logged in to reply to this topic.