## How do I adjust the decimal places used/calculated for float-numbers?

Dec 9, 2010 at 4:43am

# How do I adjust the decimal places used/calculated for float-numbers?

So I've been working on this patch that converts the value of [a/b] into a color code based on the numbers contained in each digit. In this case, i'm using the result of 1/7.

I was wondering how to calculate by more decimal places….

Let me know if this makes any sense. I have difficulty explaining math stuff.

[attachment=148490,1509]

###### Attachments:
#53823
Dec 9, 2010 at 7:47am

Hello maxer,

a naive approach like that can be a way, but because float/int artefacts i’m not sure how much decimals you can get with precision …

For example with 10 / 3 ; you find with my patch : 3333333 253860 ;-)

#P window setfont “Sans Serif” 9.;
#P window linecount 1;
#P newex 473 304 27 196617 – 1.;
#P newex 473 252 47 196617 t f i;
#P newex 349 379 35 196617 * 10.;
#P newex 349 474 27 196617 – 1.;
#P newex 567 489 32 196617 print;
#P newex 349 422 47 196617 t f i;
#P newex 349 266 40 196617 uzi;
#P newex 349 233 93 196617 t 12 f;
#P newex 349 346 36 196617 zl reg;
#P newex 349 118 43 196617 pak 0 0;
#P number 414 82 35 9 1 100 67 3 0 0 0 221 221 221 222 222 222 0 0 0;
#P number 349 80 35 9 1 100 67 3 0 0 0 221 221 221 222 222 222 0 0 0;
#P flonum 349 191 92 9 0 0 0 3 0 0 0 221 221 221 222 222 222 0 0 0;
#P newex 349 155 92 196617 / 1.;
#P connect 12 1 13 1;
#P connect 12 1 9 0;
#P connect 8 1 10 1;
#P connect 8 1 9 0;
#P connect 13 0 5 1;
#P connect 6 1 12 0;
#P connect 7 0 5 0;
#P connect 12 0 13 0;
#P connect 11 0 8 0;
#P connect 5 0 11 0;
#P connect 8 0 10 0;
#P fasten 10 0 5 1 354 501 427 501 427 336 380 336;
#P connect 1 0 6 0;
#P connect 6 0 7 0;
#P connect 4 0 0 0;
#P connect 3 0 4 1;
#P connect 2 0 4 0;
#P connect 0 0 1 0;
#P window clipboard copycount 14;

HTH

#193741
Dec 10, 2010 at 2:14pm

Max is calculating floats with 32 bit accuracy regardless of how many decimal places are displayed.

#193742
Dec 10, 2010 at 3:42pm

press edit and change that to 24. :)

#193743

You must be logged in to reply to this topic.