Re: Progressively remove decimal digits

As promised. It turns out the same input (0.99805, 5 decimal places) breaks on both patches.

If you take the time to look at 0.99805 in binary floating-point and do the arithmetic in binary, you will see why the result of 0.98804 is inevitable.

Don't they teach this stuff in schools any more ?-)

[attachment=214546,5015]

Attachments:
  1. ScreenShot20130211at13.57.48.png
Feb 11, 2013 at 1:11pm #239048