View Single Post
Old 01-13-2021, 12:36 AM   #2
Human being with feelings
mschnell's Avatar
Join Date: Jun 2013
Location: Krefeld, Germany
Posts: 10,193

What do you mean by "fixed precision" ?
64 bit float format provides a "fixed precision" of about 48 bits. That is some 10e-14. I suppose that there are not that many relevant algorithms that need more. Do check your algorithm on that behalf. Maybe some math can modify it to be less prone to the effects you see.

To get a better precision some very hard software means need to be applied, as the computer hardware can't do this.


Last edited by mschnell; 01-13-2021 at 08:37 AM.
mschnell is offline   Reply With Quote