Microsoft KB Archive/131003

= PRB: Setting DECIMALS to 15 or More Produces Incorrect Result =

Article ID: 131003

Article Last Modified on 10/15/2003

-

APPLIES TO


 * Microsoft Visual FoxPro 3.0 Standard Edition
 * Microsoft FoxPro 2.6a Standard Edition

-



This article was previously published under Q131003



SYMPTOMS
When you use SET DECIMAL to set DECIMAL to 15 or more, certain mathematical calculations produce an incorrect result.



CAUSE
Some numbers when represented in a floating point format have no exact representation. For example, this is 1/3 where as .33333333 is in fact slightly less than 1/3.

However, some numbers that do have exact floating point representations still exhibit this behavior. See the "Steps to Reproduce Behavior" section for an example.



STATUS
Microsoft is researching this issue and will post new information here in the Microsoft Knowledge Base as it becomes available.

Steps to Reproduce Problem
 Launch Visual FoxPro.  In the Command window, enter the following, one line at a time: SET DECIMALS TO 14 ? 32.16/4                        Observe the results in the active output window. You should see 8.04000000000000. This is exactly correct. Repeat step 2, but this time set decimals to 15. Observe the results. You should see 8.039999999999999. This is incorrect, and demonstrates this problem.

Additional query words: VFoxWin

Keywords: KB131003

-

[mailto:TECHNET@MICROSOFT.COM Send feedback to Microsoft]

© Microsoft Corporation. All rights reserved.