Microsoft KB Archive/193415

= PRB: Setting Decimals to 10 or More Creates Incorrect Result =

Article ID: 193415

Article Last Modified on 12/11/1999

-

APPLIES TO


 * Microsoft Visual FoxPro 5.0 Standard Edition
 * Microsoft Visual FoxPro 5.0a
 * Microsoft Visual FoxPro 6.0 Professional Edition

-



This article was previously published under Q193415



SYMPTOMS
With Decimals set to 10 or greater, the SUM function returns results that do not match the lesser decimal equivalent.



RESOLUTION
Here are two potential workarounds depending on the application's needs:
 * Use the Currency field datatype where possible. -or-


 * Use the Round function when comparing a value to another with decimal places set to > 9.

Example
IF Round(m.x3,2) <> 300.03 ? "Mismatch 3" ENDIF



STATUS
Microsoft is researching this problem and will post new information here in the Microsoft Knowledge Base as it becomes available.



Steps to Reproduce Behavior
Run the following code in a program file. *******************Begin Code**************** CLOSE ALL CLEAR SET DECIMALS TO 10

CREATE CURSOR test2 (f1 N(10,2)) CREATE CURSOR test3 (f1 N(10,2)) CREATE CURSOR test4 (f1 N(10,2)) CREATE CURSOR test5 (f1 N(10,2)) CREATE CURSOR test6 (f1 N(10,2))

INSERT INTO test2 VALUES(100.01) INSERT INTO test2 VALUES(100.01)

INSERT INTO test3 VALUES(100.01) INSERT INTO test3 VALUES(100.01) INSERT INTO test3 VALUES(100.01)

INSERT INTO test4 VALUES(100.01) INSERT INTO test4 VALUES(100.01) INSERT INTO test4 VALUES(100.01) INSERT INTO test4 VALUES(100.01)

INSERT INTO test5 VALUES(100.01) INSERT INTO test5 VALUES(100.01) INSERT INTO test5 VALUES(100.01) INSERT INTO test5 VALUES(100.01) INSERT INTO test5 VALUES(100.01)

INSERT INTO test6 VALUES(100.01) INSERT INTO test6 VALUES(100.01) INSERT INTO test6 VALUES(100.01) INSERT INTO test6 VALUES(100.01) INSERT INTO test6 VALUES(100.01) INSERT INTO test6 VALUES(100.01)

SELECT test2 SUM f1 TO m.x2

SELECT test3 SUM f1 TO m.x3

SELECT test4 SUM f1 TO m.x4

SELECT test5 SUM f1 TO m.x5

SELECT test6 SUM f1 TO m.x6

IF m.x2 <> 200.02 ? "Mismatch 2" ENDIF

IF m.x3 <> 300.03 ? "Mismatch 3" ENDIF

IF m.x4 <> 400.04 ? "Mismatch 4" ENDIF

IF m.x5 <> 500.05 ? "Mismatch 5" ENDIF

IF m.x6 <> 600.06 ? "Mismatch 6" ENDIF *******************End Code************************** Notice that there are two errors listed:
 * Mismatch 3 -and-


 * Mismatch 6

Another way to verify the problem is to step through the code in the Visual FoxPro Debugger. Place the following in the Watch window: M.X3=300.03 M.X6=600.06 Both of these entries return ".F."

If you change the number of decimals to less than 10 using the SET DECIMALS TO command, the problem does not occur. Furthermore, changing the number of decimals stored in the table does not matter.

Additional query words: kbVFp500 kbVFp600 kbVFp500a kbXBase

Keywords: kbprb kbpending kbcode KB193415

-

[mailto:TECHNET@MICROSOFT.COM Send feedback to Microsoft]

© Microsoft Corporation. All rights reserved.