Microsoft KB Archive/68537

PRB: Pointer Functions in MASM Can Hang Real Mode ID Number: Q68537

3.00 WINDOWS

SYMPTOMS Windows applications written using the Microsoft Macro Assembler (MASM) can fail when run under real mode if pointers to FAR functions are used in the code.

CAUSE This failure is caused by the following sequence of events:


 * 1) MASM breaks a FAR pointer into separate segment and offset fixup records in the OBJ file.
 * 2) The linker incorrectly resolves the offset record to point to the wrong location in Windows call thunk table.
 * 3) When the FAR function is called through the thunk table, invalid code is executed. This hangs the system.

RESOLUTION Define a double-word variable in the application’s data segment and initialize it to the function address. Instead of using the function pointer directly in any code, use the value stored in the variable. MASM will correctly create a FAR pointer fixup record for the variable. This record is handled entirely by the loader and results in correct operation.

More Information:

Normally, when a FAR function is referenced, a fixup record for the function pointer is placed in the EXE file. The fixup record is resolved at load time by the Windows Kernel to point to a call thunk. A call thunk is a short piece of code used in Windows real mode to check if the code segment containing the called function is currently in memory, and to load it from disk if necessary.

When a FAR function is used in an assembly program as a separate segment and offset, MASM creates two fixup records: a segment that is resolved by the loader and an offset, which is incorrectly resolved by the linker to point to the incorrect offset in the call thunk table.

For example, an assembly program may contain a function pointer reference in the form:

;
 * Assume wc is a WNDCLASS structure, and MAINWNDPROC is the
 * main window procedure for the application.

mov    WORD PTR wc.clsLpfnWndProc, OFFSET MAINWNDPROC mov    WORD PTR wc.clsLpfnWndProc+2, SEG MAINWNDPROC Since there are two separate references to the function, MASM generates two separate fixup records. The value for OFFSET MAINWNDPROC is incorrectly resolved by the linker.

To generate the correct fixup record, it is necessary to create a variable in the application’s data segment that references the function. Always use that variable to load memory or registers with the function address.

;
 * Define a pointer variable and initialize to the function
 * address.

data segment var_MAINWNDPROC dd     MAINWNDPROC data ends

...


 * Make all references to the function through the variable.
 * Make all references to the function through the variable.

mov    WORD PTR wc.clsLpfnWndProc, var_MAINWNDPROC.0 mov    WORD PTR wc.clsLpfnWndProc+2, var_MAINWNDPROC.2