Microsoft KB Archive/61385

PROS29109001: System Can Hang If VioPrtSc Has Been Replaced PSS ID Number: Q61385 Article last modified on 09-20-1991 PSS database name: O_Os2SDK

1.10 1.21

OS/2

Summary:

PROBLEM ID: OS29109001

SYMPTOMS If a replacement is registered for the VioPrtSc function in OS/2, and the PRINT SCREEN key is pressed repeatedly, the machine will eventually hang.

CAUSE The system creates a special thread if a replacement for the VioPrtSc or VioPrtScToggle function is registered via the VioRegister function. This thread is created only for these two replacement routines. This thread is blocked by the system until the PRINT SCREEN key is pressed, at which time thread 2 of HARDERR.EXE does some processing in preparation to printing the screen, then unblocks this special thread (that belongs to the application) to execute the replacement routine(s) that were registered with VioRegister. This thread has a rather small stack of 512 bytes that is created by the system when the replacement routine is registered with VioRegister.

After the system calls into the entry-point function with the thread created by the system when either VioPrtSc or VioPrtScToggle was registered via VioRegister, on return to the system, it does not clean up the stack of the thread that was created. This means that the stack shrinks by six USHORT parameters (12 bytes) with each call into the entry-point routine. With the small stack that the system creates for this thread (512 bytes), after about 36 calls (PRINT SCREEN key presses), the system will crash.

Also, if the PRINT SCREEN key is pressed very quickly, it will only take a few presses of the key before this small stack underflows, and the system will crash. Please note that this only happens if a replacement for VioPrtSc or VioPrtScToggle is registered. These are the only two routines the system creates a special thread for. If any other VIO routine is registered, the stack is cleaned off correctly by the system VIO router after calling the entry-point function.

RESOLUTION/STATUS There is a workaround to this problem; however, this workaround cannot be used with OS/2 version 1.1.

If an internal build number earlier than OS/2 version 12.177 is being used, the following workaround can be used. If 12 bytes are popped off when leaving the entry-point function (which according to the documentation should not be done), the stack will be cleaned off “before” the code returns to the system VIO router, and everything will function correctly.

However, there are some special considerations. Doing this means that a routine written to replace the VioPrtSc and/or VioPrtScToggle functions must be written in Assembler, since the stack must be set up C-style; however, the stack must be cleaned up in the callee (the entry-point function), which C will not do. Furthermore, the programmer must put some special coding in the entry-point routine to decide when it is necessary to do this special stack cleanup. This can easily be done by looking at the “fFunction” parameter (accessed by [bp].s_indx in PRTSCR, which is described below) to determine whether this is a VioPrtSc or VioPrtScToggle call. If it is one of these calls, a “ret 12” should be done; otherwise, a regular “ret” should be done when returning from the entry-point routine, and things should work correctly.

Microsoft has confirmed this to be a problem in OS/2 versions 1.1 and 1.21. We are researching this problem and will post new information here as it becomes available.

More Information:

When a replacement VIO routine entry-point handler is created that will be registered with VioRegister, this routine must be set up in a certain way. First of all, it must be declared as shown in the documentation for VioRegister in the “Microsoft Operating System/2 Programmer’s Reference Volume 3” for version 1.1. This entry-point function MUST NOT be of type PASCAL.

Another critical requirement is that the entry-point function must save the state of the CPU registers when it enters, and restore them when it exits. This can be done in C with the _saveregs function modifier; see the Microsoft C Compiler version 5.1 update section for more information on this topic. As stated in the documentation for VioRegister, the stack must be left in its original state when returning from the entry-point function. If this entry-point function is written in C, this will be done for the programmer.

In the Software/Data Library is a file named PRTSCR that contains a sample application with an entry-point function that shows how to accomplish these tasks in Assembler. This function can be easily adapted to handle one or more different functions registered with VioRegister.

PRTSCR can be found in the Software/Data Library by searching on the keyword PRTSCR, the Q number of this article, or S12603. PRTSCR was archived using the PKware file-compression utility.

Additional reference words: 1.10 5.10

Copyright Microsoft Corporation 1991.