Microsoft KB Archive/260901

HOWTO: Debug a GINA DLL on a Single Computer

Q260901

The information in this article applies to:

Microsoft Win32 Application Programming Interface (API), on platform(s):
- Microsoft Windows NT Server version 4.0
- Microsoft Windows NT Workstation version 4.0
- the operating system: Microsoft Windows 2000

SUMMARY

Debugging a Graphical Identification and Authentication (GINA) dynamic-link library (DLL) can be a tedious task. The process usually involves setting up a kernel debug session from a remote computer. This article offers an alternative way of debugging a GINA, which allows for source-level debugging on a single computer.

MORE INFORMATION

Typically, an application can use the built-in debug features of the operating system by placing a DebugBreak function call within the code, and by the system starting the just-in-time (JIT) debugger.

Unfortunately, a DebugBreak call in the GINA code does not typically work because the JIT debugging mechanism starts the debugger on the default desktop, whereas GINA runs on the Winlogon desktop. To overcome this limitation, you can use the stub application that is generated by the sample code in this article to start the JIT debugger on the Winlogon desktop.

Just-In-Time Debugger Settings

The operating system starts the JIT debugger based on settings within the registry. The settings are located under the following key:

HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\CurrentVersion\AeDebug

A typical AeDebug key may contain the following values:

Auto                 REG_SZ       0
Debugger             REG_SZ       C:\mssdk\Bin\windbg.exe -p %ld -e %ld
UserDebuggerHotKey   REG_DWORD    0x00000000

If an application does not handle an exception, the system looks in the AeDebug key and starts the debugger that is specified in the "Debugger" value. If "Auto" is set to 1 (one), the system will automatically start the debugger without user intervention; otherwise, the system will query the user as to whether or not they want to debug the application.

When debugging a GINA DLL, it is necessary for the system to start the debugger on the Winlogon desktop. This can be accomplished by a middle-layer stub application. The system starts this small stub, which in turn, starts the debugger on the Winlogon desktop.

The stub presented in this article is referred to as the Aedebug.exe file. The "Auto" value must be set to 1 (one) so that the stub starts automatically. To make Aedebug.exe more flexible, two additional values, "LaunchDebugger" and "Desktop", can be added to the registry. The LaunchDebugger value tells Aedebug.exe what debugger to start. The Desktop value specifies the desktop on which to create the debugger process.

An effective AeDebug key for debugging a GINA DLL might resemble the following:

Auto                 REG_SZ       1
Debugger             REG_SZ       C:\Aedebug.exe -p %ld -e %ld
UserDebuggerHotKey   REG_DWORD    0x00000000
LaunchDebugger       REG_SZ       C:\mssdk\Bin\windbg.exe
Desktop              REG_SZ       winsta0\Winlogon

Now all that needs to be done is to place a DebugBreak() in the GINA code and GINA can be debugged with the debugger of choice.

Important Note: If you place a DebugBreak function in DllMain, a deadlock may occur.

Sample Code

The following sample code can be used to create Aedebug.exe:

#include <windows.h>
#include <stdio.h>

CHAR szSubKey[] = "Software\\Microsoft\\Windows NT\\CurrentVersion\\AeDebug";
CHAR szDebugValue[] = "LaunchDebugger";
CHAR szDesktopValue[] = "Desktop";


int main(int argc, char *argv[])
{
   HKEY hKey = NULL;
   CHAR szDebugger[MAX_PATH];
   CHAR szCommandLine[MAX_PATH];
   CHAR szDesktop[256];
   BOOL fResult;
   DWORD dwSize;
   LONG lResult;
   STARTUPINFO si;
   PROCESS_INFORMATION pi;
   int n;

   if (argc < 2)
   {
      return 0;
   }

   __try
   {
      ZeroMemory(&pi, sizeof(pi));

      lResult = RegOpenKeyEx(HKEY_LOCAL_MACHINE, 
                             szSubKey, 
                             0,
                             KEY_ALL_ACCESS, 
                             &hKey);
      if (lResult != ERROR_SUCCESS)
      {
         __leave;
      }

      dwSize = MAX_PATH;
      lResult = RegQueryValueEx(hKey, 
                                szDebugValue, 
                                NULL, 
                                NULL, 
                                (LPBYTE)szDebugger, 
                                &dwSize);
      if (lResult != ERROR_SUCCESS)
      {
         __leave;
      }

      dwSize = 256;
      lResult = RegQueryValueEx(hKey, 
                                szDesktopValue, 
                                NULL, 
                                NULL,
                                (LPBYTE)szDesktop, 
                                &dwSize);
      if (lResult != ERROR_SUCCESS)
      {
         __leave;
      }

      lstrcpy(szCommandLine, szDebugger);

      for (n = 1; n < argc; n++)
      {
         lstrcat(szCommandLine, " ");
         lstrcat(szCommandLine, argv[n]);
      }

      ZeroMemory(&si, sizeof(si));
      si.cb = sizeof(si);
      si.lpDesktop = szDesktop;

      fResult = CreateProcess(NULL, 
                              szCommandLine, 
                              NULL,
                              NULL, 
                              TRUE, 
                              0, 
                              NULL,
                              NULL,
                              &si, 
                              &pi);
      if (!fResult)
      {
         __leave;
      }
   }
   __finally
   {
      if (hKey) RegCloseKey(hKey);
      if (pi.hProcess) CloseHandle(pi.hProcess);
      if (pi.hThread) CloseHandle(pi.hThread);
   }

   return 1;
}

Additional query words:

Keywords : kbDebug kbGINA kbKernBase kbOSWinNT400 kbOSWin2000 kbSecurity kbDSupport
Issue type : kbhowto
Technology : kbAudDeveloper kbWin32sSearch kbWin32API