Improved Detours logic for detection of 32bit processes (#104)

This patch improves the logic for detecting whether the process to be
patched is a 32bit or a 64bit process.

The old logic would first enumerate the modules in the process and see
if: 
    1. There is a 32bit executable module
    2. There is a 64bit DLL module

In case 1.) is true and 2.) is false, i.e. a 32bit executable but no
64bit DLL, the process was deemed to be a 32bit process.

This seems plausible, but I encountered a case in which it is not true:
I launched an IL-only .NET application (a Windows Forms GUI application) in
Windows 10. Right after the CreateProcess call, there were just two
modules in the process

- A 32bit executable
- A 32bit ntdll.dll library

I.e. the .NET runtime was not loaded yet. Hence, because there *is* a
32bit executable but there is *not* a 64bit DLL, bIs32BitProcess was set
to TRUE. However, when resuming the process and inspecting with Process
Explorer, it appears that the process executed in 64bit mode!

I suppose it would be possible to replicate the behavior of the Windows
loader and be a bit smarter about looking for 32bit executables: instead
of just looking at the 'machine' flag, also look for a potential
IMAGE_COR20_HEADER (which basically acts as the PE header for .NET
executables) and see if that requires 32bit. However, I think there is
an easier way to check if the process is 32bit or not.

The new logic performs two steps:

1. Detect whether the operating system is 64bit. If the code is compiled
as 64bit, then the OS is trivially 64bit. If the code does not have
_WIN64 defined, i.e. it is 32bit, but it is running under WOW64, then
the OS is 64bit, too.

2. Detect if the process to be patched is 32bit. If the OS is *not*
64bit, the process can't possibly be 64bit. So it must be 32bit. If the
OS *is* 64bit, we can identify 32bit processes by calling
IsWow64Process() again.

Author: frerich

src/creatwth.cpp

@@ -538,9 +538,8 @@ BOOL WINAPI DetourUpdateProcessWithDll(_In_ HANDLE hProcess,
 {
     // Find the next memory region that contains a mapped PE image.
     //
-    BOOL bHas64BitDll = FALSE;
-    BOOL bHas32BitExe = FALSE;
     BOOL bIs32BitProcess;
+    BOOL bIs64BitOS = FALSE;
     HMODULE hModule = NULL;
     HMODULE hLast = NULL;
 
@@ -558,37 +557,43 @@ BOOL WINAPI DetourUpdateProcessWithDll(_In_ HANDLE hProcess,
 
         if ((inh.FileHeader.Characteristics & IMAGE_FILE_DLL) == 0) {
             hModule = hLast;
-            if (inh.OptionalHeader.Magic == IMAGE_NT_OPTIONAL_HDR32_MAGIC
-                && inh.FileHeader.Machine != 0) {
-
-                bHas32BitExe = TRUE;
-            }
             DETOUR_TRACE(("%p  Found EXE\n", hLast));
         }
-        else {
-            if (inh.OptionalHeader.Magic == IMAGE_NT_OPTIONAL_HDR64_MAGIC
-                && inh.FileHeader.Machine != 0) {
-
-                bHas64BitDll = TRUE;
-            }
-        }
     }
 
     if (hModule == NULL) {
         SetLastError(ERROR_INVALID_OPERATION);
         return FALSE;
     }
 
-    if (!bHas32BitExe) {
-        bIs32BitProcess = FALSE;
-    }
-    else if (!bHas64BitDll) {
-        bIs32BitProcess = TRUE;
+    // Determine if the target process is 32bit or 64bit. This is a two-stop process:
+    //
+    // 1. First, determine if we're running on a 64bit operating system.
+    //   - If we're running 64bit code (i.e. _WIN64 is defined), this is trivially true.
+    //   - If we're running 32bit code (i.e. _WIN64 is not defined), test if
+    //   we're running under Wow64. If so, it implies that the operating system
+    //   is 64bit.
+    //
+#ifdef _WIN64
+    bIs64BitOS = TRUE;
+#else
+    if (!IsWow64ProcessHelper(GetCurrentProcess(), &bIs64BitOS)) {
+        return FALSE;
     }
-    else {
+#endif
+
+    // 2. With the operating system bitness known, we can now consider the target process:
+    //   - If we're running on a 64bit OS, the target process is 32bit in case
+    //   it is running under Wow64. Otherwise, it's 64bit, running natively
+    //   (without Wow64).
+    //   - If we're running on a 32bit OS, the target process must be 32bit, too.
+    //
+    if (bIs64BitOS) {
         if (!IsWow64ProcessHelper(hProcess, &bIs32BitProcess)) {
             return FALSE;
         }
+    } else {
+        bIs32BitProcess = TRUE;
     }
 
     DETOUR_TRACE(("    32BitExe=%d 32BitProcess\n", bHas32BitExe, bIs32BitProcess));