Solved VMWare Workstation: How to fix black screen on Windows XP guest

Hello!

I had a Windows XP Professional SP3 virtual machine in VMWare Workstation 17.6.3 (host is my main PC, see 2nd system specs) in order to play old Windows games from the 90's and 2000's. This worked fine, but recently it would not boot. It would show the Windows logo while loading and then the screen would go black and stay black. After a while there was no disk activity indicated, so I assumed Windows was damaged. I restarted in Safe Mode which loaded fine, tried to replace the VMWare SVGA 3D graphics drivers with Standard VGA adapter because I suspected a graphics drivers issue, but Standard VGA adapter wasn't on the list to select. So simply disabled the display adapter in Device Manager and restarted. This time it booted normally with Standard VGA adapter graphics. I downloaded the latest 32-bit VMWare Tools that support Windows XP (VMware-tools-windows-10.0.12-4448496.ISO) and updated VMWare Tools making sure the graphics driver is also installed. Upon restart, the same, black screen and then nothing after several minutes. I decided to Google the issue and it was revealed this was because I had recently upgraded my nVidia GeForce drivers. Apparently nVidia changed the way Vulkan is implemented, so VMWare could not translate legacy DirectX 9 calls to Vulkan and resulted in the black screen when 3D acceleration was enabled. I consider that very stupid on behalf of VMWare! It would make more sense to convert DirectX 9 to DirectX 11 or 12, and probably be easier to do, I don't know, than converting DirectX 9 to Vulkan. Fortunately Google AI told me to add a specific line (see below) to the virtual machine's .vmx file to force DirectX 11/12 instead of Vulkan. This worked fine! I can now load the Windows XP virtual machine with 3D enabled and I even tested it with dxdiag. It works fine again. So I wanted to share my discovery just in case someone else has the same issue. I copy-paste here from my How to document (I keep a How to document in Notepad for every workaround I find because I will not remember it afterwards):

How to fix Windows XP black screen in VMWare:

Newer versions of nVidia drivers change how DirectX and Vulkan is implemented. VMWare tries to use Vulkan by default resulting to the black screen when 3D acceleration is enabled due to these changes that break compatibility with VMWare translation to and from DirectX 9. To fix that find the virtual machines .vmx file and right-click to Edit with Notepad. Add the following line at the bottom:

mks.enableVulkanRenderer = "FALSE"

Other lines that might help are:

mks.enableDX12Renderer = "FALSE" (to disable DirectX 12 and use DirectX 11 for Windows 98 and other guests that still have issues without 3D acceleration)

svga.guestBackedPrimaryAware = "FALSE" (for old Linux guests that fail to translate from OpenGL to Vulkan)



Pro-Tip: Applying it Globally:

If you have a lot of retro virtual machines and don't want to edit every single .vmx file manually, you can disable the Vulkan renderer for all virtual machines at once:

1. Close VMware Workstation completely.
2. Navigate to C:\Users\YOUR_USERNAME\AppData\Roaming\VMware\ on your host.
3. Open the file named preferences.ini with Notepad.
4. Add the line to the bottom:

mks.enableVulkanRenderer = "FALSE"

5. Save and close.