Microsoft Surface Pro 8 and BSODs since a couple of month back

zbook · Jul 30, 2023

1) Please reinstall:

spvdport SoftPerfect Virtual Disk c:\windows\system32\drivers\spvdbus.sys

Name SoftPerfect Virtual Bus
Manufacturer KEG
Status OK
PNP Device ID ROOT\SCSIADAPTER\0000
Driver C:\WINDOWS\SYSTEM32\DRIVERS\SPVDBUS.SYS (, 97,43 KB (99 768 bytes), 2022-07-06 09:36)

2) Make a new retore point

3)Restart WDV with all customized tests except:

[ ] 0x00000004 Randomized low resources simulation.

Post a new querysettings and / or V2 into the newest post.

iorx · Jul 30, 2023

Hi!

Just curious. How did you or what made you suspect this driver?

I actually replaced SoftPerfect RAM-disk with ImDIsk since yesterday.
So a question here. Should I go back to SoftPerfect and do the above test, or keep ImDisk? I can also scrap it completely and run the machine with out it, let me know which way is the best.
Will do.
Last time I did this I encountered repated BSODs and then after a couple of them got offered to Restore from a restore point. Should I still use those settings? And, which drivers should be ?

(link for files and query settings in next post)

zbook · Jul 30, 2023

When you last had WDV on the misbehaving driver was identified in the debugging.

If the software has been completely uninstalled then it can no longer be a cause of BSOD.

iorx · Jul 30, 2023

I'll run like this then and WILL return my experience here either what the outcome is so we get a proper closure and resolution.
Can't thank you enough for this assistance. I hadn't went this deep-route in troubleshooting before with the WDV. I learned allot!

If you got time, could you point out where in which log, you found that spvdbus.sys was the misbehaving driver?

zbook · Jul 30, 2023

************* Path validation summary **************
Response Time (ms) Location
Deferred SRV*c:\symbols*Symbol information
Symbol search path is: SRV*c:\symbols*Symbol information
Executable search path is:
Windows 10 Kernel Version 22621 MP (8 procs) Free x64
Product: WinNt, suite: TerminalServer SingleUserTS
Built by: 22621.1928.amd64fre.ni_release_svc_prod3.230622-0951
Machine Name:
Kernel base = 0xfffff802`33a1e000 PsLoadedModuleList = 0xfffff802`34631100
Debug session time: Thu Jul 27 15:33:40.430 2023 (UTC - 5:00)
System Uptime: 0 days 0:00:03.698
Loading Kernel Symbols
...............................................................
.........
Loading User Symbols
Loading unloaded module list
...
*******************************************************************************
* *
* Bugcheck Analysis *
* *
*******************************************************************************

Use !analyze -v to get detailed debugging information.

BugCheck C4, {2000, fffff80801148275, 0, 0}

*** WARNING: Unable to verify timestamp for spvdbus.sys
*** ERROR: Module load completed but symbols could not be loaded for spvdbus.sys
Probably caused by : spvdbus.sys ( spvdbus+8275 )

Followup: MachineOwner
---------

3: kd> !analyze -v
*******************************************************************************
* *
* Bugcheck Analysis *
* *
*******************************************************************************

DRIVER_VERIFIER_DETECTED_VIOLATION (c4)
A device driver attempting to corrupt the system has been caught. This is
because the driver was specified in the registry as being suspect (by the
administrator) and the kernel has enabled substantial checking of this driver.
If the driver attempts to corrupt the system, bugchecks 0xC4, 0xC1 and 0xA will
be among the most commonly seen crashes.
Arguments:
Arg1: 0000000000002000, Code Integrity Issue: The caller specified an executable pool type. (Expected: NonPagedPoolNx)
Arg2: fffff80801148275, The address in the driver's code where the error was detected.
Arg3: 0000000000000000, Pool Type.
Arg4: 0000000000000000, Pool Tag (if provided).

Debugging Details:
------------------

KEY_VALUES_STRING: 1

STACKHASH_ANALYSIS: 1

TIMELINE_ANALYSIS: 1

DUMP_CLASS: 1

DUMP_QUALIFIER: 400

BUILD_VERSION_STRING: 22621.1928.amd64fre.ni_release_svc_prod3.230622-0951

TAG_NOT_DEFINED_202b: *** Unknown TAG in analysis list 202b

DUMP_TYPE: 2

BUGCHECK_P1: 2000

BUGCHECK_P2: fffff80801148275

BUGCHECK_P3: 0

BUGCHECK_P4: 0

BUGCHECK_STR: 0xc4_2000

CPU_COUNT: 8

CPU_MHZ: bb3

CPU_VENDOR: GenuineIntel

CPU_FAMILY: 6

CPU_MODEL: 8c

CPU_STEPPING: 1

CUSTOMER_CRASH_COUNT: 1

DEFAULT_BUCKET_ID: VERIFIER_ENABLED_VISTA_MINIDUMP

PROCESS_NAME: System

CURRENT_IRQL: 0

ANALYSIS_SESSION_HOST: DESKTOP-9HEBUKS

ANALYSIS_SESSION_TIME: 07-30-2023 00:52:39.0620

ANALYSIS_VERSION: 10.0.17763.132 amd64fre

LAST_CONTROL_TRANSFER: from fffff802344ea3c1 to fffff80233e4ec00

STACK_TEXT:
fffff482`c2a06f28 fffff802`344ea3c1 : 00000000`000000c4 00000000`00002000 fffff808`01148275 00000000`00000000 : nt!KeBugCheckEx
fffff482`c2a06f30 fffff802`33ff50c2 : fffff802`3462bce0 00000000`00002000 fffff808`01148275 00000000`00000000 : nt!VerifierBugCheckIfAppropriate+0x14d
fffff482`c2a06fd0 fffff802`344e0970 : 00000000`00000000 fffff802`3462bce0 fffff808`01148275 ffffa98a`00000000 : nt!VfReportIssueWithOptions+0x102
fffff482`c2a07020 fffff802`344dd05e : 00000000`00000000 00000000`686c7158 00000000`00000000 fffff802`33ca2382 : nt!VfCheckPoolType+0x90
fffff482`c2a07060 fffff808`01148275 : 00000000`00000000 00000000`00000000 ffffa98a`db8be530 ffffa98a`f1636dd0 : nt!VerifierExAllocatePoolWithTag+0x9e
fffff482`c2a070c0 00000000`00000000 : 00000000`00000000 ffffa98a`db8be530 ffffa98a`f1636dd0 ffffa98a`db8be530 : spvdbus+0x8275

iorx · Jul 31, 2023

Thanks! Found it in 072723-14984-01.dmp (I'm closing the folder share in a day or two)

In the dump analyze it mention this:
A device driver attempting to corrupt the system has been caught. This is
because the driver was specified in the registry as being suspect (by the
administrator) and the kernel has enabled substantial checking of this driver.
If the driver attempts to corrupt the system, bugchecks 0xC4, 0xC1 and 0xA will
be among the most commonly seen crashes.

Is it possible to see this, in the registry, if a drivers is suspected?

BTW. Still a stable machine!

zbook · Jul 31, 2023

1) Make a new restore point

2) Consider making free backup images and saving the images to another disk drive or the cloud

I'm not aware of methods to search the registry to confirm WDV C4 findings.

iorx · Jul 31, 2023

Celebrating the victory to soon I think...

And "no!", just BSODed.

This time I had allot running including a virtual machine in Hyper-V.

New V2 file in the share, MSP8-(2023-07-31_10-54-14).zip.
Should I repeat the last WDV, the one that had all but [ ] 0x00000004 Randomized low resources simulation. checked?

Answer to your bullets.

I do this on regular basis now.
Got a complete vhdx shadowcopy on a server locally.

iorx · Jul 31, 2023

Just went down as a "ton of bricks" in the middle och a Teams meeting...

V2 log added to the files. sp8-bsod

And... ImDisk is uninstalled now. Running without any ram-disk for the moment.

zbook · Jul 31, 2023

Please restart WDV with all customized tests except:

[ ] 0x00000004 Randomized low resources simulation.

If there is no immediate BSOD then run verifier /querysettings > post a share link into this thread using one drive, drop box, or google drive

For any BSOD post a new V2 share link into the newest post.

iorx · Aug 1, 2023

Just had a BSOD. What I had running: OpenVPN, Edge Dev (multiple profiles), Outlook, RDCman. New V2 files in the folder, MSP8-(2023-08-01_12-19-22).zip. And activating WDV again now.
A "pain in the but" is that Windows Search needs to reindex after every BSOD... machine is rather busy with this after.

OneDrive

Sign in to your OneDrive cloud storage and Office Online.

1drv.ms

Code:

verifier /querysettings

Verifier Flags: 0x03bfefbb

  Standard Flags:

    [X] 0x00000001 Special pool.
    [X] 0x00000002 Force IRQL checking.
    [X] 0x00000008 Pool tracking.
    [X] 0x00000010 I/O verification.
    [X] 0x00000020 Deadlock detection.
    [X] 0x00000080 DMA checking.
    [X] 0x00000100 Security checks.
    [X] 0x00000800 Miscellaneous checks.
    [X] 0x00020000 DDI compliance checking.

  Additional Flags:

    [ ] 0x00000004 Randomized low resources simulation.
    [X] 0x00000200 Force pending I/O requests.
    [X] 0x00000400 IRP logging.
    [X] 0x00002000 Invariant MDL checking for stack.
    [X] 0x00004000 Invariant MDL checking for driver.
    [X] 0x00008000 Power framework delay fuzzing.
    [X] 0x00010000 Port/miniport interface checking.
    [X] 0x00040000 Systematic low resources simulation.
    [X] 0x00080000 DDI compliance checking (additional).
    [X] 0x00200000 NDIS/WIFI verification.
    [X] 0x00800000 Kernel synchronization delay fuzzing.
    [X] 0x01000000 VM switch verification.
    [X] 0x02000000 Code integrity checks.

  Internal Flags:

    [X] 0x00100000 Extended Verifier flags (internal).

    [X] Indicates flag is enabled.

  Boot Mode:

    Persistent

  Rules:

    All rules are using default settings

  Extensions:

    wdm: rules.all
    ndis: rules.default

  Verified Drivers:

    npcap.sys
    ovpn-dco.sys
    wintun.sys
    tap0901.sys
    igdkmdn64.sys
    iaisp64.sys
    tbtbusdrv.sys
    ialpss2_i2c_tgl.sys
    intelthcbase.sys
    netwtw10.sys
    teedriverw10x64.sys
    ialpss2_uart2_tgl.sys
    intcaudiobus.sys
    ialpss2_gpio2_tgl.sys
    nvvad64v.sys
    nvmoduletracker.sys
    nvvhci.sys
    iactrllogic64.sys
    intcoed.sys
    ov5693.sys
    ov13858.sys
    vd55g0.sys
    rtkvhd64.sys
    intcusb.sys
    intcbtau.sys
    ibtusb.sys
    dump_dumpstorport.sys
    dump_stornvme.sys
    dump_dumpfve.sys
    iacamera64.sys
    librehardwaremonitor.sys
    ndis.sys
    netadaptercx.sys
    mbbcx.sys
    wificx.sys
    wdiwifi.sys

Restarting now...

edit/update.
It survived the reboot. Running with these settings until next BSOD.

zbook · Aug 1, 2023

The newest crash was on:

Tue Aug 1 05:06:40.889 2023 (UTC - 5:00).

This BSOD occurred without WDV.

Run WDV for approximately 48 hours with the current test settings.

iorx · Aug 1, 2023

I'll keep the settings until next BSOD. You mention 48h, should I revert/reset WDV after that even if I'm not experience a BSOD?
Are there any valuable logs from WDV while active or is it only usefull and add info to the dump-file in an event of a crash?

zbook · Aug 1, 2023

Continue using WDV for 48 hours in the background while you using the computer.

For any BSOD post a new V2 share link.

If there are no BSODs for 48 hrs and there are no major performance problems you can run it for an additional 24 hours then turn off the tool.

iorx · Aug 2, 2023

And this morning, I was not even working at the machine, it looks like it encountered a crash.

V2 log MSP8-(2023-08-02_10-34-58).zip, here: sp8-bsod

Microsoft GameInput was reconfigured right before the crash, or so it looks to me. Suspicious

.

Verifier still active.

I had a look at the dump, and this time it's ndis something that is mentioned.

Brgs,

x BlueRobot · Aug 2, 2023

iorx said:
ou mention 48h, should I revert/reset WDV after that even if I'm not experience a BSOD?

The recommendation of 48 hours is completely arbitrary, that ballpark figure was based on experience of a few different analysts from a different forum. Some of the settings you're running Driver Verifier with are actually deprecated on newer builds so probably aren't actually even running. Additionally, you should actually be including a few Microsoft drivers for verification as well, notably: fltmgr.sys, ndis.sys, storport.sys and Wdf01000.sys.

iorx said:
Are there any valuable logs from WDV while active or is it only usefull and add info to the dump-file in an event of a crash?

Driver Verifier is only "useful" when it actually crashes, which is the purpose of the tool: inject verifier.sys into several driver stacks and then check for common bugs, if it encountered one, then it will crash with a Stop 0xC4 or a Stop 0xC1. The dump file should then point out the driver which is responsible for causing the crash. Microsoft has actually implemented something similar in their new WDF (Windows Driver Framework), in that they've added some exception handling code which will deliberately throw a bugcheck (Stop 0x10D) with lots of debugging information to make crashes very easy to debug.

iorx said:
Is it possible to see this, in the registry, if a drivers is suspected?

I believe that this is simply referring to the driver verification list you specify when you start Driver Verifier. It should be available under the following registry value:

Code:

reg query "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management" /v VerifyDrivers

Source: Use Driver Verifier to identify issues - Windows Server

x BlueRobot · Aug 2, 2023

Rich (BB code):

0: kd> .exr 0xfffff08fc639efe8
ExceptionAddress: fffff8070a18e9aa (ndis!ndisMReenumerateFailedAdapterInternal+0x00000000000001ae)
   ExceptionCode: c0000420 (Assertion failure)
  ExceptionFlags: 00000000
NumberParameters: 0
Assertion: *** An NDIS miniport driver has encountered a fatal error
    This is NOT a break in NDIS.sys
    The most common causes of these types of breaks are hung/unresponsive miniport firmware,
    failed power transitions, or failing MiniportRestart.

This is one of those cases where enabling Driver Verifier with ndis.sys may have been useful. Unfortunately, this is also one of those cases where you will need to provide a full kernel memory dump and then hope that the system doesn't crash, otherwise the file will be overwritten.

The file should be available in the following path, please copy it to a .zip file and then upload it your OneDrive as you have with the other files.

Code:

%systemroot%\MEMORY.DMP

Additionally, could you please search for the following string using FRST:

FRST Registry Search
1. Click your Start button and type in cmd.
2.After you find the Command Prompt, right click on it and select Run as Administrator.
3. Copy and paste the following into the Command Prompt:

reg load HKLM\DRIVERS C:\WINDOWS\SYSTEM32\CONFIG\DRIVERS
4. Please download Farbar Recovery Scan Tool and save it to your Desktop.
Note: You need to run the 64-bit Version so please ensure you download that one.
5. Run FRST64 by Right-Clicking on the file and choosing Run as administrator.
6. Copy and paste d36e972-e325-11ce-bfc1-0 into the Search box and click the Search Registry button.
7. When the scan is complete a notepad window will open with the results. Please attach this to your next reply. It is saved on your desktop named SearchReg.txt.

I believe that the GUID is going to be part of a device instance key or something similar.

x BlueRobot · Aug 2, 2023

The bugcheck from yesterday was also network-related so there is certainly something within your network stack which is causing a problem. I suspect that it may be OpenVPN but let's wait until you've provided the MEMORY.DMP file so we can get more information.

iorx · Aug 2, 2023

Thank you so much for the elaborate answer here! I'll do my best here to follow up.

OneDrive

Sign in to your OneDrive cloud storage and Office Online.

1drv.ms

Added the MEMORY.DMP as "MEMORY-2023-08-02_1030.7z" from the crash 10:30 today (verifer did not have ndis checked). A 7z-archive.
And the FRST64 as "FRST64-2023-08-02_1653_SearchReg.txt"

I'll rerun the verifier now with previous settings specified plus the ndis driver.
I've also changed the kernel dump settings to "kernel memory dump" instead of "automatic".
Restore point created.

Should I unload the hive after the FRST64?
"reg load HKLM\DRIVERS C:\WINDOWS\SYSTEM32\CONFIG\DRIVERS"

Note. Historically I've had some problem with "Realtek USB GbE", to get a stable operation when that is active I've had to modify its power settings, "Selective Suspend". (the NIC in my eGPU, which is not attached at the moment). The machines only peripheral right now is a Bluetooth mouse.

Going for a reboot in a moment. Will reconnect here when I encounter next BSOD...

Code:

verifier /querysettings

Verifier Flags: 0x03bfefbb

  Standard Flags:

    [X] 0x00000001 Special pool.
    [X] 0x00000002 Force IRQL checking.
    [X] 0x00000008 Pool tracking.
    [X] 0x00000010 I/O verification.
    [X] 0x00000020 Deadlock detection.
    [X] 0x00000080 DMA checking.
    [X] 0x00000100 Security checks.
    [X] 0x00000800 Miscellaneous checks.
    [X] 0x00020000 DDI compliance checking.

  Additional Flags:

    [ ] 0x00000004 Randomized low resources simulation.
    [X] 0x00000200 Force pending I/O requests.
    [X] 0x00000400 IRP logging.
    [X] 0x00002000 Invariant MDL checking for stack.
    [X] 0x00004000 Invariant MDL checking for driver.
    [X] 0x00008000 Power framework delay fuzzing.
    [X] 0x00010000 Port/miniport interface checking.
    [X] 0x00040000 Systematic low resources simulation.
    [X] 0x00080000 DDI compliance checking (additional).
    [X] 0x00200000 NDIS/WIFI verification.
    [X] 0x00800000 Kernel synchronization delay fuzzing.
    [X] 0x01000000 VM switch verification.
    [X] 0x02000000 Code integrity checks.

  Internal Flags:

    [X] 0x00100000 Extended Verifier flags (internal).

    [X] Indicates flag is enabled.

  Boot Mode:

    Persistent

  Rules:

    All rules are using default settings

  Extensions:

    wdm: rules.all
    ndis: rules.default

  Verified Drivers:

    ndis.sys
    npcap.sys
    ovpn-dco.sys
    wintun.sys
    tap0901.sys
    igdkmdn64.sys
    iaisp64.sys
    tbtbusdrv.sys
    ialpss2_i2c_tgl.sys
    intelthcbase.sys
    netwtw10.sys
    teedriverw10x64.sys
    ialpss2_uart2_tgl.sys
    intcaudiobus.sys
    ialpss2_gpio2_tgl.sys
    nvvad64v.sys
    nvmoduletracker.sys
    nvvhci.sys
    iactrllogic64.sys
    intcoed.sys
    ov5693.sys
    ov13858.sys
    vd55g0.sys
    rtkvhd64.sys
    intcusb.sys
    intcbtau.sys
    ibtusb.sys
    iacamera64.sys
    librehardwaremonitor.sys
    dump_dumpstorport.sys
    dump_stornvme.sys
    dump_dumpfve.sys
    netadaptercx.sys
    mbbcx.sys
    wificx.sys
    wdiwifi.sys

x BlueRobot · Aug 2, 2023

Thanks, it looks like your Intel network adapter was the one which was experiencing problems. It was in an hibernation state when the crash occurred so there is a possibly that it failed to wake from sleep, I'm curious to see if Driver Verifier will catch anything with ndis.sys being verified as well, but if it doesn't, then we can look at updating any network related drivers and booting into Safe Mode with Networking etc.

Code:

09 fffff08fc639f2a0 fffff8070a135331 ndis!ndisQueuedReenumerateFailedAdapter+18 
    Parameter[0] = ffffc80f6cc28fb0 << NDIS IO Work Item (can be cast to _IO_WORKITEM)
    Parameter[1] = ffffc80f1b31e1a0 << NDIS handle for network adapter (Use with !netadapter)
    Parameter[2] = (unknown)       
    Parameter[3] = (unknown)

Code:

0: kd> !netadapter ffffc80f1b31e1a0

MINIPORT

    Intel(R) Wi-Fi 6 AX201 160MHz

    Ndis handle        ffffc80f1b31e1a0
    Ndis API version   v6.60
    Adapter context    ffffc80f2587ee40
    Driver             ffffc80f1b20b020 - Netwtw10  v2.1
    Network interface  ffffc80f14618a20
    Ndis verifier      [Unrecognized flags 00000800]

    Media type         802.3
    Physical medium    Native802.11
    Device instance    PCI\VEN_8086&DEV_A0F0&SUBSYS_00748086&REV_20\3&11583659&0&A3
[...]

iorx said:
I've also changed the kernel dump settings to "kernel memory dump" instead of "automatic".

You could have left it as automatic, it will produce both Minidumps and a single kernel memory dump.

iorx said:
Should I unload the hive after the FRST64?
"reg load HKLM\DRIVERS C:\WINDOWS\SYSTEM32\CONFIG\DRIVERS"

As you've rebooted the machine, then the hive is probably unloaded now. I wouldn't worry about it though, the DRIVERS hive is dynamically loaded and unloaded by Windows when needed. It won't cause any problems with it remaining loaded.