Microsoft Surface Pro 8 and BSODs since a couple of month back

iorx · Aug 2, 2023

The BSOD behavior has changed a little bit it feels like.
I had put my machine away for dinner (I just fold up the keyboard and it goes to sleep), when I later picked it up it presented me with the beautiful blue screen... checking the time for the dump file it seems to have happened while it was a sleep (confirms your findings), the time don't match when I put it to sleep or when I woke it up.

I've added a V2 log and the latest large dump file (which is from when ndis is enabled for verifier).

All drivers should be from the recommended package from Microsoft, SurfacePro8_Win11_22000_23.062.19726.0.msi Download Surface Pro 8 Drivers and Firmware. from Official Microsoft Download Center

Brgs

x BlueRobot · Aug 2, 2023

It looks like your camera driver has caused your most recent crash.

Rich (BB code):

0: kd> knL
 # Child-SP          RetAddr               Call Site
00 ffffec8c`e9ddd298 fffff805`2586f704     nt!KeBugCheckEx
01 ffffec8c`e9ddd2a0 fffff805`2580e411     nt!PspSystemThreadStartup$filt$0+0x44
02 ffffec8c`e9ddd2e0 fffff805`25859e6f     nt!_C_specific_handler+0xa1
03 ffffec8c`e9ddd350 fffff805`2568c8b3     nt!RtlpExecuteHandlerForException+0xf
04 ffffec8c`e9ddd380 fffff805`25710f9d     nt!RtlDispatchException+0x2f3
05 ffffec8c`e9dddaf0 fffff805`25789b2f     nt!RtlRaiseException+0x10d
06 ffffec8c`e9dde070 fffff805`25a31a6e     nt!RtlRaiseStatus+0x4f
07 ffffec8c`e9dde610 fffff805`3d03466a     nt!purecall+0xe << Crash here!
08 ffffec8c`e9dde640 fffff805`3c62f98e     iacamera64+0xa2466a
09 ffffec8c`e9dded00 fffff805`3d02b848     iacamera64+0x1f98e
0a ffffec8c`e9dded70 fffff805`3c6668d5     iacamera64+0xa1b848
0b ffffec8c`e9ddeda0 fffff805`3c66aa49     iacamera64+0x568d5
0c ffffec8c`e9ddee00 fffff805`3c693b64     iacamera64+0x5aa49
0d ffffec8c`e9ddef00 fffff805`3c6aac60     iacamera64+0x83b64
0e ffffec8c`e9ddefa0 fffff805`3c69244a     iacamera64+0x9ac60
0f ffffec8c`e9ddf020 fffff805`3c665b70     iacamera64+0x8244a
10 ffffec8c`e9ddf0a0 fffff805`3c62db63     iacamera64+0x55b70
11 ffffec8c`e9ddf1d0 fffff805`3c62ce82     iacamera64+0x1db63
12 ffffec8c`e9ddf220 fffff805`3c6304fe     iacamera64+0x1ce82
13 ffffec8c`e9ddf260 fffff805`25631f30     iacamera64+0x204fe
14 ffffec8c`e9ddf290 fffff805`256356a5     nt!IopProcessWorkItem+0x100
15 ffffec8c`e9ddf300 fffff805`257429f7     nt!ExpWorkerThread+0x155
16 ffffec8c`e9ddf4f0 fffff805`25853b64     nt!PspSystemThreadStartup+0x57
17 ffffec8c`e9ddf540 00000000`00000000     nt!KiStartSystemThread+0x34

This is the exception which was raised:

Code:

0: kd> !error c0000002
Error code: (NTSTATUS) 0xc0000002 (3221225474) - {Not Implemented}  The requested operation is not implemented.

Essentially, it means that your camera driver has tried to call a function which has no concrete implementation; an non-existent function effectively. You could either check for any other driver using the Intel Driver Support Assistant or remove it completely using the pnputil command.

In order to remove the driver, could you please perform a FRST registry search like you did before but with the following search term: iacamera64.

iorx said:
All drivers should be from the recommended package from Microsoft, SurfacePro8_Win11_22000_23.062.19726.0.msi Download Surface Pro 8 Drivers and Firmware. from Official Microsoft Download Center

That seems to have been released very recently so you should technically have all of the latest drivers. Your crashes are quite sporadic though, so either you're simply unlucky with a set of bad drivers or the device doesn't support hibernation properly.

zbook · Aug 2, 2023

Indicate if the free backup images had been made per post #20

If they were not yet made then when WDV is off make free backup images and save the images to another disk drive or the cloud.

(Acronis, Aoemi, EaseUS, Macrium, Paragon, etc.)

iorx · Aug 3, 2023

zbook said:
Indicate if the free backup images had been made per post #20

If they were not yet made then when WDV is off make free backup images and save the images to another disk drive or the cloud.

(Acronis, Aoemi, EaseUS, Macrium, Paragon, etc.)

Hi
Backup I've got.
Used Sysinternals disk2vhd64.exe, ShadowCopy option, at the beginning of this troubleshooting. MSP8-20230725.VHDX; 2023-07-28; ~650GB.

zbook · Aug 3, 2023

Does that method allow the relatively quick restoration of an image that includes files, drivers, applications, and partitions?
If not, then please use one of the third party software.

iorx · Aug 3, 2023

x BlueRobot said:
It looks like your camera driver has caused your most recent crash.

This makes no sense to me that BSOD originates from, all over the place, different drivers. Could it still be a memory problem or hardware fault? (I'm not as deeply experienced in this as you are guys, so just a thought

)

I've uploaded new V2 and a dump file as I came back to a unresponsive machine this morning with the fans on full speed and a black screen...
And a new FRST with search for "iacamera64"

OneDrive

Sign in to your OneDrive cloud storage and Office Online.

1drv.ms

I'm turning of WDV and check for driver updates then.

iorx · Aug 3, 2023

zbook said:
Does that method allow the relatively quick restoration of an image that includes files, drivers, applications, and partitions?
If not, then please use one of the third party software.

Not a convenient way back no

, more like a backup i one huge disk file. I've restored from VHDX a couple of time (not on this machine) but it involves booting on another machine and the write back the VHDX to the particular disk. I'll take your recommendation here and use 3:rd party tool. You mentioned a couple, any preferred?

zbook · Aug 3, 2023

Macrium was free.
The current version may no longer be free.

Backup and Restore with Macrium Reflect

How to Create and Restore a System Image Backup with Macrium Reflect

www.tenforums.com

Restart WDV selecting all customized tests.

If there is no immediate BSOD then post a new querysettings.

For any BSOD post a new V2 share link.

Find a flash drive that you can format (> or = 8 GB)

Create a bootable Windows 11 Media Creation Tool (MCT) (Window 11 iso)

Download Official Windows 11 ISO file from Microsoft Tutorial

This tutorial will show you how to download an official Windows 11 ISO file from Microsoft. Microsoft provides ISO files for Windows 11 to download. You can use these ISO files to clean install or in-place upgrade Windows 11. The Flight Hub can also help you keep updated on which Windows 11...

www.elevenforum.com

Download Windows 11

iorx · Aug 3, 2023

zbook said:
Restart WDV selecting all customized tests.
If there is no immediate BSOD then post a new querysettings.
For any BSOD post a new V2 share link.

Find a flash drive that you can format (> or = 8 GB)
Create a bootable Windows 11 Media Creation Tool (MCT) (Window 11 iso)

Download Official Windows 11 ISO file from Microsoft Tutorial

This tutorial will show you how to download an official Windows 11 ISO file from Microsoft. Microsoft provides ISO files for Windows 11 to download. You can use these ISO files to clean install or in-place upgrade Windows 11. The Flight Hub can also help you keep updated on which Windows 11...

www.elevenforum.com

Download Windows 11

Thanks!

The dump file and V2 currently in sp8-bsod are from this morning. WDV was active with all previous specified options plus ndis.

MCT. Can I ask why this task? Are you aiming for a reinstall of Windows 11?

zbook · Aug 3, 2023

Include [ ] 0x00000004 Randomized low resources simulation.

The computer likely has malfunctioning hardware.

These are some options:

a) Stop troubleshooting > contact Microsoft for a Return Merchandise Authorization (RMA) and warranty replacement

b) Monitor the computer in safe mode > monitor for unexpected shutdowns and restarts

c) Clean install Windows 11 or 10 > monitor for unexpected shutdowns and restarts

d) Remove 16 GB RAM > test 16 GB RAM for one week > test the other 16 GB RAM for one week > monitor for unexpected shutdowns and restarts

iorx · Aug 3, 2023

x BlueRobot said:
You could either check for any other driver using the Intel Driver Support Assistant or remove it completely using the pnputil command

Now running without WDV (Oh nice! a fast machine again...

verifier takes it's toll on performance it looks like).

Tried the intel update thingy. It found the below to update. Warned me about the graphic driver, if I encountered problem I should use the one provided by OEM (Microsoft) it said.

Also hunted down the driver for the camera. Found a cab at Microsoft (Microsoft Update Catalog) which had a more recent version, 2023-04-30 63.22000.3.12296. Old version was from 2022-08... but... the camera doesn't seems to work with this driver. Trying a reboot.

Intel update summary:

Code:

Windows® 10 and Windows 11* Wi-Fi Drivers for Intel® Wireless Adapters

Installation successful
8/3/2023 9:53 AM
Description:
This download record installs the Windows® 10 and Windows 11* WiFi package drivers 22.240.0 for the Intel® Wi-Fi 6E/Wi-Fi 6 and Intel® 9000 series Wireless Adapters.
Version:
22.240.0.6Release date:
July 28, 2023Size:
40.44 MB


Intel® Wireless Bluetooth® for Windows® 10 and Windows 11*

Installation successful
8/3/2023 9:41 AM
Description:
Installs Intel® Wireless Bluetooth® version 22.240.0 Driver version varies depending on the wireless adapter installed.
Version:
22.240.0.2Release date:
July 29, 2023Size:
47.55 MB


Intel® Arc™ & Iris® Xe Graphics - WHQL - Windows*

Installation successful
8/3/2023 9:32 AM
Description:
This download installs Intel® Graphics Driver 31.0.101.4577 for Intel® Arc™ Graphics, 11th-13th Gen Intel® Core™ processor graphics, and related Intel Pentium® and Celeron® processors
Version:
31.0.101.4577Release date:
July 31, 2023Size:
644.84 MB

zbook · Aug 3, 2023

To complete WDV testing please test using all customized tests or for major performance problems only select [ ] 0x00000004 Randomized low resources simulation.

x BlueRobot · Aug 3, 2023

Okay, there's been quite a few updates since my last post in this thread, to answer your questions and remarks from zbook:

iorx said:
This makes no sense to me that BSOD originates from, all over the place, different drivers. Could it still be a memory problem or hardware fault? (I'm not as deeply experienced in this as you are guys, so just a thought )

I don't see any memory related issues at the moment so it seems that it could just be a set of "bad" drivers. It is possible to have crashes caused by multiple drivers. I can only go off what I find in the dump files unfortunately.

zbook said:
Restart WDV selecting all customized tests.

There is no reason to constantly restart Driver Verifier with the same settings endlessly. Why select all the tests? Especially when "Low Resources Simulation" has consistently been considered a "taboo" option as it usually either produces false positives or simply makes the machine very difficult to use because of the performance issues it can cause.

iorx said:
MCT. Can I ask why this task? Are you aiming for a reinstall of Windows 11?

Yes probably and there is no reason to do so in my opinion.

zbook said:
d) Remove 16 GB RAM > test 16 GB RAM for one week > test the other 16 GB RAM for one week > monitor for unexpected shutdowns and restarts

This is a Surface so I very much doubt you can even access or remove the RAM since it will be soldered to the motherboard itself. This will also very likely void any warranty or RMA conditions.

zbook said:
To complete WDV testing please test using all customized tests or for major performance problems only select [ ] 0x00000004 Randomized low resources simulation.

I echo my previous statements but will also add that a few of the tests are actually deprecated on Windows 11, presumably because they never worked correctly or are no longer compatible with the newer builds due to the various API changes.

x BlueRobot · Aug 3, 2023

iorx said:
Also hunted down the driver for the camera. Found a cab at Microsoft (Microsoft Update Catalog) which had a more recent version, 2023-04-30 63.22000.3.12296. Old version was from 2022-08... but... the camera doesn't seems to work with this driver. Trying a reboot.

It looks like there should be two optional driver updates under your Windows Update settings for that camera; at least from what I can see from the FRST log you provided. It might be worthwhile trying those drivers since they should be compatible with that particular camera hardware.

It looks like the latest crash was another Stop 0x7E but for the Intel network adapter again and it was the same crash as before, so there is some consistency with your crashes at least. This would suggest to me that the issue is possibly more driver-related or a very specific hardware issue. After updating your camera drivers from Windows Update, then I would suggest booting into Safe Mode with Networking and then wait until the system crashes from there, otherwise you could also disable hibernation and any sleep settings to prevent the device from sleeping. That should hopefully help to determine if there is an issue with power transitions being handled correctly, although, it looks like the network adapter was actually up and running this time.

Do you need the OpenVPN software for work purposes? I'm been reluctant to say to remove it at the moment because I wasn't sure it was "mission critical" or not.

From the Stop 0x7E:

Code:

5: kd> .exr 0xfffff40a82eb6fe8
ExceptionAddress: fffff8003d70e9aa (ndis!ndisMReenumerateFailedAdapterInternal+0x00000000000001ae)
   ExceptionCode: c0000420 (Assertion failure)
  ExceptionFlags: 00000000
NumberParameters: 0
Assertion: *** An NDIS miniport driver has encountered a fatal error
    This is NOT a break in NDIS.sys
    The most common causes of these types of breaks are hung/unresponsive miniport firmware,
    failed power transitions, or failing MiniportRestart.

Code:

5: kd> knL
 # Child-SP          RetAddr               Call Site
00 fffff40a`82eb5f78 fffff800`3b667704     nt!KeBugCheckEx
01 fffff40a`82eb5f80 fffff800`3b606411     nt!PspSystemThreadStartup$filt$0+0x44
02 fffff40a`82eb5fc0 fffff800`3b651e6f     nt!_C_specific_handler+0xa1
03 fffff40a`82eb6030 fffff800`3b4848b3     nt!RtlpExecuteHandlerForException+0xf
04 fffff40a`82eb6060 fffff800`3b538a0e     nt!RtlDispatchException+0x2f3
05 fffff40a`82eb67d0 fffff800`3b65c3fc     nt!KiDispatchException+0x1ae
06 fffff40a`82eb6eb0 fffff800`3b65a97f     nt!KiExceptionDispatch+0x13c
07 fffff40a`82eb7090 fffff800`3d70e9aa     nt!KiRaiseAssertion+0x33f
08 fffff40a`82eb7220 fffff800`3d7c50b8     ndis!ndisMReenumerateFailedAdapterInternal+0x1ae << Crash here!
09 fffff40a`82eb72a0 fffff800`3d6b5331     ndis!ndisQueuedReenumerateFailedAdapter+0x18
0a fffff40a`82eb72d0 fffff800`3b42d6a5     ndis!ndisWorkItemHandler+0x11
0b fffff40a`82eb7300 fffff800`3b53a9f7     nt!ExpWorkerThread+0x155
0c fffff40a`82eb74f0 fffff800`3b64bb64     nt!PspSystemThreadStartup+0x57
0d fffff40a`82eb7540 00000000`00000000     nt!KiStartSystemThread+0x34

zbook · Aug 3, 2023

For option a)

Stop troubleshooting > contact Microsoft for a Return Merchandise Authorization (RMA) and warranty replacement

one or more of these links may be useful:

How to get service or repair for Surface - Microsoft Support

Learn how the service process for Microsoft Surface works and how to request service online if your Surface needs repair.

support.microsoft.com

Contact Us - Microsoft Support

Contact Microsoft Support. Find solutions to common problems, or get help from a support agent.

support.microsoft.com

Fix common Surface problems using the Surface app and Surface Diagnostic Toolkit - Microsoft Support

Access, download, or run the Surface app and the Surface Diagnostic Toolkit to help find and solve Surface problems.

support.microsoft.com

Restore or reset Surface for Windows - Microsoft Support

If you're having problems with your Surface Pro model, Surface 3, Surface Studio, or Surface Book, try fresh start, restoring, refreshing, or resetting it to its factory settings.

support.microsoft.com

For option c)

Clean install Windows 11 or 10 > monitor for unexpected shutdowns and restarts

Window 11 and Windows 10 often have different drivers.

A clean install of Windows 10 should allow testing the computer hardware in a different computer environment.

(This will remove the Windows 11 operating system and various drivers as confounding factors and allow additional testing of the hardware.)

x BlueRobot · Aug 3, 2023

Why do a clean install to Windows 10? That assumes that there is drivers available for this device. Additionally, there are better alternatives to confirm potiental hardware failure i.e. booting into Safe Mode with Networking and then seeing if it crashes there.

I still very much doubt that this is a hardware issue at the moment since we still haven't dealt with those driver issues as of yet.

zbook · Aug 4, 2023

x BlueRobot said:
Why do a clean install to Windows 10? That assumes that there is drivers available for this device.

Why not check the Microsoft website?

The laptop has Microsoft tested drivers for Windows 10.

x BlueRobot said:
Additionally, there are better alternatives to confirm potiental hardware failure i.e. booting into Safe Mode with Networking and then seeing if it crashes there.

Safe mode was one of the options offered in the prior post.

There are pro and con for every testing method.

x BlueRobot said:
I still very much doubt that this is a hardware issue at the moment since we still haven't dealt with those driver issues as of yet.

How come you did not read the log files?

The computer has malfunctioning hardware that was not yet confirmed.

The laptop is under warranty and is having or has had:

BSOD multiple types 1A, 1E, 3B, 7E, 139, D1, A, 1AA, etc.
Live kernel events
Corrected hardware errors
Possible malfunctioning hardware (not yet confirmed by testing)
Several misbehaving drivers using WDV
etc.

It is extremely unlikely that another misbehaving driver if found could fix all of the above problems.

The findings are highly consistent with an underlying hardware problem.

The computer has malfunctioning RAM (this has not yet been confirmed by testing but is consistent with the above and log file reports)

Most computer users would rather have:

a problem free new computer
a computer without corrected hardware errors in the log files
etc.

Microsoft per their link may want a reset or clean install before honoring a RMA.
Warranty replacement is recommended.

x BlueRobot · Aug 4, 2023

I do not wish to derail this thread any further so I will leave my prior analysis and recommendations as my closing post.

iorx · Aug 5, 2023

Hi!

For now I would like to thank you both immensely for your engagement and offering your time and knowledge in this case. It's been of great value and I feel like being taken care of professionals. KUDOS and many thanks again!

Knock on wood I would like to say! The machines has been stable since I ran the Intel Drivers upgrade thingy (#71) and fiddled around with the camera drivers (I used DriverStore Explorer, GitHub - lostindark/DriverStoreExplorer: Driver Store Explorer [RAPR], to clean out the old, camera drivers, and the ones not working (cab-version I found at Microsoft which installed fine but didn't work with the hardware, an amateur move maybe)

Comment on some loose ends from above.
OpenVPN. Connect to all customers and my own offices this way.

Going forward.
I've learned lots going through this with you guys. As the machine has been stable for a couple of days now I'll leave it as is and hope for the best.
Gained knowledge here have given me what I need to build a case against Microsoft for an RMA if it later shows that it actually is faulty hardware.
(When I added the WDV {badmemory} to my case there they where willing to do a RMA replacement of the device, I put that on hold as I wanted to finish this thread here.)

So, my plan if I encounter more problem is:

Re-image the device with according to Microsoft's instructions (they're very keen on that you do this before confirming hardware fault).
Re-install everything I need and have installed right now (this takes time...), and if I still encounter crashes I'll re-initiate my Microsoft Case (focusing on either faulty hardware or something bad with their re-image/driver package for this model)

I'll circle back here with info how it goes with my device if it's of interest.

(I've closed the OneDrive folder share as of now)

Best Regards!