Windows DNS - SubDomain Blackhole

penguinpages · Oct 22, 2023

First posting on this forum. Trying to find a Windows forum willing to help debug and respond. Seems this is at least active. So .. greetings.

This is a repost... posted it on various Windows forums but no responses .. Problem continues .. been a few weeks

Full Posting Here:

Windows DNS - SubDomain Blackhole - Microsoft Q&A

Sorry that this is anohter posting of same question/ topic: I have been working on this for a few weeks now. I need tool and or fresh view of how to do root cause of DNS resolution issue. System: Windows 11 All NICs and VPN services disabled except…

learn.microsoft.com

Summary:
Windows HP Laptop with fresh install (a year ago but not using OEM image) of Windows Pro 11 build 23H2

Home network has AD domain controller. All other devices and VMs and resources working fine. This is MY computer. Baseline also with other peers who also VPN via CISCO client into remote lab and I have issue resolving DNS which they can without issu. MAC and other Windows hosts.

It takes me about 2 weeks to rebuild a clean and fully functional laptop and... trying to root cause vs "reboot / reload" which is not a fix.
Sense above posting I now cannot even resolve my home domain penguinpages.local so now two domains have become blackholed.

Issue:
I can use command and resolve names for my hosts file. I can resolve other domains / get on internet and even sub domains

I have downloaded and installed windbg and trying to figure out how to use it for this task. I have taken steps to strip down functions of my laptop:
1)removed HyperV
2) remove WSL2
3) unbind ALL protocols services or bindings but ONE ipv4

And if I add hosts manually in my hosts file .. I can resolve (see example in posting / below): Ex: vcenter01.penguinpages.local But to do this for the 100s of hostnames is not scalable.

I also have a secondary DNS (for my lab) which is based on infoblox , which I use for IPAM. I redirected my laptop via DHCP to this and no change.. so its not AD DNS vs anther bind service.

Running out of ideas.

Question:
1) Does anyone have example how to use windbg to debug shell lookup...where in nslookup client it returns value but fails in shell. (aka why will it not populate DNS cache
2) Any ideas on how to further debug?

Code:

PS C:\Users\Jerem> powershell "Get-DnsClientCache | Format-Table -AutoSize" |findstr ados
PS C:\Users\Jerem> nslookup - 172.16.100.22
Default Server:  ados.penguinpages.local
Address:  172.16.100.22

> ados.penguinpages.local
Server:  ados.penguinpages.local
Address:  172.16.100.22

Name:    ados.penguinpages.local
Addresses:  172.16.100.22
          172.16.103.22
          172.16.101.22

> vcenter01.penguinpages.local
Server:  ados.penguinpages.local
Address:  172.16.100.22

Name:    vcenter01.penguinpages.local
Address:  172.16.100.31

> pandora.penguinpages.local
Server:  ados.penguinpages.local
Address:  172.16.100.22

Name:    pandora.penguinpages.local
Address:  172.16.100.110

> exit
PS C:\Users\Jerem> ping ados.penguinpages.local
Ping request could not find host ados.penguinpages.local. Please check the name and try again.
PS C:\Users\Jerem> powershell "Get-DnsClientCache | Format-Table -AutoSize" |findstr ados
PS C:\Users\Jerem> ping vcenter01.penguinpages.local

Pinging vcenter01.penguinpages.local [172.16.100.31] with 32 bytes of data:
Reply from 172.16.100.31: bytes=32 time=8ms TTL=63
Reply from 172.16.100.31: bytes=32 time=13ms TTL=63

Ping statistics for 172.16.100.31:
    Packets: Sent = 2, Received = 2, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 8ms, Maximum = 13ms, Average = 10ms
Control-C
PS C:\Users\Jerem> ping pandora.penguinpages.local
Ping request could not find host pandora.penguinpages.local. Please check the name and try again.

neemobeer · Oct 22, 2023

I would grab dig and run a that with a +trace. That should walk you through the whole recursive name resolution process.
If you can provide a clean ping w/ cache cleared and a wireshark cap and zone files for the top domain that would be helpful. Feel free to DM them if you don't want to post them to the whole forum

penguinpages · Oct 22, 2023

Thanks for response. Good ideas and I did not think about adding dig into windows.

Plot thickens.

I enabled pcap on Gb NIC. I also ran a traceroute and I can see NSLookup / dig go out for IPs that work fine (other domains and sub domains. But any domain in the current "black hole" no packets...

I can attach wireshare trace if that helps. What I am missing is how Windows takes data out of name resolution and services it up thorough to applications / shell. I think this is handled by service "DNS Client" Which translates to "c:\windows\system32\svhost.exe -k NetworkService -p"

You cannot in GUI effect service or query it. I poked around in windows "process explorer" and most all seem to call through C:\windows\System32\dnsapi.dll

You can't kill the process or effect startup etc. Trying to find other means to debug it.

Dig notes below.

Code:

PS C:\Users\nerd> ping www.googel.com


Pinging www.googel.com [142.251.15.94] with 32 bytes of data:

Reply from 142.251.15.94: bytes=32 time=5ms TTL=106


Ping statistics for 142.251.15.94:

    Packets: Sent = 1, Received = 1, Lost = 0 (0% loss),

Approximate round trip times in milli-seconds:

    Minimum = 5ms, Maximum = 5ms, Average = 5ms

Control-C

PS C:\Users\nerd> nslookup

Default Server:  penguinpages.local

Address:  172.16.110.241



> ados.penguinpages.local

Server:  penguinpages.local

Address:  172.16.110.241

Name:    ados.penguinpages.local

Address:  172.16.100.22


> exit

PS C:\Users\nerd> ping ados.penguinpages.local

Ping request could not find host ados.penguinpages.local. Please check the name and try again.

PS C:\Users\nerd> dig ados.penguinpages.local +trace

; <<>> DiG 9.16.28 <<>> ados.penguinpages.local +trace
;; global options: +cmd
.                       59514   IN      NS      a.root-servers.net.
.                       59514   IN      NS      b.root-servers.net.
.                       59514   IN      NS      c.root-servers.net.
.                       59514   IN      NS      d.root-servers.net.
.                       59514   IN      NS      e.root-servers.net.
.                       59514   IN      NS      f.root-servers.net.
.                       59514   IN      NS      g.root-servers.net.
.                       59514   IN      NS      h.root-servers.net.
.                       59514   IN      NS      i.root-servers.net.
.                       59514   IN      NS      j.root-servers.net.
.                       59514   IN      NS      k.root-servers.net.
.                       59514   IN      NS      l.root-servers.net.
.                       59514   IN      NS      m.root-servers.net.
.                       59514   IN      RRSIG   NS 8 0 518400 20231104050000 20231022040000 46780 . n7VLDTbsoxzVZeDsRWIPcCc94Mwy974spwDTMb/eEzIMf4DimquqoVWj 4z50K4XrKKhFWjMyBfcgLYgL6jkvKlmhpT5MeyM67GpI+JkxLKdsM2o2 DXYRiQZONcyy8c47WZNeDLFfo9RSyBU4hwBG+uLuUrZ2KUgXeDDaR3dH 4JiS7m1YY3Gkok8rF0i8L9PWicCK7J9Hpr3rW+C+YdeOBknnXab8QvxY szvcbLWSuGWWxMuRmiZbjUwNKq82bDxm4Xhy5vF1urXOFgpLGDQ5Ukiu VgI9Bp50QeMP7p8YslkbPrZjYEXe18Pt2AgqzvWnUL47iMWsD29flYWP TMoF7g==
;; Received 555 bytes from 172.16.100.22#53(172.16.100.22) in 6 ms

.                       86400   IN      SOA     a.root-servers.net. nstld.verisign-grs.com. 2023102201 1800 900 604800 86400
.                       86400   IN      RRSIG   SOA 8 0 86400 20231104170000 20231022160000 46780 . EI1CpBmNMSzxS5qLT84FN0FEqdfMHQlAquiC1INgohJpf97QJwbNUAON +ulYgZDQirP7eDZyEXakkBMU0/jXasJ12+VrfEnV+jX35wvhoik0pfVQ 6jXE5B83rQmrXJij3MbsNugVvIQv/nqZ4s4rms2y2iiEyWrt/O401dm1 Sce/pXA7wDHGm7ZZpON+gg82WURW6PFVkC5WcNdGDDH4qpQxly6G1PwH MxPGMPIbweFqziaWtdKEP2c4+DfTtji29X3rS9oxDn56m5Y57xWKSkNP JFdA1LOVHQpbeUQgRzDF9ZzQjYDMXwAZH8jM+bUcSM8PVGH4EuepUrjF 5N/XzA==
.                       86400   IN      NSEC    aaa. NS SOA RRSIG NSEC DNSKEY ZONEMD
.                       86400   IN      RRSIG   NSEC 8 0 86400 20231104170000 20231022160000 46780 . JjZ6m4rVZiCu8/+xWXVT87nnMnddc+I8t1dQgfBnQeqnhShj0j/4iPQE TKsqjxL/yUlvo/CJpS4EPnvbxr5k50LW77597LfRS0SpidyQlDUi4bDx SAPrIQylngg8H2C42b5hdgHq5G7Q0k6ZnQG3f4A5LumFZAjmRVw9Ax52 HLksaH3aNu3VKNA0phrUtdx153s3mnCah3ZKAVupXsb0Otu3JYEO7Ce5 pMb6KU+emm7OmOh/aTo3q9caHI87qLfIltpIpmlPcFIHMb2Xj4/rKA6J cQPA1woAdOMrf2GU1ukOvOOIh96E5VYvdgG9pi2ukB/Q85SBK5pgj66i ayO8wA==
loans.                  86400   IN      NSEC    locker. NS DS RRSIG NSEC
loans.                  86400   IN      RRSIG   NSEC 8 1 86400 20231104170000 20231022160000 46780 . N3x3GIMOY9i/OrOZPdzdbMaiC/uR+JL+ix7b3/oQo0nhgGnW03r8PbEG lpdpurTcjVLItJR3aqL72d+rXs3AAOy+pP2dhHFrWIz2s6A0RygcotVn 2rh/o+dr3Xa1aqp5xADS359x43xPvwouPp3Xino/SdspGEtOSXR7Rrna b9nESji2Avsp2BkCq7FfhVkHetIiw/4ZPx1UYMI5YC2QjKkDjQqwa2lK LA3XdBb1jhyl6z9VRaG8mM3mFEw8W1XC8NasqinfbmYwyvh3RSkFRJQ5 l6cLEd78SleVEY7DsafCUKuEeH+oQQTcQ1XQMNhwD2L8PPorezatVLgd JYbB0w==
;; Received 1045 bytes from 193.0.14.129#53(k.root-servers.net) in 17 ms

PS C:\Users\nerd>

PS C:\Users\nerd> ping ados.penguinpages.local

Ping request could not find host ados.penguinpages.local. Please check the name and try again.

PS C:\Users\nerd> ssh ansible@172.16.100.41
The authenticity of host '172.16.100.41 (172.16.100.41)' can't be established.
ED25519 key fingerprint is SHA256:VScY6Z6iCeF4nU7dMlrYwMbaXG0o8Jz5s7MlrHCanJo.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '172.16.100.41' (ED25519) to the list of known hosts.
ansible@172.16.100.41's password:
Activate the web console with: systemctl enable --now cockpit.socket

Register this system with Red Hat Insights: insights-client --register
Create an account or view all your systems at https://red.ht/insights-dashboard
Last login: Mon Jul 17 08:02:43 2023 from 172.16.100.32
[ansible@ns01 ~]$ exit
logout
Connection to 172.16.100.41 closed.
PS C:\Users\nerd> ping ns01.penguinpages.local
Ping request could not find host ns01.penguinpages.local. Please check the name and try again.
PS C:\Users\nerd> dig  ns01.penguinpages.local +trace

; <<>> DiG 9.16.28 <<>> ns01.penguinpages.local +trace
;; global options: +cmd
.                       58979   IN      NS      f.root-servers.net.
.                       58979   IN      NS      g.root-servers.net.
.                       58979   IN      NS      h.root-servers.net.
.                       58979   IN      NS      i.root-servers.net.
.                       58979   IN      NS      j.root-servers.net.
.                       58979   IN      NS      k.root-servers.net.
.                       58979   IN      NS      l.root-servers.net.
.                       58979   IN      NS      m.root-servers.net.
.                       58979   IN      NS      a.root-servers.net.
.                       58979   IN      NS      b.root-servers.net.
.                       58979   IN      NS      c.root-servers.net.
.                       58979   IN      NS      d.root-servers.net.
.                       58979   IN      NS      e.root-servers.net.
.                       58979   IN      RRSIG   NS 8 0 518400 20231104050000 20231022040000 46780 . n7VLDTbsoxzVZeDsRWIPcCc94Mwy974spwDTMb/eEzIMf4DimquqoVWj 4z50K4XrKKhFWjMyBfcgLYgL6jkvKlmhpT5MeyM67GpI+JkxLKdsM2o2 DXYRiQZONcyy8c47WZNeDLFfo9RSyBU4hwBG+uLuUrZ2KUgXeDDaR3dH 4JiS7m1YY3Gkok8rF0i8L9PWicCK7J9Hpr3rW+C+YdeOBknnXab8QvxY szvcbLWSuGWWxMuRmiZbjUwNKq82bDxm4Xhy5vF1urXOFgpLGDQ5Ukiu VgI9Bp50QeMP7p8YslkbPrZjYEXe18Pt2AgqzvWnUL47iMWsD29flYWP TMoF7g==
;; Received 747 bytes from 172.16.100.22#53(172.16.100.22) in 1 ms

.                       86400   IN      SOA     a.root-servers.net. nstld.verisign-grs.com. 2023102201 1800 900 604800 86400
.                       86400   IN      RRSIG   SOA 8 0 86400 20231104170000 20231022160000 46780 . EI1CpBmNMSzxS5qLT84FN0FEqdfMHQlAquiC1INgohJpf97QJwbNUAON +ulYgZDQirP7eDZyEXakkBMU0/jXasJ12+VrfEnV+jX35wvhoik0pfVQ 6jXE5B83rQmrXJij3MbsNugVvIQv/nqZ4s4rms2y2iiEyWrt/O401dm1 Sce/pXA7wDHGm7ZZpON+gg82WURW6PFVkC5WcNdGDDH4qpQxly6G1PwH MxPGMPIbweFqziaWtdKEP2c4+DfTtji29X3rS9oxDn56m5Y57xWKSkNP JFdA1LOVHQpbeUQgRzDF9ZzQjYDMXwAZH8jM+bUcSM8PVGH4EuepUrjF 5N/XzA==
.                       86400   IN      NSEC    aaa. NS SOA RRSIG NSEC DNSKEY ZONEMD
.                       86400   IN      RRSIG   NSEC 8 0 86400 20231104170000 20231022160000 46780 . JjZ6m4rVZiCu8/+xWXVT87nnMnddc+I8t1dQgfBnQeqnhShj0j/4iPQE TKsqjxL/yUlvo/CJpS4EPnvbxr5k50LW77597LfRS0SpidyQlDUi4bDx SAPrIQylngg8H2C42b5hdgHq5G7Q0k6ZnQG3f4A5LumFZAjmRVw9Ax52 HLksaH3aNu3VKNA0phrUtdx153s3mnCah3ZKAVupXsb0Otu3JYEO7Ce5 pMb6KU+emm7OmOh/aTo3q9caHI87qLfIltpIpmlPcFIHMb2Xj4/rKA6J cQPA1woAdOMrf2GU1ukOvOOIh96E5VYvdgG9pi2ukB/Q85SBK5pgj66i ayO8wA==
loans.                  86400   IN      NSEC    locker. NS DS RRSIG NSEC
loans.                  86400   IN      RRSIG   NSEC 8 1 86400 20231104170000 20231022160000 46780 . N3x3GIMOY9i/OrOZPdzdbMaiC/uR+JL+ix7b3/oQo0nhgGnW03r8PbEG lpdpurTcjVLItJR3aqL72d+rXs3AAOy+pP2dhHFrWIz2s6A0RygcotVn 2rh/o+dr3Xa1aqp5xADS359x43xPvwouPp3Xino/SdspGEtOSXR7Rrna b9nESji2Avsp2BkCq7FfhVkHetIiw/4ZPx1UYMI5YC2QjKkDjQqwa2lK LA3XdBb1jhyl6z9VRaG8mM3mFEw8W1XC8NasqinfbmYwyvh3RSkFRJQ5 l6cLEd78SleVEY7DsafCUKuEeH+oQQTcQ1XQMNhwD2L8PPorezatVLgd JYbB0w==
;; Received 1045 bytes from 202.12.27.33#53(m.root-servers.net) in 60 ms

PS C:\Users\nerd>

neemobeer · Oct 22, 2023

How is DNS configured on the client? I'm assuming it's pointed at an internal DNS and that domain is local only. It shouldn't even be hitting the root hint servers, because they won't know how to find the NS servers for it since it's not registered out in the ether

penguinpages · Oct 23, 2023

Standard DHCP lease from my router. Which points to local single AD node.

I can also flip it to my pair of Infoblox VMs which host the lab stuff.. both DNS which are SOA for penguinages.local (no sync just ignorant of eachother) but same results.

I have flipped between them and no change so I really don't think its external to system.

Code:

PS C:\Users\nerd> ipconfig /all

Windows IP Configuration

   Host Name . . . . . . . . . . . . : LT-01
   Primary Dns Suffix  . . . . . . . :
   Node Type . . . . . . . . . . . . : Hybrid
   IP Routing Enabled. . . . . . . . : No
   WINS Proxy Enabled. . . . . . . . : No
   DNS Suffix Search List. . . . . . : penguinpages.local

Ethernet adapter Ethernet:

   Connection-specific DNS Suffix  . : penguinpages.local
   Description . . . . . . . . . . . : Realtek USB GbE Family Controller
   Physical Address. . . . . . . . . : 80-6D-97-05-09-08
   DHCP Enabled. . . . . . . . . . . : Yes
   Autoconfiguration Enabled . . . . : Yes
   IPv4 Address. . . . . . . . . . . : 172.16.100.32(Preferred)
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Lease Obtained. . . . . . . . . . : Sunday, October 22, 2023 8:08:31 PM
   Lease Expires . . . . . . . . . . : Monday, October 23, 2023 2:09:11 AM
   Default Gateway . . . . . . . . . : 172.16.100.1
   DHCP Server . . . . . . . . . . . : 172.16.100.1
   DNS Servers . . . . . . . . . . . : 172.16.100.22
                                       8.8.8.8
   NetBIOS over Tcpip. . . . . . . . : Enabled
   Connection-specific DNS Suffix Search List :
                                       penguinpages.local
PS C:\Users\nerd>

Just as a baseline I set to have static IP DNS etc.

As expected I have local subnet zone set for default

Same result.

Code:

PS C:\Users\nerd> ipconfig /all

Windows IP Configuration

   Host Name . . . . . . . . . . . . : LT-01
   Primary Dns Suffix  . . . . . . . :
   Node Type . . . . . . . . . . . . : Hybrid
   IP Routing Enabled. . . . . . . . : No
   WINS Proxy Enabled. . . . . . . . : No
   DNS Suffix Search List. . . . . . : penguinpages.local

Ethernet adapter Ethernet:

   Connection-specific DNS Suffix  . : penguinpages.local
   Description . . . . . . . . . . . : Realtek USB GbE Family Controller
   Physical Address. . . . . . . . . : 80-6D-97-05-09-08
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes
   IPv4 Address. . . . . . . . . . . : 172.16.100.32(Preferred)
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Default Gateway . . . . . . . . . : 172.16.100.1
   DNS Servers . . . . . . . . . . . : 172.16.100.22
   NetBIOS over Tcpip. . . . . . . . : Enabled
   Connection-specific DNS Suffix Search List :
                                       penguinpages.local
PS C:\Users\nerd> ping ns01.penguinpages.local
Ping request could not find host ns01.penguinpages.local. Please check the name and try again.
PS C:\Users\nerd> nslookup
Default Server:  ados.penguinpages.local
Address:  172.16.100.22

> ns01.penguinpages.local
Server:  ados.penguinpages.local
Address:  172.16.100.22

Name:    ns01.penguinpages.local
Address:  172.16.100.41

> exit
PS C:\Users\nerd>

neemobeer · Oct 23, 2023

If the records your trying to hit are on subdomains then the zones can not be ignorant of each other. If they are then your system will always try and start with the root zone '.' which will have no way to get to the NS servers for any of your internal DNS servers

penguinpages · Oct 23, 2023

in this example.. now that its not just one subdomain connected over VPN:

Ex: VPN to lab
labs.local (top level intranet domain) -> success all a and ptr
cnan.labs.local -> success sub domain to separate AD hosts
ps.labs.local -> black hole.. no packets even show up in Wireshark

Now... a week ago.. my home lab domain penguinpages.local now blackhole.. So something is
getting messed up and VPN / remote DNS is now out of root cause.

As for the dig response.. I agree it seems like it has set in ipconfig to use local DNS 172.16.100.22 which is SOA for penguinpages.local

But it bypasses and goes right to root servers.. which have no idea of penguinpages.local

But why... nslookup does what is "normal" . why is dnscache / dnsclient bypassing lookups that are local.

Thx for noodling over ideas .. I am sure this is going to be something stupid. But not seeing trees for the forest.

neemobeer · Oct 23, 2023

I think its just the behavior of dig +trace. I saw the same behavior on my setup. I have an internal DNS server as well. Since I can't see your zones, just want to make sure you have NS records to the subdomain zones?

penguinpages · Oct 23, 2023

the zone issue is now direct within top level zone .

My home lab high level zone is "penguinpages.local" and A and PTR for that zone fail. So its now not just one odd subzone.. but my primary home lab (no more VPN in frey either).

So something I or some client is doing to blackhole the zone is not just limited to sub zones

I also now have dropped using dhcp.. Hard coding IP / DNS in windows client to rule out any weirdness in that.

And I now baselined that sub zones under the one being blackholed also fail resolution

Code:

Windows PowerShell
Copyright (C) Microsoft Corporation. All rights reserved.

Install the latest PowerShell for new features and improvements! https://aka.ms/PSWindows

PS C:\Users\nerd> nslookup
Default Server:  penguinpages.local
Address:  172.16.110.241

> exit
PS C:\Users\nerd> ipconfig /release

Windows IP Configuration

The operation failed as no adapter is in the state permissible for
this operation.
PS C:\Users\nerd> nslookup
Default Server:  ados.penguinpages.local
Address:  172.16.100.22

> ados.penguinpages.local
Server:  ados.penguinpages.local
Address:  172.16.100.22

Name:    ados.penguinpages.local
Addresses:  172.16.100.22
          172.16.101.22
          172.16.103.22

> ns01.penguinpages.local
Server:  ados.penguinpages.local
Address:  172.16.100.22

Name:    ns01.penguinpages.local
Address:  172.16.100.41

> foo.apps.ocpdev.penguinpages.local
Server:  ados.penguinpages.local
Address:  172.16.100.22

Name:    foo.apps.ocpdev.penguinpages.local
Address:  172.16.110.122

> exit
PS C:\Users\nerd> ping ados.penguinpages.local
Ping request could not find host ados.penguinpages.local. Please check the name and try again.
PS C:\Users\nerd>
PS C:\Users\nerd> ping ns01.penguinpages.local
Ping request could not find host ns01.penguinpages.local. Please check the name and try again.
PS C:\Users\nerd> ping foo.apps.ocpdev.penguinpages.local
Ping request could not find host foo.apps.ocpdev.penguinpages.local. Please check the name and try again.
PS C:\Users\nerd> dig foo.apps.ocpdev.penguinpages.local

; <<>> DiG 9.16.28 <<>> foo.apps.ocpdev.penguinpages.local
;; global options: +cmd
;; Got answer:
;; WARNING: .local is reserved for Multicast DNS
;; You are currently testing what happens when an mDNS query is leaked to DNS
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 48948
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4000
;; QUESTION SECTION:
;foo.apps.ocpdev.penguinpages.local. IN A

;; ANSWER SECTION:
foo.apps.ocpdev.penguinpages.local. 3600 IN A   172.16.110.122

;; Query time: 1 msec
;; SERVER: 172.16.100.22#53(172.16.100.22)
;; WHEN: Mon Oct 23 11:15:35 Eastern Daylight Time 2023
;; MSG SIZE  rcvd: 79

PS C:\Users\nerd>

neemobeer · Oct 23, 2023

Do you have logging enabled and have you reviewed the DNS server logs?

neemobeer · Oct 23, 2023

Also it's not clear, are the sub-domains hosted on a different system, is that on infoblox? Do you have delegated domains setup with in the parent domain of the subdomains?

penguinpages · Oct 23, 2023

Infoblox VMs / system out of picture. I just noted those as part of debug (is it that one DNS / AD node being stupid .. lets try my Infoblox local DNS servers.... Nope... no change)

As for logs from the AD DNS which is 172.16.100.22 single SOA for zone penguinpages.local

I enabled DNS debug on AD server but I don't think its working like I expect... aka giving me outputs from client queries DNS Logging and Diagnostics I don't think that collects client events.. just DNS server tasks and events.

I did install wireshare on the DC and capture queries.

Baseline ping bar.com Server on left seeing query and right is my laptop

ping vcenter01.penguinpages.local (in hosts file) .. no packets (as expected

ping ns01.penguinpages.local (valid record in SOA of zone covered by host)

Not what I expected.. I see packets head out of workstation... but nothing from DNS server.

I have wireshark trace ..tying to figure out how to DM share .

penguinpages · Oct 23, 2023

Lightbulb just went off.

WTF

100.100.100.100 for Target

UGH

Who the Heck is that... gooogled.. tailscale -> I tried this months back and removed because I did not have time to play with it and build ROI... but that was weeks agoo... it was uninstalled... but googling 100.100.100.100 handles reverse DNS requests even though override local DNS is enabled · Issue #7859 · tailscale/tailscale

https://www.reddit.com/r/Tailscale/comments/szd90w

Now digging into WTF tailscale did / setup and where its burried.

neemobeer · Oct 23, 2023

Right click on the server in DNS tool > properties > debug logging. This will log the queries. Specific a location

penguinpages · Oct 24, 2023

Post to close this out

Windows 11 23H2 - Client does not uninstall clearly for DNS · Issue #9948 · tailscale/tailscale

What is the issue? I have local DNS segment "penguinpages.local" as well as via VPN a few domains and subdomains. I installed to test tailscale and never really got that far and shelfed the product...

github.com

Thanks @neemobeer for jumping in and helping me. I was preping USB reinstall key.. which would have cost me two weeks to reset things.. so this was a help.