Bug 37687 - Windows 2008 R2 Foundation raises error popup after join
Windows 2008 R2 Foundation raises error popup after join
Product: UCS
Classification: Unclassified
Component: Samba4
UCS 4.0
Other Linux
: P5 normal (vote)
: UCS 4.0-1-errata
Assigned To: Arvid Requate
Stefan Gohmann
Depends on:
Blocks: 39254
  Show dependency treegraph
Reported: 2015-02-04 15:19 CET by Tim Petersen
Modified: 2015-08-24 10:02 CEST (History)
3 users (show)

See Also:
What kind of report is it?: ---
What type of bug is this?: ---
Who will be affected by this bug?: ---
How will those affected feel about the bug?: ---
User Pain:
Enterprise Customer affected?:
School Customer affected?:
ISV affected?:
Waiting Support:
Flags outvoted (downgraded) after PO Review:
Ticket number:
Bug group (optional):
Max CVSS v3 score:

tcpdump -i eth0 -n -s0 -w /tmp/w2k8foundation.tcpdump host (167.61 KB, application/octet-stream)
2015-02-04 15:19 CET, Tim Petersen
log.samba, debug 12 (50.13 MB, text/plain)
2015-02-04 15:20 CET, Tim Petersen
foundation_server_ar40i1.tar.bz2 (1.49 MB, application/x-bzip)
2015-02-09 19:10 CET, Arvid Requate
application_partition_referrals_and_netlogon.patch (6.22 KB, patch)
2015-02-11 23:03 CET, Arvid Requate
Details | Diff
application_partition_referrals_and_netlogon.patch (6.96 KB, patch)
2015-02-12 10:10 CET, Arvid Requate
Details | Diff
application_partition_referrals_and_netlogon.patch (7.55 KB, patch)
2015-02-12 10:14 CET, Arvid Requate
Details | Diff
application_partition_referrals_and_netlogon.patch (5.70 KB, patch)
2015-02-12 10:15 CET, Arvid Requate
Details | Diff
Server Infrastructure Licensing.log (847 bytes, application/text)
2015-02-12 12:27 CET, Arvid Requate
dont_return_forst_and_domaindnszones_for_gc_base_search.patch (2.20 KB, patch)
2015-02-13 00:06 CET, Arvid Requate
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Tim Petersen univentionstaff 2015-02-04 15:19:56 CET
Created attachment 6643 [details]
tcpdump -i eth0 -n -s0 -w /tmp/w2k8foundation.tcpdump host

First reported via 2015012621000169:
After joining a windows 2008 r2 foundation server as a member in s4 domain, it throws the following popup after login:
The server did not finish checking the license compliance. If the server is joined to a domain, make sure that the server can connect to a domain controller. If the license compliant check cannot be completed, the server will automatically shut down in 0 hour(s) 30 minute(s).

UCS Master S4 - 4.0.0-errata 48:

UCS Bakcup S4 - shut off:

Windows 2008 R2 Foundation:

S4 domain is tim.ucs4

I did the login between 15:10 and 15:20

tcpdump -i eth0 -n -s0 -w /tmp/w2k8foundation.tcpdump host
Comment 1 Tim Petersen univentionstaff 2015-02-04 15:20:28 CET
Created attachment 6644 [details]
log.samba, debug 12
Comment 2 Arvid Requate univentionstaff 2015-02-04 20:37:39 CET
From log.samba I can only find one error response about a special Kerberos request from the client (S4U2proxy), but maybe that's normal:

[2015/02/04 15:10:16.360472, 10, pid=5856, effective(0, 0), real(0, 0)] ../source4/kdc/kdc.c:226(kdc_tcp_cal
  Received krb5 TCP packet of length 2353 from ipv4:
[2015/02/04 15:10:16.360497, 10, pid=5856, effective(0, 0), real(0, 0)] ../source4/kdc/kdc.c:159(kdc_process
  Received KDC packet of length 2345 from ipv4:
[2015/02/04 15:10:16.362320,  3, pid=5856, effective(0, 0), real(0, 0)] ../source4/auth/kerberos/krb5_init_c
  Kerberos: TGS-REQ win-n4f9ol473p6$@TIM.UCS4 from ipv4: for win-n4f9ol473p6$\@TIM.UCS4@TIM.UCS4 [canonicalize, request-anonymous, renewable, forwardable]


[2015/02/04 15:10:16.379361, 10, pid=5856, effective(0, 0), real(0, 0)] ../source4/kdc/db-glue.c:1782(samba_
  samba_kdc_check_s4u2proxy: client[CN=WIN-N4F9OL473P6,CN=Computers,DC=tim,DC=ucs4] for target[win-n4f9ol473
[2015/02/04 15:10:16.379393,  3, pid=5856, effective(0, 0), real(0, 0)] ../source4/auth/kerberos/krb5_init_c
  Kerberos: Bad request for constrained delegation
[2015/02/04 15:10:16.379410,  3, pid=5856, effective(0, 0), real(0, 0)] ../source4/auth/kerberos/krb5_init_c
  Kerberos: constrained delegation from win-n4f9ol473p6$@TIM.UCS4 (win-n4f9ol473p6$@TIM.UCS4) as win-n4f9ol4
73p6$@TIM.UCS4 to win-n4f9ol473p6$\@TIM.UCS4@TIM.UCS4 not allowed
[2015/02/04 15:10:16.379455,  3, pid=5856, effective(0, 0), real(0, 0)] ../source4/auth/kerberos/krb5_init_context.c:80(smb_krb5_debug_wrapper)
  Kerberos: Failed building TGS-REP to ipv4:
[2015/02/04 15:10:16.379472,  3, pid=5856, effective(0, 0), real(0, 0)] ../source4/auth/kerberos/krb5_init_context.c:80(smb_krb5_debug_wrapper)
  Kerberos: tgs-req: sending error: -1765328371 to client

Judging from the client port (63012) this corresponds to packet 492 in the tcpdump which shows a KRB Error response (the only one) indicating KRB5KDC_ERR_BADOPTION.

Please note that the time frame covered by log.samba is between 15:10:16.519548 and 15:11:40.380267 (samba restarts)
The tcpdump covers 15:09:11.71... to 15:11:29.70... (see "Arrival Time").

Additional things found in the the trace, maybe normal:

* No result DNS query for _ldap._tcp.Default-First-Site-Name._sites.master.tim.ucs4

* No result DNS query for _ldap._tcp.master.tim.ucs4

* No result LDAP query for CN=Public Key Services,CN=Services,CN=Configuration,DC=tim,DC=ucs4

* No result LDAP query for CN=Certification Authorities,CN=Public Key Services,CN=Services,CN=Configuration,DC=tim,DC=ucs4

* No result LDAP query for CN=AIA,CN=Public Key Services,CN=Services,CN=Configuration,DC=tim,DC=ucs4

* No result DNS query for _ldap._tcp.pdc._msdcs.DomainDnsZones.tim.ucs4

One other thing: There was a second DC joind into the domain ("backup") and the client also asks for it. I don't think that this is related to the issue here, but maybe it would be cleaner to try just with a single DC. Let's just keep this in mind as a last resort.
Comment 3 Arvid Requate univentionstaff 2015-02-04 20:56:03 CET
Maybe the best idea would be to collect a network trace the join of this "Foundation" licensed server against a native 2k8R2 AD/DC and compare if some additional communication happens. Could be related to anything: PKI, KRB5 or DCERPC just to name a few.
Comment 4 Arvid Requate univentionstaff 2015-02-04 21:21:23 CET
See also http://blogs.msmvps.com/bradley/2009/08/16/windows-server-2008-r2-foundation/ , maybe the Foundation server cannot positively confirm that it is joined into the root of a domain. Quoting from the technet article linked in the original bug report:

 "This error occurs when the server cannot finish checking the requirements for the root domain, forest trust configuration, or both."
Comment 5 Arvid Requate univentionstaff 2015-02-05 17:35:16 CET
At least I tracked down the KRB5KDC_ERR_BADOPTION to the Heimdal file kdc/krb5tgs.c, which ist the source of these two corresponding messages in the log.samba:
Kerberos: Bad request for constrained delegation

Kerberos: constrained delegation from win-n4f9ol473p6$@TIM.UCS4 (win-n4f9ol473p6$@TIM.UCS4) as win-n4f9ol473p6$@TIM.UCS4 to win-n4f9ol473p6$\@TIM.UCS4@TIM.UCS4 not allowed

Two comments in the source code indicate why this fails:
     * constrained_delegation (S4U2Proxy) only works within
     * the same realm. We use the already canonicalized version
     * of the principals here, while "target" is the principal
     * provided by the client.
       /* if client delegates to itself, that ok */

Now, the realm is the same, but the principal comparison probably fails because

  win-n4f9ol473p6$@TIM.UCS4    !=     win-n4f9ol473p6$\@TIM.UCS4@TIM.UCS4

The network trace shows that the client asked for a "constrained delegation" ticket for the server "win-n4f9ol473p6$@TIM.UCS4". To me this looks like Heimdal should recognize that the realm is already ok, and it should not escape the @REALM when the client passes it (and it matches). No clue what the RFC says about this though.

Theoretically there could be a chance that the Samba4 builtin Heimdal sources may behave differently in this case. In newer UCS versions we use the standard Debian system Heimdal libraries instead.

As a result, the client cannot obtain a "constrained delegation"/S4U2proxy ticket for himself (which should always work according to the source code).
No clue why he asks for it and if this actually leads him to the conclusion that something is not OK with the AD domain.
Comment 6 Arvid Requate univentionstaff 2015-02-09 18:46:49 CET
I managed to get rid off the KRB5KDC_ERR_BADOPTION problem (Comment 5) but the Windows Server 2008R2 Foundation still displays the error message.

This is what I did:
shell# cat |ldbmodify -H /var/lib/samba/private/sam.ldb <<%EOF 
dn: CN=WIN-N4F9OL473P6,CN=Computers,DC=tim,DC=ucs4
changetype: modify
add: msDS-AllowedToDelegateTo
msDS-AllowedToDelegateTo: win-n4f9ol473p6$\@TIM.UCS4

and these are the resulting success messages in log.samba:
+[2015/02/06 21:48:41.199346, 10, pid=28365, effective(0, 0), real(0, 0)] ../source4/kdc/db-glue.c:1782(samba_kdc_check_s4u2proxy)
+  samba_kdc_check_s4u2proxy: client[CN=WIN-N4F9OL473P6,CN=Computers,DC=tim,DC=ucs4] for target[win-n4f9ol473p6$\@TIM.UCS4]
+[2015/02/06 21:48:41.199370, 10, pid=28365, effective(0, 0), real(0, 0)] ../source4/kdc/db-glue.c:1826(samba_kdc_check_s4u2proxy)
+  samba_kdc_check_s4u2proxy: client[CN=WIN-N4F9OL473P6,CN=Computers,DC=tim,DC=ucs4] allowed target[(null)]
+[2015/02/06 21:48:41.199610,  3, pid=28365, effective(0, 0), real(0, 0)] ../source4/auth/kerberos/krb5_init_context.c:80(smb_krb5_debug_wrapper)
+  Kerberos: constrained delegation for win-n4f9ol473p6$@TIM.UCS4 from win-n4f9ol473p6$@TIM.UCS4 (win-n4f9ol473p6$@TIM.UCS4) to win-n4f9ol473p6$\@TIM.UCS4@TIM.UCS4

The "allowed target[(null)]" message is ok, (null) refers to a temporary pointer that has been freed a couple of lines before the log message.
Comment 7 Arvid Requate univentionstaff 2015-02-09 19:10:30 CET
Created attachment 6657 [details]

tcpdumps (join, reboot and login) and samba logs from another test setup with a MS 2k8R2 Foundation.

Client IP:
Server IP:
Server function level: domain & forest raised to 2008R2

The server still shows the same error message, even though I

* fixed the S4U2proxy delegation
* fixed a Kerberos lookup for SPN cifs/<dnsdomain>
* added all DNS SRV records the client asked for
* removed all references to my second DC ("backup51").

The only strange message I currently see in the tcpdumps is
 AS-REP NT Status: Unknown error code 0x30345241
but that might be unrelated.
Comment 8 Arvid Requate univentionstaff 2015-02-09 19:42:17 CET
There is a message in the eventviewer on the Client:

Applications and Services Logs > Microsoft > Windows > Server Infrastructure Licensing > Operational:

09.02.2015 19:28:38 

The Forest Trust Check in the Licensing component did not pass because error 0x8007054b occurred in function f1 [PHQG].

The specified domain either does not exist or could not be contacted.
Comment 9 Arvid Requate univentionstaff 2015-02-09 19:43:35 CET
Same message at 19:00:22, which is in the time frame of reboot.pcap.txt
Comment 10 Arvid Requate univentionstaff 2015-02-11 23:03:30 CET
Created attachment 6666 [details]

Ok, in my test environment I now managed to satisfy the Forest Root check of the Windows Server 2008 R2 Foundation. It's only one proof of concept until now:

1. The attached patch adjusts the referal URLs of the ForestDnsZones and DomainDnsZones partitions.

2. The attached patch adjusts the CLDAP Netlogon response for the ForestDnsZones and DomainDnsZones application partitions.

3. Finally, and this may be the crucial point (comparing eventlog and wireshark timestamps), the Samba4 LDAP server seems to return three records instead of one when the Foundation Server asks for something like:

  ldbsearch -H ldap://samba4_fqdn \
   -UAdministrator%univention  \
   --controls=domain_scope:0 --controls=search_options:0:2 \
   '(objectCategory=Domain)' canonicalName

From this he receives *three* objects, the first of which is fqdn of DomainDnsZones.ar40i1.qa. Next he asks DNS for _ldap._tcp.pdc._msdcs.DomainDnsZones.ar40i1.qa. When he doesn't find anything for that, he is unhappy. When I artificially create that record, he performs a CLDAP query against it, and is unhappy. Next he does the same stuff for ForestDnsZones.  And finally, he asks for the domain, and the CLDAP Netlogon response seems to be satifying for him. The popup disappeared.

My impression is, that the LDAP server should probably only return one object to the LDAP query above. That's what I see when I ask a native AD server -- kind of...  Anyway, with our Samba4 LDAP reply I have to create artificial DNS records to make the client continue.

Note: In the wireshark trace I don't see that the Foundation Server passes a "search_options" control, only the "domain_scope". But in log.samba it shows that the "search_options" control is considered (but I don't know if the value is actually "2").
Comment 11 Arvid Requate univentionstaff 2015-02-11 23:17:46 CET
Note: Wireshark traces from native AD also show the KRB5KDC_ERR_BADOPTION for the "constrained delegation" (S4U2proxy) Ticket request, so all the discussions about that in Comment 2, Comment 5 and Comment 6 are obsolete.

In my case the error popup disappeared even without the msDS-AllowedToDelegateTo workaround proposed in Comment 6.
Comment 12 Arvid Requate univentionstaff 2015-02-12 10:10:20 CET
Created attachment 6668 [details]

Patch containing the adjusted dns_update_list.
Comment 13 Arvid Requate univentionstaff 2015-02-12 10:14:23 CET
Created attachment 6669 [details]

Updated patch.
Comment 14 Arvid Requate univentionstaff 2015-02-12 10:15:26 CET
Created attachment 6670 [details]

Updated patch (need another coffee..)
Comment 15 Arvid Requate univentionstaff 2015-02-12 12:27:53 CET
Created attachment 6671 [details]
Server Infrastructure Licensing.log

Ok, fixed reproducably after reverting the Samba4 DC:

* Since my Samba4 DC was suspended tonight, the (running) Foundation Server showed three popups now, apparently the License Check is performed on a regular basis.

* After reverting my Samba4 DC, installing the patched Samba debian packages, adding the two DNS SRV records and re-joining the Foundation Server the License Check was again successful, the relevant exported MS eventviewer log messages are attached.

* Removing the DNS records and re-booting triggers the license error again.

* Adding the DNS records instead to an unpatched Samba is not enough, the license error is triggered again.

* The Samba patches are in two sections: 1. Referal URLs and 2. CLDAP Netlogon. Im quite confident about the first of those, since it makes Samba4 respond like AD. The second one is a bit of guess-work, some cleanup might be needed here to reduce possible side effects. The changes only affect requests for application partitions, so I guess that should not be an issue in "normal" installations.

* I guess it's better now to simply add the additional DNS records in the specific customer domain instead of trying to track down why the Windows Server Foundation asks for them.
Comment 16 Arvid Requate univentionstaff 2015-02-13 00:06:22 CET
Created attachment 6672 [details]

This alternative patch is actually enough:

1. No need to fix the referral URLs for the ForestDnsZones and DomainDnsZones partitions. The Windows Server 2008 R2 Foundation Forest Check doesn't care.

2. No need to adjust the CLDAP Netlogon response for the ForestDnsZones and DomainDnsZones application partitions either.

3. But when the Foundation client connects to the GC LDAP port 3268 to search for "(objectCategory=Domain)", a native AD server doesn't return any results from the application partitions (like ForestDnsZones and DomainDnsZones). The patch adjusts the behaviour of the Samba LDAP response at this point, so it only contains the single entry the client expects to find.

Due to this adjustment the Foundation client doesn't ask any longer for the additional non-standard DNS SRV records and the Forest Root check performed by the Server Infrastructure Licensing service is successful.
Comment 17 Arvid Requate univentionstaff 2015-02-16 14:54:59 CET
Built in errata4.0-1.

Advisory: 2015-02-16-samba.yaml
Comment 18 Stefan Gohmann univentionstaff 2015-03-05 16:43:31 CET
YAML: OK (r58687, fixed package version. I had to rebuild the samba package because the patch was disabled r14430)

Code review: OK

Foundation server join: OK

Samba ucs-test: OK

Win7 join + logon: OK

Win8 join + logon: OK
Comment 19 Moritz Muehlenhoff univentionstaff 2015-03-11 15:07:29 CET
Comment 20 Arvid Requate univentionstaff 2015-05-27 12:30:15 CEST
No reaction upstream. Filed bug https://bugzilla.samba.org/show_bug.cgi?id=11292 for this.