Univention Bugzilla – Bug 39207
Logon failures on Windows Clients that have changed their password after domain/forest function level was raised to 2008r2
Last modified: 2020-07-02 17:19:45 CEST
Ticket#2015072421000189 Customer raised domain and forest function level vom 2003 to 2008R2. On the next password change of the Windows (7+) clients domain users are unable to login ("Username or password is wrong") until the next reboot of the client. After clients run into this once they are able to do subsequent password changes without the need to reboot. So in short it looks as if the clients need a reboot to accept the first new key/password after domain/forest function level was raised to 2008r2. There's a technet article that looks quite similar although we have not seen the error messages quoted there: http://blogs.technet.com/b/askds/archive/2014/07/23/it-turns-out-that-weird-things-can-happen-when-you-mix-windows-server-2003-and-windows-server-2012-r2-domain-controllers.aspx Workaround: Force all clients to change their passwords and reboot after raising the function level: nltest /SC_CHANGE_PWD:<DomainName> /server:<CLIENT_FQDN> nltest /SHUTDOWN:PasswordChange /server:<CLIENT_FQDN>
Since it is not planned to raise the domain level automatically, we should move it into a SDB article.
Is this still useful? I have never came across this issue
(In reply to Christina Scheinig from comment #2) > Is this still useful? I have never came across this issue With UCS 4.1-0 we started using "2008 R2" as default function level. That means: → Domains installed with 4.1-0 or later are fine → Domains installed with a UCS Version prior to 4.1-0 most probably still use "2003" Upgrading the function level is probably a rare case, but every customer that attempts it, will hit the nasty problem described here.
Any clarifications on that? Although Windows 10 doesn't yet require higher functional levels, MS says that (as of writing this message) "No new Active Directory schema updates or specific functional levels are currently required for core Windows 10 product functionality, although subsequent upgrades could require these to support new features." (https://docs.microsoft.com/en-us/windows/deployment/planning/windows-10-infrastructure-requirements) There is also little written about it other than that fresh installs with 4.1-0 or newer start with 2008 R2 forest and domain level. We'd really like to see a known-good documented way to do this properly since otherwise it's going to be considered to scary to touch. Really looking forward to see an article on this topic. If we can work something up together I'd be willing to assist. - Both on a professional as well as purely based on personal interest.
I think we should test it. Reading the original bug description carefully suggests to me that a simple reboot of the clients after the level raise may be sufficient to avoid the issue. At least that we should test. Also, for UCS 4.2 we changed the preference order of krb5 encryption types (Bug 43850 Comment 3), that may change something too. Regarding the blog article Janis cited above, Microsoft released a hotfix for Windows Server 2012 R2: https://support.microsoft.com/en-za/help/2989971/can-t-log-on-after-changing-machine-account-password-in-mixed-windows . But that doesn't help us directly. Also it's unclear if there is any direct relation the issue reported here.
More info on nltest: https://technet.microsoft.com/de-de/library/cc731935(v=ws.10).aspx > Nltest can test and reset the secure channel that the NetLogon service > establishes between clients and the domain controller that logs them on. > Clients using Kerberos authentication cannot use this secure channel. This also means that nltest will only work if NetLogon is available including the NetBios ports 137-139: https://support.microsoft.com/en-us/help/832017/service-overview-and-network-port-requirements-for-windows#method31
(In reply to Michael Grandjean from comment #6) > This also means that nltest will only work if NetLogon is available > including the NetBios ports 137-139: Hm, according to Microsoft, NetBios (137-139) is optional. For future reference: make sure that nltest is executed with Domain Admin or local Admin privileges when automating this step.
We have a similar case: Ticket #2018010821000607. Maybe the reason is that krbtgt has still old DES keys and no AES keys: ldbsearch -H /var/lib/samba/private/sam.ldb samaccountname=krbtgt supplementalcredentials --show-binary This could help: samba-tool user setpassword --random-password --filter sAMAccountName=krbtgt
Some more infos about the krbtgt password change: * https://cloudblogs.microsoft.com/microsoftsecure/2015/02/11/krbtgt-account-password-reset-scripts-now-available-for-customers/ * https://social.technet.microsoft.com/Forums/windowsserver/en-US/53033b4d-766b-4588-95fc-aadd93d8a053/impact-of-resetting-the-password-of-the-krbtgt-account?forum=winserverDS * https://visualplanet.org/blog/?p=20 * https://imav8n.wordpress.com/2007/12/19/replication-version-number-for-your-krbtgt-account-password/
https://lists.samba.org/archive/samba/2015-July/192909.html And here is the script: https://git.samba.org/?p=samba.git;a=blob_plain;f=source4/scripting/devel/chgkrbtgtpass;hb=HEAD
The script changed slightly, so better link to the corresponding branch: https://git.samba.org/?p=samba.git;a=blob_plain;f=source4/scripting/devel/chgkrbtgtpass;hb=refs/heads/v4-7-stable
Created attachment 10041 [details] raise_function_level_to_2008R2.sh In the attached script I collected all these steps: * print a warning and ask for confirmation to continue * check/raise function level of all locally known DCs * raise domain & forest function level in local samba domain * print instructions how to change the DNS-Service accounts on all DCs * print instructions how to change the machine passwords of all Windows Clients * print instructions how to handle UCS@school Slave PDCs (if found in LDAP) * print domain function level I'm still unsure about changing all UCS Samba/AD DC server passwords too. I hope that's not necessary, as there may be some timing issue regarding DRS replication and Kerberos if it is done to quickly on all DCs. Testing required.
Created attachment 10042 [details] raise_function_level_to_2008R2.sh After discussing with Felix I think we can and should recommend the server password change. The attached version also prints detailed instructions how to do that.
Let's QA the script first before we continue with the SDB. Please reopen in any case.
* no python3 in python-samba yet Changing password of KRBTGT Traceback (most recent call last): File "<stdin>", line 33, in <module> ImportError: No module named 'samba' changed python3 to python in the script * more Kerberos service accounts, do we need a password change for e.g. proxy-*, ucs-sso.$fqdn too? * --privatekeytab option is required for /usr/share/univention-samba4/scripts/create_spn_account.sh * keyVNo the same after service princiopal password change? Looks strange but not sure if this is relevant. ktutil --keytab=/var/lib/samba/private/dns.keytab list \ --keys /var/lib/samba/private/dns.keytab: Vno Type Principal Key 2 des-cbc-crc DNS/master.four.four@FOUR.FOUR 8f54d608406eb67f 2 des-cbc-crc dns-master@FOUR.FOUR 8f54d608406eb67f 2 des-cbc-md5 DNS/master.four.four@FOUR.FOUR 8f54d608406eb67f 2 des-cbc-md5 dns-master@FOUR.FOUR 8f54d608406eb67f 2 arcfour-hmac-md5 DNS/master.four.four@FOUR.FOUR 5fbd3e3341783f1c0994babad03e9aee 2 arcfour-hmac-md5 dns-master@FOUR.FOUR 5fbd3e3341783f1c0994babad03e9aee 2 aes128-cts-hmac-sha1-96 DNS/master.four.four@FOUR.FOUR 545fcb04125d85b93e5d78e7e393502a 2 aes128-cts-hmac-sha1-96 dns-master@FOUR.FOUR 545fcb04125d85b93e5d78e7e393502a 2 aes256-cts-hmac-sha1-96 DNS/master.four.four@FOUR.FOUR 687256487e6aea508c623ba8008db762d098b91bd1a75cbccfb2b42f5be64371 2 aes256-cts-hmac-sha1-96 dns-master@FOUR.FOUR 687256487e6aea508c623ba8008db762d098b91bd1a75cbccfb2b42f5be64371 ktutil --keytab=/var/lib/samba/private/dns.keytab.bak list \ --keys/var/lib/samba/private/dns.keytab.bak: Vno Type Principal Key 2 des-cbc-crc DNS/master.four.four@FOUR.FOUR 1a862fd97046049d 2 des-cbc-crc dns-master@FOUR.FOUR 1a862fd97046049d 2 des-cbc-md5 DNS/master.four.four@FOUR.FOUR 1a862fd97046049d 2 des-cbc-md5 dns-master@FOUR.FOUR 1a862fd97046049d 2 arcfour-hmac-md5 DNS/master.four.four@FOUR.FOUR 4402611b758ef43b48fdf4a06b5517eb 2 arcfour-hmac-md5 dns-master@FOUR.FOUR 4402611b758ef43b48fdf4a06b5517eb 2 aes128-cts-hmac-sha1-96 DNS/master.four.four@FOUR.FOUR be5419b8e714d148a592c805003771a0 2 aes128-cts-hmac-sha1-96 dns-master@FOUR.FOUR be5419b8e714d148a592c805003771a0 2 aes256-cts-hmac-sha1-96 DNS/master.four.four@FOUR.FOUR 258fbdf1215eb6ab3efe4d685243431b2671e0a4cfd538d33daf3282db6d277c 2 aes256-cts-hmac-sha1-96 dns-master@FOUR.FOUR 258fbdf1215eb6ab3efe4d685243431b2671e0a4cfd538d33daf3282db6d277c * create_spn_account.sh needs domain credentials on slave systems ssh slave '/usr/share/univention-samba4/scripts/create_spn_account.sh --samaccountname dns-slave --serviceprincipalname DNS/slave.four.four --privatekeytab dns.keytab' Password: Permission denied. ERROR: could not create user account dns-slave so + -binddn uid=Administrator,cn=users,dc=four,dc=four and --bindpwdfile /tmp/univention or maybe re-run the samba4-dns join script? * "shutdown /r /t 0 /c PasswordChage" message typo -> shutdown /r /t 0 /c PasswordChange OK - drs repl OK - windows client logon OK - user password change via windows client OK - univention-system-check OK - ucs-test checks (including diagnostic checks)
Created attachment 10049 [details] raise_function_level_to_2008R2.sh Ok, I adjusted the script to * not use python3 * simply re-run 98univention-samba4-dns.inst * fix the typo I agree that updating/re-creating the keys for ucs-sso and proxy-* would probably be good, but I couldn't find a way yet to do this for ucs-sso. Simply running 91univention-saml.inst and 98univention-samba4-saml-kerberos.inst again (even removing the account before) leaves me with a non-functional keytab, no clue why yet. I'll have to experiment a bit with actual sso. Re: The not changing KVNO you mentioned: that's probably because I don't change the password but delete and recreate the dns-* accounts, and they get the same KVNO.
While trying the script in a test environment (clone of prod at customers site I got the following: # ./raise_function_level.sh | tee raise.log Continue? [yN]: WARNING: running this script may change the function level of this domain and possibly update the krbtgt password hash WARNING: If the function level actually needed to be raised, all joined Microsoft Windows systems will have to rotate their machine passwords! WARNING: Please first check this in a representative (i.e. cloned) test environment before proceeding. y Setting domain function level to 2008_R2 Domain function level changed! All changes applied successfully! Setting forest function level to 2008_R2 Forest function level changed! All changes applied successfully! INFO: KRBTGT account doesn't have AES keys yet Changing password of KRBTGT Traceback (most recent call last): File "<stdin>", line 66, in <module> TypeError: update_krbtgt_account_password() takes exactly 1 argument (2 given) A transaction is still active in ldb context [0x2511d10] on /var/lib/samba/private/sam.ldb A transaction is still active in ldb context [0x2cb34d0] on /var/lib/samba/private/idmap.ldb A transaction is still active in ldb context [0x26094a8] on /var/lib/samba/private/secrets.ldb A transaction is still active in ldb context [0x25eef18] on /var/lib/samba/private/privilege.ldb The displayed domain and forest functional level was 2003 before and "2008 R2" after running the script.
Changes and improvements for SDB entries aren't tracked in Bugzilla anymore, so I close these entries. Please comment on help.univention.com or get in touch with the Univention Support team in case you have any suggestions for the SDB.