Bug 50492 - Windows login fails in UCS Samba/AD domain after changing password in MS AD domain
Windows login fails in UCS Samba/AD domain after changing password in MS AD d...
Status: CLOSED FIXED
Product: UCS
Classification: Unclassified
Component: AD Connector
UCS 4.4
Other Linux
: P5 normal (vote)
: UCS 4.4-3-errata
Assigned To: Julia Bremer
Arvid Requate
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2019-11-13 17:54 CET by Arvid Requate
Modified: 2020-03-18 12:27 CET (History)
8 users (show)

See Also:
What kind of report is it?: Bug Report
What type of bug is this?: 5: Major Usability: Impairs usability in key scenarios
Who will be affected by this bug?: 2: Will only affect a few installed domains
How will those affected feel about the bug?: 5: Blocking further progress on the daily work
User Pain: 0.286
Enterprise Customer affected?: Yes
School Customer affected?:
ISV affected?:
Waiting Support: Yes
Flags outvoted (downgraded) after PO Review:
Ticket number: 2019110721000375
Bug group (optional):
Max CVSS v3 score:


Attachments
adapted patch 92_allow_missing_des-cbc to allow every combination of keys (4.11 KB, patch)
2020-03-06 17:53 CET, Julia Bremer
Details | Diff
Alternative 14_univention-k5pwd-use-correct-salt.quilt (2.67 KB, text/plain)
2020-03-09 19:25 CET, Arvid Requate
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Arvid Requate univentionstaff 2019-11-13 17:54:06 CET
Setup:

UCS Samba/AD domain with AD-Connector configured to sync with MS AD domain.

Trigger:

User changes his AD password, e.g. while logged in at a Windows client machine that is joined into the MS AD domain (or admin resets AD password via MS ADUC gui).

Problem:

After that the user login against UCS Samba/AD fails with new password, e.g. when logging in with a second Windows client that is joined into the UCS domain. Instead Windows login is possible with the old password.
Comment 1 Arvid Requate univentionstaff 2019-11-13 18:03:39 CET
Other symptoms visible for Support:

* kinit in UCS only possible with *old* PW
* simple bind against OpenLDAP only possible with *old* PW
  (because userPassword = {K5KEY})
* smbclient (w/o kerberos) possible with *new* PW

Analysis:

The AD-Connector only synchronizes the NT-Hash from/to MS AD. It does not synchronize Kerberos hashes. That's fine for OpenLDAP, but causes the S4-Connector to not update the Kerberos-Keys stored in the Samba/AD attribute "supplementalCredentials". It cannot, because OpenLDAP doesn't have the required AES and/or DES keys.
Comment 2 Arvid Requate univentionstaff 2019-11-13 18:47:43 CET
Idea for a workaround: Make the KDC prefer the arcfour hashes:

ucr set \
kerberos/defaults/enctypes/permitted="arcfour-hmac-md5 aes256-cts-hmac-sha1-96 aes128-cts-hmac-sha1-96 des-cbc-crc des-cbc-md5 des-cbc-md4 des3-hmac-sha1 des3-cbc-sha1"
ucr unset kerberos/defaults/enctypes/tgs kerberos/defaults/enctypes/tkt
/etc/init.d/samba restart

Caveat: Worked in my artifical test-VM with Samba domain function level 2008-R2. Not tested in production yet.



Idea for a solution:

1. Make the AD-Connector read the supplementalCredentials from MS AD and write them into OpenLDAP krb5key attributes. They are salted with the UPN from the MS AD domain.

2. Check that the S4-Connector synchronizes these "alien" keys properly to Samba/AD (worked in my test setup).

3. Adjust our OpenLDAP patch 12_k5pwd.patch to use the salt stored in the krb5key(s) when checking the user password during simple bind.


This seems to work with Heimdal with and without Samba. Heimdal seems to use the salt that is stored in the krb5Key/supplementalCredentials rather than the actual principal name of the user.

There is one caveat though: If the MS AD domain has a different set of Kerberos keys than the UCS Samba/AD domain, this could result in logon errors. E.g. I had issues when implating DES-only hashes from a Samba/AD domain with function level 2003 into a 2008-R2 level Samba/AD domain account. In this case I was able to authenticate but kerberized logon e.g. to a Samba/AD share failed with NT_STATUS_LOGON_FAILURE. But that's a condition we could communicate or even check in the AD-Connector.
Comment 3 Christian Völker univentionstaff 2019-12-04 17:02:19 CET
Increasing as customer can not update their passwords properly. Have not found a working workaround.

Getting urgent
Comment 4 Julia Bremer univentionstaff 2020-03-05 09:21:49 CET
Successful build
Package: univention-ad-connector
Version: 13.0.0-30A~4.4.0.202003041103
Branch: ucs_4.4-0
Scope: errata4.4-3
User: jbremer

Successful build
Package: openldap
Version: 2.4.45+dfsg-1~bpo9+1A~4.4.0.202003031624
Branch: ucs_4.4-0
Scope: errata4.4-3
User: jbremer

The Ad-Connector has been adjusted to sync the Kerberos Hashes from supplementalCredentials to the ldap attribute krb5Key.

The k5pwd openldap overlay module has been adjusted to use the correct salt, found in the keys itself instead of the default salt.
These adjustments are in svn in patch:  
14_univention-k5pwd-use-correct-salt.quilt

Test 55_adconnector/050sync_password_sync has been adapted to check kinit with every etype after changing the password in AD.

The UCR Variable con.*/ad/mapping/user/password/kerberos/enabled has been added to enable/disable the sync of kerberos hashes. 
If unset, the new kerberos hashes won't be synced, which is the default right now.

A WIP of sdb article can be found here: 
https://help.univention.com/t/activating-the-synchronization-of-kerberos-hashes-with-ad-connector/14454
Comment 5 Erik Damrose univentionstaff 2020-03-05 16:23:22 CET
e21d950f97 Bug #50429: yaml
286de9e7f1 Bug #50429: Add ucr variable to disable kerberos hash sync
d3594dfdab Bug #50429: version bump
cfa65eb5c2 Bug #50429: yaml
7339eac89e Bug #50491: Get krb5 hashes from ad
d84d4a862c Bug #50492: openldap yaml
1659845ca2 Bug #50492: Test Kerberos hash sync with kinit
28011cd5fd Bug #50492: version bump
cbf3f17dd9 Bug #50492: Merge branch 'jbremer/bug50492' into 4.4-3
4894950664 Bug #50492: Version bump
8830bcc099 Bug #50492: yaml

Commits were done with different bugnumbers, i hope i found everything.
Comment 6 Erik Damrose univentionstaff 2020-03-06 15:00:39 CET
We discussed some implementation details yesterday

More notes:
The ucs-test 55_adconnector/050sync_password_sync now always sets the kerberos pw sync to yes, the behavior without syncing kerberos hashses is not tested anymore. We should have a separate section in the test for that.
Also, the AD connector is not restarted at the end after resetting the UCRv.


In the tests we also found that the s4 connector can not sync the KRB hashes synced from an AD into S4, because we have to sync exactly 2 respective 4 keys. Traceback:

in password_sync_ucs_to_s4
06.03.2020 14:59:00.112 LDAP        (WARNING): sync failed, saved as rejected
        /var/lib/univention-connector/s4/1583501026.171020
06.03.2020 14:59:00.113 LDAP        (WARNING): Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/univention/s4connector/__init__.py", line 891, in __sync_file_from_ucs
    if ((old_dn and not self.sync_from_ucs(key, mapped_object, pre_mapped_ucs_dn, unicode(old_dn, 'utf8'), old, new)) or (not old_dn and not self.sync_from_ucs(key, mapped_object, pre_mapped_ucs_dn, old_dn, old, new))):
  File "/usr/lib/python2.7/dist-packages/univention/s4connector/s4/__init__.py", line 2610, in sync_from_ucs
    f(self, property_type, object)
  File "/usr/lib/python2.7/dist-packages/univention/s4connector/s4/password.py", line 639, in password_sync_ucs_to_s4
    s4connector.lo_s4.lo.modify_ext_s(compatible_modstring(object['dn']), modlist, serverctrls=[ctrl_bypass_password_hash])
  File "/usr/lib/python2.7/dist-packages/ldap/ldapobject.py", line 374, in modify_ext_s
    resp_type, resp_data, resp_msgid, resp_ctrls = self.result3(msgid,all=1,timeout=self.timeout)
  File "/usr/lib/python2.7/dist-packages/ldap/ldapobject.py", line 514, in result3
    resp_ctrl_classes=resp_ctrl_classes
  File "/usr/lib/python2.7/dist-packages/ldap/ldapobject.py", line 521, in result4
    ldap_result = self._ldap_call(self._l.result4,msgid,all,timeout,add_ctrls,add_intermediates,add_extop)
  File "/usr/lib/python2.7/dist-packages/ldap/ldapobject.py", line 106, in _ldap_call
    result = func(*args,**kwargs)
CONSTRAINT_VIOLATION: {'info': '0000202F: PrimaryKerberos num_keys != 2 at ../../source4/dsdb/samdb/ldb_modules/password_hash.c:414', 'desc': 'Constraint violation'}
Comment 7 Julia Bremer univentionstaff 2020-03-06 17:53:27 CET
Created attachment 10304 [details]
adapted patch 92_allow_missing_des-cbc to allow every combination of keys


For some reason a local security policy called
Network security: Configure encryption types allowed for Kerberos
was set on my machine, to allow all enctypes, which is why it "worked on my machine"(tm). 

By using this policy, we can get a variety of different amounts of hashes, 
while Samba wants exactly 4 hashes of the kind:
aes256-cts-hmac-sha1-96, aes128-cts-hmac-sha1-96, des-cbc-crc, des-cbc-md5.

We can "fill" the key lists with dummy hashes (hashes without any key value)
in the S4-Connector, to make Samba happy, 
OR
we could loosen the Restrictions in Samba, which seem outdated, and do not seem to have any purpose but to make it harder to write "bad" data into the password attributes. 
In my first tests with this patch applied, it did not make a difference if these constraints are there or not.
Comment 8 Julia Bremer univentionstaff 2020-03-09 10:14:20 CET
454f7ad8bf Bug #50492: yaml
83695bd9f4 Bug #50492: yaml
53db07bb4d Bug #50492: version bump
b34deeaed5 Bug #50492: version bump
ef4a9325e5 Bug #50492: Changelog
5f58121f85 Bug #50492: Adjust testcase for password sync
78de0ee599 Bug #50492: Pass "dummy" hashes to samba if etypes are not supported


Successful build
Package: univention-ad-connector
Version: 13.0.0-32A~4.4.0.202003081641
Branch: ucs_4.4-0
Scope: errata4.4-3
User: jbremer


Successful build
Package: univention-s4-connector
Version: 13.0.2-65A~4.4.0.202003081635
Branch: ucs_4.4-0
Scope: errata4.4-3
User: jbremer


In default configuration, Windows Server 2008r2 and up do not send any des-cbc-crc by default.
Samba needs an exact number of keys though, which is why a "dummy" hash is now created in the S4-Connector if the DES-CBC-CRC is missing.
Customers can set the allowed encryption types for kerberos, which mean they can disable the other des-key too. 
I tried passing a dummy hash in this case too, but if the whole Primary:Kerberos package (des-cbc-crc and des-cb-md5) is replaced with dummy-hashes, it raises a "Subcontext" (?) Error on ndr_pack . 
I documented in the sdb article, that at least the enctypes
DES_CBC_MD5, RC4_HMAC_MD5, AES128_HMAC_SHA1and AES256_HMAC_SHA1 need to be enabled on the AD DC for the kerberos hash sync.
https://help.univention.com/t/activating-the-synchronization-of-kerberos-hashes-with-ad-connector/14454

I adjusted the test case, so that the password sync without the hash sync enabled is still tested.
The new Jenkins Job, which tests s4connector AND adconnector failed once at setup, and the second time seemed to hang after 15+ hours.
Comment 9 Julia Bremer univentionstaff 2020-03-09 10:25:43 CET
For QA:
Setup a UCS Master with Active Directory compatible Domain Controller and Ad Connection installed, 
Create (bidirectional) Ad-Connection in syncmode
 
After installing the new packages :
ucr set 'connector/ad/mapping/user/password/kerberos/enabled=yes'
/etc/init.d/univention-ad-connector restart

create a user in UCS, reset the password in RSAT "Users and Computers" by right-clicking on the user object.
If the sync was succesfull, there are 3 krb5key entries on the user-object.
Test kinit with the encryption types.
kinit -e <enctype> <username> 
It needs to work at least with (des-cbc-crc is disabled by default):
aes256-cts-hmac-sha1-96 aes128-cts-hmac-sha1-96 arcfour-hmac-md5 des-cbc-md5

Check all the boxes in GPO_name**\Computer Configuration\Windows Settings\Security Settings\Local Policies\Security Options\“Network security: Configure encryption types allowed for Kerberos”

reset password again and check if authentication also works with des-cbc-crc.

Test if ldapsearch with simplebind works with the new password, since openldap was not able to handle foreign hashes with foreign salts before.
Comment 10 Arvid Requate univentionstaff 2020-03-09 17:06:27 CET
FYI: https://lists.samba.org/archive/cifs-protocol/2008-September/000469.html and https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-samr/fa61e5fc-f8fb-4d5b-9695-c724af0c3829  explains the origin of the keytype -140 dummy hash and states that the key value in that case is the NT-Hash. I'm unsure if that matters though. If Samba is fine with a "None" value then I don't object.
Comment 11 Arvid Requate univentionstaff 2020-03-09 19:25:35 CET
Created attachment 10306 [details]
Alternative 14_univention-k5pwd-use-correct-salt.quilt

QA for the adjusted OpenLDAP patch: Ok, works. Maybe we could have kept the old krb5_get_pw_salt default Salt strategy in case the ekey.salt is empty. This is what we had in very early stages of UCS: krb5Keys that where salted with the defult salt but the didn't explicitely contain this salt. Heimdal falls back to the default salt in that case.

Long story short: The slapd works and doesn't crash even if the krb5Key doesn't explicitly mention the salt. In that case the simplebind just simply fails -- and the S4-Connector rejects those theoretical types of krb5Key anyway with a traceback. No regression.

The attached quilt patch would be an alternative, just in case.
Comment 12 Arvid Requate univentionstaff 2020-03-09 19:30:19 CET
QA:
* Code review: Ok
* Functional test: Ok
* SDB-article: Ok, I've done some wording changes in cooperation with Julia. Basically it's better to always explicitly write "Microsoft AD", that makes it easier to understand which side we are talking about.
Comment 13 Julia Bremer univentionstaff 2020-03-12 22:33:50 CET
Found some tracebacks in the ucs-test.log,  which seem to happen because of this change in the s4connector

14 times:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/univention/s4connector/__init__.py", line 891, in __sync_file_from_ucs
    if ((old_dn and not self.sync_from_ucs(key, mapped_object, pre_mapped_ucs_dn, unicode(old_dn, 'utf8'), old, new)) or (not old_dn and not self.sync_from_ucs(key, mapped_object, pre_mapped_ucs_dn, old_dn, old, new))):
  File "/usr/lib/python2.7/dist-packages/univention/s4connector/s4/__init__.py", line 2610, in sync_from_ucs
    f(self, property_type, object)
  File "/usr/lib/python2.7/dist-packages/univention/s4connector/s4/password.py", line 609, in password_sync_ucs_to_s4
    supplementalCredentials_new = calculate_supplementalCredentials(krb5Key, supplementalCredentials)
  File "/usr/lib/python2.7/dist-packages/univention/s4connector/s4/password.py", line 399, in calculate_supplementalCredentials
    krb3_blob = ndr_pack(krb)
  File "/usr/lib/python2.7/dist-packages/samba/ndr.py", line 33, in ndr_pack
    return ndr_pack()
 RuntimeError: (7, 'Subcontext Error')

=====================================================================
The change that the s4-connector now always creates the Kerberos:Primary package seems to be the issue here, which I reverted.
I also set the nt_hash as the value of the dummy key to match the samba documentation.

Package: univention-s4-connector
Version: 13.0.2-66A~4.4.0.202003122223
Branch: ucs_4.4-0
Scope: errata4.4-3
= /var/univention/buildsystem2/logs/ucs_4.4-0-0-errata4.4-3/univention-s4-connector_13.0.2-66A~4.4.0.202003122223_all.deb'.


9fda6b32a0 Bug #50492: Write nt-hash into new dummy hash
Comment 14 Arvid Requate univentionstaff 2020-03-13 17:21:34 CET
Ok, confirmed.