Bug 47025 - AD Connector crash after MemoryError exception
AD Connector crash after MemoryError exception
Status: CLOSED FIXED
Product: UCS
Classification: Unclassified
Component: AD Connector
UCS 4.2
All Linux
: P4 normal (vote)
: UCS 4.3-1-errata
Assigned To: Felix Botner
Arvid Requate
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2018-05-17 13:45 CEST by Arvid Requate
Modified: 2018-08-15 13:14 CEST (History)
2 users (show)

See Also:
What kind of report is it?: Bug Report
What type of bug is this?: 7: Crash: Bug causes crash or data loss
Who will be affected by this bug?: 1: Will affect a very few installed domains
How will those affected feel about the bug?: 5: Blocking further progress on the daily work
User Pain: 0.200
Enterprise Customer affected?: Yes
School Customer affected?:
ISV affected?:
Waiting Support:
Flags outvoted (downgraded) after PO Review:
Ticket number: 2018051721000548
Bug group (optional):
Max CVSS v3 score:
requate: Patch_Available+


Attachments
find_binary_samba_ad_schema_attributes.sh (1.31 KB, application/x-shellscript)
2018-08-01 11:49 CEST, Arvid Requate
Details
find_binary_samba_ad_schema_attributes.log (3.26 KB, text/x-log)
2018-08-01 11:51 CEST, Arvid Requate
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Arvid Requate univentionstaff 2018-05-17 13:45:12 CEST
I just worked on a case where the AD Connector process repeatedly crashed. After some debugging we saw that a MemoryError exception was raised, apparently in

 ad.poll() -> __search_ad() -> encode_ad_resultlist() ->encode_ad_object()


Before that we saw a couple (>100) of connector.log messages like

17.05.2018 12:00:22  LDAP        ( WARN    ) : encode_ad_object: encode attrib msPKIAccountCredentials failed, ignored!


We were able to fix the AD-Connector crashes by extending the list of non-utf8 attributes. This is the line we finally used:


elif key in ['objectGUID', 'ipsecData', 'repsFrom',
'replUpToDateVector', 'userCertificate', 'dNSProperty', 'dnsRecord',
'securityIdentifier', 'mS-DS-CreatorSID', 'logonHours', 'mSMQSites',
'mSMQSignKey', 'currentLocation', 'dSASignature', 'linkTrackSecret',
'mSMQDigests', 'mSMQEncryptKey', 'mSMQSignCertificates', 'may',
'sIDHistory', 'msExchMailboxSecurityDescriptor', 'msExchMailboxGuid',
'msExchMasterAccountSid', 'replicationSignature', 'repsTo', 'msRTCSIP-
UserRoutingGroupId', 'msPKIRoamingTimeStamp', 'msDFS-GenerationGUIDv2', 
'msDFS-LinkSecurityDescriptorv2', 'msDFS-LinkIdentityGUIDv2', 'msDFS-
NamespaceIdentityGUIDv2', 'msDFS-TargetListv2',
'msPKIAccountCredentials', 'msPKIDPAPIMasterKeys']:

in univention-ad-connector/modules/univention/connector/ad/__init__.py.

From a quick glance at Bug #9674 I think pKTGuid should be added too.
Comment 1 Tobias Birkefeld univentionstaff 2018-05-30 18:12:40 CEST
I saw some more encode errors in a customer environment (UCS 4.3-0):

30.05.2018 17:14:20,973 LDAP        (WARNING): encode_ad_object: encode attrib msExchBlockedSendersHash failed, ignored!
30.05.2018 17:14:20,991 LDAP        (WARNING): encode_ad_object: encode attrib msExchSafeSendersHash failed, ignored!
Comment 2 Tobias Birkefeld univentionstaff 2018-06-05 12:45:15 CEST
Two more:

05.06.2018 11:47:01,415 LDAP        (WARNING): encode_ad_object: encode attrib msExchSafeRecipientsHash failed, ignored!
05.06.2018 11:47:01,416 LDAP        (WARNING): encode_ad_object: encode attrib msExchDisabledArchiveGUID failed, ignored!
Comment 3 Felix Botner univentionstaff 2018-07-18 14:38:10 CEST
Die Liste der Binärattribute muss einfach auf einen aktuellen Stand erweitert werden. Nice to have wäre, wenn sie per UCR erweiterbar wäre,
Comment 4 Felix Botner univentionstaff 2018-08-01 10:44:56 CEST
please set the bug to resolved if you think you are done

remove the tab after the +ATTRIBUTE_LIST line

make ATTRIBUTE_LIST configurable with ucr
Comment 5 Felix Botner univentionstaff 2018-08-01 11:00:24 CEST
always create/update univention-ad-connector.yaml (source package name.yaml) after building a package, so that we do not accidentally release a untested package
Comment 6 Arvid Requate univentionstaff 2018-08-01 11:49:42 CEST
Created attachment 9615 [details]
find_binary_samba_ad_schema_attributes.sh

With the attaches script I've scanned the Samba/AD schema (Samba 4.7.5) and looked up the attributeSyntax of the attributes listed above. Then I've searched for all attributes that also have one of those attributeSyntax.

I found this list of AD attribute syntaxes but I can't quite make sense of that:
https://msdn.microsoft.com/en-us/library/cc223177.aspx

I'll attach the output of my script.
Comment 7 Arvid Requate univentionstaff 2018-08-01 11:51:40 CEST
Created attachment 9616 [details]
find_binary_samba_ad_schema_attributes.log
Comment 8 Felix Botner univentionstaff 2018-08-02 15:06:20 CEST
ok, compared your list against a w2k12 binary attribute list

-> ldbsearch --paged -H ldap://WIN-M1LHUHEJFSI.w2k12.test -U Administrator%Univention.99 --cross-ncs '(|(attributeSyntax=2.5.5.15)(attributeSyntax=2.5.5.10)(attributeSyntax=2.5.5.17)(attributeSyntax=2.5.5.7))' lDAPDisplayName | sed -ne 's|lDAPDisplayName: ||p' | sort

found these additional attributes in w2k12

+msAuthz-CentralAccessPolicyID
+msDNS-DNSKEYRecords
+msDNS-SigningKeyDescriptors
+msDNS-SigningKeys
+msDS-AllowedToActOnBehalfOfOtherIdentity
+msDS-GenerationId
+msDS-GroupMSAMembership
+msDS-ManagedPassword
+msDS-ManagedPasswordId
+msDS-ManagedPasswordPreviousId
+msDS-TransformationRulesCompiled
+msImaging-ThumbprintHash
+msKds-KDFParam
+msKds-RootKeyData
+msKds-SecretAgreementParam
+msSPP-ConfigLicense
+msSPP-CSVLKSkuId
+msSPP-IssuanceLicense
+msSPP-KMSIds
+msSPP-OnlineLicense
+msSPP-PhoneLicense
+msTPM-SrkPubThumbprint
+netbootDUID

so your list and these attributes is the new connecot binary attributes listr
Comment 9 Arvid Requate univentionstaff 2018-08-06 16:50:30 CEST
Looks like resolved-fixed.
Comment 10 Arvid Requate univentionstaff 2018-08-07 20:04:32 CEST
List complete, code review ok, advisory too.

The new UCR variable (family) "con.*/ad/binary_attributes" allows extending the list of binary attributes.
Comment 11 Arvid Requate univentionstaff 2018-08-15 13:14:30 CEST
<http://errata.software-univention.de/ucs/4.3/168.html>