Bug 51647 - No synchronisation of mail/mailPrimaryAddress to UCS LDAP from AD anymore
No synchronisation of mail/mailPrimaryAddress to UCS LDAP from AD anymore
Status: CLOSED FIXED
Product: UCS
Classification: Unclassified
Component: AD Connector
UCS 4.4
Other Linux
: P5 normal (vote)
: UCS 4.4-5-errata
Assigned To: Arvid Requate
Julia Bremer
:
Depends on: 37289
Blocks: 52208
  Show dependency treegraph
 
Reported: 2020-07-08 12:09 CEST by stefan.bauer
Modified: 2020-10-12 13:38 CEST (History)
8 users (show)

See Also:
What kind of report is it?: Bug Report
What type of bug is this?: 5: Major Usability: Impairs usability in key scenarios
Who will be affected by this bug?: 3: Will affect average number of installed domains
How will those affected feel about the bug?: 3: A User would likely not purchase the product
User Pain: 0.257
Enterprise Customer affected?: Yes
School Customer affected?:
ISV affected?:
Waiting Support: Yes
Flags outvoted (downgraded) after PO Review:
Ticket number: 2020100221000338
Bug group (optional): Regression
Max CVSS v3 score:
requate: Patch_Available+


Attachments
connector.log (16.98 KB, text/x-log)
2020-07-08 14:42 CEST, stefan.bauer
Details
mailPrimaryAddress_depends_on_mail.patch (3.06 KB, patch)
2020-07-13 16:13 CEST, Arvid Requate
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description stefan.bauer 2020-07-08 12:09:12 CEST
Latest UCS 4.4-4 does not sync mail-Attribute from AD to local UCS LDAP anymore.

However a change in AD to the mail-attribute is noticed.

UCS is member server in server 2016 windows AD. AD-sync works but local UCS ldap does not show attribute mailPrimaryAddress even though connector logs a change. 

I set mustermann@customerdomain.de in field E-Mail in AD. Domain is set in UCS:
mail/hosteddomains: customerdomain.de customer.local

If i change users mail-addy in AD, connector.log even shows:

    LDAP (PROCESS): sync to ucs: [ user] [ modify] uid=mustermann,cn=users,dc=CUSTOMER,dc=local

So it’s noticed, but not written to LDAP:

univention-ldapsearch uid=mustermann

does not show any attribute with mail (not mail, not mailPrimaryAddress).

Reported to forum: https://help.univention.com/t/mailprimaryaddress-not-synced-to-local-ldap-from-activedirectory/15560
Comment 1 Ingo Steuwer univentionstaff 2020-07-08 13:22:23 CEST
Has our support team been able to analyze this yet? Are logfiles available?
Comment 2 stefan.bauer 2020-07-08 13:27:00 CEST
@Ingo - we did not had any response from your side yet.

I'm happy to provide log files on request.
Comment 3 Ingo Steuwer univentionstaff 2020-07-08 13:50:24 CEST
(In reply to stefan.bauer from comment #2)
> @Ingo - we did not had any response from your side yet.

Did you contact our support team?
https://www.univention.com/products/support/enterprise-support/

> I'm happy to provide log files on request.

Can you attach a logfile excerpt with connector debug level 4?

Feel free to anonymize by replacing usernames and domain names with other fixed strings.
Comment 5 stefan.bauer 2020-07-08 14:42:56 CEST
attached. Had also to encode base64 content to obfuscate.
Comment 6 Arvid Requate univentionstaff 2020-07-08 16:59:10 CEST
The output of

ucr search --brief connector/ad/mapping/user/primarymail connector/ad/mapping/user/alternativemail

would be interesting.
Comment 7 Arvid Requate univentionstaff 2020-07-08 17:20:58 CEST
From digging though the history: Bug 40357 made bigger changes, as described in this commit message:

https://git.knut.univention.de/univention/ucs/-/commit/06f6e81eb8ac8bdace8cb080c7ee97b5f22f0bd5

From that I thing "mail" is only synchronized from UCS to AD, but not the other way.
A change of the "SMTP:" value in the AD attribute "proxyAddresses" should get synchronized to mailPrimaryAddress if connector/ad/mapping/user/primarymail active, which e.g. happens when a UCS system is configured as AD-Member.
Comment 8 Florian Best univentionstaff 2020-07-08 17:23:50 CEST
(In reply to Arvid Requate from comment #7)
> https://git.knut.univention.de/univention/ucs/-/commit/
> 06f6e81eb8ac8bdace8cb080c7ee97b5f22f0bd5
https://github.com/univention/univention-corporate-server/commit/06f6e81eb8ac8bdace8cb080c7ee97b5f22f0bd5
Comment 9 stefan.bauer 2020-07-08 17:58:29 CEST
root@mailserver:/home/admin# ucr search --brief connector/ad/mapping/user/primarymail connector/ad/mapping/user/alternativemail
con.*/ad/mapping/user/alternativemail: <empty>
con.*/ad/mapping/user/primarymail: <empty>
connector/ad/mapping/user/primarymail: true

We did not ever touch any of the above settings.

It was working in the past. Just simply can not tell which UCS update broke it.
Comment 10 Arvid Requate univentionstaff 2020-07-08 18:06:28 CEST
After

ucr set connector/debug/level=4; /etc/init.d/univention-ad-connector restart

and triggering a change of proxyAddresses and mail in AD lines with "ALL"
are expected in the connector.log, which I don't see in the file attached to
comment 4. In my test I see e.g.: 
========================================================
08.07.2020 17:58:49.808 LDAP        (INFO   ): object_from_element: olddn: CN=user6,CN=Users,DC=autotestwin,DC=local
08.07.2020 17:58:49.808 LDAP        (INFO   ): _ignore_object: Do not ignore CN=user6,CN=Users,DC=autotestwin,DC=local (key: user)
[...]
08.07.2020 17:58:49.812 LDAP        (INFO   ): sync_to_ucs: old_ad_object: {...}
08.07.2020 17:58:49.812 LDAP        (INFO   ): sync_to_ucs: new_ad_object: {...}
[...]
08.07.2020 17:58:49.815 LDAP        (PROCESS): sync to ucs:   [          user] [    modify] uid=user6,cn=users,dc=autotest,dc=local
[...]
08.07.2020 17:58:49.818 LDAP        (INFO   ): __set_values: Set: proxyAddresses
08.07.2020 17:58:49.818 LDAP        (INFO   ): __set_values: set attribute, ucs_key: mailAlternativeAddress - value: []
08.07.2020 17:58:49.818 LDAP        (ALL    ): proxyAddesses: values1: []
08.07.2020 17:58:49.819 LDAP        (ALL    ): proxyAddesses: values2: []
[...]
08.07.2020 17:58:49.820 LDAP        (INFO   ): __set_values: Set: proxyAddresses
08.07.2020 17:58:49.820 LDAP        (INFO   ): __set_values: set attribute, ucs_key: mailPrimaryAddress - value: [u'user6b@autotest.local']
08.07.2020 17:58:49.820 LDAP        (ALL    ): proxyAddesses: values1: user6@autotest.local
08.07.2020 17:58:49.821 LDAP        (ALL    ): proxyAddesses: values2: user6b@autotest.local
08.07.2020 17:58:49.821 LDAP        (INFO   ): set key in ucs-object: mailPrimaryAddress
========================================================
Comment 11 stefan.bauer 2020-07-08 20:43:19 CEST
Just checked, also the sync from 2012 servers are broken now. We have several installations with all now on latest UCS 4.4.4.

we only used sync from AD to UCS in the past. Setup users in AD and had it in UCS for kopano.

the active directory LDAP attribute is and ever has been for primary mail address -> 'mail'.

There is no exchange or whatsover, that uses 'proxyAddresses' attribut.

See: https://docs.microsoft.com/de-de/windows/win32/adschema/a-proxyaddresses?redirectedfrom=MSDN
Comment 12 stefan.bauer 2020-07-09 09:14:05 CEST
Just to make my last comment more clear. Exchange uses proxyAddresses as attribute for one or more mail-addresses.

'mail' is the default attribute and also what AD distributes, when filling in an address in the users E-Mail field.
Comment 13 Arvid Requate univentionstaff 2020-07-09 10:28:18 CEST
* In which UCS version did you last see the behavior which you expect,
  i.e. the synchronization of AD "mail" to UCS/OpenLDAP "proxyMailAddress"?

* Do you remember the action that may have caused the change in your environment?
  E.g. did you update UCS recently? If so from which version?

I understand your point regarding AD "mail" vs "proxyAddresses", and that was the behavior of the AD-Connector mapping before Bug 40357 got fixed in 4.1-1-errata in response to a customer request, to support multiple addresses in the context of MS-exchange. In cases where customers adjusted their mapping manually, package updates will not have replaced the adjusted mapping so that the previous behavior remained, otherwise it changed with http://errata.software-univention.de/ucs/4.1/126.html .
Comment 14 Arvid Requate univentionstaff 2020-07-09 10:35:56 CEST
Update: More code digging reveals another change with Bug #43216:
http://errata.software-univention.de/ucs/4.1/386.html

That's the one where the mapping actually changed from "mail" to "proxyAddresses", so that's 4.1-4-errata.
Comment 15 stefan.bauer 2020-07-09 10:40:29 CEST
I'm almost certain, that it was working even in 4.4.4. We added in March users to AD and mail was synchec correctly to mailPrimaryAddress. We have here around 40 customer installations, all on 4.4.4 with broken sync now. Windows servers are 2012,2016,2019.

mail from AD was in the past synced to mailPrimaryAddress in UCS, - not/never to proxyMailAddress as the E-Mail-field in AD is only for a single mail address.

The entries are still present for all of our users in UCS. 
New users do not get mailPrimaryAddress in UCS.
Updates to existing users mail-address will not be synced.
Comment 16 stefan.bauer 2020-07-10 13:05:59 CEST
Can i provide further infos to help solve this issue?
Comment 17 stefan.bauer 2020-07-13 15:29:44 CEST
Please notice that currently - without an Microsoft Exchange Server -  it's not even possible to set ANY mail address in active directory, as the proxyAddress field does not exist, and the mail-attribute will not be synced :(
Comment 18 Arvid Requate univentionstaff 2020-07-13 16:12:33 CEST
Ok, we now have an idea what's going on.

Since Bug #43216 the AD attribute "mail" was not supposed
to get synchronized from AD to UDM/OpenLDAP, but due to
some 'interesting' code flow, it in fact got synchronized
and thus has become a feature users expect and depend on.

Now since fixing Bug 18501, this "Bug/Feature" is gone,
causing a regression, especially for AD-Membermode.

I'll attach a patch proposal.
Comment 19 Arvid Requate univentionstaff 2020-07-13 16:13:23 CEST
Created attachment 10427 [details]
mailPrimaryAddress_depends_on_mail.patch
Comment 20 stefan.bauer 2020-07-14 08:01:06 CEST
Wow. Much appreciated! It works. Thank you very much. just tested your patch.

14.07.2020 07:57:45.816 LDAP        (INFO   ): __set_values: Set: proxyAddresses
14.07.2020 07:57:45.817 LDAP        (INFO   ): __set_values: set attribute, ucs_key: mailPrimaryAddress - value: [u'maxi@domain.de']
14.07.2020 07:57:45.817 LDAP        (ALL    ): proxyAddesses: values1: 
14.07.2020 07:57:45.817 LDAP        (ALL    ): proxyAddesses: values2: maxi@domain.de
14.07.2020 07:57:45.817 LDAP        (INFO   ): set key in ucs-object: mailPrimaryAddress



root@mailserver:/usr/lib/python2.7/dist-packages/univention/connector# univention-ldapsearch uid=mustermann | grep mail
univentionFetchmailProtocol: IMAP
objectClass: univentionFetchmail
mailPrimaryAddress: maxi@domain.de


I can also confirm ,that it does not touch/break existing users/ldap attributes.
Comment 21 Arvid Requate univentionstaff 2020-07-14 10:47:42 CEST
Thanks for the feedback!

We also need to extend our tests 55_adconnector/*_user_mail_attributes.
Comment 22 Arvid Requate univentionstaff 2020-07-15 21:32:13 CEST
cc3470cf26 | patch
047eed9bcc | advisory

The test case is still elusive.
Comment 23 Arvid Requate univentionstaff 2020-07-16 18:39:17 CEST
8101514d6a | Support ad/member=true in ucs-test-adconnector
8ebb41d1f7 | Test synchronization of AD mail attributes in read/AD-Member mode
e15a4fb138 | Test should also run in read/AD-Member mode
083ef0c55a | Extend and cleanup mail attribute tests
e5cbd7a944 | Rename mail tests that deal with ucs user objects
294b1fb0a0 | Test AD user mail attribute changes in ad/member mode

Package: ucs-test                                                                                                                                                              
Version: 9.0.4-6A~4.4.0.202007161832                                                                                                                                           
Branch: ucs_4.4-0                                                                                                                                                              
Scope: errata4.4-5

And another commit, that didn't make it into that build yet:

a51e183fd4 | Rename 230read_user_mail_attributes
             to 230read_ucs_user_change_mail_attributes
Comment 24 Julia Bremer univentionstaff 2020-07-20 12:36:03 CEST
AD Membermode: OK
sync of mail -> primaryMailAddress: OK
Tests local and in Jenkins: OK
YAML: OK

Verified
Comment 26 Yuri 2020-09-23 22:52:43 CEST
Hello!

Any instructions how to use this patch?

In new UCS releases the problem will stay?
Comment 27 Ingo Steuwer univentionstaff 2020-09-25 12:15:22 CEST
(In reply to Yuri from comment #26)
> Hello!
> 
> Any instructions how to use this patch?
> 
> In new UCS releases the problem will stay?

The patch is already part of the latest UCS Releases, so you only need to update. If you have any problems I suggest to discuss in help.univention.com
Comment 28 Yuri 2020-09-25 12:46:48 CEST
Hello!

The currently installed release version is 4.4-6 errata750.

I using MS Windows server 2016 for AD

The problem still in place.
Comment 29 Dirk Schnick univentionstaff 2020-10-02 16:14:41 CEST
Customer reports same problem with
root@ucs:# univention-app info
UCS: 4.4-6 errata758
Installed: adconnector=12.0
Upgradable:

and 

connector/ad/mapping/user/primarymail: true
connector/ad/mapping/user/alternativemail: true

Windows Server2019 AD und Mode 2016

See attached Ticket. I set waiting for support as the customer needs it and there is no workaround.
Comment 30 Florian Best univentionstaff 2020-10-02 16:15:39 CEST
(In reply to Dirk Schnick from comment #29)
Please do not reopen CLOSED bugs! Create a new one.