Bug 45205 - Postfix does not consider all configured mail addresses
Postfix does not consider all configured mail addresses
Status: CLOSED FIXED
Product: UCS
Classification: Unclassified
Component: Mail
UCS 4.3
Other Linux
: P5 normal (vote)
: UCS 4.4-7
Assigned To: Tobias Wenzel
Daniel Tröder
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2017-08-15 11:12 CEST by Sönke Schwardt-Krummrich
Modified: 2020-12-07 11:10 CET (History)
8 users (show)

See Also:
What kind of report is it?: Bug Report
What type of bug is this?: 4: Minor Usability: Impairs usability in secondary scenarios
Who will be affected by this bug?: 3: Will affect average number of installed domains
How will those affected feel about the bug?: 3: A User would likely not purchase the product
User Pain: 0.206
Enterprise Customer affected?: Yes
School Customer affected?: Yes
ISV affected?:
Waiting Support:
Flags outvoted (downgraded) after PO Review:
Ticket number: 2017081521000164, 2019011721000642
Bug group (optional): API change
Max CVSS v3 score:


Attachments
Test sending mails to user.mailAlternativeAddress (see comment) (6.22 KB, patch)
2020-02-18 12:29 CET, Tobias Wenzel
Details | Diff
45205_explain_mailboxes_in_aliases.patch (1.52 KB, patch)
2020-10-06 12:41 CEST, Daniel Tröder
Details | Diff
45205_pytest.patch (1.25 KB, patch)
2020-10-06 12:41 CEST, Daniel Tröder
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Sönke Schwardt-Krummrich univentionstaff 2017-08-15 11:12:15 CEST
The postfix version (2.11) of UCS 4.2 (and earlier) is unable to aggregate the results of several lookup tables. Postfix does a lookup for the first entry, if a result is returned by the first lookup entry, the evaluation is stopped, otherwise the next entry is evaluated, and so on...

e.g. default for virtual_alias_maps:

virtual_alias_maps = hash:/etc/postfix/virtual,
    ldap:/etc/postfix/ldap.groups, 
    ldap:/etc/postfix/ldap.distlist, 
    ldap:/etc/postfix/ldap.virtual, 
    ldap:/etc/postfix/ldap.external_aliases, 
    ldap:/etc/postfix/ldap.sharedfolderremote, 
    ldap:/etc/postfix/ldap.sharedfolderlocal_aliases

This leads to the problem, that e.g. mailAlternativeAddresses are not evaluated (ldap.virtual) if ldap.groups already returns at least one match.

The admin is unaware of this problem, because UMC and UDM do not present a warning.

Starting with postfix 3.0 there is a special lookup table "unionmap", that merges the result of several other lookup tables. This would fix this problem in a optimal way.
→ http://www.postfix.org/announcements/postfix-3.0.0.html
→ unionmap:{map1,map2,...}
Comment 1 Sönke Schwardt-Krummrich univentionstaff 2017-08-15 12:17:13 CEST
Additional information from support:
The customer configured a user account a@example.com and gave a second user B the mailAlternativeAddress a@example.com.

Reason: OX only allows the user to send mails from mailPrimaryAddress and mailAlternativeAddresses.

Problem: a mail to a@example.com is only delivered to user B but not to user A.

Workaround: 
set mailPrimaryAddress=a@example.com AND mailAlternativeAddress=a@example.com at user A
Comment 2 Sönke Schwardt-Krummrich univentionstaff 2017-08-15 12:18:53 CEST
(In reply to Sönke Schwardt-Krummrich from comment #1)
> Additional information from support:
> The customer configured a user account a@example.com and gave a second user
> B the mailAlternativeAddress a@example.com.
> 
> Reason: OX only allows the user to send mails from mailPrimaryAddress and
> mailAlternativeAddresses.
> 
> Problem: a mail to a@example.com is only delivered to user B but not to user
> A.

This seems to have been introduced with errata bug 42249.
Comment 3 Daniel Tröder univentionstaff 2017-09-04 10:16:54 CEST
(In reply to Sönke Schwardt-Krummrich from comment #1)
> The customer configured a user account a@example.com and gave a second user
> B the mailAlternativeAddress a@example.com.
I'm pretty sure this is not (or should not be) allowed. The same name is used for two different things. Results will not be deterministic.

The correct thing(TM) to do would be to create a forwarding entry in the user account A that points to B and activate the "forward plus copy to self" option.
Comment 4 Sönke Schwardt-Krummrich univentionstaff 2017-09-04 10:42:32 CEST
(In reply to Daniel Tröder from comment #3)
> (In reply to Sönke Schwardt-Krummrich from comment #1)
> > The customer configured a user account a@example.com and gave a second user
> > B the mailAlternativeAddress a@example.com.
> I'm pretty sure this is not (or should not be) allowed. The same name is
> used for two different things. Results will not be deterministic.

I agree on "should not" but not on "is not" :-)
 
> The correct thing(TM) to do would be to create a forwarding entry in the
> user account A that points to B and activate the "forward plus copy to self"
> option.

This is only possible since a few weeks.
Comment 5 Sönke Schwardt-Krummrich univentionstaff 2018-03-02 15:57:26 CET
This problem has been triggered by a customer:
https://help.univention.com/t/mailrouting-mailalternativeaddress-mailprimaryaddress/8096
Comment 6 Christina Scheinig univentionstaff 2019-01-21 09:15:54 CET
Maybe we could transfer the following query into the product as a workaround. This query has been used as a workaround by another customer, too.

The rewritten query:
From

query_filter =
(&(objectClass=univentionMail)(|(mailAlternativeAddress=%s)(oxResourceMailAddress=%s))(!(univentionCanonicalRecipientRewriteEnabled=1)))

to

query_filter =
(&(objectClass=univentionMail)(|(mailAlternativeAddress=%s)(mailPrimaryAddress=%s)(oxResourceMailAddress=%s))(!(univentionCanonicalRecipientRewriteEnabled=1)))

Ticket#2019011721000642
Comment 7 Valentin Heidelberger univentionstaff 2019-04-11 11:21:26 CEST
A customer uses mailAlternativeAddress to enable users to answer mails in shared folders (with those mailAlternativeAddress as the folders primary mail address) with another mail address besides their own. Seemingly because of this bug, this is not possible anymore, because if a mail address of a shared folder is set as an mailAlternativeAddress, mails to that address are forwarded to the users personal mailboxes and not to the shared folder anymore.
For the customer this is a rather important feature because some (official)  institutions *require* mails to be sent from a certain address.
Comment 8 Sönke Schwardt-Krummrich univentionstaff 2019-12-17 17:16:11 CET
I think another customer is affected. When using a mailing list that sends mails to a group (with mail address), no mail is delivered to the group members, at least because groups and distlists are different lookup tables.
Comment 9 Tobias Wenzel univentionstaff 2020-02-18 12:29:16 CET
Created attachment 10296 [details]
Test sending mails to user.mailAlternativeAddress (see comment)

I tried to reproduce the scenarios and wrote a test for this:

- user_b.mailAlternativeAddress = user_a.mailPrimaryAddress -> fails
- user.mail_alternative_address = shared_folder.mailAddress -> fails 
- group.mailAddress = user_a.mail_alternative_address -> passed
- mailing_list.mailAddress = user_a.mail_alternative_address -> passed
- sending mails to a mailing_list which sends mails to a group_mail -> passed

Adding the unionmap-suggestion to /etc/postfix/main.cf  did not change anything.
The test should pass all cases.


UCS Version 4.4.3
mail_version = 3.1.12
Comment 11 Tobias Wenzel univentionstaff 2020-09-30 16:07:34 CEST
I added a fix, a test & a ucrv description with:

[twenzel/45205_postfix_mapping] 1ab6721bf6 Bug #45205: add ucrv description
[twenzel/45205_postfix_mapping] 0193d69f19 fixup! Bug #45205: add new virtual_alias_maps
[twenzel/45205_postfix_mapping] cefea630d8 Bug #45205: add new virtual_alias_maps
[twenzel/45205_postfix_mapping] 7cc7793731 Bug #45205: add ucs-test

The default_virtual_alias_maps was extended by "ldap:/etc/postfix/ldap.virtual_mailbox, ldap:/etc/postfix/ldap.sharedfolderlocal" and evaluated with a unionmap.
Luckily the test already covered most of the cases.

Since UCR-V mail/postfix/virtual/alias/maps allows to set the list, I added an UCR-V description mentioning the said changes. I will propably have to change the version. As discussed we will release this a an erratum. 

The ox-fork has a slightly different naming, something like
ldap:/etc/postfix/ldap.ox-virtual_mailbox, ldap:/etc/postfix/ldap.ox-sharedfolderlocal

This has to be fixed in a different bug (TODO clone this bug).
Comment 12 Daniel Tröder univentionstaff 2020-10-06 12:40:29 CEST
REOPEN:

47_send_mail_to_alternative_address:
  * Please change to pytest as test runner (see attachment 45205_pytest.patch).
  * Unused import "mail_delivered".
  * Python 3 incompatible use of print and iteritems.
  * In test_group_mail_equal_user_mail_alt and test_mail_list_equal_user_mail_alt 
  do not put "user_a" into the group. Otherwise it's not clear from where (mAA or group) the user got the email.
  * Missing "u" in eqals of test_user_mail_alt_eqals_shared_folder_mail_address.
  * In test_group_mail_in_mailing_list, in "udm.create_object", "members" is set two times, and not as a list.

univention-mail-postfix.univention-config-registry-variables:
  * as a union <OF> all

.../main.cf.d/30_maps:
  * Please explain to future code readers why mailbox maps are part of the alias maps, see for example attached patch (45205_explain_mailboxes_in_aliases.patch).

OK: manual test of the functionality
OK: test fails in current UCS, succeeds with code from branch

Currently running mail-tests on a dev-VM and in a Jenkins branch test.
Comment 13 Daniel Tröder univentionstaff 2020-10-06 12:41:16 CEST
Created attachment 10508 [details]
45205_explain_mailboxes_in_aliases.patch
Comment 14 Daniel Tröder univentionstaff 2020-10-06 12:41:30 CEST
Created attachment 10509 [details]
45205_pytest.patch
Comment 15 Daniel Tröder univentionstaff 2020-10-06 14:19:24 CEST
> Currently running mail-tests on a dev-VM and in a Jenkins branch test.
All mail-tests in my dev-VM succeeded, waiting for Jenkins...
Comment 16 Tobias Wenzel univentionstaff 2020-10-06 16:22:46 CEST
Thanks for the QA!

I implemented the changes with

[twenzel/45205_postfix_mapping] 8d9105c971 fixup! Bug #45205: add new virtual_alias_maps
[twenzel/45205_postfix_mapping] fa26134d1c fixup! Bug #45205: add ucrv description
[twenzel/45205_postfix_mapping] f3107f8d48 fixup! Bug #45205: add ucs-test

47_send_mail_to_alternative_address.py::test_user_b_mail_alt_equal_user_a_mail_primary PASSED
47_send_mail_to_alternative_address.py::test_group_mail_equal_user_mail_alt PASSED
47_send_mail_to_alternative_address.py::test_mail_list_equal_user_mail_alt PASSED
47_send_mail_to_alternative_address.py::test_user_mail_alt_equals_shared_folder_mail_address PASSED
47_send_mail_to_alternative_address.py::test_group_mail_in_mailing_list PASSED

Please check if the .py extension is ok. We need this for pytest. 

reopen for merge&build
Comment 17 Daniel Tröder univentionstaff 2020-10-08 15:01:23 CEST
Reopend for discussion about a failing use case:
user_a.mAA == user_b.mailForwardAddress + mailForwardCopyToSelf=0
Comment 18 Tobias Wenzel univentionstaff 2020-10-21 16:37:04 CEST
As mentioned in comment #17 the bug was reopened. Some explainations & following steps:

Consider the follwing scenario:

demo_student
mailPrimaryAddress: demo_student@autotest201.local
mailForwardAddress: myforward@autotest201.local
mailForwardAddress: demo_student@autotest201.local
mailForwardCopyToSelf: 0


demo_teacher
mailPrimaryAddress: demo_teacher@autotest201.local
mailAlternativeAddress: demo_student@autotest201.local


So far, mailForwardCopyToSelf is not an attribute in ldap. It is set to '1' if the multi-value mailForwardAddress has mailPrimaryAddress as a second entry. Thus it is not part of any ldap-query. For this we need an ldap -> udm mapping. This is easier than and easier to read than rewriting the queries with the described logic.

In the previous fix (comment #16) I had to prepend the external_aliases query to the rest of the unionmap. This can be included again, see below.
If there is time: The mailbox_maps actually should not be inside of virtual_alias_maps. If there is a way, it would be cleaner to exclude them. 



---- PoC ----

To demonstrate the feasibility, I mocked the attribute mailForwardCopyToSelf with the existing postalCode and setting it to 1 or 0 accordingly.
virtual_alias_maps now also includes ldap.external_aliases:


virtual_alias_maps = unionmap:{hash:/etc/postfix/virtual,
        ldap:/etc/postfix/ldap.groups,
        ldap:/etc/postfix/ldap.distlist,
        ldap:/etc/postfix/ldap.virtual,
        ldap:/etc/postfix/ldap.sharedfolderremote,
        ldap:/etc/postfix/ldap.sharedfolderlocal_aliases,
        ldap:/etc/postfix/ldap.virtual_mailbox,
        ldap:/etc/postfix/ldap.sharedfolderlocal,
        ldap:/etc/postfix/ldap.external_aliases}

In my PoC external_aliases and virtual_mailbox both only return addresses if postalCode!=0. I changed the queries accordingly for the virtual_mailbox

query_filter = (&(&(objectClass=univentionMail)(mailPrimaryAddress=%s)(!(univentionCanonicalRecipientRewriteEnabled=1)))(!(postalCode=0)))

and external_aliases 

query_filter = (&(&(objectClass=univentionMail)(|(mailAlternativeAddress=%s)(mailPrimaryAddress=%s))(!(univentionCanonicalRecipientRewriteEnabled=1)))(!(postalCode=0)))

I tested the queries with postmap and also with swaks: 

# after setting postcode=0
$ swaks --to myforward@autotest201.local --from someone@autotest201.local --server 127.0.0.1
$ grep -r "test Wed, 21 Oct 2020 11:06:01 +0200" /var/spool/dovecot/private/autotest201.local/
/var/spool/dovecot/private/autotest201.local/demo_teacher/Maildir/new/1603271162.M850681P6823.master201,S=1463,W=1494:Subject: test Wed, 21 Oct 2020 11:06:01 +0200
root@master201:~/40_mail# swaks --to myforward@autotest201.local --from someone@autotest201.local --server 127.0.0.1
# after setting postcode=1
$ grep -r "test Wed, 21 Oct 2020 11:06:51 +0200" /var/spool/dovecot/private/autotest201.local/
/var/spool/dovecot/private/autotest201.local/demo_teacher/Maildir/new/1603271211.M644629P6823.master201,S=1307,W=1336:Subject: test Wed, 21 Oct 2020 11:06:51 +0200
/var/spool/dovecot/private/autotest201.local/demo_student/Maildir/new/1603271211.M641748P6823.master201,S=1307,W=1336:Subject: test Wed, 21 Oct 2020 11:06:51 +0200


Additionally, I provided a test-case in 47_send_mail_to_alternative_address.py which sets the postcode. After the actual fix, this should still work after removing the line setting the postcode to '1'/'0'.

[twenzel/45205_postfix_mapping] 83ac15c10d Bug #45205: poc mailforward

the queries of course have to be changed as described above.

Please feel free to discuss my PoC and comment if I missed something :)
Comment 19 Tobias Wenzel univentionstaff 2020-11-10 14:26:49 CET
I implemented a fix in:

[twenzel/45205_implementation] d98567691d Bug #45205: migration script
[twenzel/45205_implementation] b375f5a81e fixup! fixup! Bug #45205: add mailForwardAddressCopy2Self to ldap and udm
[twenzel/45205_implementation] 32683d9fc0 fixup! Bug #45205: add mailForwardAddressCopy2Self to ldap and udm
[twenzel/45205_implementation] f08e86a6ea Bug #45205: add mailForwardAddressCopy2Self to ldap and udm

the migration fix refers to a help article which can be found here: https://help.univention.com/t/migration-of-ldap-attribute-mailforwardcopytoself/16509 (the errata level has to be adjusted here & in the script before the release).
As discussed the migration is not done automatically but has to be triggered manually by calling the script.


Other than described in comment #18 changing the virtual_mailbox was enough, external_aliases was not needed. 
I tested the code manually with postmap & swaks as well as by running the test. A branch test is currently running.
Comment 20 Daniel Tröder univentionstaff 2020-11-13 08:22:46 CET
(In reply to Tobias Wenzel from comment #19)
> the migration fix refers to a help article which can be found here:
> https://help.univention.com/t/migration-of-ldap-attribute-
> mailforwardcopytoself/16509 (the errata level has to be adjusted here & in
> the script before the release).
Please change the first paragraph to be less low-level technical.
* It should first describe _why_ something is done at a high level (the problem being solved: fix corner case situations where all possible recipients of emails were evaluated by the old Postfix configuration).
* Then it should describe _what_ is done at a high level to fix the problem (1. data is migrated to allow more precise LDAP queries, 2. all query results are merged).
* Next is missing a section with an overview what the user must do: 1. the migration is not done automatically - nothing changes until the user begins the process, 2. it must be done in the whole domain at the same time (meaning on the master and all mail servers): 2a: run a data migration script on the master, 2b: change UCRV and restart Postfix on all mail servers. If those steps are not all done on all servers, there will be mail loss!
* Then it should describe what is being done in the migration (in case of mPA in the mAA, it is removed there and new property mailForwardCopyToSelf=1 → text exists as 2nd paragraph of existing "Migration" section).
* Next the migration instructions (1st paragraph of existing "Migration" section).
* Please add to the migration instructions that it is possible to run the script as often as one likes (I assume that's true).
* I thought that it is required to set a UCRV on the mail servers… I don't see this in the instructions.
Comment 21 Daniel Tröder univentionstaff 2020-11-13 08:42:40 CET
Please remove _set_default_mail_forward_copy_to_self() from the user handler.
Saving a default value to LDAP adds no information to the system but grows the database.
Additionally it means, that a previous no-op modify() will now be an actual modify(), leading to for example 10 times more LDAP modifications on the next UCS@school import.

Haven't tested it, but I think the 2nd half of _modlist_mail_forward() can be removed. Without the old hack of having the mPA in the mailForwardAddress (and removing it in open()), it does only recreate an unchanged modlist entry.
Comment 22 Daniel Tröder univentionstaff 2020-11-13 09:43:11 CET
migrate_mailForwardCopyToSelf:
* Copyright should start on 2020.
* mixed tabs and spaces

mail.schema:
* please use a boolean attribute

47_mailForwardAddress.py:
* test_user_b_mail_alt_equal_user_a_mail_forward() should be named differently. For one userB.mAA != userA.mfa, and also it tests two things:
  1. if the setting of userA.mailForwardCopyToSelf actually works (userA receives a copy or not)
  2. if userB with userB.mAA==userA.mPA gets a copy of emails sent to userA.mPA, regardless of the value of userA.mailForwardCopyToSelf
That is a useful test case, it does however _not_ test user_a.mAA == user_b.mailForwardAddress + user_b.mailForwardCopyToSelf={0, 1}. Please _add_ such a test.
* test_user_mail_alt_equals_shared_folder_mail_address() to many indents l.148 l. 152+153

47_mailForwardAddress (without .py):
* Why was it removed?
* It is the only test that checks if the forwarding functionality actually works.

47_udm_mailForwardAddress:
* Please verify the LDAP value of mailForwardCopyToSelf in all tests.
Comment 23 Daniel Tröder univentionstaff 2020-11-13 09:51:27 CET
With the migration not being automatic, UDMs behavior must be switched by a UCRV on all UCS systems of the domain, minimum on the master and mail servers.
Comment 24 Florian Best univentionstaff 2020-11-16 13:16:54 CET
The current changes look like an API change. It provides a migration script of LDAP data, it changes the schema, etc.
Also activating UDM mapping via an UCR variable does not really sound nice.
Before merging this, we should discuss this. But maybe these things are still going to be changed nevertheless. Please tell when you thing your changes are finished.
Comment 25 Tobias Wenzel univentionstaff 2020-11-17 16:39:30 CET
Thanks for the QA!

I added quite some changes with

[twenzel/45205_implementation] 20e12fb518 fixup! Bug #45205: qa improvements
[twenzel/45205_implementation] 2686bba518 fixup! fixup! Bug #45205: add new ucr-v
[twenzel/45205_implementation] 8160db8def fixup! Bug #45205: add new ucr-v
[twenzel/45205_implementation] ffeaf16d9e fixup! fixup! Bug #45205: fix udm mailForwardAddress test
[twenzel/45205_implementation] a0bfbef40c fixup! Bug #45205: fix udm mailForwardAddress test
[twenzel/45205_implementation] 9e24c2a6db Bug #45205: qa improvements
[twenzel/45205_implementation] dcae769a80 Bug #45205: add new ucr-v
[twenzel/45205_implementation] 81c5fe1d9c Bug #45205: fix udm mailForwardAddress test
[twenzel/45205_implementation] e1cdb07f03 Bug #45205: fix branch-test cfg
[twenzel/45205_implementation] 0178b68288 Bug #45205: migration script
[twenzel/45205_implementation] 04a58c4222 Bug #45205: add mailForwardAddressCopy2Self to ldap and udm


https://help.univention.com/t/migration-of-ldap-attribute-mailforwardcopytoself/16509

Rem: The errata/release level has to be adjusted accordingly.

For old installations, the behaviour is not changed.
For new installations, this will be default.
-> has to be discussed.

I added some tests to tests before and after the ucr-vs are set to make sure both settings are ok. Most of the needed explainations can be found in the help-article.

The joinscript still has to be modified.
Comment 26 Daniel Tröder univentionstaff 2020-11-18 18:35:33 CET
The following UCRVs will be set to True in the case of a fresh installation and to False in the case of an update:

directory/manager/user/activate_ldap_attribute_mailForwardCopyToSelf
mail/postfix/activate_unionmap_in_virtual_alias_maps  mail/postfix/activate_ldap_attribute_mailForwardCopyToSelf_in_virtual_alias_maps

Both the UDM users/user handler code and the Postfix UCR templates will handle both the old and the new configuration, depending on the UCR settings.

Tests have been adapted to test the feature and all corner cases.
The help article has been improved.

[4.4-6] d3976b8d52 Bug #45205: add mailForwardAddressCopy2Self to ldap and udm
[4.4-6] ee073d4516 Bug #45205: migration script
[4.4-6] 0b4d799df6 Bug #45205: fix udm mailForwardAddress test
[4.4-6] addcb1603e Bug #45205: add new ucr-v
[4.4-6] 39b13b1794 Bug #45205: qa improvements
[4.4-6] fb6d186b9e Bug #45205: set ucrv on new installations
[4.4-6] 2199ee7d35 Bug #45205: changelogs and advisories
[4.4-6] 1e278be811 Bug #45205: advisory updates

univention-ldap (15.0.2-1)
univention-directory-manager-modules (14.0.17-1)
univention-mail-postfix (13.0.3-1)
ucs-test (9.0.5-26)
Comment 27 Daniel Tröder univentionstaff 2020-11-18 18:38:30 CET
OK: code changes
OK: manual tests
OK: automated tests in dev machine
OK: advisories
OK: as decided by PO: backwards compatible API through UDM, although not through LDAP
OK: help article and migration script

Waiting for Jenkins tests.
Comment 28 Florian Best univentionstaff 2020-11-18 18:55:56 CET
REOPEN: Please see my gitlab comments.
REOPEN: Please create the UCS 5 merge request.
Comment 29 Ole Schwiegert univentionstaff 2020-11-19 09:12:19 CET
Package: univention-directory-manager-modules
Version: 14.0.17-3A~4.4.0.202011190908
Branch: ucs_4.4-0
Scope: errata4.4-6

Use description instead of usage for the migration script.
Comment 31 Daniel Tröder univentionstaff 2020-11-19 09:56:00 CET
(In reply to Florian Best from comment #30)
> The test case 40_mail.47_mailForwardAddress_corner_cases.master091 is
> failing:
> https://jenkins.knut.univention.de:8181/job/UCS-4.4/job/UCS-4.4-6/job/
> AutotestJoin/lastCompletedBuild/SambaVersion=s4,Systemrolle=master-part-II/
> testReport/40_mail/47_mailForwardAddress_corner_cases/master091/
I have just retested it in a branch test with a sleep(60) and it suceeded.
It is a timing issue in the test.
Comment 32 Daniel Tröder univentionstaff 2020-11-20 09:48:58 CET
If a master (4.4-6) is not yet updated to the new univention-ldap package and another UCS system is joined to the domain with a higher errata level (but also 4.4-6) the new LDAP attribute will not yet be known, but the new UDM code will be active. This will lead to errors in UDM users/user and can prevent the join.
Thus it was decided to publish the new schema and code with 4.4-7.
For patch level releases it is enforced, that the master must be updated before other systems.
So all changes in 4.4-6 have been reverted and will be added to UCS once the development for 4.4-7 starts:

[4.4-6] 348ecf7f30 Revert "Bug #45205: Use description instead of usage"
[4.4-6] 1a08845b4f Revert "Bug #45205: fix typo"
[4.4-6] a14b2aadea Revert "Bug #45205: changelogs and advisories"
[4.4-6] 275978f335 Revert "Bug #45205: set ucrv on new installations"
[4.4-6] 8663c70fad Revert "Bug #45205: qa improvements"
[4.4-6] 3081c5b69b Revert "Bug #45205: add new ucr-v"
[4.4-6] 14f65d078a Revert "Bug #45205: fix udm mailForwardAddress test"
[4.4-6] c7a305423f Revert "Bug #45205: migration script"
[4.4-6] 8fbb782b40 Revert "Bug #45205: add mailForwardAddressCopy2Self to ldap and udm"
[4.4-6] 688f66d04c Bug #45205: changelogs and advisory for revert
[4.4-6] b249e771e1 Bug #45205: version bump

Packages have been rebuild:
* univention-directory-manager-modules (14.0.16-9)
* ucs-test (9.0.5-30)
Packages have *not* been rebuild:
* univention-ldap
* univention-mail-postfix
Comment 33 Daniel Tröder univentionstaff 2020-11-20 09:52:09 CET
Installation and update tests with systems freshly installed from DVD have been successful with 4.4-6 test systems.
Comment 34 Daniel Tröder univentionstaff 2020-11-20 09:55:22 CET
TODOs left:

* cherry-pick commits into 4.4-7, build, advisories
* installation and update tests with 4.4-7 DVDs
* remove flakiness from 47_mailForwardAddress_corner_cases (see comment31)
* merge code into 5.0-0
Comment 35 Florian Best univentionstaff 2020-11-20 09:57:40 CET
(In reply to Daniel Tröder from comment #32)
> Packages have *not* been rebuild:
> * univention-ldap
> * univention-mail-postfix
Why not?

Question: can we enforce running the migration script e.g. in the 4.4-7 postup.sh (and therefore get rid of the UCR variables in UDM)?
Comment 36 Daniel Tröder univentionstaff 2020-11-20 12:52:23 CET
(In reply to Florian Best from comment #35)
> (In reply to Daniel Tröder from comment #32)
> > Packages have *not* been rebuild:
> > * univention-ldap
> > * univention-mail-postfix
> Why not?
Because the code in them has been reverted and the binary packages would thus be the same as those already published.

> Question: can we enforce running the migration script e.g. in the 4.4-7
> postup.sh (and therefore get rid of the UCR variables in UDM)?
No: the migration for existing systems must be done manually. It is necessary to change UCRVs and run a data migration "atomically" on the DC master and all mail server systems. We don't think we can assume, that a customer updates all its mail and OX servers when updating the DC master. I'll ask the PO what he thinks. See help article regarding data migration.
Comment 37 Florian Best univentionstaff 2020-11-20 13:00:15 CET
(In reply to Daniel Tröder from comment #36)
> (In reply to Florian Best from comment #35)
> > (In reply to Daniel Tröder from comment #32)
> > > Packages have *not* been rebuild:
> > > * univention-ldap
> > > * univention-mail-postfix
> > Why not?
> Because the code in them has been reverted and the binary packages would
> thus be the same as those already published.
But our VM's and the Jenkins tests now run with the un-reverted packages.
Comment 38 Daniel Tröder univentionstaff 2020-11-20 14:52:17 CET
(In reply to Florian Best from comment #37)
> (In reply to Daniel Tröder from comment #36)
> > (In reply to Florian Best from comment #35)
> > > (In reply to Daniel Tröder from comment #32)
> > > > Packages have *not* been rebuild:
> > > > * univention-ldap
> > > > * univention-mail-postfix
> > > Why not?
> > Because the code in them has been reverted and the binary packages would
> > thus be the same as those already published.
> But our VM's and the Jenkins tests now run with the un-reverted packages.
You're right - I hadn't thought about that. I have built management/univention-ldap (15.0.1-5) and mail/univention-mail-postfix (13.0.2-3) in scope errata4.4-6 now.
Comment 39 Daniel Tröder univentionstaff 2020-11-26 11:55:48 CET
Code has been merged to the "4.4-7" git branch and built into the "ucs4.4-7" release (non-errata) scope, advisories have been updated:

[4.4-7] cd66564436 Bug #45205: add mailForwardAddressCopy2Self to ldap and udm
[4.4-7] 0b3f0601da Bug #45205: migration script
[4.4-7] d052ecebf3 Bug #45205: fix udm mailForwardAddress test
[4.4-7] 67ec390551 Bug #45205: add new ucr-v
[4.4-7] 45c037a647 Bug #45205: qa improvements
[4.4-7] 358fd9f723 Bug #45205: set ucrv on new installations
[4.4-7] d4ac886cc3 Bug #45205: changelogs and advisories
[4.4-7] 215f5e4477 Bug #45205: fix typo
[4.4-7] 0c1704f5f6 Bug #45205: Use description instead of usage
[4.4-7] 0e360c1c16 Bug #45205: changelogs and advisory for revert
[4.4-7] d655450910 Bug #45205: changelogs for readded email forwarding code
[4.4-7] aa96ebca8d Bug #52383: Update required target release
[4.4-7] 12c24673e1 Bug #45205: advisory updates

univention-ldap (15.0.1-6)
univention-directory-manager-modules (14.0.17-4)
univention-mail-postfix (13.0.2-4)
ucs-test (9.0.5-34)
Comment 40 Florian Best univentionstaff 2020-11-26 12:13:35 CET
FYI: I commited PEP8 fixes in: fdccc12913149a76a5ef7a49904cf05dc30c3b64
REOPEN: test/ucs-test/tests/40_mail/47_mailForwardAddress_corner_cases.py contains test_user_b_mail_alt_equal_user_a_mail_primary twice.
Comment 41 Daniel Tröder univentionstaff 2020-11-26 13:54:59 CET
Double "the" in:
--------------
... specifying the
the LDAP value ...
--------------
Comment 42 Daniel Tröder univentionstaff 2020-11-27 08:05:45 CET
In Jenkins 40_mail/47_mailForwardAddress_corner_cases failed. Probably because of a timing issue. Will check again tomorrow with more robust test code:

[4.4-7 d99ef66202] Bug #45205: make IMAP folder testing more rebust against timing issues
Comment 43 Daniel Tröder univentionstaff 2020-11-27 09:06:16 CET
A merge request for branch 5.0-0 was created: https://git.knut.univention.de/univention/ucs/-/merge_requests/40
Comment 44 Tobias Wenzel univentionstaff 2020-11-27 11:43:14 CET
Thanks for the changes!
Fixes for comment 40 & comment 41 are implemented in:

[4.4-7] b378d1ea8b Bug #45205: changelogs
[4.4-7] 4a045858db Bug #45205: rename test and remove duplicate word


Package: univention-directory-manager-modules
Version: 14.0.17-5A~4.4.0.202011271134
Branch: ucs_4.4-0
Scope: ucs4.4-7

Package: ucs-test
Version: 9.0.5-37A~4.4.0.202011271137
Branch: ucs_4.4-0
Scope: ucs4.4-7
Comment 45 Tobias Wenzel univentionstaff 2020-11-27 11:53:27 CET
Adjust ucs-version in migration script text.

Package: univention-directory-manager-modules
Version: 14.0.17-6A~4.4.0.202011271151
Branch: ucs_4.4-0
Scope: ucs4.4-7
Comment 46 Florian Best univentionstaff 2020-11-30 11:53:31 CET
Hm, I am thinking about REOPENING the bug.
The debian/changelog changes did not increase the `C` of version `A.B.C-D` but actually `D`. This makes any further changes in UCS 4.4-6 impossible. This happened e.g. to Christian for ucs-test.
To fix this, we would have to increase C twice, so that we can increase it once in 4.4-6 and be able to import packages again.
Comment 47 Tobias Wenzel univentionstaff 2020-11-30 12:37:33 CET
I increased 'C' twice.

Package: ucs-test
Version: 9.0.7-1A~4.4.0.202011301231
Branch: ucs_4.4-0
Scope: ucs4.4-7
Comment 48 Florian Best univentionstaff 2020-11-30 12:40:06 CET
(In reply to Tobias Wenzel from comment #47)
> I increased 'C' twice.
> 
> Package: ucs-test
> Version: 9.0.7-1A~4.4.0.202011301231
> Branch: ucs_4.4-0
> Scope: ucs4.4-7

OK, for ucs-test. But this would be the same for every other changed package as well.
Comment 49 Tobias Wenzel univentionstaff 2020-11-30 13:07:49 CET
Sorry, here is the rest:

Package: univention-ldap
Version: 15.0.3-1A~4.4.0.202011301300
Branch: ucs_4.4-0
Scope: ucs4.4-7

Package: univention-directory-manager-modules
Version: 14.0.19-1A~4.4.0.202011301303
Branch: ucs_4.4-0
Scope: ucs4.4-7

Package: univention-mail-postfix
Version: 13.0.4-1A~4.4.0.202011301306
Branch: ucs_4.4-0
Scope: ucs4.4-7
Comment 50 Daniel Tröder univentionstaff 2020-11-30 15:50:16 CET
OK: rename test
OK: text fixes
OK: manual tests fail in 4.4-6 and succeed in 4.4-7
OK: update behavior as described (no change if mail server was already installed, new behavior if not)
OK: migration procedure (copy & paste)-able from help article
OK: automatic tests fail in 4.4-6 and succeed in 4.4-7
OK: 4.4-7 release changelog entry
OK: no advisories for non-errata release
OK: changelog version bump
OK: package rebuilds
Comment 52 Erik Damrose univentionstaff 2020-12-07 11:10:50 CET
UCS 4.4-7 has been released:
 https://docs.software-univention.de/release-notes-4.4-7-en.html

If this error occurs again, please use the 'Clone This Bug' option.