Bug 38471 - Enable shared IMAP folder feature with Dovecot
Enable shared IMAP folder feature with Dovecot
Status: CLOSED FIXED
Product: UCS
Classification: Unclassified
Component: Mail - Dovecot
UCS 4.0
Other Linux
: P5 normal (vote)
: ---
Assigned To: Daniel Tröder
Sönke Schwardt-Krummrich
:
Depends on: 38475 38573
Blocks: 38792 34839 38472 38600 38791
  Show dependency treegraph
 
Reported: 2015-05-06 19:17 CEST by Daniel Tröder
Modified: 2015-07-09 18:13 CEST (History)
3 users (show)

See Also:
What kind of report is it?: ---
What type of bug is this?: ---
Who will be affected by this bug?: ---
How will those affected feel about the bug?: ---
User Pain:
Enterprise Customer affected?:
School Customer affected?:
ISV affected?:
Waiting Support:
Flags outvoted (downgraded) after PO Review:
Ticket number:
Bug group (optional):
Max CVSS v3 score:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Daniel Tröder univentionstaff 2015-05-06 19:17:52 CEST
Users can read and write from/to the same folder. Use namespace, not symlinks.
Comment 1 Daniel Tröder univentionstaff 2015-05-15 16:50:19 CEST
Changes in the ACL file path and Per-user \Seen flag in v2.2.
Comment 2 Daniel Tröder univentionstaff 2015-05-15 20:13:16 CEST
* A basic configuration to enable public mailboxes with ACLs is in r60748.
* User flags ("important", "work" etc) are shared, \Seen flag is
per-user. Per-user \Seen is implemented with <2.2 mechanism ("dovecot-shared") until #38475 is fixed.

To add a public mailbox run: 
# doveadm mailbox create Public/team5
(ignore error msg, check list of existing mailboxes: with "doveadm mailbox list -A")

Subscribe the public mailbox team1 for one user:
# doveadm mailbox subscribe -u test1m@uni.dtr Public/team1
Or for all users:
# doveadm mailbox subscribe -A Public/team1
Comment 3 Daniel Tröder univentionstaff 2015-05-18 16:28:54 CEST
r60763: switched to v2.2 per-user \Seen flag index storage: Creates index files at ~/Maildir/public/<shared_mailbox>/dovecot.index.pvt.log, doesn't need dovecot-shared file any more.
Comment 4 Daniel Tröder univentionstaff 2015-05-18 16:37:05 CEST
Btw: keywords ("important", color etc) are shared, but clients don't update their view in realtime.
Thunderbird does it only when subscribing the 1st time to a folder (deleting cache helps somewhat), OX updates the labels only when logging in (again).
Comment 5 Daniel Tröder univentionstaff 2015-06-01 17:39:21 CEST
Commit: 60968

We use two different kinds of shared folders:
- "public mailboxes" (http://wiki2.dovecot.org/SharedMailboxes/Public)
- "shared mailboxes" (http://wiki2.dovecot.org/SharedMailboxes/Shared)

In the case of the creation of a shared IMAP folder _without_ an email
address, a mailbox in the "public" namespace will be used. We create a
separate namespace for each public mailbox, as the quota is the same for all
mailboxes of each public namespace. The namespace name is the cn.
Those mailboxes' name will be <namespace>/<mailbox>, eg "pub1@uni.dtr/INBOX".

In the case of the creation of a shared IMAP folder _with_ an email
address, a private mailbox will be created and its INBOX will be shared.
The mailbox will live in the "private" namespace, but the shared folder
(the INBOX) will be accessed through the "shared" namespace. Being a
common private mailbox, the user quota from LDAP will be used just like
with a common email account. The only difference is, that a LDAP object of
objectClass univentionMailSharedFolder contains the mailPrimaryAddress
attribute, not a common user with objectClass univentionMail.
Those mailboxes' name will be shared/<email>, eg "shared/pub1@uni.dtr". Just
like any shared mailbox from a user.

If multiple shared folders of both type are present, then in the users
frontend it may look like this:

/
+--INBOX
|
+--pub1@uni.dtr
|    +--INBOX
|
+--pub2@uni.dtr
|    +--INBOX
|
+--shared
    +--pub3@uni.dtr
    +--pub4@uni.dtr


In both cases it would be possible to create subfolders if a user gets the
"k" (create) and "x" (delete) ACL flags. Currently those are not given to
users even when in UDM the "all" permission is chosen. For creation "k" is
sufficient, "x" is only needed for renaming, but it does allow mailbox
deletion too.


The listener module:
* adds mailboxes with the type depending on existence of mailPrimaryAddress.
* removes mailboxes of both types, deletes from disk only
  if mail/dovecot/mailbox/delete=True.
* renames mailboxes of both types, renames/moves on disk only
  if mail/dovecot/mailbox/rename=True.
* modifies of existing public mailboxes:
  - add a mailPrimaryAddress: move mb from public to private/shared
    namespace (or just create the private/shared mb if
    mail/dovecot/mailbox/rename=False).
  - rm a mailPrimaryAddress: move mb from private/shared to public
    namespace (or just create the public mb
    if mail/dovecot/mailbox/rename=False).
  - change mailPrimaryAddress: rename private/shared mb (on disk only
    if mail/dovecot/mailbox/rename=True).
* tries to unsubscribes potential mb subscribers if a mb was renamed or
  removed.
* sets ACLs according to UDM. Uses "$ doveadm acl ..." for public mb,
  imaplib.setacl() for private/shared mb.
* uses a master-user to login as another user via IMAP. Credentials are
  stored in clear text in /etc/dovecot/master-users.
* sets the quota for public folders in the configuration file
  /etc/dovecot/conf.d/10-mail.conf, the quota for private/shared mb is
  taken from LDAP. Info is in objectClass=univentionMailSharedFolder
   → univentionMailUserQuota. The quota info (and the name) for the public
  folders are cached in the UCRV mail/dovecot/internal/_shared.
* runss "$ doveadm" commands as root, using the "Administrator" user as
  login.
* assumes Dovecot runs as dovecot:dovecot.
* stores the list of shared (private) mailboxes in a flat text file:
  /var/lib/dovecot/shared-mailboxes. It gets only updated when a ACL is
  changed through IMAP. For large installations this should be changed to
  a SQL dictionary.
Comment 6 Daniel Tröder univentionstaff 2015-06-01 18:34:35 CEST
Tests
=====
→ copy & paste
→ tail -f /var/log/univention/listener.log)

# ucr set mail/dovecot/mailbox/rename=True
> Setting mail/dovecot/mailbox/rename
# ucr set mail/dovecot/mailbox/delete=True
> Setting mail/dovecot/mailbox/delete

# DOMPOSITION=cn=domain,cn=mail,$(ucr get ldap/base)
# DOM1=abcde.fgh 
# udm mail/domain create --position $DOMPOSITION --set name=$DOM1
> Object created: cn=abcde.fgh,cn=domain,cn=mail,dc=uni,dc=dtr

# FOLPOSITION=cn=folder,cn=mail,$(ucr get ldap/base)
# NAME=pub1
# udm mail/folder create --position $FOLPOSITION --set mailDomain=$DOM1 --set mailHomeServer=$(hostname -f) --append sharedFolderUserACL="test1m@uni.dtr write" --append sharedFolderUserACL="test2m@uni.dtr read" --set cyrus-userquota=12 --set name=$NAME
> Object created: cn=pub1@abcde.fgh,cn=folder,cn=mail,dc=uni,dc=dtr

# ls -a /var/spool/dovecot/public/$DOM1/$NAME/
> .  ..  dovecot-acl-list  .INBOX

# LOC=pub1new
# EMAIL=$LOC@$DOM1
# udm mail/folder modify --dn cn=$NAME@$DOM1,$FOLPOSITION --set mailPrimaryAddress=$EMAIL
> Object modified: cn=pub1@abcde.fgh,cn=folder,cn=mail,dc=uni,dc=dtr

# ls -a /var/spool/dovecot/private/$DOM1/$LOC/
> .  ..  dovecot-acl-list  Maildir

# swaks --server $(hostname -f) --port 25 --from test@example.com --to $EMAIL
> ...
# ls -a /var/spool/dovecot/private/$DOM1/$LOC/Maildir/new/*.*.$(hostname -s),*
> /var/spool/dovecot/private/abcde.fgh/pub1new/Maildir/new/1433175074.M610992P3977.mail-cyr-1,S=1086,W=1110

# udm mail/folder remove --dn cn=$NAME@$DOM1,$FOLPOSITION
> Object removed: cn=pub1@abcde.fgh,cn=folder,cn=mail,dc=uni,dc=dtr

# ls -a /var/spool/dovecot/public/$DOM1 /var/spool/dovecot/private/$DOM1 
> /var/spool/dovecot/private/abcde.fgh:
> .  ..
> 
> /var/spool/dovecot/public/abcde.fgh:
> .  ..

# NAME=pub2
# LOC=pub2m
# EMAIL=$LOC@$DOM1
# udm mail/folder create --position $FOLPOSITION --set mailDomain=$DOM1 --set mailHomeServer=$(hostname -f) --append sharedFolderUserACL="test1m@uni.dtr write" --append sharedFolderUserACL="test2m@uni.dtr read" --set cyrus-userquota=12 --set mailPrimaryAddress=$EMAIL --set name=$NAME
> Object created: cn=pub2@abcde.fgh,cn=folder,cn=mail,dc=uni,dc=dtr

# doveadm acl get -u test1m@uni.dtr shared/$EMAIL
> ID                  Global Rights
> user=test1m@uni.dtr  insert lookup post read write write-deleted write-seen
> user=test2m@uni.dtr  lookup read write write-seen

# cat /var/spool/dovecot/private/abcde.fgh/pub2m/Maildir/dovecot-acl
> user=test1m@uni.dtr ilprwts
> user=test2m@uni.dtr lrws

# doveadm quota get -u $EMAIL
> Quota name Type    Value           %
> User quota STORAGE     0           0
> User quota MESSAGE     0           0

# udm mail/folder modify --dn cn=$NAME@$DOM1,$FOLPOSITION --set mailPrimaryAddress=
> Object modified: cn=pub2@abcde.fgh,cn=folder,cn=mail,dc=uni,dc=dtr

# doveadm acl get -u test1m@uni.dtr pub2@abcde.fgh/INBOX
> ID          Global Rights                          
> user=test1m@uni.dtr   insert lookup post read write write-deleted write-seen
> user=test2m@uni.dtr   lookup read write write-seen

# cat /var/spool/dovecot/public/abcde.fgh/pub2/.INBOX/dovecot-acl
> user=test1m@uni.dtr ilprwts
> user=test2m@uni.dtr lrws

# doveadm quota get -u test1m@uni.dtr
Quota name           Type    Value Limit      %
User quota           STORAGE 18689 20480      91
User quota           MESSAGE    10     -      0
pub2@abcde.fgh quota STORAGE     0 12288      0
pub2@abcde.fgh quota MESSAGE     0     -      0

# grep $EMAIL /var/lib/dovecot/shared-mailboxes 
> shared/shared-boxes/user/test2m@uni.dtr/pub2m@abcde.fgh
> shared/shared-boxes/user/test1m@uni.dtr/pub2m@abcde.fgh

Also test with
# ucr set mail/dovecot/mailbox/delete=False
and
# ucr set mail/dovecot/mailbox/rename=False
Comment 7 Daniel Tröder univentionstaff 2015-06-03 09:13:28 CEST
r61016 (addendum r60968) adds the needed master-user support.

The master-user is used to login as another user and set its ACLs via IMAP command. Only then will the list of shared private mailboxes (flat text file at /var/lib/dovecot/shared-mailboxes) be updated.
Comment 8 Daniel Tröder univentionstaff 2015-06-17 09:40:42 CEST
Mail-tests found: "anyone" user is disabled, but can be chosen in UMC. Enable in Dovecot.
Comment 9 Daniel Tröder univentionstaff 2015-06-17 10:09:06 CEST
Fixed configuration disallowing "anyone" identifier in commit 61301.
Comment 10 Sönke Schwardt-Krummrich univentionstaff 2015-06-30 00:11:37 CEST
(In reply to Daniel Tröder from comment #9)
> Fixed configuration disallowing "anyone" identifier in commit 61301.
→ OK

> * renames mailboxes of both types, renames/moves on disk only
>   if mail/dovecot/mailbox/rename=True.
> * modifies of existing public mailboxes:
>   - add a mailPrimaryAddress: move mb from public to private/shared
>     namespace (or just create the private/shared mb if
>     mail/dovecot/mailbox/rename=False).
>   - rm a mailPrimaryAddress: move mb from private/shared to public
>     namespace (or just create the public mb
>     if mail/dovecot/mailbox/rename=False).
>   - change mailPrimaryAddress: rename private/shared mb (on disk only
>     if mail/dovecot/mailbox/rename=True).

→ This has to be clearly stated in manual. Haven't found this yet.
→ see also Bug #38471

→ see bug #38792 for automatic tests
Comment 11 Sönke Schwardt-Krummrich univentionstaff 2015-07-02 16:17:19 CEST
Please rename mail/dovecot/internal/_shared to 
"mail/dovecot/internal/sharedfolders".
Comment 12 Sönke Schwardt-Krummrich univentionstaff 2015-07-03 11:25:54 CEST
(In reply to Sönke Schwardt-Krummrich from comment #10)
> → This has to be clearly stated in manual. Haven't found this yet.
> → see also Bug #38471

Wrong bug number. Bug #38791 was meant.
Comment 13 Daniel Tröder univentionstaff 2015-07-04 11:31:17 CEST
(In reply to Sönke Schwardt-Krummrich from comment #11)
> Please rename mail/dovecot/internal/_shared to 
> "mail/dovecot/internal/sharedfolders".
Done in r61767.
Comment 14 Sönke Schwardt-Krummrich univentionstaff 2015-07-06 21:41:16 CEST
(In reply to Daniel Tröder from comment #13)
> (In reply to Sönke Schwardt-Krummrich from comment #11)
> > Please rename mail/dovecot/internal/_shared to 
> > "mail/dovecot/internal/sharedfolders".
> Done in r61767.

→ VERIFIED
Comment 15 Janek Walkenhorst univentionstaff 2015-07-09 18:09:57 CEST
<http://errata.univention.de/ucs/4.0/238.html>
Comment 16 Janek Walkenhorst univentionstaff 2015-07-09 18:13:19 CEST
<http://errata.univention.de/ucs/4.0/237.html>