Bug 35144 - Migrate listener / notifier to UCS 4
Migrate listener / notifier to UCS 4
Status: CLOSED FIXED
Product: UCS
Classification: Unclassified
Component: Listener (univention-directory-listener)
UCS 4.0
Other Linux
: P5 enhancement (vote)
: UCS 4.0
Assigned To: Felix Botner
Arvid Requate
: interim-1
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2014-06-18 06:35 CEST by Stefan Gohmann
Modified: 2014-11-26 06:55 CET (History)
4 users (show)

See Also:
What kind of report is it?: ---
What type of bug is this?: ---
Who will be affected by this bug?: ---
How will those affected feel about the bug?: ---
User Pain:
Enterprise Customer affected?:
School Customer affected?:
ISV affected?:
Waiting Support:
Ticket number:
Bug group (optional):
Max CVSS v3 score:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Stefan Gohmann univentionstaff 2014-06-18 06:35:06 CEST
Listener and notifier should be migrated to UCS 4. Please consider the package migration list:
 https://hutten.knut.univention.de/mediawiki/index.php/UCS-4.0_Univention_Package_Update
Comment 1 Stefan Gohmann univentionstaff 2014-06-18 06:40:46 CEST
Packsges:

univention-directory-listener
univention-directory-notifier
univention-directory-replication
Comment 2 Stefan Gohmann univentionstaff 2014-06-18 06:51:13 CEST
We have to think about the local LDAP cache. Currently we use libdb3. I think at least for the first milestone we should use libdb3 again.

Do we need to change it for the final UCS 4 release? If we need to change, should we switch to sqlite or db4?
Comment 3 Philipp Hahn univentionstaff 2014-06-18 09:12:35 CEST
(In reply to Stefan Gohmann from comment #2)
> Do we need to change it for the final UCS 4 release? If we need to change,
> should we switch to sqlite or db4?

The listener always look-up entries through their DN, which is hierarchical by nature.
SQLlite on the other hand is relational by nature, so those two data modules do not match. (Implementing a hierarchical model in SQL is not fun and I have personal experience with one project, were an application crash destroyed the complete DB.)
It don't give us any benefit (expect perhaps implementation robustness). A simple key-value-store like DB is AFAIK enough.

There are some additional settings (last-id,...), which are stored in one value per file; code exists to store that in DB4 as well, but it's IMHO over-kill to use SQLite to store those values.

IMHO it would make more sense to use the local LDAP as the cache on DCs and only use a custom implementation on Member-Servers (and other external system roles). AFAIK the only extra information, which is stored in the listener cache but not in the local LDAP, is the list of modules handling an entry, which is only needed by handlers_delete() to call the handlers for previously handled objects.
Comment 4 Arvid Requate univentionstaff 2014-06-23 11:28:38 CEST
> use a custom implementation on Member-Servers (and other external system roles).

How about running a local LDAP server on every system role and just adjust its configuration on memberservers to only bind to IPv4 localhost (or even just use LDAPI).
Comment 5 Stefan Gohmann univentionstaff 2014-06-30 08:01:59 CEST
(In reply to Arvid Requate from comment #4)
> > use a custom implementation on Member-Servers (and other external system roles).
> 
> How about running a local LDAP server on every system role and just adjust
> its configuration on memberservers to only bind to IPv4 localhost (or even
> just use LDAPI).

OK, that sounds good. I think we should start with libdb3 and re-check this issue later in the UCS 4 development. Then we have to compare this with other issues.
Comment 6 Sönke Schwardt-Krummrich univentionstaff 2014-07-21 09:51:27 CEST
In univention-ssl the dependency to univention-directory-listener has been disabled temporarily. If univention-directory-listener has been built successfully, the dependency has to be reenabled!
Comment 7 Felix Botner univentionstaff 2014-08-19 17:13:50 CEST
migrated univention-directory-reports, univention-directory-listener, univention-directory-notifier and univention-directory-replication to UCS 4.0

added olcMdbConfig OID to BUILTIN_OIDS in replication.py

readded dependency to univention-directory-listener in  univention-ssl
Comment 8 Stefan Gohmann univentionstaff 2014-08-20 13:31:56 CEST
The following traceback occurs:

20.08.14 12:37:57.728  LISTENER    ( ERROR   ) : import of filename=/usr/lib/univention-directory-listener/system/faillog.py failed in module_import()
20.08.14 12:37:57.732  LISTENER    ( ERROR   ) : import of filename=/usr/lib/univention-directory-listener/system/bind.py failed
Traceback (most recent call last):
  File "/usr/lib/univention-directory-listener/system/bind.py", line 40, in <module>
    import listener
ImportError: No module named listener
Comment 9 Stefan Gohmann univentionstaff 2014-08-20 14:19:18 CEST
(In reply to Stefan Gohmann from comment #8)
> The following traceback occurs:
> 
> 20.08.14 12:37:57.728  LISTENER    ( ERROR   ) : import of
> filename=/usr/lib/univention-directory-listener/system/faillog.py failed in
> module_import()
> 20.08.14 12:37:57.732  LISTENER    ( ERROR   ) : import of
> filename=/usr/lib/univention-directory-listener/system/bind.py failed
> Traceback (most recent call last):
>   File "/usr/lib/univention-directory-listener/system/bind.py", line 40, in
> <module>
>     import listener
> ImportError: No module named listener

Fixed with r52868
Comment 10 Stefan Gohmann univentionstaff 2014-08-25 08:32:14 CEST
I've added a changelog for all migrated packages: r52983
Comment 11 Arvid Requate univentionstaff 2014-08-28 17:29:59 CEST
There are additional builtin OIDs to be considered in replication.py for MDB:

1.3.6.1.4.1.4203.666.11.1.3.2.12.1   # olcDbMaxReaders
1.3.6.1.4.1.4203.666.11.1.3.2.12.2   # olcDbMaxSize
1.3.6.1.4.1.4203.666.11.1.3.2.12.3   # olcDbEnvFlags

See openldap-2.4.35/servers/slapd/back-mdb/config.c
Comment 12 Felix Botner univentionstaff 2014-08-29 11:18:55 CEST
(In reply to Arvid Requate from comment #11)
> There are additional builtin OIDs to be considered in replication.py for MDB:
> 
> 1.3.6.1.4.1.4203.666.11.1.3.2.12.1   # olcDbMaxReaders
> 1.3.6.1.4.1.4203.666.11.1.3.2.12.2   # olcDbMaxSize
> 1.3.6.1.4.1.4203.666.11.1.3.2.12.3   # olcDbEnvFlags
> 
> See openldap-2.4.35/servers/slapd/back-mdb/config.c

do you mean 

  1.3.6.1.4.1.4203.1.12.2.3.2.12.1
  1.3.6.1.4.1.4203.1.12.2.3.2.12.2
  1.3.6.1.4.1.4203.1.12.2.3.2.12.3
  (OLcfgDbAt + 12.1 etc.)

Added those three to BUILTIN_OIDS in replication.py
Comment 13 Arvid Requate univentionstaff 2014-09-01 11:31:53 CEST
Yeah, right, there is our 31_avoid_OID_changes.patch, which patches the OpenLDAP internal OIDs back so the OpenLDAP 2.3 conventions. We might consider to drop that patch at some point. I added a comment to Bug 35697.
Comment 14 Arvid Requate univentionstaff 2014-09-01 11:34:22 CEST
But as 31_avoid_OID_changes.patch patches

-       { "OLcfg", "1.3.6.1.4.1.4203.1.12.2" },
+       { "OLcfg", "1.3.6.1.4.1.4203.666.11.1" },

I guess we need to add the .666. variants as well. Better keep both conventions, in case we drop the patch.
Comment 15 Felix Botner univentionstaff 2014-09-01 14:44:44 CEST
(In reply to Arvid Requate from comment #14)
> But as 31_avoid_OID_changes.patch patches
> 
> -       { "OLcfg", "1.3.6.1.4.1.4203.1.12.2" },
> +       { "OLcfg", "1.3.6.1.4.1.4203.666.11.1" },
> 
> I guess we need to add the .666. variants as well. Better keep both
> conventions, in case we drop the patch.

ok
Comment 16 Arvid Requate univentionstaff 2014-09-01 15:08:55 CEST
Ok.
Comment 17 Stefan Gohmann univentionstaff 2014-11-26 06:55:27 CET
UCS 4.0-0 has been released:
 http://docs.univention.de/release-notes-4.0-0-en.html
 http://docs.univention.de/release-notes-4.0-0-de.html

If this error occurs again, please use "Clone This Bug".