Index: listener/listener.xml =================================================================== --- listener/listener.xml (Revision 48130) +++ listener/listener.xml (Arbeitskopie) @@ -10,10 +10,8 @@ Univention Directory ListenerDirectory Listener - PMH: Bug #29420 - - Replication of the directory data within a UCS domain occurs via the Univention + Replication of the directory data within a UCS domain is provided by the Univention Directory Listener/Notifier mechanism: @@ -24,7 +22,7 @@ On the &ucsMaster; (and possibly existing &ucsBackup; systems) the &ucsUDN; service monitors changes in the LDAP directory and makes the selected changes available to the &ucsUDL; - services on the other UCS systems. + services on all UCS systems joined into the domain. @@ -33,11 +31,11 @@ The active &ucsUDL; instances in the domain connect to a &ucsUDN; service. If an LDAP change is performed on the &ucsMaster; (all other LDAP servers in the domain are read-only), this is registered by the &ucsUDN; - and notified to the listener instances. + and reported to the listener instances. - Each &ucsUDL; instance uses a range of &ucsUDL; + Each &ucsUDL; instance hosts a range of &ucsUDL; modules. These modules are shipped by the installed applications; the print server package includes, for example, listener modules which generate the CUPS configuration. @@ -48,17 +46,17 @@ read from the LDAP, but instead from the file /etc/cups/printers.conf. Now, if a printer is saved in the printer management of the &ucsUMC;, it is stored in the LDAP directory. This change is detected by the &ucsUDL; module - cups-printers and an entry added to, modified or deleted in - /etc/cups/printers.conf based on the data in the LDAP. + cups-printers and an entry gets added to, modified in or deleted from + /etc/cups/printers.conf based on the modification in the LDAP directory.
- Structure of Listener modules + Structure of Listener Modules Each listener module must declare several string constants. - They are required by the &ucsUDL; to handle and process each module. - They should be defined in the header at the beginning of the module. + They are required by the &ucsUDL; to handle each module. + They should be defined at the beginning of the module. it is case sensitive - it only support equal matches + it only supports equal matches - The name filter was historically poorly chosen, since it overwrite the built-in Python function filter(). - If that function is required for implementing the listener module, a alias-reference must be defined before overwriting the reference or sys.modules["__builtin__"].filter can be used to reference the function. + The name filter has the drawback that it shadows the Python built-in function filter(), but its use has historical reasons. + If that function should be required for implementing the listener module, an alias-reference may be defined before overwriting the name or it may be explicitly accessed via the Python __builtin__ module. @@ -110,9 +108,9 @@ attributes - A Python list of attribute names as string, which further filters the condition, when the Listener module should be called. - By default the empty list invokes the module on all changes. - Otherwise the module is only invoked, when at least one attribute mentioned in the list is changed. + A Python list of LDAP-attribute names which further narrows down the condition under which the listener module gets called. + By default the module is called on all attribute changes of objects matching the filter. + If the list is specified, the module is only invoked when at least one of the listed attributes is changed. @@ -128,7 +126,7 @@ - In addition to the static string a module must implement several functions. + In addition to the static strings a module must implement several functions. They are called in different situations of the live-cycle of the module. handler(dn, old, new, command='') - This function is called for each change matching the filter and attributes expressions as declared in the header above. + This function is called for each change matching the filter and attributes as declared in the header of the module. The distinguished name (dn) of the object is supplied as the first argument dn. - Depending on the type of modification, old and new are either None or reference a dictionary of arrays, representing the multi-valued attributes of the object. - The &ucsUDL; uses a local cache to cache the latest values of each object. - This cache is used to supply the values for old, while the values in new are those retrieved from the LDAP directory service. + Depending on the type of modification, old and new may each independently either be None or reference a Python dictionary of arrays. Each array represents one of the multi-valued attributes of the object. + The &ucsUDL; uses a local cache to cache the values of each object as it was seen most recently. + This cache is used to supply the values for old, while the values in new are those retrieved from that LDAP directory service which is running on the same server as the &ucsUDN; (&ucsMaster; or &ucsBackup; servers in the domain). If and only if the global modrdn setting is enabled, command is passed as a fourth argument. It contains a single letter, which indicates the type of modification. - This can be used to distinguish an modrdn operation from a delete operation followed by a create operation. + This can be used to distinguish an modrdn operation from a delete operation followed by a create operation. - modify (m) + m (modify) Signals a modify operation, where an existing object is changed. @@ -168,7 +166,7 @@ - add (a) + a (add) Signals the addition of a new object. @@ -177,7 +175,7 @@ - delete (d) + d (delete) Signals the removal of a previously existing object. @@ -186,28 +184,19 @@ - modify relative distinguished name (modrdn, r) + r (rename: modification of distinguished name via modrdn) - Signals a change in the distinguished name, which might be caused by renaming a object or moving the object from one container into one other. - The module is called with this command instead of the delete command, so that the module can optimize that case and skip the deletion of the object. - The module will be called again with the add command just after the modrdn command, where it should perform the rename or move operation. - The module is responsible for keeping track of the rename-case by internally storing the previous distinguished name. + Signals a change in the distinguished name, which might be caused by renaming a object or moving the object from one container into another. + The module is called with this command instead of the delete command, so that modules can recognize this special case and skip the deletion of the object. + The module will be called again with the add command just after the modrdn command, where it should process the rename or move operation. + Each module is responsible for keeping track of the rename-case by internally storing the previous distinguished name during the modrdn phase of this two phased operation. - z + n (new or schema change) - - This should be ignored. - PMH: Unknown - - - - - new or schema change (n) - This command can signal two changes: @@ -214,7 +203,7 @@ If dn is cn=Subschema, it signals that a schema change occurred. - All other cases signal the initialization of a new object, which should be handles just like a normal add operation. + All other cases signal the initialization of a new object, which should be handled just like a normal add operation. @@ -231,7 +220,7 @@ This is recorded persistently in the file /var/lib/univention-directory-listener/name, where name equals the value from the header. - If for whatever reason the Listener module should be reset and re-run for all matching objects, the state can be reset by running the command univention-directory-listener-ctrl resync name. + If for whatever reason the listener module should be reset and re-run for all matching objects, the state can be reset by running the command univention-directory-listener-ctrl resync name. In that case the function initialize() will be called again. @@ -250,14 +239,14 @@ The opening is signaled by the invocation of the function prerun() and the closing by postrun(). - The function postrun() is most often used to restart services, as restarting a service takes some time and makes the service unavailable during that time: - The stream of changes is only used to generate new configuration files, which are only activated by restarting the service in the postrun() handler. + The function postrun() is most often used to restart services, as restarting a service takes some time and makes the service unavailable during that time. + It's best practice to use the handler() only to process the stream of changes, set UCR variables or generate new configuration files and delay a restart of associated services to the postrun() function. The function postrun() is only called, when no change happens for 15 seconds. This is not on a per-module basis, but globally. - In an ever changing system, where the stream of changes never pauses for 15 seconds, the functions would be called never! + In an ever changing system, where the stream of changes never pauses for 15 seconds, the functions may never be called! @@ -266,21 +255,21 @@ setdata(key, value) - This function is called several times by the main &ucsUDL; to pass configuration data into the modules. + This function is called several times by the &ucsUDL; main process to pass configuration data into the modules. The following keys are supplied: ldapserver - The host-name of the LDAP server the &ucsUDL; is using. + The hostname of the LDAP server the &ucsUDL; is currently reading from. binddn - The bind distinguished name the &ucsUDL; is using to authenticate with the LDAP. + The distinguished name the &ucsUDL; is using to authenticate to the LDAP server (via simple bind). bindpw - The simple bind password the &ucsUDL; is using to authenticate with the LDAP. + The password the &ucsUDL; is using to authenticate to the LDAP server. basedn @@ -293,7 +282,7 @@
- Listener tasks and examples + <title>Listener Tasks and Examples <indexterm><primary>Directory Listener</primary><secondary>Example module</secondary></indexterm> @@ -302,10 +291,10 @@
- Basic example + Basic Example - The following boiler-plate delegates each change type to a separate function. - It does not handle renames and moves explicitly, but only through the removal of the object at the old dn and the following addition at the new dn. + The following boilerplate code delegates each change type to a separate function. + It does not handle renames and moves explicitly, but only as the removal of the object at the old dn and the following addition at the new dn. Source code: @@ -314,11 +303,11 @@
- Rename and move + <title>Rename and Move <indexterm><primary>Directory Listener</primary><secondary>modrdn</secondary></indexterm> - In case rename and move actions should be handled separately, use the following code: + In case rename and move actions should be handled separately, the following code may be used: Source code: @@ -326,14 +315,14 @@ - Please be aware that tracking the two subsequent calls for modrdn in memory might cause duplicates, if the &ucsUDL; is terminated while such an operation is performed. - If this is problematic, saving the state on a persistent disk mist be implemented. + Please be aware that tracking the two subsequent calls for modrdn in memory might cause duplicates, in case the &ucsUDL; is terminated while such an operation is performed. + If this is critical, the state should be stored persistently into a temporary file.
- Full example with packaging + Full Example with Packaging The following example shows a listener module, which logs all changes to users into the file /tmp/UserList.txt. @@ -342,18 +331,19 @@ - Some comments: + Some comments on the code: - Overwriting __package__ is currently necessary, as the &ucsUDL; imports the Listener module by its own mechanism, which is incompatible with the mechanism normally used by Python itself. - Be aware, that this might cause problems when use pickle. + Overwriting __package__ is currently necessary, as the &ucsUDL; imports the listener module by its own mechanism, which is incompatible with the mechanism normally used by Python itself. + Be aware, that this might cause problems when using pickle. + AREQ: And what's the recommendation? - The LDAP filter is cleverly chosen to only match user objects, but not computer objects, which end on $. + The LDAP filter is specifically chosen to only match user objects, but not computer objects, which have a uid characteristically terminated by a $-sign. @@ -364,13 +354,14 @@ Writing a file as root in a public location like /tmp/ should be avoided. + AREQ: Why? What's the point here? Relevant? It is only done here to show how it works. - For testing run a command like tail -f /tmp/UserList.txt &. - Then create a new user or modify the lastname of an existing user to trigger the module. + To test this run a command like tail -f /tmp/UserList.txt &. + Then create a new user or modify the lastname of an existing one to trigger the module. @@ -402,9 +393,9 @@
- A little bit more object oriented + A Little Bit more Object Oriented - For larger modules it is recommended to use a more object-oriented design like the following example, which logs referential integrity violations into a file. + For larger modules it might be preferrable to use a more object oriented design like the following example, which logs referential integrity violations into a file. Source code: @@ -414,7 +405,7 @@
- Technical details + <title>Technical Details <indexterm><primary>Directory Listener</primary><secondary>Debug</secondary></indexterm> @@ -429,18 +420,18 @@ -->
- User-ID and credentials + <title>User-ID and Credentials <indexterm><primary>Directory Listener</primary><secondary>Credentials</secondary></indexterm> The Listener runs with the effective permissions of the user listener. - If root-privileges are required, listener.setuid() can be used to switch the effective UID. - When done, listener.unsetuid() should be called to drop back to the listener UID. + If root-privileges are required, listener.setuid() can be used to switch the effective UID. + When done, listener.unsetuid() should be called to drop back to the listener UID. It's best practice to code this as try/finally clauses in Python.
- Internal cache + <title>Internal Cache <indexterm><primary>Directory Listener</primary><secondary>Cache</secondary></indexterm> @@ -459,9 +450,9 @@ This is required when a new module is added, which is invoked for all already existing objects when the &ucsUDL; is restarted. - On Domaincontrollers the cache could be replaced by doing a query to the local LDAP server, before the new values are written into it. + On domain controllers the cache could be replaced by doing a query to the local LDAP server, before the new values are written into it. But &ucsMember; do not have a local LDAP server, so there the cache is needed. - Also note that the cache keeps track of the associated Listener modules, which is not available from the LDAP. + Also note that the cache keeps track of the associated listener modules, which is not available from the LDAP. @@ -469,7 +460,7 @@ notifier_id - This file contains the notifier id as last queried from the &ucsUDN;. + This file contains the last notifier ID read from the &ucsUDN;. @@ -478,7 +469,7 @@ For each module the directory contains a text file consisting of a single number. - The name of the file is derived from the values of the variable name as defined in each Listener module. + The name of the file is derived from the values of the variable name as defined in each listener module. The number is to be interpreted as a bit-field of HANDLER_INITIALIZED=0x1 and HANDLER_READY=0x2. If both bits are set, it indicates that the module was successfully initialized by running the function initialize(). Otherwise both bits are unset. @@ -488,15 +479,15 @@ The package univention-directory-listener contains several commands useful for controlling and debugging problems with the &ucsUDL;. - This can be useful for debugging Listener cache inconsistencies. + This can be useful for debugging listener cache inconsistencies.
univention-directory-listener-ctrl - The command univention-directory-listener-ctrl resync name can be used to reset and re-initialize a Module. + The command univention-directory-listener-ctrl resync name can be used to reset and re-initialize a module. It stops any currently running listener process, removes the state file for the specified module and starts the listener process again. - This forces the functions clean() and initialize() to be called one after each other. + This forces the functions clean() and initialize() to be called one after the other.
@@ -505,7 +496,7 @@ The command univention-directory-listener-dump can be used to dump the cache file /var/lib/univention-directory-listener/cache.db. The &ucsUDL; must be stopped first by invoking service univention-directory-listener stop. - It outputs the cache in a LDAP Data Interchange Format (LDIF) compatible format. + It outputs the cache in format compatible to the LDAP Data Interchange Format (LDIF).
@@ -517,7 +508,7 @@ The command univention-directory-listener-verify can be used to compare the content of the cache file /var/lib/univention-directory-listener/cache.db to the content of an LDAP server. The &ucsUDL; must be stopped first by invoking service univention-directory-listener stop. LDAP credentials must be supplied at the command line. - For example, the following command would use the machine secrets: + For example, the following command would use the machine password: univention-directory-listener-verify \ -b "$(ucr get ldap/base)" \ @@ -535,7 +526,7 @@ The command /usr/share/univention-directory-listener/get_notifier_id.py can be used to get the latest ID from the notifier. This is done by querying the &ucsUDN; running on the LDAP server configured through the &ucsUCRV; ldap/master. The returned value should be equal to the value currently stored in the file /var/lib/univention-directory-listener/notifier_id. - Otherwise the &ucsUDL; might still be processing a transaction or might indicate a problem with the &ucsUDL; + Otherwise the &ucsUDL; might still be processing a transaction or it might indicate a problem with the &ucsUDL;
@@ -549,7 +540,7 @@ The Listener/Notifier mechanism is used to trigger arbitrary actions when changes occur in the LDAP directory service. In addition to the LDAP server slapd it consists of two other services: The &ucsUDN; service runs next to the LDAP server and broadcasts change information to interested parties. - The &ucsUDL; service listens for those notifications, downloads the changes and runs arbitrary local actions like storing the data in a local LDAP server for replication or generating configuration files for non-LDAP-aware local services. + The &ucsUDL; service listens for those notifications, downloads the changes and runs listener modules performing arbitrary local actions like storing the data in a local LDAP server for replication or generating configuration files for non-LDAP-aware local services.
Listener/Notifier mechanism @@ -561,19 +552,19 @@ Listener/Notifier prcoedure - The LDAP object is modified on the &ucsMaster;. + An LDAP object is modified on the &ucsMaster;. Changes initiated on all other system roles are re-directed to the master. - The overlay-module translog appends the DN to the file /var/lib/univention-ldap/listener/listenerFILE_NAME_LISTENER, TRANSACTION_FILE. + The UCS-specific overlay-module translog appends the DN to the file /var/lib/univention-ldap/listener/listenerReferred to as FILE_NAME_LISTENER, TRANSACTION_FILE in the source code. - The &ucsUDN; watches that file, picks up and removes the modification. - It assigns the next transaction number and writes it into the file /var/lib/univention-ldap/notify/transactionFILE_NAME_TF, including the DN and change-type. + The &ucsUDN; watches that file, picks up and removes each line it processed. + It assigns the next transaction number and writes it into the file /var/lib/univention-ldap/notify/transactionReferred to as FILE_NAME_TF in the source code, including the DN and change type. For efficient access by transaction ID the index transaction.index is updated. @@ -584,51 +575,48 @@ - Each Listener such notified queries the Notifier for the latest transaction ID, DN and change-type. - The ID is written into the file /var/lib/univention-directory-listener/notifier_id. + Each Listener triggered in this way queries the Notifier for the latest transaction ID, DN and change type. + The ID is written into the local file /var/lib/univention-directory-listener/notifier_id. - Each Listener opens a connection to their LDAP server configured through &ucsUCRV; ldap/master and ldap/server/addition:. + Each Listener opens a connection to the LDAP server running on the UCS system which was used to query the Notifier (configured through &ucsUCRV; ldap/master and ldap/server/addition:). It tries to retrieve the latest state of the object identified through the DN, which might be blocked by selective replication. If that fails and nothing is returned, the process stops here and closes the LDAP connection. + AREQ: Uh, really? A d operation should be processed as well? Maybe rephrase "the process"? - On a &ucsBackup; the &ucsUDL; writes the transaction data to the file /var/lib/univention-ldap/listener/listenerFILE_NAME_LISTENER, TRANSACTION_FILE to allow the &ucsUDN; to be cascaded. - This can be configured with the option of univention-directory-listener and is done for load balancing and reliability reasons. + On a &ucsBackup; the &ucsUDL; writes the transaction data to the file /var/lib/univention-ldap/listener/listenerReferred to as FILE_NAME_LISTENER, TRANSACTION_FILE in the source code to allow the &ucsUDN; to be cascaded. + This is automatically internally configured with the option of univention-directory-listener and is done for load balancing and failover reasons. - The old state of the object is fetched from the Listener cache. + The old state of the object is fetched from the local listener cache. - For each module it is checked, if the module matches either the old or new state according to the variables filter and attributes. + For each module it is checked, if either the old or new state of the object matches the filter and attributes specified in the corresponding Python variables. If not, the module is skipped. - If the function prerun() of module was not yet called, this is done to signal the start of changes. + If the function prerun() of module was not called yet, this is done to signal the start of changes. - The function handler() is called for the module, passing in the DN with the old and new state. - In previous releases of the &ucsUDL; modules could modify the new state, which was written to the local LDAP server by the last module. - This lead to some very strange problems and was disabled therefore. - Each module now gets its own new copy of the values and thus can no longer communicate with subsequent modules. - + The function handler() specified in the module is called, passing in the DN and the old and new state. - The cache is updated with the new values, including the names of the modules which successfully handled that object. + The main listener process updates its cache with the new values, including the names of the modules which successfully handled that object. This guarantees that the module is still called, even when the filter criteria would no longer match the object after modification. @@ -635,7 +623,7 @@ After 15 seconds of inactivity the function postrun() is invoked for all prepared modules. - This signals the end of changes and requests the module to release its resources and/or start pending operations. + This signals a break in the stream of changes and requests the module to release its resources and/or start pending operations. Index: listener/modrdn.py =================================================================== --- listener/modrdn.py (Revision 48130) +++ listener/modrdn.py (Arbeitskopie) @@ -24,17 +24,15 @@ handler_schema() else: handler_add(dn, new) - elif "z" == command: - pass # ignore else: - pass # error + pass # ignore, reserved for future use def handler_move(old_dn, old, new_dn, dn): """Handle rename or move of object.""" - pass + pass ## replace this def handlee_schema(): """Handle change in LDAP schema.""" - pass + pass ## replace this Index: listener/simple.py =================================================================== --- listener/simple.py (Revision 48130) +++ listener/simple.py (Arbeitskopie) @@ -6,19 +6,19 @@ elif not new and old: handler_remove(dn, old) else: - pass # error + pass # ignore def handler_add(dn, new): """Handle addition of object.""" - pass + pass # replace this def handler_modify(dn, old, new): """Handle modification of object.""" - pass + pass # replace this def handler_remove(dn, old): """Handle removal of object.""" - pass + pass # replace this