Attachment #5859 for bug #29420

View | Details | Raw Unified | Return to bug 29420
Collapse All | Expand All




		<indexterm><primary>Univention Directory Listener</primary><see>Directory Listener</see></indexterm>
	</title>

	<remark>PMH: Bug #29420</remark>

	<para>
		Replication of the directory data within a UCS domain is provided by the Univention
		Directory Listener/Notifier mechanism:

		<itemizedlist>

			<listitem><simpara>
				On the &ucsMaster; (and possibly existing &ucsBackup; systems) the <emphasis>&ucsUDN;</emphasis> service monitors
				changes in the LDAP directory and makes the selected changes available to the &ucsUDL;
				services on all UCS systems joined into the domain.
			</simpara></listitem>
		</itemizedlist>
	</para>

		The active &ucsUDL; instances in the domain connect to a &ucsUDN;
		service. If an LDAP change is performed on the &ucsMaster; (all other LDAP
		servers in the domain are read-only), this is registered by the &ucsUDN;
		and reported to the listener instances.
	</para>

	<para>
		Each &ucsUDL; instance hosts a range of &ucsUDL;
		modules. These modules are shipped by the installed applications; the print server package
		includes, for example, listener modules which generate the CUPS configuration.
	</para>

		read from the LDAP, but instead from the file <filename>/etc/cups/printers.conf</filename>.
		Now, if a printer is saved in the printer management of the &ucsUMC;, it is stored
		in the LDAP directory. This change is detected by the &ucsUDL; module
		<emphasis>cups-printers</emphasis> and an entry gets added to, modified in or deleted from
		<filename>/etc/cups/printers.conf</filename> based on the modification in the LDAP directory.
	</para>

	<section id="listener:handler">
		<title>Structure of Listener Modules</title>

		<para>
			Each listener module must declare several string constants.
			They are required by the &ucsUDL; to handle each module.
			They should be defined at the beginning of the module.
		</para>
		<programlisting language="python"><![CDATA[
name = "module_name"

					</para>
					<itemizedlist>
						<listitem><simpara>it is case sensitive</simpara></listitem>
						<listitem><simpara>it only supports equal matches</simpara></listitem>
					</itemizedlist>
					<note>
						<para>
							The name <literal>filter</literal> has the drawback that it shadows the Python built-in function <function>filter()</function>, but its use has historical reasons.
							If that function should be required for implementing the listener module, an alias-reference may be defined before overwriting the name or it may be explicitly accessed via the Python <literal>__builtin__</literal> module.
						</para>
					</note>
				</listitem>

				<term><varname>attributes</varname></term>
				<listitem>
					<para>
						A Python list of LDAP-attribute names which further narrows down the condition under which the listener module gets called.
						By default the module is called on all attribute changes of objects matching the filter.
						If the list is specified, the module is only invoked when at least one of the listed attributes is changed.
					</para>
				</listitem>
			</varlistentry>

		</variablelist>

		<para>
			In addition to the static strings a module must implement several functions.
			They are called in different situations of the live-cycle of the module.
		</para>
		<programlisting language="python"><![CDATA[

				<term><function>handler(dn, old, new, command='')</function></term>
				<listitem>
					<para>
						This function is called for each change matching the <varname>filter</varname> and <varname>attributes</varname> as declared in the header of the module.
						The distinguished name (<abbrev>dn</abbrev>) of the object is supplied as the first argument <varname>dn</varname>.
					</para>
					<para>
						Depending on the type of modification, <varname>old</varname> and <varname>new</varname> may each independently either be <literal>None</literal> or reference a Python dictionary of arrays. Each array represents one of the multi-valued attributes of the object.
						The &ucsUDL; uses a local cache to cache the values of each object as it was seen most recently.
						This cache is used to supply the values for <varname>old</varname>, while the values in <varname>new</varname> are those retrieved from that LDAP directory service which is running on the same server as the &ucsUDN; (&ucsMaster; or &ucsBackup; servers in the domain).
					</para>
					<para>
						If and only if the global <varname>modrdn</varname> setting is enabled, <varname>command</varname> is passed as a fourth argument.
						It contains a single letter, which indicates the type of modification.
						This can be used to distinguish an <firstterm>modrdn</firstterm> operation from a delete operation followed by a create operation.
					</para>
					<variablelist>
						<varlistentry>
							<term><literal>m</literal> (modify)</term>
							<listitem>
								<para>
									Signals a modify operation, where an existing object is changed.

							</listitem>
						</varlistentry>
						<varlistentry>
							<term><literal>a</literal> (add)</term>
							<listitem>
								<para>
									Signals the addition of a new object.

							</listitem>
						</varlistentry>
						<varlistentry>
							<term><literal>d</literal> (delete)</term>
							<listitem>
								<para>
									Signals the removal of a previously existing object.

							</listitem>
						</varlistentry>
						<varlistentry>
							<term><literal>r</literal> (rename: modification of distinguished name via <literal>modrdn</literal>)</term>
							<listitem>
								<para>
									Signals a change in the distinguished name, which might be caused by renaming a object or moving the object from one container into another.
									The module is called with this command instead of the <emphasis>delete</emphasis> command, so that modules can recognize this special case and skip the deletion of the object.
									The module will be called again with the <emphasis>add</emphasis> command just after the <emphasis>modrdn</emphasis> command, where it should process the rename or move operation.
									Each module is responsible for keeping track of the rename-case by internally storing the previous distinguished name during the <emphasis>modrdn</emphasis> phase of this two phased operation.
								</para>
							</listitem>
						</varlistentry>
						<varlistentry>
							<term><literal>n</literal> (new or schema change)</term>
							<listitem>
								<para>
									This should be ignored.
									<remark>PMH: Unknown</remark>
								</para>
							</listitem>
						</varlistentry>
						<varlistentry>
							<term>new or schema change (<literal>n</literal>)</term>
							<listitem>
								<para>This command can signal two changes:</para>
								<itemizedlist>
									<listitem>

										<simpara>If <varname>dn</varname> is <literal>cn=Subschema</literal>, it signals that a schema change occurred.</simpara>
									</listitem>
									<listitem>
										<simpara>All other cases signal the initialization of a new object, which should be handled just like a normal <function>add</function> operation.</simpara>
									</listitem>
								</itemizedlist>
							</listitem>

						This is recorded persistently in the file <filename>/var/lib/univention-directory-listener/<replaceable>name</replaceable></filename>, where <varname>name</varname> equals the value from the header.
					</para>
					<para>
						If for whatever reason the listener module should be reset and re-run for all matching objects, the state can be reset by running the command <command>univention-directory-listener-ctrl resync <replaceable>name</replaceable></command>.
						In that case the function <function>initialize()</function> will be called again.
					</para>
					<para>

						The opening is signaled by the invocation of the function <function>prerun()</function> and the closing by <function>postrun()</function>.
					</para>
					<para>
						The function <function>postrun()</function> is most often used to restart services, as restarting a service takes some time and makes the service unavailable during that time.
						It's best practice to use the <function>handler()</function> only to process the stream of changes, set UCR variables or generate new configuration files and delay a restart of associated services to the <function>postrun()</function> function.
					</para>
					<warning>
						<para>
							The function <function>postrun()</function> is only called, when no change happens for 15 seconds.
							This is not on a per-module basis, but globally.
							In an ever changing system, where the stream of changes never pauses for 15 seconds, the functions may never be called!
						</para>
					</warning>
				</listitem>

				<term><function>setdata(key, value)</function></term>
				<listitem>
					<para>
						This function is called several times by the &ucsUDL; main process to pass configuration data into the modules.
						The following <varname>key</varname>s are supplied:
					</para>
					<variablelist>
						<varlistentry>
							<term><literal>ldapserver</literal></term>
							<listitem><simpara>The hostname of the <abbrev>LDAP</abbrev> server the &ucsUDL; is currently reading from.</simpara></listitem>
						</varlistentry>
						<varlistentry>
							<term><literal>binddn</literal></term>
							<listitem><simpara>The distinguished name the &ucsUDL; is using to authenticate to the <abbrev>LDAP</abbrev> server (via <literal>simple bind</literal>).</simpara></listitem>
						</varlistentry>
						<varlistentry>
							<term><literal>bindpw</literal></term>
							<listitem><simpara>The password the &ucsUDL; is using to authenticate to the <abbrev>LDAP</abbrev> server.</simpara></listitem>
						</varlistentry>
						<varlistentry>
							<term><literal>basedn</literal></term>

	</section>

	<section id="listener:example">
		<title>Listener Tasks and Examples
			<indexterm><primary>Directory Listener</primary><secondary>Example module</secondary></indexterm>
		</title>
		<para>

		</para>

		<section id="listener:example:simple">
			<title>Basic Example</title>
			<para>
				The following boilerplate code delegates each change type to a separate function.
				It does not handle renames and moves explicitly, but only as the removal of the object at the old <abbrev>dn</abbrev> and the following addition at the new <abbrev>dn</abbrev>.
			</para>
			<para>
				Source code: <ulink url="&websvn;doc/developer-reference/listener/simple.py"/>

		</section>

		<section id="listener:example:modrdn">
			<title>Rename and Move
				<indexterm><primary>Directory Listener</primary><secondary>modrdn</secondary></indexterm>
			</title>
			<para>
				In case rename and move actions should be handled separately, the following code may be used:
			</para>
			<para>
				Source code: <ulink url="&websvn;doc/developer-reference/listener/modrdn.py"/>

			<programlisting language="python"><xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="modrdn.py" parse="text"/></programlisting>
			<warning>
				<para>
					Please be aware that tracking the two subsequent calls for <literal>modrdn</literal> in memory might cause duplicates, in case the &ucsUDL; is terminated while such an operation is performed.
					If this is critical, the state should be stored persistently into a temporary file.
				</para>
			</warning>
		</section>

		<section id="listener:example:user">
			<title>Full Example with Packaging</title>
			<para>
				The following example shows a listener module, which logs all changes to users into the file <filename>/tmp/UserList.txt</filename>.
			</para>

			</para>
			<programlisting language="python"><xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="printusers/printusers.py" parse="text"/></programlisting>
			<para>
				Some comments on the code:
			</para>
			<itemizedlist>
				<listitem>
					<simpara>
						Overwriting <varname>__package__</varname> is currently necessary, as the &ucsUDL; imports the listener module by its own mechanism, which is incompatible with the mechanism normally used by Python itself.
						Be aware, that this might cause problems when using <package>pickle</package>.
						<remark>AREQ: And what's the recommendation?</remark>
					</simpara>
				</listitem>
				<listitem>
					<simpara>
						The <abbrev>LDAP</abbrev> filter is specifically chosen to only match user objects, but not computer objects, which have a uid characteristically terminated by a <literal>$</literal>-sign.
					</simpara>
				</listitem>
				<listitem>

				<listitem>
					<simpara>
						Writing a file as <emphasis>root</emphasis> in a public location like <filename class="directory">/tmp/</filename> should be avoided.
						<remark>AREQ: Why? What's the point here? Relevant?</remark>
						It is only done here to show how it works.
					</simpara>
				</listitem>
				<listitem>
					<simpara>
						To test this run a command like <command>tail -f /tmp/UserList.txt &amp;</command>.
						Then create a new user or modify the <emphasis>lastname</emphasis> of an existing one to trigger the module.
					</simpara>
				</listitem>
			</itemizedlist>

		</section>

		<section id="listener:example:setdata">
			<title>A Little Bit more Object Oriented</title>
			<para>
				For larger modules it might be preferrable to use a more object oriented design like the following example, which logs referential integrity violations into a file.
			</para>
			<para>
				Source code: <ulink url="&websvn;doc/developer-reference/listener/obj.py"/>

	</section>

	<section id="listener:details">
		<title>Technical Details
			<indexterm><primary>Directory Listener</primary><secondary>Debug</secondary></indexterm>
		</title>


		-->

		<section id="listener:details:credentials">
			<title>User-ID and Credentials
				<indexterm><primary>Directory Listener</primary><secondary>Credentials</secondary></indexterm>
			</title>
			<para>
				The Listener runs with the effective permissions of the user <literal>listener</literal>.
				If <literal>root</literal>-privileges are required, <function>listener.setuid()</function> can be used to switch the effective UID.
				When done, <function>listener.unsetuid()</function> should be called to drop back to the <literal>listener</literal> UID. It's best practice to code this as <literal>try</literal>/<literal>finally</literal> clauses in Python.
			</para>
		</section>

		<section id="listener:details:cache">
			<title>Internal Cache
				<indexterm><primary>Directory Listener</primary><secondary>Cache</secondary></indexterm>
			</title>
			<para>

							This is required when a new module is added, which is invoked for all already existing objects when the &ucsUDL; is restarted.
						</para>
						<para>
							On domain controllers the cache could be replaced by doing a query to the local LDAP server, before the new values are written into it.
							But &ucsMember; do not have a local LDAP server, so there the cache is needed.
							Also note that the cache keeps track of the associated listener modules, which is not available from the LDAP.
						</para>
					</listitem>
				</varlistentry>

					<term><filename>notifier_id</filename></term>
					<listitem>
						<para>
							This file contains the last <firstterm>notifier ID</firstterm> read from the &ucsUDN;.
						</para>
					</listitem>
				</varlistentry>

					<listitem>
						<para>
							For each module the directory contains a text file consisting of a single number.
							The name of the file is derived from the values of the variable <varname>name</varname> as defined in each listener module.
							The number is to be interpreted as a bit-field of <constant>HANDLER_INITIALIZED=0x1</constant> and <constant>HANDLER_READY=0x2</constant>.
							If both bits are set, it indicates that the module was successfully initialized by running the function <function>initialize()</function>.
							Otherwise both bits are unset.

			</variablelist>
			<para>
				The package <package>univention-directory-listener</package> contains several commands useful for controlling and debugging problems with the &ucsUDL;.
				This can be useful for debugging listener cache inconsistencies.
			</para>

			<section id="listener:commands:ctrl">
				<title>univention-directory-listener-ctrl</title>
				<para>
					The command <command>univention-directory-listener-ctrl resync <replaceable>name</replaceable></command> can be used to reset and re-initialize a module.
					It stops any currently running listener process, removes the state file for the specified module and starts the listener process again.
					This forces the functions <function>clean()</function> and <function>initialize()</function> to be called one after the other.
				</para>
			</section>


				<para>
					The command <command>univention-directory-listener-dump</command> can be used to dump the cache file <filename>/var/lib/univention-directory-listener/cache.db</filename>.
					The &ucsUDL; must be stopped first by invoking <command>service univention-directory-listener stop</command>.
					It outputs the cache in format compatible to the <abbrev>LDAP</abbrev> Data Interchange Format (<abbrev>LDIF</abbrev>).
				</para>
			</section>


					The command <command>univention-directory-listener-verify</command> can be used to compare the content of the cache file <filename>/var/lib/univention-directory-listener/cache.db</filename> to the content of an <abbrev>LDAP</abbrev> server.
					The &ucsUDL; must be stopped first by invoking <command>service univention-directory-listener stop</command>.
					<abbrev>LDAP</abbrev> credentials must be supplied at the command line.
					For example, the following command would use the machine password:
					<programlisting language="sh">
univention-directory-listener-verify \
	-b "$(ucr get ldap/base)" \

					The command <command>/usr/share/univention-directory-listener/get_notifier_id.py</command> can be used to get the latest ID from the notifier.
					This is done by querying the &ucsUDN; running on the <abbrev>LDAP</abbrev> server configured through the &ucsUCRV; <envar>ldap/master</envar>.
					The returned value should be equal to the value currently stored in the file <filename>/var/lib/univention-directory-listener/notifier_id</filename>.
					Otherwise the &ucsUDL; might still be processing a transaction or it might indicate a problem with the &ucsUDL;
				</para>
			</section>
		</section>

				The Listener/Notifier mechanism is used to trigger arbitrary actions when changes occur in the <abbrev>LDAP</abbrev> directory service.
				In addition to the <abbrev>LDAP</abbrev> server <command>slapd</command> it consists of two other services:
				The &ucsUDN; service runs next to the <abbrev>LDAP</abbrev> server and broadcasts change information to interested parties.
				The &ucsUDL; service listens for those notifications, downloads the changes and runs listener modules performing arbitrary local actions like storing the data in a local <abbrev>LDAP</abbrev> server for replication or generating configuration files for non-<abbrev>LDAP</abbrev>-aware local services.
			</para>
			<figure id="listener:schema"><title>Listener/Notifier mechanism</title>
				<graphic scalefit="1" width="80%" fileref="illustrations/ListenerNotifier.png"/>

				<title>Listener/Notifier prcoedure</title>
				<step>
					<para>
						An <abbrev>LDAP</abbrev> object is modified on the &ucsMaster;.
						Changes initiated on all other system roles are re-directed to the master.
					</para>
				</step>
				<step>
					<para>
						The UCS-specific overlay-module <filename>translog</filename> appends the <abbrev>DN</abbrev> to the file <filename>/var/lib/univention-ldap/listener/listener</filename><footnote><simpara>Referred to as <varname>FILE_NAME_LISTENER</varname>, <varname>TRANSACTION_FILE</varname> in the source code</simpara></footnote>.
					</para>
				</step>
				<step>
					<para>
						The &ucsUDN; watches that file, picks up and removes each line it processed.
						It assigns the next transaction number and writes it into the file <filename>/var/lib/univention-ldap/notify/transaction</filename><footnote><simpara>Referred to as <varname>FILE_NAME_TF</varname> in the source code</simpara></footnote>, including the <abbrev>DN</abbrev> and change type.
						For efficient access by transaction ID the index <filename>transaction.index</filename> is updated.
					</para>
				</step>

				</step>
				<step>
					<para>
						Each Listener triggered in this way queries the Notifier for the latest transaction ID, <abbrev>DN</abbrev> and change type.
						The ID is written into the local file <filename>/var/lib/univention-directory-listener/notifier_id</filename>.
					</para>
				</step>
				<step>
					<para>
						Each Listener opens a connection to the LDAP server running on the UCS system which was used to query the Notifier (configured through &ucsUCRV; <envar>ldap/master</envar> and <envar>ldap/server/addition:</envar>).
						It tries to retrieve the latest state of the object identified through the <abbrev>DN</abbrev>, which might be blocked by <firstterm>selective replication</firstterm>.
						If that fails and nothing is returned, the process stops here and closes the <abbrev>LDAP</abbrev> connection.
						<remark>AREQ: Uh, really? A d operation should be processed as well? Maybe rephrase "the process"?</remark>
					</para>
				</step>
				<step>
					<para>
						On a &ucsBackup; the &ucsUDL; writes the transaction data to the file <filename>/var/lib/univention-ldap/listener/listener</filename><footnote><simpara>Referred to as <varname>FILE_NAME_LISTENER</varname>, <varname>TRANSACTION_FILE</varname> in the source code</simpara></footnote> to allow the &ucsUDN; to be cascaded.
						This is automatically internally configured with the option <option>-o</option> of <command>univention-directory-listener</command> and is done for load balancing and failover reasons.
					</para>
				</step>
				<step>
					<para>
						The old state of the object is fetched from the local listener cache.
					</para>
				</step>
				<step>
					<para>
						For each module it is checked, if either the old or new state of the object matches the <varname>filter</varname> and <varname>attributes</varname> specified in the corresponding Python variables.
						If not, the module is skipped.
					</para>
				</step>
				<step>
					<para>
						If the function <function>prerun()</function> of module was not called yet, this is done to signal the start of changes.
					</para>
				</step>
				<step>
					<para>
						The function <function>handler()</function> specified in the module is called, passing in the <abbrev>DN</abbrev> and the old and new state.
								In previous releases of the &ucsUDL; modules could modify the new state, which was written to the local <abbrev>LDAP</abbrev> server by the last module.
								This lead to some very strange problems and was disabled therefore.
								Each module now gets its own new copy of the values and thus can no longer communicate with subsequent modules.
						</simpara></footnote>
					</para>
				</step>
				<step>
					<para>
						The main listener process updates its cache with the new values, including the names of the modules which successfully handled that object.
						This guarantees that the module is still called, even when the filter criteria would no longer match the object after modification.
					</para>
				</step>

				<step>
					<para>
						After 15 seconds of inactivity the function <function>postrun()</function> is invoked for all prepared modules.
						This signals a break in the stream of changes and requests the module to release its resources and/or start pending operations.
					</para>
				</step>
			</procedure>




			handler_schema()
		else:
			handler_add(dn, new)
	elif "z" == command:
		pass  # ignore
	else:
		pass  # ignore, reserved for future use


def handler_move(old_dn, old, new_dn, dn):
	"""Handle rename or move of object."""
	pass	## replace this


def handlee_schema():
	"""Handle change in LDAP schema."""
	pass	## replace this




	elif not new and old:
		handler_remove(dn, old)
	else:
		pass  # ignore


def handler_add(dn, new):
	"""Handle addition of object."""
	pass	# replace this


def handler_modify(dn, old, new):
	"""Handle modification of object."""
	pass	# replace this


def handler_remove(dn, old):
	"""Handle removal of object."""
	pass	# replace this

Return to bug 29420