Univention Bugzilla – Bug 47316
create simple and extensible API for UDM
Last modified: 2023-05-31 19:55:48 CEST
The Python interface for UDM is difficult and error prone to use. Create a Python API for UDM clients that can be quickly learned and is easy to use. The implementation should allow the backend to evolve without exposing the change to the client. → Define a Python API. → Write an implementation of the interface as a wrapper for the existing UDM code. → Write API documentation. → Write ucs-tests for the API implementation.
A proposed API can be found in git branch dtroeder/simple_udm43 (branched off 4.3-1). The code does not modify existing UDM code, but adds new classes, tests and initial documentation. * management/univention-directory-manager-modules/modules/univention/udm/ * management/univention-directory-manager-modules/doc/ * test/ucs-test/tests/59_udm/5?_udm_api_* The function signatures are fully documented with sphinx-compatible docstrings and mypy annotations. A class diagram shows how which classes are derived from each other and which are composited. (Please ignore UDM_Layers.*. It is a fantasy of a remote UDM architecture and has nothing to do with this bug.) The implementation uses a variety of factory concepts to abstract the creation of * LDAP connection objects * UDM modules * UDM objects The *API* looks like this: from univention.udm import Udm # Get a UDM module factory using the specified LDAP connection method: # ------------------------------------------------------------------ udm = Udm.using_admin() | Udm.using_machine() | Udm.using_credentials() # Get a UDM module (a UDM object factory) of the specified type: # ------------------------------------------------------------------ user_mod = udm.get('users/user') # Get existing UDM objects: # ------------------------------------------------------------------ obj = user_mod.get(dn) objs = user_mod.search('pwdChangeNextLogin=1') # Create a new UDM object: # ------------------------------------------------------------------ obj = user_mod.new() # Change a UDM objects property: # ------------------------------------------------------------------ obj.props.firstname = 'foo' obj.save() # A UDM object has the following attributes: # ------------------------------------------------------------------ dn # type: str props # type: BaseUdmObjectProperties (a simple container) options # type: List[str] policies # type: List[str] position # type: str # When continuing to use the UDM object, reload it from LDAP, in case UDM hooks changed it: # ------------------------------------------------------------------ obj.refresh() # Move an UDM object: # ------------------------------------------------------------------ obj.position = 'cn=users,cn=example,dc=com' obj.save() # Delete an UDM object: # ------------------------------------------------------------------ obj.delete() The UDM module has a `meta` attribute, which has the following attributes: # ------------------------------------------------------------------ identifying_property # type: str - 'cn' for a group, 'uid' for users, etc auto_open # type: bool (True) - when loading, open() UDM objects lookup_filter # LDAP filter for this module mapping # UDM props to LDAP attrs mapping and vice versa Function calls can be chained: # ------------------------------------------------------------------ obj = Udm.using_credentials('s3cr3t', 'myuser').get('users/user').get(dn) obj.save().refresh()
Please be aware, that in the concrete implementation of the API in the git branch two Module/Object factories exist: * GenericUdm1O* works for all UDM module types, is a UDM 1 wrapper * UsersUserUdm1* works only for users/user, it is derived from GenericUdm1O* UsersUserUdm1* is a showcase for the dynamic factory feature. It is registered in the postinst with: ----- config_storage.register_configuration(UdmModuleFactoryConfiguration(r"^users/user$", "univention.udm.users_user", "UsersUserUdm1Module")) ----- This specialization does one thing: it decodes the property "homePostalAddress" from a list of str to a dict (and encodes it back when saving). As the UDM 1 API should not be changed, this presents the possibility to fix all properties that are... strange or not intuitive. After all that is the purpose of this API! So I'd like to discuss if it wouldn't make sense to look through all properties of all modules and identify those that are really hard to use (like for example translation*Description in settings/extended_attribute etc). IMHO that are not so many. Converting them to well usable dicts is very easy - see UsersUserUdm1*. This is the one chance to make this right.
The UsersUserUdm1 now converts more users/user UDM properties to&from sane values: * disabled: '0' / '1' (strings!) → False / True (bool) * sambaLogonHours: [0, 1, 167] → ['Sun 0-1', 'Sun 1-2', 'Sat 23-24'] (text as seen in the UMC) * birthday: '2018-09-16' → datetime.date(2018, 9, 16) * userexpiry: '2018-09-16' → datetime.date(2018, 9, 16) * jpegPhoto: '/9j/4AAQSkZJRgABAQAAAQABAAD/2wBDAAgGBgcGBQgHBwcJCQgKDB....' (very long string) → Base64BinaryProperty('jpegPhoto') Base64BinaryProperty objects have getters and setters 'encoded' and 'decoded' which will deliver the UDM properties value as Base64 encoded or raw (decoded), and will accept a Base64 encoded string or a raw string, respectively. The programmer can do both: user.props.jpegPhoto.encoded = open('pic.jpg').read() user.save() and user.props.jpegPhoto.decoded = base64.b64encode(open('pic.jpg').read()) user.save() And it will always be saved as a base64 encoded string to LDAP. The getters deliver: obj.props.jpegPhoto.encoded == base64.b64encode(obj.props.jpegPhoto.decoded) True [dtroeder/simple_udm43] ad41037af4 use absolute imports [dtroeder/simple_udm43] 9bcd32e6c7 convert properties disabled, sambaLogonHours, birthday and userexpiry to native Python objects [dtroeder/simple_udm43] 84a59e2c99 transparently handle encoding and decoding of binary values (jpegPhoto)
Properties in settings/portal_entry UDM objects are now also converted to Python objects: * activated: 'TRUE'/'FALSE' → True/False * description: List(List(str))) -> Dict(str): [['fr_FR', 'Page web du portail central du domaine UCS'], ['en_US', 'Central portal web page for the UCS domain']] → {'fr_FR': 'Page web du portail central du domaine UCS', 'en_US': 'Central portal web page for the UCS domain'} * same for displayName * icon: Base64BinaryProperty (see comment3) [dtroeder/simple_udm43 177bb47de5] better attributes for settings/portal_entry
Property conversion can now be done in a concise and simple declarative style. [dtroeder/simple_udm43] acb88fddd5 refactor property encoders, make usage of them declarative
A convenience function was added that allows to get a UDM object with one line: Udm.using_admin().get_obj(dn) It will automatically load the UDM module with the name stored in the "univentionObjectType" LDAP attribute. Udm.using_admin().get_obj('cn=grp,cn=groups,dc=uni,dc=dtr') → GroupsGroupUdm1Object('groups/group', 'cn=grp,cn=groups,dc=uni,dc=dtr') Udm.using_admin().get_obj('cn=UNIVENTION_PING,cn=nagios,dc=uni,dc=dtr') → NagiosServiceUdm1Object('nagios/service', 'cn=UNIVENTION_PING,cn=nagios,dc=uni,dc=dtr') Udm.using_admin().get_obj('uid=francoise,cn=users,dc=uni,dc=dtr') → UsersUserUdm1Object('users/user', 'uid=francoise,cn=users,dc=uni,dc=dtr') Udm.using_admin().get_obj('cn=m125,cn=dc,cn=computers,dc=uni,dc=dtr') → GenericUdm1Object('computers/domaincontroller_master', 'cn=m125,cn=dc,cn=computers,dc=uni,dc=dtr') [dtroeder/simple_udm43] 23432273c9 free namespace for a property named 'encoders' [dtroeder/simple_udm43] 172c69b8d1 add nagios/service [dtroeder/simple_udm43] 5fe04403ca add spcific groups/group module [dtroeder/simple_udm43] 11e9a28603 add convenience function to load a UDM object with one call
It is now possible to navigate through referenced DNs! A property (or policy) that is a DN / list of DNs can be encoded with a subclass of DnPropertyEncoder / DnListPropertyEncoder. That will lead to a string/list object that has an additional member "obj" / "objs". That member is a lazy object! Only when accessed will it create the UDM object(s). The lazy initialization is usually configured to statically create a specific subclass for each UDM module. Additionally a module name "auto" exists, that will make the encoder detect the kind of UDM object through a LDAP query (like Udm().get_obj(DN) does). Because of the additional LDAP query, 'auto' should be avoided if possible. It must however be used in some cases, because a list may contain objects of different UDM modules (eg computer/* or policies/*). It is important that the object is loaded lazily, because a) each object costs memory, b) its costs an LDAP query and c) to prevent an infinit recursion: user_obj.props.groups.objs[0].props -> infinit recursion, if this would now immediately resolve the user DNs of the group (as they contain user_obj.dn). Usage examples: obj = Udm.using_admin().get_obj('uid=Administrator,cn=users,dc=uni,dc=dtr') obj.props.groups # list of DNs → ['cn=Domain Admins,cn=groups,dc=uni,dc=dtr', 'cn=Domain Users,cn=groups,dc=uni,dc=dtr', 'cn=DC Backup Hosts,cn=groups,dc=uni,dc=dtr', ..] obj.props.groups.objs # list of UDM objects → [GroupsGroupUdm1Object('groups/group', 'cn=Domain Admins,cn=groups,dc=uni,dc=dtr'), GroupsGroupUdm1Object('groups/group', 'cn=Domain Users,cn=groups,dc=uni,dc=dtr'), GroupsGroupUdm1Object('groups/group', 'cn=DC Backup Hosts,cn=groups,dc=uni,dc=dtr'), ..] obj.policies → ['cn=default-settings,cn=pwhistory,cn=users,cn=policies,dc=uni,dc=dtr', 'cn=default-udm-self,cn=UMC,cn=policies,dc=uni,dc=dtr'] obj.policies.objs → [GenericUdm1Object('policies/pwhistory', 'cn=default-settings,cn=pwhistory,cn=users,cn=policies,dc=uni,dc=dtr'), GenericUdm1Object('policies/umc', 'cn=default-udm-self,cn=UMC,cn=policies,dc=uni,dc=dtr')] obj.props.memberOf.objs[0].props.users.objs → [UsersUserUdm1Object('users/user', 'uid=Administrator,cn=users,dc=uni,dc=dtr')] obj.props.memberOf.objs[0].props.users.objs[0].props.username → 'Administrator' grp = obj.props.memberOf.objs[0].props.users.objs[0].props.groups.objs[-1] grp → GroupsGroupUdm1Object('groups/group', 'cn=wusbo-1a,cn=klassen,cn=schueler,cn=groups,ou=wusbo,dc=uni,dc=dtr') grp.props.hosts → ['cn=m125,cn=dc,cn=computers,dc=uni,dc=dtr'] grp.props.hosts.objs → [GenericUdm1Object('computers/domaincontroller_master', 'cn=m125,cn=dc,cn=computers,dc=uni,dc=dtr')] [dtroeder/simple_udm43] 44229fff8f add navigation through 'obj'/'objs' member for DNs/lists of DNs
The lazy object loading introduces a package requirement for "python-lazy-object-proxy", which is currently in unmaintained.
* API version support was added. * The factory configuration module now verifies modules before adding them. Version 0 is without encoders. Version 1 is with encoder support. Each module bears a list of API versions it supports. That is checked, when a module is requested. The documentation tells the developer to use a fixed API version. The tests were adapted to the module move. [dtroeder/simple_udm43] 13dd886f53 add API version support [dtroeder/simple_udm43] 228215209c move UDM modules to separate package [dtroeder/simple_udm43] ab7376f462 adapt tests to modules move
Work on ucs-tests [dtroeder/simple_udm43] daf5524b89 remove '.py' from test file names [dtroeder/simple_udm43] c704a0a5fa reduce docstring by moving code to test [dtroeder/simple_udm43] 42999e5169 cleanup tests [dtroeder/simple_udm43] 039f27e28b fix typo, fix missed move [dtroeder/simple_udm43] 2b41e3fd79 add test for LDAP connection initialization
I made a mistake with the dtroeder/simple_udm43, so that a merge to 4.3-2 would have required a lot of work. So I make a new branch "dtroeder/47316_simple_udm_api", which is branched of freshly from origin/4.3-2. All previous commits have been cherry-picked into branch dtroeder/47316_simple_udm_api. All commit messages have been rewritten to start with "Bug #47316: ".
8180fa1a2a Bug #47316: handle inconsistent univentionObjectType usage on OX systems c395ca8f0d Bug #47316: add API v1 support for all computers/* modules 1216e2560b Bug #47316: add convenience function to object load by uid/cn/etc 01995de6ac Bug #47316: handle deepcopy in udm handlers 6736bc46c6 Bug #47316: add support for sambaGroupType in groups/group ce2d60884d Bug #47316: API v1 module for appcenter/app 4e30b00be5 Bug #47316: handle non-existent UDM module dac9c1908f Bug #47316: encoder that accepts case insensitive str(true/false) values and writes configurable upper/lower case (syntax of TrueFalseUp and TrueFalseUpper does not validate input) 9ef78e4e8f Bug #47316: use case insensitive encoder f70c6a9551 Bug #47316: add API v1 support for saml/serviceprovider and settings/directory a4164dfdbb Bug #47316: add API v1 support for container/* (data container/msgpo requires no conversion) 41780a76cc Bug #47316: add API v1 support for policies/* (all other policies/* require no conversion) bdb2aa3a45 Bug #47316: small fixes 78d7de383d Bug #47316: add support for mail/*
* Changed the API for setting the API version to be more in line with the existing API (using a chained method). * Fixed a caching error that lead to using always the same API version. * Log messages with level >= INFO are now printed to the terminal when using the interactive console. Bug #47316: fix caching error (ignoring API version) Bug #47316: print log messages to terminal when in interactive mode and level>=INFO Bug #47316: set API version through chainable method
Bug #47971 lead to objects created by new() already having displayName (and oxDisplayName etc) set to empty strings, which where later save()ed as such. Implemented a workaround until Bug #47971 is fixed. Bug #47316: workaround Bug #47971: _udm1_object.items() changes object
As a consequence of the previous commit, the properties were all initialized with None. That's just how it is with UDM1 and really bad. Getting the types and defaults into fresh objects without modifying the underlying UDM1 object is a bit tricky, but works now: Bug #47316: part2 of workaround Bug #47971: get type and default value without changing object I added a test to verify initial types (list) and values: Bug #47316: test default values
* Original-UDM errors (univention.admin.uexceptions.*) are now caught and reraised as UdmError (univention.udm.exceptions) retaining the original traceback. * Support for superordinate objects was added. Modules that require them, check that the argument to new() is a valid superordinate object, other modules ignore them. * The LDAP connection configuration method names were shortend. * univention.admin.uldap is not imported in udm.py and base classes anymore. This allows to create UDM modules that use other means (e.g. HTTP) to connect to UDM to not depend on a local UDM installation. * Static type annotations (mypy) were moved to stub files. * Generic types have been added to the static type annotations. * The module config factory has been removed and a simpler and more robust replacement to load modules has been added. * The use of an explicit API version is now enforced. Bug #47316: str2int for mail quota Bug #47316: catch and reraise all UDM errors Bug #47316: Documentation fixes Bug #47316: Documentation fixes Bug #47316: auto_reload after save Bug #47316: Documentation fixes Bug #47316: auto_reload after save Bug #47316: Add default position to modules Bug #47316: * Bug #47316: Reworking using_credentials to use fewer parameters and connect to the UCS master Bug #47316: Renaming some classes Bug #47316: Documentation fixes Bug #47316: auto_reload after save Bug #47316: Add default position to modules Bug #47316: Renaming some classes Bug #47316: Reworking using_credentials to use fewer parameters and connect to the UCS master Bug #47316: Documentation fixes Bug #47316: auto_reload after save Bug #47316: Add default position to modules Bug #47316: Renaming some classes Bug #47316: Reworking using_credentials to use fewer parameters and connect to the UCS master Bug #47316: Documentation fixes Bug #47316: auto_reload after save Bug #47316: Add default position to modules Bug #47316: Renaming some classes Bug #47316: Reworking using_credentials to use fewer parameters and connect to the UCS master Bug #47316: Documentation fixes Bug #47316: auto_reload after save Bug #47316: Add default position to modules Bug #47316: Renaming some classes Bug #47316: Reworking using_credentials to use fewer parameters and connect to the UCS master Bug #47316: Documentation fixes Bug #47316: auto_reload after save Bug #47316: Add default position to modules Bug #47316: Renaming some classes Bug #47316: Reworking using_credentials to use fewer parameters and connect to the UCS master Bug #47316: Documentation fixes Bug #47316: auto_reload after save Bug #47316: Add default position to modules Bug #47316: Renaming some classes Bug #47316: Reworking using_credentials to use fewer parameters and connect to the UCS master Bug #47316: Documentation fixes Bug #47316: auto_reload after save Bug #47316: Add default position to modules Bug #47316: Move LDAP_connection Bug #47316: Renaming some classes Bug #47316: Reworking using_credentials to use fewer parameters and connect to the UCS master Bug #47316: Documentation fixes Bug #47316: auto_reload after save Bug #47316: Add default position to modules Bug #47861: make error messages more accurate Bug #47316: remove unused argument Bug #47316: move mypy annotations to stub file, add stubs for missing methods Bug #47316: Remove RuntimeErrors Bug #47316: Move LDAP_connection Bug #47316: Renaming some classes Bug #47316: Reworking using_credentials to use fewer parameters and connect to the UCS master Bug #47316: Documentation fixes Bug #47316: auto_reload after save Bug #47316: Add default position to modules Bug #47316: make source of error messages distinguishable Bug #47316: Remove RuntimeErrors Bug #47316: Add DeleteError; fix and cache default_position Bug #47316: Documentation fixes Bug #47316: * Bug #47316: reraise with stacktrace Bug #47316: some fixes Bug #47316: prevent recursion Bug #47316: remove uldap and thus concrete UDM backend references from base classes Bug #47316: adapt tests to API changes Bug #47316: fixup Bug #47316: add helpers: get_all_udm_module_names() Bug #47316: move static type annotations into stub files Bug #47316: improve static type annotations Bug #47316: Mark modules for their corresponding udm module Bug #47316: Rename udm.identify_object_by_dn -> udm.obj_by_dn Bug #47316: Remove module config factory Bug #47316: Fix encoders Bug #47316: use generic types in static type hints Bug #47316: workaround ox udm module claiming non-ox objects Bug #47316: handle empty string for an int Bug #47316: style fixes Bug #47316: follow api change Bug #47316: fix annotations Bug #47316: fix AttributeError Bug #47316: adapt ucs-tests Bug #47316: Python modules do not need to be cached anymore Bug #47316: readd caching for UDM modules Bug #47316: enforce use of API version Bug #47316: remove accidentally added file of other project Bug #47316: More generic plugins Bug #47316: Do not pass UDM objects down the line Bug #47316: Default positions for computers Bug #47316: Default positions for computers Bug #47316: add docstrings, annotations Bug #47316: Expose ConnectionError Bug #47316: rename Udm to UDM Bug #47316: shorten LDAP configuration method names Bug #47316: support superordinate Bug #47316: remove "Udm" from class names, as it is already in namespace
(In reply to Daniel Tröder from comment #8) > The lazy object loading introduces a package requirement for > "python-lazy-object-proxy", which is currently in unmaintained. I created a separate bug for this: Bug #48086.
Added support for settings/data (see Bug #47944). Added a property 'content_type' to BinaryProperty objects: list(UDM.admin().version(1).get('settings/data').search())[0].props.data.content_type → namedtuple(mime_type='text/plain', encoding='us-ascii', text='ASCII text') list(UDM.admin().version(1).get('settings/portal').search())[0].props.background.content_type → namedtuple(mime_type='image/png', encoding='binary', text='PNG image data, 280 x 280, 8-bit/color RGBA, non-interlaced') Bug #47316: add module for settings/data Bug #47316: add property 'content_type' to BinaryProperty objects
Bug #47316: fix superordinate passed as GenricObject Bug #47316: use 'default_containers' if they are configured for a UDM module
Bug #47316: allow assigning binary data directly to unset property (In reply to Daniel Tröder from comment #3) > Base64BinaryProperty objects have getters and setters 'encoded' and > 'decoded' which will deliver the UDM properties value as Base64 encoded or > raw (decoded), and will accept a Base64 encoded string or a raw string, > respectively. > > The programmer can do both: > > user.props.jpegPhoto.encoded = open('pic.jpg').read() > user.save() > > and > > user.props.jpegPhoto.decoded = base64.b64encode(open('pic.jpg').read()) > user.save() Binary (not base64 encoded!) data can now be assigned directly to the UDM property: user.props.jpegPhoto = open('picture.jpg').read() user.save() Reading from user.props.jpegPhoto will still yield a Base64BinaryProperty instance.
Added the new API: https://git.knut.univention.de/univention/ucs/compare/3c4c69d239a206e00d9a87f9903c188c12408e76...98b63603bd6aa3ec551b05a2983c1f5cab59af03 See ucs/management/univention-directory-manager-modules/modules/univention/udm/__init__.py for a starting point regarding documentation. There is also a reasonable test coverage: test/ucs-test/tests/59_udm/5*.py
Excellent test coverage! [4.3-2 aecff39362] Bug #47316: handle loading of modules in ucs-test that require a superordinate
[4.3-2 6fb56ecf65] Bug #47316: handle NoObject in Dn*PropertyEncoders
[4.3-2] 74359d5546 Bug #47316: improve repr() [4.3-2] a5d35ed0d7 Bug #47316: move metaclass from generic to base
2f40d0fc76aff4cada37ba3c0564b262624fddce breaks the docker tests! There are tests which remove /etc/ldap.secret before the test and restore the file after the test, but with wrong permissions and user/group ownership. correct -> ls -la /etc/ldap.secret -rw-r----- 1 root DC Backup Hosts 20 Mär 14 2018 /etc/ldap.secret after the test ls -la /etc/machine.secret -rw------- 1 root root 20 Nov 8 14:27 /etc/machine.secret which such a ldap.secret a join into the domain as Administrator is no longer possible (univention-join -> ssh Administrator@master -> udm , udm as Administrator works because Administrator is member of "DC Backup Hosts" and can read ldap.secret) fixed 54_udm_api_ldap_connection.py by rename ldap.secret instead of remove
[4.3-2 63f047533e] Bug #47316: handle invalid DN when moving object
[4.3-2] 50d27eb115 Bug #47316: fix handling of superordinate in API v0 and v1
This Bug blocks the release of a fix for Bug #47872, with user pain close to 0.3. Can we have a quick fix&QA or a revert?
Q: Is the current state of this bug releasable without causing regressions in productive parts of UCS? If so, then please split off remaining improvements to another bug and get this out of the door.
Built univention-directory-manager-modules 13.0.25-13A~4.3.0.201811141246 which should at least not cause any regressions as no parts of UCS use the new files.
Package installed: OK Changelog & Advisory: OK Due to the need to release the package and the fact that only new files were added, I verify this ticket now. Should problems with the new API arise they have to be solved in new tickets. Since no existing functionality was changed or removed, this should be safe for release.
<http://errata.software-univention.de/ucs/4.3/313.html>