Bug 46765 - replace legacy create_ou with script using ucsschool.lib
replace legacy create_ou with script using ucsschool.lib
Product: UCS@school
Classification: Unclassified
Component: Import scripts
UCS@school 4.3
Other Linux
: P5 normal (vote)
: UCS@school 4.3 v6
Assigned To: Ole Schwiegert
Jürn Brodersen
: 42676 (view as bug list)
Depends on: 47408
Blocks: 47967
  Show dependency treegraph
Reported: 2018-04-04 11:13 CEST by Daniel Tröder
Modified: 2019-03-01 21:41 CET (History)
5 users (show)

See Also:
What kind of report is it?: Development Internal
What type of bug is this?: ---
Who will be affected by this bug?: ---
How will those affected feel about the bug?: ---
User Pain:
Enterprise Customer affected?:
School Customer affected?:
ISV affected?:
Waiting Support:
Flags outvoted (downgraded) after PO Review:
Ticket number:
Bug group (optional):
Max CVSS v3 score:


Note You need to log in before you can comment on or make changes to this bug.
Description Daniel Tröder univentionstaff 2018-04-04 11:13:26 CEST
Currently /usr/share/ucs-school-import/scripts/create_dc is a symlink to ucs-school-import, a legacy import script for all sorts of things, that does not use the ucsschool.lib.

The ucsschool.lib contains the functionality to create school OUs, but it is used only  by the UMC wizards.

ucs-tests verify that both legacy code and ucsschool.lib code produce valid OUs.

Remove the legacy script code and replace it with a cmdline tool that uses the ucsschool.lib code.
Comment 1 Daniel Tröder univentionstaff 2018-04-04 11:15:44 CEST
Path in 1st line of description should be ".../create_ou".
Comment 2 Ole Schwiegert univentionstaff 2018-07-09 12:18:41 CEST
As discussed in the meeting the script will be extended to be able to read in a csv file of ou's to create
Comment 3 Ole Schwiegert univentionstaff 2018-07-10 11:06:03 CEST
Little progress note for dev:

When creating ou with ucschool.lib and an existing educational_dc following Exception occurs:

./create_ou TEST master --displayName=TT
Traceback (most recent call last):
  File "./create_ou", line 139, in <module>
  File "./create_ou", line 132, in main
  File "/usr/lib/pymodules/python2.7/ucsschool/lib/models/base.py", line 435, in create
    success = self.create_without_hooks(lo, validate)
  File "/usr/lib/pymodules/python2.7/ucsschool/lib/models/school.py", line 389, in create_without_hooks
  File "/usr/lib/pymodules/python2.7/ucsschool/lib/models/school.py", line 280, in add_host_to_dc_group
  File "/usr/lib/pymodules/python2.7/ucsschool/lib/models/base.py", line 435, in create
    success = self.create_without_hooks(lo, validate)
  File "/usr/lib/pymodules/python2.7/ucsschool/lib/models/base.py", line 463, in create_without_hooks
    self.do_create(udm_obj, lo)
  File "/usr/lib/pymodules/python2.7/ucsschool/lib/models/computer.py", line 87, in do_create
    return super(SchoolDCSlave, self).do_create(udm_obj, lo)
  File "/usr/lib/pymodules/python2.7/ucsschool/lib/models/base.py", line 483, in do_create
  File "/usr/lib/pymodules/python2.7/univention/admin/handlers/__init__.py", line 538, in create
    dn = self._create(response=response, serverctrls=serverctrls)
  File "/usr/lib/pymodules/python2.7/univention/admin/handlers/__init__.py", line 1136, in _create
  File "/usr/lib/pymodules/python2.7/univention/admin/handlers/computers/domaincontroller_slave.py", line 584, in _ldap_modlist
    raise univention.admin.uexceptions.uidAlreadyUsed(': %s' % requested_uid)
univention.admin.uexceptions.uidAlreadyUsed: : master$
Comment 4 Ole Schwiegert univentionstaff 2018-07-30 09:42:02 CEST
The script itself is done, but not fully functional yet since there is a problem with the ucs school lib and the creation of school dc's when they already exist. I created a new bug which this one depends on.
Comment 5 Ole Schwiegert univentionstaff 2018-07-31 08:30:34 CEST
A new script has been created in oschwieg/4.3/46765
It needs thorough testing before merging though since it has to replace the old script as precisely as possible.
Comment 6 Sönke Schwardt-Krummrich univentionstaff 2018-08-30 14:31:50 CEST
Please do *NOT* merge this issue to "4.3" branch before the "4.3v5" release has been made!
Comment 7 Jürn Brodersen univentionstaff 2018-09-12 11:21:39 CEST
Please change the old "## INIT LDAP CONNECTION" to use the uldap lib:
lo, po = univention.admin.uldap.getAdminConnection()

Not that important for this script, but could you throw exceptions in the create_ou function instead of just printing errors, also print the error to stderr.

Some mixed tabs and spaces. (Use tabs...) :)

I would like to see the school validation moved into the schoollib. But I think that needs to be discussed with the others.
Comment 8 Sönke Schwardt-Krummrich univentionstaff 2018-09-27 17:13:47 CEST
(In reply to Jürn Brodersen from comment #7)
> I would like to see the school validation moved into the schoollib. But I
> think that needs to be discussed with the others.

"school validation" → "OU name validation" or "all items for an OU are present and correct"?

In both cases I would also prefer to integrate this code in the UCS@school lib.
Comment 9 Ole Schwiegert univentionstaff 2018-10-02 09:38:48 CEST
Package: ucs-school-lib
Version: 11.0.1-23A~
Comment 10 Ole Schwiegert univentionstaff 2018-10-02 09:39:37 CEST
sorry, closed the wrong bug!
Comment 11 Ole Schwiegert univentionstaff 2018-10-04 12:35:18 CEST
Package: ucs-school-import
Version: 16.0.2-57A~

Package: ucs-school-lib
Version: 11.0.1-24A~
Comment 12 Ole Schwiegert univentionstaff 2018-10-04 12:38:02 CEST
There remained some validation code in the create_ou script to determine if given share file servers for classes and home dirs exist. This code should be reconsidered once the work on https://trello.com/c/qOTS44Wn/265-schulservernamen-beim-anlegen-einer-schule-nicht-mehr-abfragen-default-f%C3%BCr-sambahomepath-%C3%A4ndern is done
Comment 13 Daniel Tröder univentionstaff 2018-10-05 08:37:17 CEST
The change broke the jenkins tests (no idea if the tests or the code is wrong): http://jenkins.knut.univention.de:8080/job/UCSschool-4.3/job/Upgrade%20Singleserver/266/#showFailuresLink
Comment 14 Ole Schwiegert univentionstaff 2018-10-05 09:43:20 CEST
Package: ucs-school-import
Version: 16.0.2-58A~

90_ucsschool.34_import-users_via_cli.test fixed

One function call was missing a parameter. The other tests fail because of the new additional validation step included in the School class. The hostname of educational and administrative dc's must not be equal and produce validation errors now. This was discussed with Sönke. The remaining tests will be adapted.
Comment 15 Ole Schwiegert univentionstaff 2018-10-05 10:21:29 CEST
Package: ucs-school-lib
Version: 11.0.1-25A~

On my local test machine the tests aborting with School verification erros due to equal names for educative and administrative dcs all pass. I have no idea where the error comes from, since the schools seem to be created with distinct names. Only thing imaginable is that both fields are None and thus seem to be 'equal'. I added an additional condition to the verification to prevent None entries to produce an error.
Comment 16 Jürn Brodersen univentionstaff 2018-10-30 13:11:38 CET
Small change to the script

[4.3 6e96cd6e8] Bug #46765: Don't print OU was skipped if it wasn't in create_ou script
[4.3 0cec61b7a] Bug #46765: YAML

What I tested:
Create ou on single server -> OK
Create ou on multi sever -> OK
Use existing dc -> OK
Use csv file -> OK
Comment 17 Daniel Tröder univentionstaff 2018-11-01 13:13:10 CET
90_ucsschool/30_import-create_ou_via_cli (and 34_import-users_via_*) still fail.
The "--sharefileserver" argument produces an error.

root@m126:~# udm computers/domaincontroller_slave create --set name=server1
Object created: cn=server1,dc=uni,dc=dtr

root@m126:~# /usr/share/ucs-school-import/scripts/create_ou myou --sharefileserver=wlfr47qvux
need to create container ou=my,dc=uni,dc=dtr
creating object ou=my,dc=uni,dc=dtr

root@m126:~# udm computers/domaincontroller_slave create --set name=server2
Object created: cn=server2,dc=uni,dc=dtr

root@m126:~# /usr/share/ucs-school-import/scripts/create_ou myou2 --sharefileserver=server2

Create OU: myou2
The following fields reported errors during validation:
class_share_file_server: ['Not a valid LDAP DN']
home_share_file_server: ['Not a valid LDAP DN']

Skip OU: myou2

root@m126:~# echo $?

The exit code must be !=0 if there was an error.
Comment 18 Ole Schwiegert univentionstaff 2018-11-07 09:35:55 CET
Package: ucs-school-import
Version: 16.0.2-61A~
Branch: ucs_4.3-0
Scope: ucs-school-4.3

Exit code is now !=0 if there are errors for at least one ou
Comment 19 Jürn Brodersen univentionstaff 2018-11-08 23:52:12 CET
are still failing

The old script resolved the shareserver hostname to a dn.
Comment 20 Ole Schwiegert univentionstaff 2018-11-12 10:44:28 CET
Oi, I overlooked the handling of the share file servers!

Package: ucs-school-import
Version: 16.0.2-62A~
Branch: ucs_4.3-0
Scope: ucs-school-4.3

I recreated now the handling from the old script:

IF NOT share_name: share_name = (final) edu_name

search for existing dc and use that dn
ELSE IF: share_name == dc{ouname}-01 or edu_name: create dn under the ou (object has to be created later)
ELSE: Use configRegistry.get('ldap/hostdn') as share_name
Comment 21 Ole Schwiegert univentionstaff 2018-11-12 10:46:37 CET
Also: I realized that if the ucsschool.lib is used to create a School and the *_share_file_servers are set to non existing dn's the servers are changed to the edu_server upon creation/modification. If that is unintended behavior we should open a new bug for that.
Comment 22 Ole Schwiegert univentionstaff 2018-11-13 12:57:04 CET
*** Bug 42676 has been marked as a duplicate of this bug. ***
Comment 23 Ole Schwiegert univentionstaff 2018-11-13 14:13:46 CET
Package: ucs-school-import
Version: 16.0.2-63A~
Branch: ucs_4.3-0
Scope: ucs-school-4.3

Included code for shell script hooks in create_ou script. (Bug #48141)
Comment 24 Jürn Brodersen univentionstaff 2018-11-14 09:49:31 CET
Tests look good: OK :)
Creating OUs is possible: OK
Comment 25 Sönke Schwardt-Krummrich univentionstaff 2018-11-16 11:48:22 CET
UCS@school 4.3 v6 has been released.


If this error occurs again, please clone this bug.