Univention Bugzilla – Bug 45023
[RESTful Import API] create UMC module to manage imports
Last modified: 2017-12-11 15:42:08 CET
Create a UMC module that uses the Python client library to manage imports through the RESTful Import API.
While in the future the UMC module should run on all server roles, for the MVP it will run on the DC master only! So please don't spend time on making it usable everywhere, but try to design it in a way that it will not be to hard to do so. And if possible mark the places in the code that will need to be changed.
UI description: * If a user is authorized for importing for more than 1 school, the school to import to should be requested (drop-down). * The user type should be requested (drop-down). It should be possible to have multiple user types in one CSV file, so there must either be an additional option in the drop-down or a separate checkbox that disables the drop-down. * File upload button * A dry-run must always be executed before the actual import. The results of the dry-run must be shown to the user to allow a continue/abort decision. * The results of all executed import jobs (that the user has access to) should be retrievable: - logfiles - user-summary-CSV - passwords-CSV * Display a progress report on the currently running import job. * Display queue content (job names) and length
(In reply to Daniel Tröder from comment #2) > UI description: > > * If a user is authorized for importing for more than 1 school, the school > to import to should be requested (drop-down). A request to the Import API will give me the available schools? Or are all schools the user is part of allowed? > * The user type should be requested (drop-down). It should be possible to > have multiple user types in one CSV file, so there must either be an > additional option in the drop-down or a separate checkbox that disables the > drop-down. OK. The API should return an empty list or an pseudo entry for all user types. > * File upload button OK > * A dry-run must always be executed before the actual import. The results of > the dry-run must be shown to the user to allow a continue/abort decision. OK > * The results of all executed import jobs (that the user has access to) > should be retrievable: > - logfiles > - user-summary-CSV > - passwords-CSV OK > * Display a progress report on the currently running import job. There will be a possibility to get information about the currently running dry-run via a request where I can get an information about what steps are currently performed and a rough percentage? > * Display queue content (job names) and length OK. What is the length?
(In reply to Florian Best from comment #3) > (In reply to Daniel Tröder from comment #2) > > UI description: > > > > * If a user is authorized for importing for more than 1 school, the school > > to import to should be requested (drop-down). > A request to the Import API will give me the available schools? Or are all > schools the user is part of allowed? See below. > > * The user type should be requested (drop-down). It should be possible to > > have multiple user types in one CSV file, so there must either be an > > additional option in the drop-down or a separate checkbox that disables the > > drop-down. > OK. The API should return an empty list or an pseudo entry for all user > types. IMHO this information disclosure is OK, as the user has already logged in. But the list of schools and user-types does only make sense in combination (as a structure), because a user may be allowed for (type1+type2)@schoolA and type2@schoolB but not for type1@schoolB. So this would be something like: {schoolA: [type1, type2], schoolB: [type1]} I suggest to either retrieve this information from a separate endpoint "/schools/permissions/" or include it in the data of "/schools/" and "/schools/{ou}/" (list and get). > > * Display a progress report on the currently running import job. > There will be a possibility to get information about the currently running > dry-run via a request where I can get an information about what steps are > currently performed and a rough percentage? > > > * Display queue content (job names) and length > OK. What is the length? names = names of jobs in state (waiting|running). length = number of jobs in state (waiting|running). But this point may be mute, as the UI design may not display any queue.
The first version is build: Package: ucs-school-umc-import Version: 0.0.0-1A~4.2.0.201708011600 Branch: ucs_4.2-0 Scope: ucs-school-4.2
ucs-school-import (15.0.0-33): r81797 | Bug #45023: added Recommends: ucs-school-umc-import ucs-school-umc-import (0.0.0-6): r81968 | Bug #45023: don't display the success page as own page r81909 | Bug #45023: Update to API changes r81900 | Bug #45023: improve error handling r81898 | Bug #45023: improove error handling r81874 | Bug #45023: translations r81862 | Bug #45023: text descriptions, download icons, enhancements r81802 | Bug #45023: enhancements r81796 | Bug #45023: Update to API changes r81766 | Various enhancements (Bug #45023) r81752 | Adapt to API change (Bug #45023) r81664 | Improove error handling (Bug #45023) r81663 | Hide "next" button on file upload page (Bug #45023) r81661 | Parse the status and display in translated form (Bug #45023) r81659 | Bug #45023: add more security r81650 | Fix school display name (Bug #45023) r81596 | Fix closing of error page (Bug #45023) r81595 | Error handling (Bug #45023) r81594 | Initial release (Bug #45023) r81589 | Bug #45023: adapt to API change r81587 | Bug #45023: first skeletton
When the language settings of the UMC is English, the job list is empty. In german it is filled. Apart from that all is fine.
40ucs-school-import-http-api.inst Doesn't set the option. udm .. create has no argument --append-option. - --set description="Default group for UCS@school user imports" \ + --set description="Default group for UCS@school user imports" || die + + udm groups/group modify "$@" \ + --dn "cn=$school-import-all,cn=groups,ou=$school${district},$ldap_base"
* also in 53importgroup_create * please reverse the default for ucsschool/import/generate/import/group: the group creation should be on by default.
* The OU is lowercased in 53importgroup_create (not in the join script). Please modify it to not change upper/lowercase. * 53importgroup_create tries to use "$@", but that's not available. The script is run as root however, so the credentials can be retrieved from disk. * Please add the email address (if possible a clickable mailto: link) to the error window "Please notify the administrator about this via email.".
(In reply to Daniel Tröder from comment #10) > * Please add the email address (if possible a clickable mailto: link) to the > error window "Please notify the administrator about this via email.". This is already the case. Did you set ucsschool/import/error/mail-address? I added a UCR variable description.
Thank you, I addresses all points. ucs-school-umc-import (0.0.0-7): a0a8993e52e9 | Bug #45023: fix locale parsing ec4ac53d6faf | Bug #45023: add UCR variable description ucs-school-import (15.0.0-45): 601c56a36615 | Bug #45023: fix various errors during import group creation
(In reply to Florian Best from comment #11) > (In reply to Daniel Tröder from comment #10) > > * Please add the email address (if possible a clickable mailto: link) to the > > error window "Please notify the administrator about this via email.". > This is already the case. Did you set ucsschool/import/error/mail-address? > I added a UCR variable description. Ah yes - thank you. Everything looks good and the UMC module works nicely. I just found some minor things to fix: * There is a small error in the link resulting in a mail message body: --- Please examine the reason for this error: %s --- To: and Subject: are fine. * Line 220 in 40ucs-school-import-http-api.inst is missing a "\". * A suggestion for ucs-school-umc-import.univention-config-registry-variables: -Description[en]=This e-mail address is displayed to the user when an error during the import happens. +Description[en]=This e-mail address is displayed to the user when an error happens during the import. * Please create in 40ucs-school-import-http-api.inst a UMC policy to allow the use of the UMC module and attach it to all $school-import-all groups. See section "create a UMC policy" in ucs-school-import/README.md.
Everything done: ucs-school-import (15.0.0-46): fe35e1bfe77e | Bug #45023: create UMC policy with permissions to schoolimport module for the import groups ucs-school-umc-import (0.0.0-8): a19ba29aa6dc | Bug #45023: fix error messages in mail templates
All OK, except this didn't change: --- Please examine the reason for this error: %s ---
(In reply to Daniel Tröder from comment #15) > All OK, except this didn't change: > --- > Please examine the reason for this error: %s > --- You did not remove your browser cache.
(In reply to Florian Best from comment #16) > (In reply to Daniel Tröder from comment #15) > > All OK, except this didn't change: > > --- > > Please examine the reason for this error: %s > > --- > > You did not remove your browser cache. That's it. Excellent.
*** Bug 45456 has been marked as a duplicate of this bug. ***
UCS@school 4.2 v4 has been released. http://docs.software-univention.de/changelog-ucsschool-4.2v4-de.html If this error occurs again, please clone this bug.