Bug 45023 - [RESTful Import API] create UMC module to manage imports
[RESTful Import API] create UMC module to manage imports
Status: CLOSED FIXED
Product: UCS@school
Classification: Unclassified
Component: HTTP-API
UCS@school 4.2
Other Linux
: P5 normal (vote)
: UCS@school 4.2 (HTTP-API-MVP)
Assigned To: Florian Best
Daniel Tröder
:
: 45456 (view as bug list)
Depends on: 45022 45045
Blocks: 45019 45456 45657 45659 45660 45662 45664 45844
  Show dependency treegraph
 
Reported: 2017-07-17 14:42 CEST by Daniel Tröder
Modified: 2017-12-11 15:42 CET (History)
4 users (show)

See Also:
What kind of report is it?: Feature Request
What type of bug is this?: ---
Who will be affected by this bug?: ---
How will those affected feel about the bug?: ---
User Pain:
Enterprise Customer affected?:
School Customer affected?: Yes
ISV affected?:
Waiting Support:
Ticket number:
Bug group (optional):
Max CVSS v3 score:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Daniel Tröder univentionstaff 2017-07-17 14:42:36 CEST
Create a UMC module that uses the Python client library to manage imports through the RESTful Import API.
Comment 1 Daniel Tröder univentionstaff 2017-07-17 15:08:49 CEST
While in the future the UMC module should run on all server roles, for the MVP it will run on the DC master only!

So please don't spend time on making it usable everywhere, but try to design it in a way that it will not be to hard to do so. And if possible mark the places in the code that will need to be changed.
Comment 2 Daniel Tröder univentionstaff 2017-07-17 15:50:24 CEST
UI description:

* If a user is authorized for importing for more than 1 school, the school to import to should be requested (drop-down).
* The user type should be requested (drop-down). It should be possible to have multiple user types in one CSV file, so there must either be an additional option in the drop-down or a separate checkbox that disables the drop-down.
* File upload button
* A dry-run must always be executed before the actual import. The results of the dry-run must be shown to the user to allow a continue/abort decision.
* The results of all executed import jobs (that the user has access to) should be retrievable:
  - logfiles
  - user-summary-CSV
  - passwords-CSV
* Display a progress report on the currently running import job.
* Display queue content (job names) and length
Comment 3 Florian Best univentionstaff 2017-07-24 17:20:59 CEST
(In reply to Daniel Tröder from comment #2)
> UI description:
> 
> * If a user is authorized for importing for more than 1 school, the school
> to import to should be requested (drop-down).
A request to the Import API will give me the available schools? Or are all schools the user is part of allowed?

> * The user type should be requested (drop-down). It should be possible to
> have multiple user types in one CSV file, so there must either be an
> additional option in the drop-down or a separate checkbox that disables the
> drop-down.
OK. The API should return an empty list or an pseudo entry for all user types.

> * File upload button
OK
> * A dry-run must always be executed before the actual import. The results of
> the dry-run must be shown to the user to allow a continue/abort decision.
OK

> * The results of all executed import jobs (that the user has access to)
> should be retrievable:
>   - logfiles
>   - user-summary-CSV
>   - passwords-CSV
OK

> * Display a progress report on the currently running import job.
There will be a possibility to get information about the currently running dry-run via a request where I can get an information about what steps are currently performed and a rough percentage?

> * Display queue content (job names) and length
OK. What is the length?
Comment 4 Daniel Tröder univentionstaff 2017-07-25 09:30:30 CEST
(In reply to Florian Best from comment #3)
> (In reply to Daniel Tröder from comment #2)
> > UI description:
> > 
> > * If a user is authorized for importing for more than 1 school, the school
> > to import to should be requested (drop-down).
> A request to the Import API will give me the available schools? Or are all
> schools the user is part of allowed?
See below.

> > * The user type should be requested (drop-down). It should be possible to
> > have multiple user types in one CSV file, so there must either be an
> > additional option in the drop-down or a separate checkbox that disables the
> > drop-down.
> OK. The API should return an empty list or an pseudo entry for all user
> types.
IMHO this information disclosure is OK, as the user has already logged in.

But the list of schools and user-types does only make sense in combination (as a structure), because a user may be allowed for (type1+type2)@schoolA and type2@schoolB but not for type1@schoolB. So this would be something like:
{schoolA: [type1, type2], schoolB: [type1]}

I suggest to either retrieve this information from a separate endpoint "/schools/permissions/" or include it in the data of "/schools/" and "/schools/{ou}/" (list and get).

> > * Display a progress report on the currently running import job.
> There will be a possibility to get information about the currently running
> dry-run via a request where I can get an information about what steps are
> currently performed and a rough percentage?
> 
> > * Display queue content (job names) and length
> OK. What is the length?
names  = names  of jobs in state (waiting|running).
length = number of jobs in state (waiting|running).

But this point may be mute, as the UI design may not display any queue.
Comment 5 Florian Best univentionstaff 2017-08-01 16:03:08 CEST
The first version is build:

Package: ucs-school-umc-import
Version: 0.0.0-1A~4.2.0.201708011600
Branch: ucs_4.2-0
Scope: ucs-school-4.2
Comment 6 Florian Best univentionstaff 2017-08-14 13:11:10 CEST
ucs-school-import (15.0.0-33):
r81797 | Bug #45023: added Recommends: ucs-school-umc-import

ucs-school-umc-import (0.0.0-6):
r81968 | Bug #45023: don't display the success page as own page
r81909 | Bug #45023: Update to API changes
r81900 | Bug #45023: improve error handling
r81898 | Bug #45023: improove error handling
r81874 | Bug #45023: translations
r81862 | Bug #45023: text descriptions, download icons, enhancements
r81802 | Bug #45023: enhancements
r81796 | Bug #45023: Update to API changes
r81766 | Various enhancements (Bug #45023)
r81752 | Adapt to API change (Bug #45023)
r81664 | Improove error handling (Bug #45023)
r81663 | Hide "next" button on file upload page (Bug #45023)
r81661 | Parse the status and display in translated form (Bug #45023)
r81659 | Bug #45023: add more security
r81650 | Fix school display name (Bug #45023)
r81596 | Fix closing of error page (Bug #45023)
r81595 | Error handling (Bug #45023)
r81594 | Initial release (Bug #45023)
r81589 | Bug #45023: adapt to API change
r81587 | Bug #45023: first skeletton
Comment 7 Daniel Tröder univentionstaff 2017-09-08 16:49:43 CEST
When the language settings of the UMC is English, the job list is empty. In german it is filled.

Apart from that all is fine.
Comment 8 Daniel Tröder univentionstaff 2017-09-08 16:55:05 CEST
40ucs-school-import-http-api.inst Doesn't set the option.
udm .. create has no argument --append-option.

-               --set description="Default group for UCS@school user imports" \
+               --set description="Default group for UCS@school user imports" || die
+
+       udm groups/group modify "$@" \
+               --dn "cn=$school-import-all,cn=groups,ou=$school${district},$ldap_base"
Comment 9 Daniel Tröder univentionstaff 2017-09-08 17:08:32 CEST
* also in 53importgroup_create
* please reverse the default for ucsschool/import/generate/import/group: the group creation should be on by default.
Comment 10 Daniel Tröder univentionstaff 2017-09-08 17:17:11 CEST
* The OU is lowercased in 53importgroup_create (not in the join script). Please modify it to not change upper/lowercase.

* 53importgroup_create tries to use "$@", but that's not available. The script is run as root however, so the credentials can be retrieved from disk.

* Please add the email address (if possible a clickable mailto: link) to the error window "Please notify the administrator about this via email.".
Comment 11 Florian Best univentionstaff 2017-09-25 14:38:05 CEST
(In reply to Daniel Tröder from comment #10)
> * Please add the email address (if possible a clickable mailto: link) to the
> error window "Please notify the administrator about this via email.".
This is already the case. Did you set ucsschool/import/error/mail-address?
I added a UCR variable description.
Comment 12 Florian Best univentionstaff 2017-09-25 15:09:24 CEST
Thank you, I addresses all points.

ucs-school-umc-import (0.0.0-7):
a0a8993e52e9 | Bug #45023: fix locale parsing
ec4ac53d6faf | Bug #45023: add UCR variable description

ucs-school-import (15.0.0-45):
601c56a36615 | Bug #45023: fix various errors during import group creation
Comment 13 Daniel Tröder univentionstaff 2017-09-26 17:07:57 CEST
(In reply to Florian Best from comment #11)
> (In reply to Daniel Tröder from comment #10)
> > * Please add the email address (if possible a clickable mailto: link) to the
> > error window "Please notify the administrator about this via email.".
> This is already the case. Did you set ucsschool/import/error/mail-address?
> I added a UCR variable description.
Ah yes - thank you.

Everything looks good and the UMC module works nicely. I just found some minor things to fix:

* There is a small error in the link resulting in a mail message body:
---
Please examine the reason for this error: %s
---
To: and Subject: are fine.

* Line 220 in 40ucs-school-import-http-api.inst is missing a "\".

* A suggestion for ucs-school-umc-import.univention-config-registry-variables:
-Description[en]=This e-mail address is displayed to the user when an error during the import happens.
+Description[en]=This e-mail address is displayed to the user when an error happens during the import.

* Please create in 40ucs-school-import-http-api.inst a UMC policy to allow the use of the UMC module and attach it to all $school-import-all groups. See section "create a UMC policy" in ucs-school-import/README.md.
Comment 14 Florian Best univentionstaff 2017-09-26 19:44:27 CEST
Everything done:

ucs-school-import (15.0.0-46):
fe35e1bfe77e | Bug #45023: create UMC policy with permissions to schoolimport module for the import groups

ucs-school-umc-import (0.0.0-8):
a19ba29aa6dc | Bug #45023: fix error messages in mail templates
Comment 15 Daniel Tröder univentionstaff 2017-09-27 08:56:17 CEST
All OK, except this didn't change:
---
Please examine the reason for this error: %s
---
Comment 16 Florian Best univentionstaff 2017-09-27 11:45:52 CEST
(In reply to Daniel Tröder from comment #15)
> All OK, except this didn't change:
> ---
> Please examine the reason for this error: %s
> ---

You did not remove your browser cache.
Comment 17 Daniel Tröder univentionstaff 2017-09-27 11:53:40 CEST
(In reply to Florian Best from comment #16)
> (In reply to Daniel Tröder from comment #15)
> > All OK, except this didn't change:
> > ---
> > Please examine the reason for this error: %s
> > ---
> 
> You did not remove your browser cache.
That's it. Excellent.
Comment 18 Florian Best univentionstaff 2017-10-11 12:54:09 CEST
*** Bug 45456 has been marked as a duplicate of this bug. ***
Comment 19 Sönke Schwardt-Krummrich univentionstaff 2017-10-16 21:32:09 CEST
UCS@school 4.2 v4 has been released.

http://docs.software-univention.de/changelog-ucsschool-4.2v4-de.html

If this error occurs again, please clone this bug.