Univention Bugzilla – Bug 45022
[RESTful Import API] create Python client library
Last modified: 2017-12-11 15:42:08 CET
Create a Python client library to ease access to the HTTP API.
Is there already anything which can be used?
Not yet, but soon. I'll post updates here as soon as I have commited.
This should have been trivial, as the framework implements the generation of CoreAPI, and thus clients can be auto-generated: http://www.django-rest-framework.org/topics/api-clients/#client-side-core-api But it turns out, that the package version in django-rest-framework in Debian is (once again!) completely outdated and it doesn't work. So we'll have to write a client lib using requests ourselves :/
r81417: created a small Python client library to use the currently available API functions from ucsschool.http_api.client import Client client = Client('Administrator', 'univention', log_level=Client.LOG_RESPONSE) client.get_schools() [{u'displayName': u'Schule Eins', u'url': u'https://m90s4.uni.dtr/api/v1/schools/SchuleEinz/'}, {u'displayName': u'Schule Zwei', u'url': u'https://m90s4.uni.dtr/api/v1/schools/SchuleZwei/'}, {u'displayName': u'testschule', u'url': u'https://m90s4.uni.dtr/api/v1/schools/testschule/'}] client.get_schools('SchuleZwei') {u'displayName': u'Schule Zwei', u'url': u'https://m90s4.uni.dtr/api/v1/schools/SchuleZwei/', u'user_import': u'https://m90s4.uni.dtr/api/v1/schools/SchuleZwei/imports/users'} client.create_importjob(filename='/sync/test_users_small.csv', source_uid='abd213', school='https://m90s4.uni.dtr/api/v1/schools/SchuleZwei/') {u'created': u'2017-07-26T16:05:07.903426Z', u'dryrun': True, u'hooks': [], u'input_file': u'uploads/2017-07-26/test_users_small.csv', u'input_file_type': u'csv', u'log_file': None, u'principal': u'Administrator', u'progress': u'{}', u'result': None, u'school': u'https://m90s4.uni.dtr/api/v1/schools/SchuleZwei/', u'source_uid': u'abd213', u'status': u'Scheduled', u'url': u'https://m90s4.uni.dtr/api/v1/imports/users/9/'} client.get_importjobs() [{u'created': u'2017-07-26T16:05:07.903426Z', u'dryrun': True, u'hooks': [], u'input_file': u'uploads/2017-07-26/test_users_small.csv', u'input_file_type': u'csv', u'log_file': None, u'principal': u'Administrator', u'progress': u'{}', u'result': None, u'school': u'https://127.0.0.1/api/v1/schools/SchuleZwei/', u'source_uid': u'abd213', u'status': u'Scheduled', u'url': u'https://127.0.0.1/api/v1/imports/users/9/'}, {u'created': u'2017-07-26T16:01:03.428138Z', u'dryrun': True, [..] Not marking bug as resolved, as I expect more calls to follow.
Can you give me something like .get_user_types() ?
How do I set the language of the request?
(In reply to Florian Best from comment #5) > Can you give me something like .get_user_types() ? As discussed yesterday, the user types should be hard coded to students, teachers and staff. (In reply to Florian Best from comment #6) > How do I set the language of the request? There is no translatable data except for "progress". That we'll have to represent in a language neutral way, so that the UI can localize it. The general problem with i18n is, that the data is created and results stored in the backend independently from the logged in user. When later retrieving the data any language used in the backend might be wrong. So we should use only language neutral data in the results.
(In reply to Daniel Tröder from comment #7) > (In reply to Florian Best from comment #5) > > Can you give me something like .get_user_types() ? > As discussed yesterday, the user types should be hard coded to students, > teachers and staff. Please use one of ucsschool/http_api/import_api/models.py line 63: USER_STAFF = 'Staff' USER_STUDENT = 'Student' USER_TEACHER = 'Teacher' USER_TEACHER_AND_STAFF = 'Teacher and Staff' USER_ROLES = (USER_STAFF, USER_STUDENT, USER_TEACHER, USER_TEACHER_AND_STAFF) The role attribute is not yet supported. It will be evaluated case insensitive.
>>> from ucsschool.http_api.client import Client Traceback (most recent call last): File "<stdin>", line 1, in <module> ImportError: No module named client # dpkg -L ucs-school-import-http-api-client | grep client.py /usr/share/pyshared/ucsschool/client.py
Using the library I get the following warning: /usr/lib/python2.7/dist-packages/urllib3/connectionpool.py:732: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html (This warning will only appear once by default.) InsecureRequestWarning)
(In reply to Daniel Tröder from comment #7) > (In reply to Florian Best from comment #5) > > Can you give me something like .get_user_types() ? > As discussed yesterday, the user types should be hard coded to students, > teachers and staff. Yes, but not in the client / UMC module. I need a result which only queries the available user types for the user which makes the import. Otherwise he sees a list of user types which he isn't allowed to import.
(In reply to Daniel Tröder from comment #4) > client.create_importjob(filename='/sync/test_users_small.csv', > source_uid='abd213', > school='https://m90s4.uni.dtr/api/v1/schools/SchuleZwei/') Please remove the source_uid parameter, the client can't know the source uid. Why do I need to pass a URI as argument for "school" instead of the selected school? I need a method which I can call to check if the dry-run is finished.
Please split client.get_importjobs() into two methods: * client.get_importjobs() * client.get_importjob(job) Please do the same for client.get_schools() / client.get_school().
The exception string contains the user credentials! Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib/pymodules/python2.7/ucsschool/http_api/client.py", line 172, in get_importjobs return self._call_api('get', url) File "/usr/lib/pymodules/python2.7/ucsschool/http_api/client.py", line 124, in _call_api raise ApiError('Received status_code={!r} with reason={!r} for requests.{}(**{}).'.format(response.status_code, response.reason, method, ', '.join('{}={!r}'.format(k, v) for k, v in request_kwargs.items()))) ucsschool.http_api.client.ApiError: Received status_code=404 with reason='NOT FOUND' for requests.get(**files=None, params=None, url=u'https://master.school.local/api/v1/imports/users/4/', verify=False, data=None, auth=('Administrator', 'univention')). Please add a reasonable exception hierarchy.
The API is completely missing URL encoding.
There is no argument for the user type when starting a import.
(In reply to Florian Best from comment #9) > >>> from ucsschool.http_api.client import Client > Traceback (most recent call last): > File "<stdin>", line 1, in <module> > ImportError: No module named client > > # dpkg -L ucs-school-import-http-api-client | grep client.py > /usr/share/pyshared/ucsschool/client.py There is a typo in debian/ucs-school-import-http-api-client.install, fix will be commited with next client release. (In reply to Florian Best from comment #10) > Using the library I get the following warning: > > /usr/lib/python2.7/dist-packages/urllib3/connectionpool.py:732: > InsecureRequestWarning: Unverified HTTPS request is being made. Adding > certificate verification is strongly advised. See: > https://urllib3.readthedocs.org/en/latest/security.html (This warning will > only appear once by default.) > InsecureRequestWarning) verify=False is easier during development. It will be an option later. (In reply to Florian Best from comment #11) > (In reply to Daniel Tröder from comment #7) > > (In reply to Florian Best from comment #5) > > > Can you give me something like .get_user_types() ? > > As discussed yesterday, the user types should be hard coded to students, > > teachers and staff. > Yes, but not in the client / UMC module. > I need a result which only queries the available user types for the user > which makes the import. Otherwise he sees a list of user types which he > isn't allowed to import. This information can be obtained through a library call that lists the available school/type options for a user. This library call agreed upon to be implemented in Bug #45026 (ext attr for groups to manage remote import ACLs). (In reply to Florian Best from comment #12) > (In reply to Daniel Tröder from comment #4) > > client.create_importjob(filename='/sync/test_users_small.csv', > > source_uid='abd213', > > school='https://m90s4.uni.dtr/api/v1/schools/SchuleZwei/') > Please remove the source_uid parameter, the client can't know the source uid. If the sourceUID should be hardcoded into the configuration files, that means that for each OU there *must* be a dedicated configuration. Is this desired? → question to product owner → @Jan, @Tobias - RFC: we cannot say there is a "fallback to a default config, if there is no school-specific one", if the sourceUID is to be hardcoded into the configuration. > Why do I need to pass a URI as argument for "school" instead of the selected > school? My mistake, will be the schools name only. > I need a method which I can call to check if the dry-run is finished. get_importjobs(importjob_id)['status'] == 'Finished' (In reply to Florian Best from comment #13) > Please split client.get_importjobs() into two methods: > * client.get_importjobs() > * client.get_importjob(job) > > Please do the same for client.get_schools() / client.get_school(). Will be list_*() → list(dict) and get_*() → dict. list_*() should support filtering in the future. (In reply to Florian Best from comment #14) > The exception string contains the user credentials! The client is very much WIP. Thanks for the pointers though :) > Please add a reasonable exception hierarchy. I don't want to copy all of the requests libraries exceptions. But for some common errors it should be done. What come to mind are: * 404 * 5xx * Connection refused What other errors do you want a dedicated exception for? (In reply to Florian Best from comment #15) > The API is completely missing URL encoding. Everything is UTF-8 encoded. You can do: $ vi /var/spool/ucs-school-import/jobs/2017/1/user_import_summary.csv # add some äòü open('/tmp/summary1.csv', 'wb').write( client._call_api('get', 'imports/users/1/summary')['text'] ) $ file /tmp/summary1.csv /tmp/summary1.csv: UTF-8 Unicode text, with CRLF line terminators (In reply to Florian Best from comment #16) > There is no argument for the user type when starting a import. Yes - I wrote this in the email: WIP.
(In reply to Daniel Tröder from comment #17) > (In reply to Florian Best from comment #12) > > (In reply to Daniel Tröder from comment #4) > > > client.create_importjob(filename='/sync/test_users_small.csv', > > > source_uid='abd213', > > > school='https://m90s4.uni.dtr/api/v1/schools/SchuleZwei/') > > Please remove the source_uid parameter, the client can't know the source uid. > If the sourceUID should be hardcoded into the configuration files, that > means that for each OU there *must* be a dedicated configuration. Is this > desired? > → question to product owner @Jan Chrostoph @Tobias RFC: we cannot say there is a "fallback to a default config, if there is no school-specific one", if the sourceUID is to be hardcoded into the configuration. Please comment.
(In reply to Daniel Tröder from comment #17) > > I need a method which I can call to check if the dry-run is finished. > get_importjobs(importjob_id)['status'] == 'Finished' Yes, I know this. But the importjob_id is nowhere returned. > > Please add a reasonable exception hierarchy. > I don't want to copy all of the requests libraries exceptions. But for some > common errors it should be done. What come to mind are: > * 404 > * 5xx > * Connection refused > What other errors do you want a dedicated exception for? 400 Bad Request. 404 Not Found. 5XX Connection errors What else error codes is the API using? > (In reply to Florian Best from comment #15) > > The API is completely missing URL encoding. > Everything is UTF-8 encoded. You can do: No, you can never use UTF-8 in a URI. Especially for ASCII characters like '%' '&' '?', … must be url encoded.
A job should also contain the display name of the school.
(In reply to Florian Best from comment #19) > (In reply to Daniel Tröder from comment #17) > > > I need a method which I can call to check if the dry-run is finished. > > get_importjobs(importjob_id)['status'] == 'Finished' > Yes, I know this. But the importjob_id is nowhere returned. Sorry - still thinking in HTTP-API terms: the ID is the 'url'. The Python client will return a numeric ID. > > > Please add a reasonable exception hierarchy. > > I don't want to copy all of the requests libraries exceptions. But for some > > common errors it should be done. What come to mind are: > > * 404 > > * 5xx > > * Connection refused > > What other errors do you want a dedicated exception for? > 400 Bad Request. > 404 Not Found. > 5XX > Connection errors ACK > What else error codes is the API using? No others yet. > > (In reply to Florian Best from comment #15) > > > The API is completely missing URL encoding. > > Everything is UTF-8 encoded. You can do: > No, you can never use UTF-8 in a URI. Especially for ASCII characters like > '%' '&' '?', … must be url encoded. Misunderstanding: I thought it was about the data. The URL will be correctly encoded by the requests library, as arguments are passed separately.
r81582: complete makeover of client API * fix module installation path * handle resources in separate classes * accept ID instead of URL as object identifier * add exception classes for most common errors * add support for user type * make SSL certificate verification optional from ucsschool.http_api.client import Client client = Client('Administrator', 'univention') client.School.list() client.School.get('testschule') client.UserImportJob.list() client.UserImportJob.get(3) School resource now contains a 'name' property, which can be used to get() it. UserImportJob resource now contains an 'id' property, which can be used to get() it.
r81586: lower case resource attributes client.School.list -> client.school.list client.UserImportJob.get -> client.userimportjob.get
importjob = self.client.userimportjob.create(filename=import_file, source_uid=usertype, school=school, user_role=usertype, dryrun=True) File "/usr/lib/pymodules/python2.7/ucsschool/http_api/client.py", line 257, in create school_resource = self.client.School.get(school) TypeError: unbound method get() must be called with School instance as first argument (got str instance instead)
Fixed that traceback in: ucs-school-import (15.0.0-17): r81588 | Bug #45022: fix typos
(In reply to Daniel Tröder from comment #21) > > > (In reply to Florian Best from comment #15) > > > > The API is completely missing URL encoding. > > > Everything is UTF-8 encoded. You can do: > > No, you can never use UTF-8 in a URI. Especially for ASCII characters like > > '%' '&' '?', … must be url encoded. > Misunderstanding: I thought it was about the data. The URL will be correctly > encoded by the requests library, as arguments are passed separately. No. Quoting ucs-school-import/modules/ucsschool/http_api/client.py: 218 » » » url = urljoin(self.RESOURCE_URL, str(pk)) → a primary key containing e.g. "?" will cause a 404 Error instead of returning the correct school.
(In reply to Florian Best from comment #26) > (In reply to Daniel Tröder from comment #21) > > > > (In reply to Florian Best from comment #15) > > > > > The API is completely missing URL encoding. > > > > Everything is UTF-8 encoded. You can do: > > > No, you can never use UTF-8 in a URI. Especially for ASCII characters like > > > '%' '&' '?', … must be url encoded. > > Misunderstanding: I thought it was about the data. The URL will be correctly > > encoded by the requests library, as arguments are passed separately. > No. > Quoting ucs-school-import/modules/ucsschool/http_api/client.py: > 218 » » » url = urljoin(self.RESOURCE_URL, str(pk)) > > → a primary key containing e.g. "?" will cause a 404 Error instead of > returning the correct school. 1) We have no primary keys (OU names oder integers) containing "?". 2) The requests lib does encode the URL: client = Client('Administrator', 'univention', log_level=Client.LOG_RESPONSE) client.school.get('föö') 2017-08-01 16:51:48 REQUEST client.<lambda>:154 get(files=None, data=None, params=None, url=u'https://m90s4.uni.dtr/api/v1/schools/f\xf6\xf6/', auth=('Administrator', u'**********')) What is put into the request lib call: '.../api/v1/schools/f\xf6\xf6/'. 2017-08-01 16:51:50 RESPONSE client.<lambda>:155 https://m90s4.uni.dtr/api/v1/schools/f%C3%B6%C3%B6/ -> NOT FOUND (404): '{"detail":"Not found."}' Request lib converts it to: '.../api/v1/schools/f%C3%B6%C3%B6/'.
(In reply to Daniel Tröder from comment #27) > (In reply to Florian Best from comment #26) > > (In reply to Daniel Tröder from comment #21) > > > > > (In reply to Florian Best from comment #15) > > > > > > The API is completely missing URL encoding. > > > > > Everything is UTF-8 encoded. You can do: > > > > No, you can never use UTF-8 in a URI. Especially for ASCII characters like > > > > '%' '&' '?', … must be url encoded. > > > Misunderstanding: I thought it was about the data. The URL will be correctly > > > encoded by the requests library, as arguments are passed separately. > > No. > > Quoting ucs-school-import/modules/ucsschool/http_api/client.py: > > 218 » » » url = urljoin(self.RESOURCE_URL, str(pk)) > > > > → a primary key containing e.g. "?" will cause a 404 Error instead of > > returning the correct school. > 1) We have no primary keys (OU names oder integers) containing "?". > 2) The requests lib does encode the URL: > > client = Client('Administrator', 'univention', log_level=Client.LOG_RESPONSE) > client.school.get('föö') > > 2017-08-01 16:51:48 REQUEST client.<lambda>:154 get(files=None, data=None, > params=None, url=u'https://m90s4.uni.dtr/api/v1/schools/f\xf6\xf6/', > auth=('Administrator', u'**********')) > > What is put into the request lib call: '.../api/v1/schools/f\xf6\xf6/'. > > 2017-08-01 16:51:50 RESPONSE client.<lambda>:155 > https://m90s4.uni.dtr/api/v1/schools/f%C3%B6%C3%B6/ -> NOT FOUND (404): > '{"detail":"Not found."}' > > Request lib converts it to: '.../api/v1/schools/f%C3%B6%C3%B6/'. Sorry, you are still wrong. requests doesn't do anything for you here. This can cause a security hole in the worst case!: Assuming you have got the school "schule1": client = Client('Administrator', 'univention', log_level=Client.LOG_RESPONSE) client.school.get('schule1?someparam=foo&') Will cause a request to: https://master.school.local/api/v1/schools/schule1/?someparam=foo&/ This request is successfully answered with the contents of schule1 insteaf of a 404 HTTP Error. Instead the request should go to: https://master.school.local/api/v1/schools/schule1%3Fsomeparam%3Dfoo/ and return 404 Not Found. The other way round is also possible in theory (you have got a school/etc. names 'schul?bla' which you are requesting but instead of an valid object you will receive a HTTP 404 Error.
(In reply to Florian Best from comment #28) > client.school.get('schule1?someparam=foo&') This is not how the requests lib is used. Parameters are passed separately. It should be used like this: client.call_api('get', 'schools/schule1', params={'someparam': 'foo'}) The requests-lib fixes your mistake and requests a valid URL. I don't see a security hole here.
r81678 (ucs-school-import 15.0.0-18): * make client lib pythonic * dynamically find resource URLs * auto-register resource classes * expicitely request JSON r2 = client.userimportjob.get(2) r2.date_created → datetime.datetime(2017, 8, 1, 14, ...) r2._resource → {..} r2.result → ResultResource(status=u'FAILURE') r2.result.date_done → datetime.datetime(2017, 8, 1, 14, ...) r2.school → SchoolResource(schools, name=u'SchuleEinz') r2.school.name → u'SchuleEinz' r2.school.user_imports → [UserImportJobResource(imports/users, id=4), UserImportJobResource(imports/users, id=3), UserImportJobResource(imports/users, id=2), UserImportJobResource(imports/users, id=1)] r2.school.user_imports[0].school → SchoolResource(schools, name=u'SchuleEinz')
r81679: * add support for text artifacts → r2 = client.userimportjob.get(2) → r2.log_file → r2.password_file → r2.summary_file * add get_last() method to retrieve newest resource → client.userimportjob.list() [UserImportJobResource(imports/users, id=6), UserImportJobResource(imports/users, id=5)] → r2.get_last() UserImportJobResource(imports/users, id=6) r81680 (15.0.0-20): * adapt create() to API change * rename get_last() to latest()
(In reply to Daniel Tröder from comment #29) > (In reply to Florian Best from comment #28) > > client.school.get('schule1?someparam=foo&') > This is not how the requests lib is used. Parameters are passed separately. > It should be used like this: > > client.call_api('get', 'schools/schule1', params={'someparam': 'foo'}) > > The requests-lib fixes your mistake and requests a valid URL. I don't see a > security hole here. Please read the comment 28 again. I don't want to pass parameters. I want to request a resource which contains special characters but the client.py doesn't escape them.
(In reply to Daniel Tröder from comment #30) > r81678 (ucs-school-import 15.0.0-18): > * make client lib pythonic > * dynamically find resource URLs > * auto-register resource classes > * expicitely request JSON > > r2 = client.userimportjob.get(2) > r2.date_created → datetime.datetime(2017, 8, 1, 14, ...) > r2._resource → {..} > r2.result → ResultResource(status=u'FAILURE') > r2.result.date_done → datetime.datetime(2017, 8, 1, 14, ...) > r2.school → SchoolResource(schools, name=u'SchuleEinz') > r2.school.name → u'SchuleEinz' > r2.school.user_imports → [UserImportJobResource(imports/users, id=4), > UserImportJobResource(imports/users, id=3), > UserImportJobResource(imports/users, id=2), > UserImportJobResource(imports/users, id=1)] > r2.school.user_imports[0].school → SchoolResource(schools, > name=u'SchuleEinz') The latest changes are very intransparent because they are not consistent: Sometimes I now get a object where I have to access school.displayName (which also violates pep8 naming), sometimes I get a dict where I have to access importjob['id'].
(In reply to Florian Best from comment #32) > (In reply to Daniel Tröder from comment #29) > > (In reply to Florian Best from comment #28) > > > client.school.get('schule1?someparam=foo&') > > This is not how the requests lib is used. Parameters are passed separately. > > It should be used like this: > > > > client.call_api('get', 'schools/schule1', params={'someparam': 'foo'}) > > > > The requests-lib fixes your mistake and requests a valid URL. I don't see a > > security hole here. > Please read the comment 28 again. I don't want to pass parameters. I want to > request a resource which contains special characters but the client.py > doesn't escape them. The special characters you are talking about are not allowed in resource names. How such an error by the client (here UMC module) is handled is not specified. The current implementation decides, that instead of escaping illegal characters it will separate the remaining URL into parameters. This ends up in a API call that may have more parameters than intended by the lib, but that is still OK. As the user must be authenticated, the user could do it directly against the HTTP interface anyway. Please provide a concrete scenario in which this poses a security thread.
(In reply to Florian Best from comment #33) > The latest changes are very intransparent because they are not consistent: > Sometimes I now get a object where I have to access school.displayName > (which also violates pep8 naming), sometimes I get a dict where I have to > access importjob['id']. That shouldn't be. Please provide an example in your bug report. The attribute names have been chosen to fit JavaScript naming conventions. That's common for HTTP APIs. For transparency reasons they should not be changed to snake case by the Python client lib.
r81780: * add support for parameters (paging, filtering, ordering) * fix userimportjob.create() result * escape path argument * check if resource URLs returned at API root match request-URL * use ordering and limit in latest() to fetch one 1 resource * <resource>.list() now returns an iterator that transparently handles paging Package: ucs-school-import Version: 15.0.0-21A~4.2.0.201708031931 Branch: ucs_4.2-0 Scope: ucs-school-4.2
r81781: * allow parameters for latest() * fix method doc Use filtering, ordering and limiting like this: client.userimportjob.list(status=['Aborted', 'Finished'], dryrun=False, ordering='id', limit=1) Latest now fetches one 1 resource. It also accepts parameters: client.userimportjob.latest(dryrun=True) will return the last ImportJob which was a dryrun. ucs-school-import 15.0.0-22A~4.2.0.201708031943
No import is possible anymore: File "/usr/lib/pymodules/python2.7/univention/management/console/modules/schoolimport/__init__.py", line 187, in jobs } for job in self.client.userimportjob.list() if not job.dryrun] File "/usr/lib/pymodules/python2.7/ucsschool/http_api/client.py", line 234, in school return self._resource_client.client.school.get(self._resource['school']) File "/usr/lib/pymodules/python2.7/ucsschool/http_api/client.py", line 388, in get return self._to_python(self._get_resource(pk)) File "/usr/lib/pymodules/python2.7/ucsschool/http_api/client.py", line 374, in _get_resource return self._resource_from_url(url, **params) File "/usr/lib/pymodules/python2.7/ucsschool/http_api/client.py", line 370, in _resource_from_url return self.client.call_api('get', url, params=params) File "/usr/lib/pymodules/python2.7/ucsschool/http_api/client.py", line 351, in call_api raise exc(msg, status_code=response.status_code) ObjectNotFound: Received status_code=404 with reason='NOT FOUND' for requests.get(**files=None, url=u'https://master.school.local/api/v1/schools/https%3A//master.school.local/api/v1/schools/newschool/', auth=('Administrator', u'**********'), headers={u'Accept': u'application/json'}, params={}, data=None).
Commited something in r81799 to fix it. Should be good?
(In reply to Florian Best from comment #39) > Commited something in r81799 to fix it. Should be good? Yes, thanks. r81805: * add update() method to fetch fresh data from API * enhance repr output ucs-school-import (15.0.0-24)
From my point of view, the lib is usable and used in the UMC module. All required functionality is implemented.
UCS@school 4.2 v4 has been released. http://docs.software-univention.de/changelog-ucsschool-4.2v4-de.html If this error occurs again, please clone this bug.