Bug 34498 - enhance UMCConnection
enhance UMCConnection
Status: CLOSED FIXED
Product: UCS
Classification: Unclassified
Component: univention-lib
UCS 4.1
Other Linux
: P5 enhancement (vote)
: UCS 4.2
Assigned To: Florian Best
Jürn Brodersen
: interim-2
: 34490 (view as bug list)
Depends on:
Blocks: 38720 39731
  Show dependency treegraph
 
Reported: 2014-04-08 12:00 CEST by Florian Best
Modified: 2017-04-04 18:28 CEST (History)
3 users (show)

See Also:
What kind of report is it?: Development Internal
What type of bug is this?: ---
Who will be affected by this bug?: ---
How will those affected feel about the bug?: ---
User Pain:
Enterprise Customer affected?:
School Customer affected?:
ISV affected?:
Waiting Support:
Flags outvoted (downgraded) after PO Review:
Ticket number:
Bug group (optional):
Max CVSS v3 score:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Florian Best univentionstaff 2014-04-08 12:00:15 CEST
For a test case it was necessary to get the result of a UMCP request which does not return JSON (instead returns a PDF file).

With the current API this is not possible because the response mimetype is not checked. JSON is always assumed.

There is also no possibility to get the response headers which is necessary for the test script, too.
Comment 1 Alexander Kläser univentionstaff 2014-04-30 16:21:39 CEST
Agreed, for some test cases, we had similar problems.

Simple suggestion for a workaround could be via an additional parameter "raw_response"; if set to True, the response object is returned.
Comment 2 Florian Best univentionstaff 2014-07-14 16:21:26 CEST
In general the class should provide this functionality:
* setting the language
* get the whole UMCP response including UMCP message, status and result
* get the response HTTP headers (e.g. Content-Type)
* get the plain response (as bytes)
* (set the request HTTP method)
* possibility to upload non UMCP-json files
* setting HTTP request header (which aren't bound to all requests which are done with this connection)
* the error handler should get the original exception (sys.exc_info()) instead of str(exc)
* Bug #34490
Comment 3 Florian Best univentionstaff 2015-10-20 11:27:11 CEST
Another error in the library if the response doesn't contain 'result' in the returned object.
It shows also "Internal server error" while this is a bug in the lib and server returned 200 OK.


500 Internal Server Error

The server encountered an unexpected condition which prevented it from fulfilling the request.

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/cherrypy/_cprequest.py", line 656, in respond
    response.body = self.handler()
  File "/usr/lib/python2.7/dist-packages/cherrypy/lib/encoding.py", line 188, in __call__
    self.body = self.oldhandler(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/cherrypy/_cpdispatch.py", line 34, in __call__
    return self.callable(*self.args, **self.kwargs)
  File "/usr/share/univention-self-service/web/setpassword/wsgi.py", line 59, in set_password
    return self.umc_request(connection, url, data, command='set')
  File "/usr/lib/pymodules/python2.7/univention/selfservice/frontend.py", line 109, in umc_request
    result = connection.request(url, data, command=command)
  File "/usr/lib/pymodules/python2.7/univention/lib/umc_connection.py", line 143, in request
    return loads(content)['result']
KeyError: 'result'
Comment 4 Florian Best univentionstaff 2015-10-20 11:29:11 CEST
Another wrong thing:
It returns the exact Set-Cookie header as Cookie in the requests while the cookie should only contain the cookie values.
Comment 5 Florian Best univentionstaff 2017-01-20 16:52:48 CET
*** Bug 34490 has been marked as a duplicate of this bug. ***
Comment 6 Florian Best univentionstaff 2017-01-24 19:53:35 CET
There is a new class now:
univention.lib.umc.Client(hostname=localhost, username=None, password=None, language='en-US', timeout=None, automatic_reauthentication=False):
  authenticate(username, password) → Response
  reauthenticate() → Response
  request(method, path, data=None, headers=None) → Response
  umc_command(path, options=None, flavor=None) → Response
  umc_set(options) → Response
  umc_get(path, options=None) → Response
…
The Response class contains some usefull attributes/properties
class univention.lib.umc.Response(status, reason, body, headers, _response):
  get_header() → str
  data → decoded body

The body of a response is automatically decoded if the response contains "Content-Type: application/json" header.

If you set a request header with "Content-Type: application/json" your body is automatically json.dumps().

(In reply to Florian Best from comment #2)
> In general the class should provide this functionality:
> * setting the language
This is now done with the Accept-Language header:
Client(language='de-DE')

> * get the whole UMCP response including UMCP message, status and result
Client.umc_command('foo').{result,message,status}

> * get the response HTTP headers (e.g. Content-Type)
Client.umc_command('foo').get_header('Content-Type')

> * get the plain response (as bytes)
Client.umc_command('foo').body

> * (set the request HTTP method)
Client.request(method, path_relative_to_umc, data, headers)

> * possibility to upload non UMCP-json files
Client.request(POST, 'upload', multipart_body, {'Content-Type: 'multipart/form-data'})
→ for example done in:
90_ucsschool/essential/distribution.py, 90_ucsschool/essential/exam.py, 90_ucsschool/essential/importcsv.py

> * setting HTTP request header (which aren't bound to all requests which are
> done with this connection)
Client.request(method, path_relative_to_umc, data, headers)

> * the error handler should get the original exception (sys.exc_info())
> instead of str(exc)
→ The stupid error handler parameter has been removed. use try-except instead!

> * Bug #34490
There is a exception hierarchy:
ConnectionError
→ wraps every socket.error, IOError, OSError
HTTPError (and subclasses, for each non-successful HTTP response)
* BadRequest
* Unauthorized
* Forbidden
* NotFound
* MethodNotAllowed
* NotAcceptable
* UnprocessableEntity
* InternalServerError
* BadGateway
* ServiceUnavailable

try:
 client.request(...)
except HTTPError as exc:
 print exc.response.status
 print exc.request
 print exc

(In reply to Florian Best from comment #0)
> For a test case it was necessary to get the result of a UMCP request which
> does not return JSON (instead returns a PDF file).
This is now possible and done in a UCS@school test case.
90_ucsschool/07_printermoderation_check:
  129 »   response = client.umc_command('printermoderation/download', param, print_response=False)
  130 »   assert response.get_header('Content-Type') == 'application/pdf'

(In reply to Florian Best from comment #3)
> Another error in the library if the response doesn't contain 'result' in the
> returned object.
> It shows also "Internal server error" while this is a bug in the lib and
> server returned 200 OK.
> 
> 
> 500 Internal Server Error
> 
> The server encountered an unexpected condition which prevented it from
> fulfilling the request.
> 
> Traceback (most recent call last):>   File "/usr/lib/pymodules/python2.7/univention/lib/umc_connection.py", line
> 143, in request
>     return loads(content)['result']
> KeyError: 'result'
client.umc_get('modules/list').result e.g. is None

(In reply to Florian Best from comment #4)
> Another wrong thing:
> It returns the exact Set-Cookie header as Cookie in the requests while the
> cookie should only contain the cookie values.
Has also been fixed.

r75987 | Changelog Bug #34498
r75986 | Bug #34498: use univention.lib.umc.Client

ucs-ec2-tools (3.0.2-1):
r75989 | Bug #34498: use univention.lib.umc.Client

univention-lib (6.0.6-4):
r76069 | Bug #34498: remove unused umc_connection
r76067 | Bug #34498: deprecate univention.lib.umc_connection
r76066 | Bug #34498: fix cookie handling
r76042 | Bug #34498: make options parameter optional
r76029 | Bug #34498: add reconnection parameter
r75993 | Bug #34498: fix parameters
r75990 | Bug #34498: add univention.lib.umc
r75982 | Bug #34498: add univention.lib.umc

ucs-test (7.0.10-5):
r76069 | Bug #34498: remove unused umc_connection
r76068 | Bug #34498: use univention.lib.umc.Client
r76043 | Bug #34498: add missing .result
r76030 | Bug #34498: rename parameter
r76012 | Bug #34498: add verbosity for requests
r76010 | Bug #34498: use univention.lib.umc.Client
r75992 | Bug #34498: fix import error
r75991 | Bug #34498: use univention.lib.umc.Client

univention-ldb-modules (5.0.9-2):
r76058 | Bug #34498: use univention.lib.umc.Client
r75983 | Bug #34498: use univention.lib.umc.Client

univention-system-setup (10.0.3-1):
r75985 | Bug #34498: use univention.lib.umc.Client

univention-appcenter (6.0.1-4):
r75984 | Bug #34498: use univention.lib.umc.Client
Comment 7 Jürn Brodersen univentionstaff 2017-03-02 11:44:34 CET
from univention.lib.umc import Client
Client(username="Administrator", password="univention")

Unauthorized: 401 on m42.univention.intranet (auth): {"status": 401, "message": "Cross Site Request Forgery attack detected. Please provide the \"UMCSessionId\" cookie value as HTTP request header \"X-Xsrf-Protection\".", "location": "https://m42.univention.intranet/univention/auth"}

Related to Bug 39733 ?
Comment 8 Florian Best univentionstaff 2017-03-03 16:49:23 CET
Fixed.
Comment 9 Jürn Brodersen univentionstaff 2017-03-14 12:25:07 CET
---Anonymous commands are not working:---
"""
client = Client()
client.umc_command("passwordreset/get_reset_methods", {"username": "foo"}).data
"""

Result:
"""
Forbidden: 403 on m42.univention.intranet (command/passwordreset/get_reset_methods): {"status": 403, "message": "Cross Site Request Forgery attack detected. Please provide the \"UMCSessionId\" cookie value as HTTP request header \"X-Xsrf-Protection\".", "location": "https://m42.univention.intranet/univention/command"}
"""


---umc_logout() is not working---
"""
client = Client(username="Administrator", password="univention")
client.umc_logout()
"""

Result:
"""HTTPError                                 Traceback (most recent call last)
<ipython-input-48-cc0ef8d28a37> in <module>()
----> 1 client.umc_logout()

/usr/lib/pymodules/python2.7/univention/lib/umc.pyc in umc_logout(self)
    267 
    268         def umc_logout(self):
--> 269                 return self.request('GET', 'logout')
    270 
    271         def request(self, method, path, data=None, headers=None):

/usr/lib/pymodules/python2.7/univention/lib/umc.pyc in request(self, method, path, data, headers)
    272                 request = Request(method, path, data, headers)
    273                 try:
--> 274                         return self.send(request)
    275                 except Unauthorized:
    276                         if not self._automatic_reauthentication:

/usr/lib/pymodules/python2.7/univention/lib/umc.pyc in send(self, request)
    290                 response = Response._from_httplib_response(response)
    291                 if self._raise_errors and response.status > 299:
--> 292                         raise HTTPError(request, response, self.hostname)
    293                 return response
    294 

HTTPError: 303 on m42.univention.intranet (logout): This resource can be found at <a href="https://m42.univention.intranet/univention/">https://m42.univention.intranet/univention/</a>.
"""
Comment 10 Florian Best univentionstaff 2017-03-14 13:03:17 CET
r77687 | Bug #34498: fix requests for anonymous commands; fix logout
Comment 11 Jürn Brodersen univentionstaff 2017-03-14 18:57:48 CET
Test 1.1
from univention.lib.umc import Client
from univention.lib.umc import HTTPError
client = Client(username="Administrator", password="univention")
client.umc_command("setup/lang/locales", {'pattern':'*'}).{result,status,data,body} -> All OK
client.umc_command("setup/lang/locales", {'pattern':'*'}).get_header('Content-Type') -> OK

try:
    client.umc_command("setup/lang/locales").message
except HTTPError as exc:
    print exc.response.status -> OK
    print exc.response.message -> OK
client.authenticate_with_machine_account()
client.umc_command("setup/lang/locales", {'pattern':'*'}).data -> OK
client = Client()
client.umc_command("setup/lang/locales", {'pattern':'*'}).data -> OK
client.umc_command("passwordreset/get_reset_methods", {"username": "foo"}).data -> OK
client = Client(username="Administrator", password="univention")
client.umc_logout()
client.umc_command("setup/lang/locales", {'pattern':'*'}).data -> OK (forbidden)
client = Client(username="Administrator", password="univention", language='de-DE')
try:
    client.umc_command("setup/lang/locales").message
except HTTPError as exc:
    print exc.response.status -> OK
    print exc.response.message -> OK (deutsch)

Test 1.2
For multipart tests see ucs@school tests:
e.g. 90_ucsschool/essential/importcsv.py


Test 2 (r75985)
from univention.management.console.modules.setup.util import domain_has_activated_license
domain_has_activated_license('10.200.41.200', 'Administrator', 'univention') -> OK

Test 3 (r75984)
installed an app remotely

changelog: ok

-> Verified :)
Comment 12 Stefan Gohmann univentionstaff 2017-04-04 18:28:30 CEST
UCS 4.2 has been released:
 https://docs.software-univention.de/release-notes-4.2-0-en.html
 https://docs.software-univention.de/release-notes-4.2-0-de.html

If this error occurs again, please use "Clone This Bug".