Bug 50240 - [UDM HTTP API] support pagination
[UDM HTTP API] support pagination
Status: NEW
Product: UCS
Classification: Unclassified
Component: UDM - REST API
UCS 4.4
Other Linux
: P5 normal (vote)
: ---
Assigned To: UMC maintainers
UMC maintainers
https://git.knut.univention.de/univen...
:
Depends on: 49666 54786
Blocks:
  Show dependency treegraph
 
Reported: 2019-09-22 21:03 CEST by Daniel Tröder
Modified: 2024-01-04 20:30 CET (History)
4 users (show)

See Also:
What kind of report is it?: Development Internal
What type of bug is this?: ---
Who will be affected by this bug?: ---
How will those affected feel about the bug?: ---
User Pain:
Enterprise Customer affected?:
School Customer affected?:
ISV affected?:
Waiting Support:
Flags outvoted (downgraded) after PO Review:
Ticket number:
Bug group (optional):
Max CVSS v3 score:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Daniel Tröder univentionstaff 2019-09-22 21:03:45 CEST
When searching, the result set can be very large, especially in UCS@school scenarios. Producing, downloading and parsing megabytes of data takes a lot of time and is potentially blocking.

The UDM REST API should paginate resource collections (lists/searches).
Comment 1 Florian Best univentionstaff 2019-09-22 23:21:30 CEST
It already supports pagination, but I excluded it from the openapi schema because not all modules support this (e.g. computers/computer doesn't).
This also probably depends on the patch in Bug #49666.
And we should add the lvl module in slapd, so that results can be sorted.
Comment 2 Daniel Tröder univentionstaff 2019-09-23 08:57:47 CEST
(In reply to Florian Best from comment #1)
> It already supports pagination, but I excluded it from the openapi schema
> because not all modules support this (e.g. computers/computer doesn't).
Please don't just remove things from the OpenAPI schema - that breaks the API!
Things missing in the OpenAPI schema cannot be used by OpenAPI clients - at all.

Either add pagination server side to the modules not supporting it, or add it selectively to the schema only for the modules that support it.
Comment 3 Florian Best univentionstaff 2019-09-23 10:16:49 CEST
(In reply to Daniel Tröder from comment #2)
> (In reply to Florian Best from comment #1)
> > It already supports pagination, but I excluded it from the openapi schema
> > because not all modules support this (e.g. computers/computer doesn't).
> Please don't just remove things from the OpenAPI schema - that breaks the
> API!
> Things missing in the OpenAPI schema cannot be used by OpenAPI clients - at
> all.
> 
> Either add pagination server side to the modules not supporting it, or add
> it selectively to the schema only for the modules that support it.

Yes, but for the first version I removed this "experimental" feature. We can easily add it later on.
I can currently only vague detect which modules support this.
We should add it when we support it in a stable manner (i.e. sorting is implemented on ldap server side) and we currently don't have shared memory between processes, for this we should probably find a solution as well.

Or should we re-add it again and fix/improove it later on?
Comment 4 Daniel Tröder univentionstaff 2019-09-23 11:24:29 CEST
(In reply to Florian Best from comment #3)
> Or should we re-add it again and fix/improove it later on?
No, it's OK to add it later in a consistent manner.
In the current state it is not worse than what Python UDM currently does (plus the HTTP+JSON overhead).
Comment 5 Lukas Zumvorde univentionstaff 2023-08-17 15:53:16 CEST
In the professional services we have one customer that needs this feature right now and another one were we can already see the request for this feature coming. I have marked them internally.

As a user i would look at https://docs.software-univention.de/developer-reference/5.0/en/udm/rest-api.html#api-usage-examples and read

"... These properties contain links which can be used to traverse the API. For example the “_links” property of the response to a paginated query could contain the “next” property which points to the next page."

as, there exists something like a paginated query for the UDM REST API.
Comment 6 Florian Best univentionstaff 2023-10-04 16:30:39 CEST
To clarify the state of this bug:
* UDM REST API is already capable of returning sorted paginated results (by specifying: ?by=uid&dir=ASC&limit=50&page=1). This is unsupported and marked as "Broken/Experimental" in the OpenAPI spec.
* But this requires enabling the SSSLVL overlay module in the LDAP server → Bug #54786
* So enabling the overlay modules opens possibility for DoS, as mentioned in Bug #54786 comment 1
* We also would have to configure good enough defaults for the maximum parallel search requests. If that size is reached the API returns with an error instead of a reliable result.

Necessary changes in the UDM REST API:
* return appropriate HTTP status if maximum size of parallel paginated searches is reached (e.g. 502 Service Unavailable)
* change the (sort) "by" query string to contain a UDM property name instead of a LDAP attribute name
* each UDM property should be able to specify if the request can be sorted by it, which LDAP attribute should be used for sorting then, and which matching rule is applied then (e.g.:caseIgnoreOrderingMatch) 
* each UDM module should specify the default sorting property
* adjust the default client to include easy navigation between pages and make it possible to explicitly enabled/disable it.