Bug 51197 - New UDM modules and attributes for the portal
New UDM modules and attributes for the portal
Status: CLOSED FIXED
Product: UCS
Classification: Unclassified
Component: Portal
UCS 4.4
Other Linux
: P5 normal (vote)
: UCS 5.0
Assigned To: Dirk Wiesenthal
Florian Best
: interim-1
Depends on: 41072 51157 52984
Blocks:
  Show dependency treegraph
 
Reported: 2020-04-30 23:09 CEST by Dirk Wiesenthal
Modified: 2021-05-25 15:59 CEST (History)
3 users (show)

See Also:
What kind of report is it?: Feature Request
What type of bug is this?: ---
Who will be affected by this bug?: ---
How will those affected feel about the bug?: ---
User Pain:
Enterprise Customer affected?:
School Customer affected?:
ISV affected?:
Waiting Support:
Flags outvoted (downgraded) after PO Review:
Ticket number:
Bug group (optional): API change
Max CVSS v3 score:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Dirk Wiesenthal univentionstaff 2020-04-30 23:09:12 CEST
We need new UDM attributes. We want ordered portal entry objects to render items in the menu of the new portal. We do not need footer links anymore.

We want these changes in the App so we would need to register the new modules if the App is installed on a Memberserver.

We should copy settings/portal and adjust it, renaming it to portal/portal in the process. (and the other object types accordingly)
Comment 1 Florian Best univentionstaff 2021-02-11 09:29:46 CET
REOPEN: 
 24865 portalentries                    21-02-11 00:19:33 [   DEBUG]: Processing cn=ucs-local-to-domain,cn=entry,cn=portals,cn=univention,dc=autotest071c,dc=local
 24865 portalentries                    21-02-11 00:19:33 [   DEBUG]: Attrs: {'name': 'ucs-local-to-domain', 'description': [('en_US', 'Central portal web page for the UCS domain'), ('de_DE', 'Zentrale Portal-Webseite für die UCS-Domäne'), ('fr_FR', 'Page web du portail central du domaine UCS')], 'displayName': [('en_US', 'Univention Portal'), ('de_DE', 'Univention Portal'), ('fr_FR', 'Portail Univention')]}
 24865 portalentries                    21-02-11 00:19:33 [   DEBUG]: Links: ['https://master071c.autotest071c.local/univention/portal/', 'http://10.207.34.137/univention/portal/', 'http://master071c.autotest071c.local/univention/portal/', 'https://10.207.34.137/univention/portal/']
 24865 portalentries                    21-02-11 00:19:33 [   ERROR]: Exception in UCR module create_portal_entries
Traceback (most recent call last):
  File "/etc/univention/templates/modules/create_portal_entries.py", line 241, in handler
    _handler(ucr, changes)
  File "/etc/univention/templates/modules/create_portal_entries.py", line 202, in _handler
    obj = init_object('portals/entry', lo, pos, dn)
  File "/usr/lib/python3/dist-packages/univention/appcenter/udm.py", line 75, in init_object
    module = _get_module(module, lo, pos)
  File "/usr/lib/python3/dist-packages/univention/appcenter/udm.py", line 69, in _get_module
    udm_modules.init(lo, pos, mod)
  File "/usr/lib/python3/dist-packages/univention/admin/modules.py", line 184, in init
    univention.admin.ucr_overwrite_properties(module, lo)
  File "/usr/lib/python3/dist-packages/univention/admin/__init__.py", line 66, in ucr_overwrite_properties
    ucr_prefix = ucr_property_prefix % module.module
AttributeError: 'NoneType' object has no attribute 'module'
Comment 2 Florian Best univentionstaff 2021-02-11 09:30:20 CET
See also Bug #52125 git:46004743beb974a9bcc7ea070c61ec58abe96174
Comment 3 Florian Best univentionstaff 2021-02-13 20:58:44 CET
REOPEN: During UCS installation this error happens.

Traceback (most recent call last):
  File "/usr/lib/univention-directory-listener/system/license_uuid.py", line 55, in handler
    univention.config_registry.handler_set(ucrVars)
  File "/usr/lib/python3/dist-packages/univention/config_registry/frontend.py", line 176, in handler_set
    _run_changed(ucr, changed, None if quiet else 'W: %s is overridden by scope "%s"')
  File "/usr/lib/python3/dist-packages/univention/config_registry/frontend.py", line 232, in _run_changed
    handlers(list(changed.keys()), (ucr, changed))
  File "/usr/lib/python3/dist-packages/univention/config_registry/handler.py", line 1074, in __call__
    handler(arg)
  File "/usr/lib/python3/dist-packages/univention/config_registry/handler.py", line 622, in __call__
    run_module(self.module, 'generate', ucr, changed)
  File "/usr/lib/python3/dist-packages/univention/config_registry/handler.py", line 204, in run_module
    arg2meth[arg](module)(ucr, changes)
  File "/etc/univention/templates/modules/univention_blog.py", line 72, in handler
    process('Creating blog entry failed', cmd)
  File "/etc/univention/templates/modules/univention_blog.py", line 88, in process
    raise Exception('%s: %d: %s %r' % (msg, process.returncode, stdout, cmd))
Exception: Creating blog entry failed: 3: unknown module portals/entry.
Comment 4 Florian Best univentionstaff 2021-03-24 19:46:56 CET
The univention blog entry is not shown as tile anymore. It's only present in the help menu.
Comment 5 Felix Botner univentionstaff 2021-04-01 10:40:05 CEST
There seems to be a timing issue during the upgrade (4.4->5.0). See https://jenkins.knut.univention.de:8181/job/UCS-5.0/job/UCS-5.0-0/job/Update%20Tests/test_system=update-from-2.4-start-4.4-7/

This is a master and after the update i get

univention-check-join-status 
Warning: 'univention-portal' is not configured.
Error: Not all install files configured: 1 missing

join.log
RUNNING 33univention-portal.inst
2021-04-01 04:19:50.010397072+02:00 (in joinscript_init)
...
Waiting for activation of the extension object portals/entry: OK
Waiting for file /usr/lib/python3/dist-packages/univention/admin/handlers/portals/entry.py: OK
Waiting for UDM module 'portals/entry' to be present: OK
Terminating running univention-cli-server processes.
Object exists: cn=udm_module,cn=univention,dc=old,dc=test
Object created: cn=portals/folder,cn=udm_module,cn=univention,dc=old,dc=test
...
unknown module portals/entry.
...
EXITCODE=3
...
RUNNING 35univention-nagios-squid.inst
EXITCODE=already_executed
RUNNING 35univention-server-overview.inst
2021-04-01 04:20:32.006259509+02:00 (in joinscript_init)
...


listener.log
01.04.21 04:20:20.861  LISTENER    ( PROCESS ) : updating 'cn=portals/entry,cn=udm_module,cn=univention,dc=old,dc=test' command a
01.04.21 04:20:21.452  LISTENER    ( PROCESS ) : updating 'cn=portals/entry,cn=udm_module,cn=univention,dc=old,dc=test' command m


manually running univention-run-join-scripts works
Comment 6 Felix Botner univentionstaff 2021-04-09 09:53:27 CEST
added a workaround in the tests cfg file ucs/test/scenarios/update-testing/update-from-2.4-start-4.4-7.cfg, please remove once this is fixed
ea88b43bc61fdf4e30b508799887172c07d50ae9
Comment 7 Dirk Wiesenthal univentionstaff 2021-04-21 15:26:47 CEST
The new portals/* modules are in place. They are registered in the join script. We added a workaround for "update-from-2.4-start-4.4-7" (see Comment#5) as for unknown reasons, there is an issue with this particular scenario. It should work fine otherwise.

try-excepts were added in univention-appcenter and univention-portal in case UDM is used before the join script ran.

/usr/lib/univention-portal/migrate-to-new-portal is run on every 4.4 -> 5.0 update.
Comment 8 Florian Best univentionstaff 2021-04-23 15:24:46 CEST
Summary of your changes:

univention-portal (4.0.5-34)
5e72576242d0 | Bug #51197: Re-add ensureLogin

univention-portal (4.0.5-27)
009e3d4e5dd6 | Bug #51197: Forgot one file where the old attributes are still used

univention-portal (4.0.5-26)
891844fc2966 | Bug #51197: Removed some now unneeded attributes from portals/portal

univention-portal (4.0.5-25)
3f7b99bc3ec6 | Bug #51197: Prevent tracebacks, print warnings

univention-portal (4.0.5-13)
f0bf5bb3f2f0 | Bug #51197: Soften up error message; also fix the position of the blog (was menu, should be category)

univention-portal (4.0.1-5)
7f0f01c37fe1 | Bug #51197: fix reading sorted results of X-ORDERED values

univention-portal (4.0.0-2)
4fe1c07835e1 | Bug #51197: Restart server after UDM modules have been added
e361e352e056 | Bug #51197: Fix migrate script for non-primary systems
b615c40c4319 | Bug #51197 UDM portal/category: fix ldap filter escaping
312f0ec56edf | Bug #51197 UDM portal/category: fix _ldap_post_remove
7cf1d96879de | Bug #51197: User links
791b00d4ff03 | Bug #51197: Local portal
9c3f25efcd33 | Bug #51197: Migration to new portal
07265f212733 | Bug #51197: UMC module
80dc81d42408 | Bug #51197: Translatiions
7bbbce03db56 | Bug #51197: Use new UDM modules
953cfba66792 | Bug #51197: New UDM modules, with own syntax, own schema, ...
c41b039ae14e | Bug #51197: Copy of old UDM modules as a starting point

univention-directory-manager-rest (10.0.1-3)
8fe03c27f13a | Bug #51197: change settings/portal_all to portal/all

univention-appcenter (9.0.2-30)
993aa4376d09 | Bug #51197: Prevent tracebacks, print warnings (2)

ucs-test (10.0.5-2)
93b74e786d44 | Bug #51197: adjust tests/86_selenium/186_portal_administration_umc

ucs-test (10.0.4-59)
ce8f777120e9 | Bug #51197: Adjust tests to removed attributes in portals/portal
Comment 9 Florian Best univentionstaff 2021-04-23 15:26:52 CEST
REOPEN: "Redirect anonymous visitors to the login" is not translated.
Comment 10 Florian Best univentionstaff 2021-04-23 15:33:24 CEST
REOPEN:
The representation of links must not be a dictionary but a list in the UDM REST API if you allow to specify more than one link per language code.
Comment 11 Florian Best univentionstaff 2021-04-26 17:13:21 CEST
management/univention-management-console-module-udm/umc/udm.xml still contains <flavor priority="50" id="settings/portal_all"> but not one for the new portal module.
Comment 12 Florian Best univentionstaff 2021-04-28 15:33:35 CEST
(In reply to Florian Best from comment #10)
> REOPEN:
> The representation of links must not be a dictionary but a list in the UDM
> REST API if you allow to specify more than one link per language code.

I fixed this in:

univention-portal (4.0.5-42)
82fcf359468e | Bug #51197: do not represent portals/entry:link as dictionary in the UDM REST API
Comment 13 Dirk Wiesenthal univentionstaff 2021-04-30 01:19:41 CEST
(In reply to Florian Best from comment #9)
> REOPEN: "Redirect anonymous visitors to the login" is not translated.

Done in commit 75476795b878414d17e5314001b3c4fdb564ac50

(In reply to Florian Best from comment #11)
> management/univention-management-console-module-udm/umc/udm.xml still
> contains <flavor priority="50" id="settings/portal_all"> but not one for the
> new portal module.

Fixed in 1ff6ca68

(In reply to Florian Best from comment #12)
> (In reply to Florian Best from comment #10)
> > REOPEN:
> > The representation of links must not be a dictionary but a list in the UDM
> > REST API if you allow to specify more than one link per language code.
> 
> I fixed this in:
> 
> univention-portal (4.0.5-42)
> 82fcf359468e | Bug #51197: do not represent portals/entry:link as dictionary
> in the UDM REST API

Thanks. Works.
I have reset the name of the syntax. It broke UDM on update. Or it would have if you had increased the join script version (as one needs to if something changed in the UDM integration)
Comment 14 Florian Best univentionstaff 2021-05-03 12:34:21 CEST
/usr/lib/univention-portal/umc/univention-portal.xml is registered for UCS 4.4. Should be restricted to UCS 5.0.

Please also move the translationId to the <flavor>, not the <module>.
Comment 15 Dirk Wiesenthal univentionstaff 2021-05-04 17:32:56 CEST
(In reply to Florian Best from comment #14)
> /usr/lib/univention-portal/umc/univention-portal.xml is registered for UCS
> 4.4. Should be restricted to UCS 5.0.

Now all UDM modules are registered starting at 5.0. Schema and ACLs are also in 4.4.
We may have to take a second look at the preview App and its update behaviour.

> 
> Please also move the translationId to the <flavor>, not the <module>.

This is not my choice. The listener puts
  --messagecatalog /usr/lib/univention-portal/udm/de.mo
to
/usr/share/locale/de/LC_MESSAGES/univention-admin-handlers-portals-portal.mo
Comment 16 Dirk Wiesenthal univentionstaff 2021-05-04 17:33:29 CEST
[5.0-0 66582a6c2b] Bug #51197: The new UDM modules only need to be present starting with 5.0-0
 2 files changed, 13 insertions(+), 6 deletions(-)
Comment 17 Dirk Wiesenthal univentionstaff 2021-05-04 17:46:01 CEST
(In reply to Dirk Wiesenthal from comment #15)
> This is not my choice. The listener puts
>   --messagecatalog /usr/lib/univention-portal/udm/de.mo
> to
> /usr/share/locale/de/LC_MESSAGES/univention-admin-handlers-portals-portal.mo

I was confused. I changed the XML file in

[5.0-0 bc415230c6] Bug #51197: Proper XML file
 1 file changed, 1 insertion(+), 2 deletions(-)

But: The translationId is not used anyway, so I removed it completely. (Bug#35061)
Comment 18 Florian Best univentionstaff 2021-05-05 15:20:14 CEST
REOPEN: missing changelog entry

REOPEN: workaround has not been reverted:
(In reply to Felix Botner from comment #6)
> added a workaround in the tests cfg file
> ucs/test/scenarios/update-testing/update-from-2.4-start-4.4-7.cfg, please
> remove once this is fixed
> ea88b43bc61fdf4e30b508799887172c07d50ae9
Comment 19 Dirk Wiesenthal univentionstaff 2021-05-05 22:04:54 CEST
(In reply to Florian Best from comment #18)
> REOPEN: missing changelog entry

Done

> 
> REOPEN: workaround has not been reverted:
> (In reply to Felix Botner from comment #6)
> > added a workaround in the tests cfg file
> > ucs/test/scenarios/update-testing/update-from-2.4-start-4.4-7.cfg, please
> > remove once this is fixed
> > ea88b43bc61fdf4e30b508799887172c07d50ae9

This is not really a workaround. Running the join scripts is actually correct there. It just was not necessary before. But it is documented like that.
Comment 20 Florian Best univentionstaff 2021-05-05 22:47:18 CEST
OK
Comment 21 Florian Best univentionstaff 2021-05-25 15:59:10 CEST
UCS 5.0 has been released:
 https://docs.software-univention.de/release-notes-5.0-0-en.html
 https://docs.software-univention.de/release-notes-5.0-0-de.html

If this error occurs again, please use "Clone This Bug".