Bug 53581 - Appcenter [python3]: RuntimeError: dictionary changed size during iteration
Appcenter [python3]: RuntimeError: dictionary changed size during iteration
Status: CLOSED FIXED
Product: UCS
Classification: Unclassified
Component: UDM (Generic)
UCS 5.0
Other Linux
: P5 normal (vote)
: UCS 5.0-1-errata
Assigned To: Juan Pedro Torres
Arvid Requate
https://git.knut.univention.de/univen...
: python3-migration
Depends on:
Blocks: 54597
  Show dependency treegraph
 
Reported: 2021-07-15 11:20 CEST by Florian Best
Modified: 2022-03-27 00:13 CET (History)
2 users (show)

See Also:
What kind of report is it?: Development Internal
What type of bug is this?: ---
Who will be affected by this bug?: ---
How will those affected feel about the bug?: ---
User Pain:
Enterprise Customer affected?:
School Customer affected?:
ISV affected?:
Waiting Support:
Flags outvoted (downgraded) after PO Review:
Ticket number:
Bug group (optional): bitesize, Regression
Max CVSS v3 score:


Attachments
PoC fix (1.28 KB, patch)
2022-03-21 18:42 CET, Jürn Brodersen
Details | Diff
53581_proposal2.patch (1.17 KB, patch)
2022-03-21 20:01 CET, Arvid Requate
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Florian Best univentionstaff 2021-07-15 11:20:32 CEST
Traceback (most recent call last):                                                 
  File "/usr/lib/python3/dist-packages/univention/management/console/base.py", line 344, in __error_handling
    six.reraise(etype, exc, etraceback)                                              
  File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise
    raise value
  File "/usr/lib/python3/dist-packages/univention/management/console/base.py", line 247, in execute
    function.__func__(self, request, *args, **kwargs)      
  File "/usr/lib/python3/dist-packages/univention/management/console/modules/decorators.py", line 321, in _response
    result = _multi_response(self, request)                    
  File "/usr/lib/python3/dist-packages/univention/management/console/modules/decorators.py", line 181, in _response
    return function(self, request)              
  File "/usr/lib/python3/dist-packages/univention/management/console/modules/decorators.py", line 443, in _response                                                                                                                           
    return list(function(self, iterator, *nones))                                                                                                                                                                                             
  File "/usr/lib/python3/dist-packages/univention/management/console/modules/decorators.py", line 289, in _fake_func                                                                                                                          
    yield function(self, *args)                                                                                                                                                                                                               
  File "/usr/lib/python3/dist-packages/univention/management/console/modules/appcenter/__init__.py", line 389, in query                                                                                                                       
    info = domain.to_dict(apps)                                                                                                                                                                                                               
  File "/usr/lib/python3/dist-packages/univention/appcenter/actions/domain.py", line 98, in to_dict                                                                                                                                           
    hosts = self.get_appcenter_hosts(lo, pos)                                                                                                                                                                                                 
  File "/usr/lib/python3/dist-packages/univention/appcenter/actions/domain.py", line 80, in get_appcenter_hosts                                                                                                                               
    objs = search_objects('computers/%s' % role, lo, pos)                                                                                                                                                                                     
  File "/usr/lib/python3/dist-packages/univention/appcenter/udm.py", line 117, in search_objects                                                                                                                                              
    module = _get_module(_module, _lo, _pos)                                                                                                                                                                                                  
  File "/usr/lib/python3/dist-packages/univention/appcenter/udm.py", line 66, in _get_module                   
    udm_modules.init(lo, pos, mod)                           
  File "/usr/lib/python3/dist-packages/univention/admin/modules.py", line 214, in init            
    for pname, prop in module.property_descriptions.items():    
RuntimeError: dictionary changed size during iteration
Comment 1 Florian Best univentionstaff 2021-10-21 12:38:56 CEST
Also in UCS@school within a diagnostic module (running in a thread):

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/univention/management/console/modules/diagnostic/__init__.py", line 280, in execute
    result = execute(umc_module, **kwargs)
  File "/usr/lib/python3/dist-packages/univention/management/console/modules/diagnostic/plugins/909_ucsschool_check_ox_support.py", line 69, in run 
    info = domain.to_dict([ox_app])
  File "/usr/lib/python3/dist-packages/univention/appcenter/actions/domain.py", line 98, in to_dict
    hosts = self.get_appcenter_hosts(lo, pos)
  File "/usr/lib/python3/dist-packages/univention/appcenter/actions/domain.py", line 80, in get_appcenter_hosts
    objs = search_objects('computers/%s' % role, lo, pos)
  File "/usr/lib/python3/dist-packages/univention/appcenter/udm.py", line 117, in search_objects
    module = _get_module(_module, _lo, _pos)
  File "/usr/lib/python3/dist-packages/univention/appcenter/udm.py", line 66, in _get_module
    udm_modules.init(lo, pos, mod)
  File "/usr/lib/python3/dist-packages/univention/admin/modules.py", line 269, in init
    univention.admin.syntax.update_choices()
  File "/usr/lib/python3/dist-packages/univention/admin/syntax.py", line 128, in update_choices
    func()
  File "/usr/lib/python3/dist-packages/univention/admin/syntax.py", line 4958, in update_choices
    cls.choices = [(key, x.short_description) for key, x in users.options.items() if key != 'default']
  File "/usr/lib/python3/dist-packages/univention/admin/syntax.py", line 4958, in <listcomp>
    cls.choices = [(key, x.short_description) for key, x in users.options.items() if key != 'default']
RuntimeError: dictionary changed size during iteration
Comment 2 Juan Pedro Torres univentionstaff 2022-03-10 12:11:07 CET
Modifications done to shared resources (dictionaries) where causing RuntimeErrors on multithread environments. Solved the issue with the proposed solution by Florian.



univention-directory-manager-modules.yaml
ce582691f9c9 | Bug #53581: update YAML for univention-directory-manager-modules
17264e5da41c | Bug #53581: Merge branch 'jtorres/53581-Python3-RuntimeError-dictionary-changed-size' into 5.0-1
b74a2e5e9b28 | Bug #53581: Python3: RuntimeError at resource sharing dictionaries on multithread enviroments

univention-directory-manager-modules (15.0.11-33)
17264e5da41c | Bug #53581: Merge branch 'jtorres/53581-Python3-RuntimeError-dictionary-changed-size' into 5.0-1
b74a2e5e9b28 | Bug #53581: Python3: RuntimeError at resource sharing dictionaries on multithread enviroments
Comment 3 Jürn Brodersen univentionstaff 2022-03-21 12:34:33 CET
Commit b74a2e5e9b28a0da859efd9a91020064dd048a20 seems to break a lot of the exam school tests:

https://jenkins.knut.univention.de:8181/job/UCSschool-5.0/job/Install%20Multiserver/lastCompletedBuild/testReport/

https://jenkins.knut.univention.de:8181/job/UCSschool-5.0/job/Install%20Multiserver/lastCompletedBuild/Config=s4,TestGroup=base1,UCSRelease=public/testReport/90_ucsschool/21_computerroom_module_base_checks/test_computerroom_module_base_checks/

Could you please take a look what broke here? Thanks :)

It also looks like the normal udm tests didn't caught this? Please also check if we need an additional test for this scenario.
Comment 4 Arvid Requate univentionstaff 2022-03-21 13:25:13 CET
> It also looks like the normal udm tests didn't caught this?

I think it's the multi-threaded code in UCS@school that triggers this.

We now copy the dicts before changing them. We should check how the UCS@school specific module "computerroom" triggers this behavior, where the key `ucsschoolRole` cannot be found. Either we fix it in UDM or we need to create another bugfix in UCS@school, but that would be more complex to coordinate the release of both. So perferrably we can identify how UCS@school triggers this but create a workaround/fix in UDM.
Comment 5 Jürn Brodersen univentionstaff 2022-03-21 18:42:26 CET
Created attachment 10927 [details]
PoC fix

(In reply to Arvid Requate from comment #4)
> > It also looks like the normal udm tests didn't caught this?
> 
> I think it's the multi-threaded code in UCS@school that triggers this.
> 
> We now copy the dicts before changing them. We should check how the
> UCS@school specific module "computerroom" triggers this behavior, where the
> key `ucsschoolRole` cannot be found. Either we fix it in UDM or we need to
> create another bugfix in UCS@school, but that would be more complex to
> coordinate the release of both. So perferrably we can identify how
> UCS@school triggers this but create a workaround/fix in UDM.

I'm not sure about the specific UCS@school code that triggered this, but there seems to be a reference to the old, changed out, dict. See the attached patch. Please note I only tested this with the "21_computerroom_module_base_checks.py" test where it did worked, but it is more of a PoC to show the problem.

Maybe property_descriptions could be implemented using a @property decorator? That reference change would catch me off guard. To be fair I only realize this in retrospective.
Comment 6 Arvid Requate univentionstaff 2022-03-21 20:01:17 CET
Created attachment 10928 [details]
53581_proposal2.patch

Discussed with Florian:
* "dunder" (__) would mangle the name to _simpleLdap__description
* We don't need to store _description locally since we get it every time
* We don't need and should not offer a setter as public API
Comment 7 Juan Pedro Torres univentionstaff 2022-03-23 10:54:17 CET
Applied the patch (53581_proposal2.patch) suggested by Arvid in Comment 6.
Comment 8 Arvid Requate univentionstaff 2022-03-23 12:28:25 CET
Verified:
* Package update
* Jenkins Tests UCS@school
* Advisory