Bug 53256 - Traceback from univention-pkgdb-scan: Out of memory
Traceback from univention-pkgdb-scan: Out of memory
Status: NEW
Product: UCS
Classification: Unclassified
Component: pkgdb
UCS 5.0
Other Linux
: P5 normal (vote)
: ---
Assigned To: UCS maintainers
UCS maintainers
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2021-05-12 19:15 CEST by Arvid Requate
Modified: 2021-05-12 19:15 CEST (History)
0 users

See Also:
What kind of report is it?: Development Internal
What type of bug is this?: ---
Who will be affected by this bug?: ---
How will those affected feel about the bug?: ---
User Pain:
Enterprise Customer affected?:
School Customer affected?:
ISV affected?:
Waiting Support:
Flags outvoted (downgraded) after PO Review:
Ticket number:
Bug group (optional): Error handling, Regression
Max CVSS v3 score:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Arvid Requate univentionstaff 2021-05-12 19:15:33 CEST
This traceback from univention-pkgdb-scan was found in the jenkins test

https://jenkins.knut.univention.de:8181/job/UCS-5.0/job/UCS-5.0-0/job/AutotestJoin/SambaVersion=no-samba,Systemrolle=master-part-II/ws/test/appcenter.log

==========================================================================
Traceback (most recent call last):
  File "/usr/sbin/univention-pkgdb-scan", line 37, in <module>
    univention.pkgdb.main()
  File "/usr/lib/python2.7/dist-packages/univention/pkgdb.py", line 620, in main
    return action_scan(connection, cursor, config_registry)
  File "/usr/lib/python2.7/dist-packages/univention/pkgdb.py", line 528, in action_scan
    scan_and_store_packages(cursor, sysname, fake_null, architecture)
  File "/usr/lib/python2.7/dist-packages/univention/pkgdb.py", line 478, in scan_and_store_packages
    cursor.execute(insert_statement)
  File "/usr/lib/python2.7/dist-packages/pgdb.py", line 1030, in execute
    return self.executemany(operation, [parameters])
  File "/usr/lib/python2.7/dist-packages/pgdb.py", line 1054, in executemany
    rows = self._src.execute(sql)
pg.OperationalError: ERROR:  out of memory
DETAIL:  Failed on request of size 262144 in memory context "ErrorContext".
==========================================================================

Maybe the insert statement is too large? We could probably break that up easily. That number is 256k and there are similar reports on the net with 512k etc and for different memory contexts. Maybe the exact details depend on the particular situation when the memory limit is hit.

See also: https://www.google.com/search?q=postgresql+"Failed+on+request+of+size"+"ErrorContext"


Context from the log:
==========================================================================
  5635 packages                         21-05-12 00:54:45 [    INFO]: Entfernen von test-install-conflicting-packages (1.0) ...
  5635 packages                         21-05-12 00:54:46 [    INFO]: 
Reading package lists... 0%

Reading package lists... 100%

Reading package lists... Done
  5635 packages                         21-05-12 00:54:46 [    INFO]: 
Building dependency tree... 0%

Building dependency tree... 0%

Building dependency tree... 50%

Building dependency tree... 50%

Building dependency tree       
  5635 packages                         21-05-12 00:54:46 [    INFO]: 
Reading state information... 0%

Reading state information... 0%

Reading state information... Done
  5635 packages                         21-05-12 00:55:20 [ WARNING]: Traceback (most recent call last):
  5635 packages                         21-05-12 00:55:20 [ WARNING]:   File "/usr/sbin/univention-pkgdb-scan", line 37, in <module>
  5635 packages                         21-05-12 00:55:20 [ WARNING]:     univention.pkgdb.main()
  5635 packages                         21-05-12 00:55:20 [ WARNING]:   File "/usr/lib/python2.7/dist-packages/univention/pkgdb.py", line 620, in main
  5635 packages                         21-05-12 00:55:20 [ WARNING]:     return action_scan(connection, cursor, config_registry)
  5635 packages                         21-05-12 00:55:20 [ WARNING]:   File "/usr/lib/python2.7/dist-packages/univention/pkgdb.py", line 528, in action_scan
  5635 packages                         21-05-12 00:55:20 [ WARNING]:     scan_and_store_packages(cursor, sysname, fake_null, architecture)
  5635 packages                         21-05-12 00:55:20 [ WARNING]:   File "/usr/lib/python2.7/dist-packages/univention/pkgdb.py", line 478, in scan_and_store_packages
  5635 packages                         21-05-12 00:55:20 [ WARNING]:     cursor.execute(insert_statement)
  5635 packages                         21-05-12 00:55:20 [ WARNING]:   File "/usr/lib/python2.7/dist-packages/pgdb.py", line 1030, in execute
  5635 packages                         21-05-12 00:55:20 [ WARNING]:     return self.executemany(operation, [parameters])
  5635 packages                         21-05-12 00:55:20 [ WARNING]:   File "/usr/lib/python2.7/dist-packages/pgdb.py", line 1054, in executemany
  5635 packages                         21-05-12 00:55:20 [ WARNING]:     rows = self._src.execute(sql)
  5635 packages                         21-05-12 00:55:20 [ WARNING]: pg.OperationalError: ERROR:  out of memory
  5635 packages                         21-05-12 00:55:20 [ WARNING]: DETAIL:  Failed on request of size 262144 in memory context "ErrorContext".
  5635 packages                         21-05-12 00:55:20 [ WARNING]: 
  5635 actions.remove.progress          21-05-12 00:55:23 [   DEBUG]: 45
  5635 actions.remove                   21-05-12 00:55:23 [    INFO]: Removing localhost from LDAP object
  5635 actions.remove                   21-05-12 00:55:27 [   DEBUG]: Calling /etc/init.d/apache2 reload
  5635 actions.remove                   21-05-12 00:55:27 [    INFO]: Reloading apache2 configuration (via systemctl): apache2.service.
  5635 actions.remove                   21-05-12 00:55:28 [   DEBUG]: /etc/init.d/apache2 returned with 0
  5635 actions.remove.progress          21-05-12 00:55:28 [   DEBUG]: 55
==========================================================================