Univention Bugzilla – Bug 54880
SAML login causes "_dbm.error: cannot add item to database"
Last modified: 2024-05-10 16:21:52 CEST
The following traceback shows, if you are login with saml on a school replica. UCS: 5.0-1 errata310 Installed: cups=2.2.1 dhcp-server=12.0 prometheus-node-exporter=2.0.1 radius=5.0 samba4=4.13 squid=3.5 ucsschool=5.0 v1 4.4/ucsschool-veyon-proxy=1.1 Upgradable: 17.06.22 15:28:30.238 MAIN ( ERROR ) : Traceback (most recent call last): File "/usr/lib/python3/dist-packages/cherrypy/_cprequest.py", line 670, in respond response.body = self.handler() File "/usr/lib/python3/dist-packages/cherrypy/lib/encoding.py", line 220, in __call__ self.body = self.oldhandler(*args, **kwargs) File "/usr/lib/python3/dist-packages/cherrypy/_cpdispatch.py", line 60, in __call__ return self.callable(*self.args, **self.kwargs) File "/usr/sbin/univention-management-console-web-server", line 1258, in index return acs(binding, message, relay_state) File "/usr/sbin/univention-management-console-web-server", line 1281, in attribute_consuming_service_iframe response = self.acs(message, binding) File "/usr/sbin/univention-management-console-web-server", line 1395, in acs response = self.sp.parse_authn_request_response(message, binding, self.outstanding_queries) File "/usr/lib/python3/dist-packages/saml2/client_base.py", line 717, in parse_authn_request_response self.users.add_information_about_person(resp.session_info()) File "/usr/lib/python3/dist-packages/saml2/population.py", line 27, in add_information_about_person session_info["not_on_or_after"]) File "/usr/lib/python3/dist-packages/saml2/cache.py", line 129, in set self._db[cni] = data File "/usr/lib/python3.7/shelve.py", line 125, in __setitem__ self.dict[key.encode(self.keyencoding)] = f.getvalue() _dbm.error: cannot add item to database I have no idea were this comes from and what the root cause is. Restarting the services does not fix the problem.
1. Login with SAML must be done on a primary or backup node (ucs-sso.$domain). 2. Please describe the complete use case: where did the user log in? What tile was clicked next? What module did the user try to use? Where did the traceback appear?
The customer cannot use any module, because he is not able to login on the portal of the school replica anymore. The traceback is directly shown after the Login. No modules could be seen at all.
That may be a broken BDB. I think it's /var/cache/univention-management-console/saml-8090.bdb.db Try: db_verify /var/cache/univention-management-console/saml-8090.bdb.db Or ask a UMC developer. This is not a UCS@school specific error.
The verication failed: [...] db_verify: BDB0540 Page 5247: invalid next_pgno 5727 db_verify: BDB0539 Page 5255: invalid prev_pgno 5494 db_verify: /var/cache/univention-management-console/saml-8090.bdb.db: BDB0090 DB_VERIFY_BAD: Database verification failed BDB5105 Verification of /var/cache/univention-management-console/saml-8090.bdb.db failed. ------ Workaround: Because this file is located in /var/cache/univention-management-console/, so in a cache directory, we tried to move it somewhere else, and it was recreated when the umc-{web,}-server were restarted. This fixed the problem
I reopen this bug, because the workaround involves a manual step by an administrator. From my point of view, the univention-management-console should detect that the cache file is corrupt, and recreate it by itself. But we have to make sure that losing the information in the file is not an issue.
Another Customer affected. Restart of univention-management-console-web-server helped. Added Ticket to bug.
Another Customer is affected. He noted that the SAML login stopped working after the /etc/cron.d/univention-maintenance cronjob.
Another customer is affected. In this case, it's a backup domain controller
An other customer affected (2022101421000339)
Another costumer is effected on the Master System = 2022103121000637
An other customer 2022110321000732
to prevent it completely (doesn't work with multiprocessing) this patch can be applied: diff --git management/univention-management-console/univention-management-console-web-server management/univention-management-console/univention-management-console-web-server index 3352b8b32f..a64d3b6f36 100755 --- management/univention-management-console/univention-management-console-web-server +++ management/univention-management-console/univention-management-console-web-server @@ -1236,7 +1236,7 @@ class SAML(Ressource): CORE.info('Reloading SAML service provider configuration') sys.modules.pop(os.path.splitext(os.path.basename(cls.configfile))[0], None) try: - cls.SP = Saml2Client(config_file=cls.configfile, identity_cache=cls.identity_cache % (PORT,), state_cache=cls.state_cache) + cls.SP = Saml2Client(config_file=cls.configfile, identity_cache=None, state_cache=cls.state_cache) return True except Exception: CORE.warn('Startup of SAML2.0 service provider failed:\n%s' % (traceback.format_exc(),))
Better format and the workaround from Comment 4 was used! Debugging information from a customer environment :~# ls -lah /var/cache/univention-management-console/ │insgesamt 12M │drwxr-xr-x 3 root root 4,0K Aug 29 22:35 . │drwxr-xr-x 28 root root 4,0K Aug 29 22:06 .. │drwxr-xr-x 2 root root 176K Nov 10 09:52 acls │-rw------- 1 root root 19M Okt 25 15:16 saml-8090.bdb.db :~# db_verify /var/cache/univention-management-console/saml-8090.bdb.db │db_verify: BDB0539 Page 3: invalid prev_pgno 490 │db_verify: /var/cache/univention-management-console/saml-8090.bdb.db: BDB0090 DB_VERIFY_BAD: Database verification failed │BDB5105 Verification of /var/cache/univention-management-console/saml-8090.bdb.db failed. After applying the workaround from comment 4 in this Bug, we only get a saml-8090.bdb (.db is missing and the verify still fails) :~# ls -lah /var/cache/univention-management-console/ │insgesamt 204K │drwxr-xr-x 3 root root 4,0K Nov 11 16:38 . |drwxr-xr-x 28 root root 4,0K Aug 29 22:06 .. │drwxr-xr-x 2 root root 176K Nov 10 09:52 acls |-rw------- 1 root root 16K Nov 11 16:38 saml-8090.bdb :~# db_verify /var/cache/univention-management-console/saml-8090.bdb │db_verify: BDB0641 __db_meta_setup: /var/cache/univention-management-console/saml-8090.bdb: unexpected file type or format │db_verify: BDB0524 Page 0: pgno incorrectly set to 4096 │db_verify: BDB0525 Page 0: bad magic number 0 │db_verify: BDB0527 Page 0: bad page size 9 |db_verify: /var/cache/univention-management-console/saml-8090.bdb: BDB0090 DB_VERIFY_BAD: Database verification failed Is this really a workaround, or is something else breaking
With Bug#55424 closed, I downvote this bug until it re-emerges in an environment that needs to have in-memory cache disabled.
Some findings from earlier investigations: saml-8090.bdb.db means that the file was initially created as a dbm.ndbm database. saml-8090.bdb means that the file was initially created as a dbm.gdbm database. The latter happens since we included python3-gdbm on every UCS some time ago for the group cache. If you remove the ndbm file which has been created with UCS 5.0-1, it will be re-created on UMC-web-server startup as a gdbm file. And verify_db fails on empty databases. The root cause is still unknown. We suspect parallel writers but it is hard to reproduce under laboratory conditions.
The customer used the ucr Variable umc/saml/in-memory-identity-cache=false to get rid of the "_dbm.error: cannot add item to database" messages, but they still occur and the univention-management-console-web-server needs manual restarting The customer gets this on all of his Portal servers on different days. We need a fix here.
The traceback from comment 30 has been moved to bug 56303, it affects only UCS 5.0-4
#2023082921000402 1) Schritte, um den Fehler zu reproduzieren- einloggen (Benutzer), Passwort eingegeben, dann erscheint eine schwarze Seite mit diversen Fehlermeldungen , die zu dieser Mail letztendlich leitet. 2) erwartetes Ergebnis- ich möchte mich einloggen, um dann zur zweiten Authentifizierung zu gelangen. 3) beobachtetes Ergebnis- Siege unter 1, kein Einloggen möglich, nicht zum ersten, sondern wiederholtem Male auf verschiedenen Geräten
Reported over feedback@univention.de Traceback (most recent call last): File "/usr/lib/python3/dist-packages/tornado/web.py", line 1595, in _execute result = yield result File "/usr/lib/python3/dist-packages/tornado/gen.py", line 1133, in run value = future.result() File "/usr/lib/python3/dist-packages/univention/management/console/saml.py", line 232, in get await acs(binding, message, relay_state) File "/usr/lib/python3/dist-packages/univention/management/console/saml.py", line 237, in attribute_consuming_service response = self.parse_authn_response(message, binding) File "/usr/lib/python3/dist-packages/univention/management/console/saml.py", line 324, in parse_authn_response response = self.sp.parse_authn_request_response(message, binding, self.outstanding_queries) File "/usr/lib/python3/dist-packages/saml2/client_base.py", line 717, in parse_authn_request_response self.users.add_information_about_person(resp.session_info()) File "/usr/lib/python3/dist-packages/saml2/population.py", line 27, in add_information_about_person session_info["not_on_or_after"]) File "/usr/lib/python3/dist-packages/saml2/cache.py", line 129, in set self._db[cni] = data File "/usr/lib/python3.7/shelve.py", line 125, in __setitem__ self.dict[key.encode(self.keyencoding)] = f.getvalue() _dbm.error: cannot add item to database
Customer affected Ticket#2023120521000289 UCS: 5.0-5 errata880 Installed: ox-connector=2.2.7 privacyidea-saml=2.1.2 self-service=5.0 self-service-backend=5.0 ucsschool=5.0 v4 4.4/itslearning=5.0 Upgradable: privacyidea-saml itslearning root@pucs:/# ucr get umc/saml/in-memory-identity-cache false root@pucs:/# ucr get umc/http/processes 10 [Error description]: When logging in to the school portal, after entering the correct login data and clicking on the login button, the error message: cannot add item to database root@pucs:/var/cache/univention-management-console# db_verify saml-18202.bdb.db db_verify: BDB0540 Page 1: invalid next_pgno 8780 db_verify: BDB0540 Page 2: invalid next_pgno 6952 db_verify: saml-18202.bdb.db: BDB0090 DB_VERIFY_BAD: Database verification failed BDB5105 Verification of saml-18202.bdb.db failed. /var/cache/univention-management-console# for f in ./*.bdb*; do db_verify $f; done db_verify: BDB0540 Page 2: invalid next_pgno 19150 db_verify: ./saml-18200.bdb.db: BDB0090 DB_VERIFY_BAD: Database verification failed BDB5105 Verification of ./saml-18200.bdb.db failed. db_verify: BDB0540 Page 1: invalid next_pgno 17991 db_verify: BDB0540 Page 2: invalid next_pgno 17992 db_verify: ./saml-18201.bdb.db: BDB0090 DB_VERIFY_BAD: Database verification failed BDB5105 Verification of ./saml-18201.bdb.db failed. db_verify: BDB0540 Page 1: invalid next_pgno 8780 db_verify: BDB0540 Page 2: invalid next_pgno 6952 db_verify: ./saml-18202.bdb.db: BDB0090 DB_VERIFY_BAD: Database verification failed BDB5105 Verification of ./saml-18202.bdb.db failed. db_verify: BDB0540 Page 1: invalid next_pgno 10219 db_verify: BDB0540 Page 2: invalid next_pgno 7562 db_verify: ./saml-18203.bdb.db: BDB0090 DB_VERIFY_BAD: Database verification failed BDB5105 Verification of ./saml-18203.bdb.db failed. db_verify: BDB0540 Page 1: invalid next_pgno 6217 db_verify: ./saml-18204.bdb.db: BDB0090 DB_VERIFY_BAD: Database verification failed BDB5105 Verification of ./saml-18204.bdb.db failed. db_verify: BDB0540 Page 1: invalid next_pgno 4901 db_verify: BDB0540 Page 2: invalid next_pgno 6802 db_verify: ./saml-18205.bdb.db: BDB0090 DB_VERIFY_BAD: Database verification failed BDB5105 Verification of ./saml-18205.bdb.db failed. db_verify: BDB0540 Page 1: invalid next_pgno 11187 db_verify: ./saml-18206.bdb.db: BDB0090 DB_VERIFY_BAD: Database verification failed BDB5105 Verification of ./saml-18206.bdb.db failed. db_verify: BDB0540 Page 1: invalid next_pgno 4326 db_verify: BDB0540 Page 2: invalid next_pgno 5509 db_verify: ./saml-18207.bdb.db: BDB0090 DB_VERIFY_BAD: Database verification failed BDB5105 Verification of ./saml-18207.bdb.db failed. BDB5105 Verification of ./saml-18208.bdb.db succeeded. BDB5105 Verification of ./saml-18209.bdb.db succeeded. BDB5105 Verification of ./saml-8090.bdb.db succeeded. db_verify: BDB0641 __db_meta_setup: ./saml.bdb: unexpected file type or format db_verify: BDB0524 Page 0: pgno incorrectly set to 4096 db_verify: BDB0525 Page 0: bad magic number 0 db_verify: BDB0527 Page 0: bad page size 9 db_verify: ./saml.bdb: BDB0090 DB_VERIFY_BAD: Database verification failed BDB5105 Verification of ./saml.bdb failed. Traceback (most recent call last): File "/usr/lib/python3/dist-packages/tornado/web.py", line 1595, in _execute result = yield result File "/usr/lib/python3/dist-packages/tornado/gen.py", line 1133, in run value = future.result() File "/usr/lib/python3/dist-packages/univention/management/console/saml.py", line 232, in get await acs(binding, message, relay_state) File "/usr/lib/python3/dist-packages/univention/management/console/saml.py", line 237, in attribute_consuming_service response = self.parse_authn_response(message, binding) File "/usr/lib/python3/dist-packages/univention/management/console/saml.py", line 324, in parse_authn_response response = self.sp.parse_authn_request_response(message, binding, self.outstanding_queries) File "/usr/lib/python3/dist-packages/saml2/client_base.py", line 717, in parse_authn_request_response self.users.add_information_about_person(resp.session_info()) File "/usr/lib/python3/dist-packages/saml2/population.py", line 27, in add_information_about_person session_info["not_on_or_after"]) File "/usr/lib/python3/dist-packages/saml2/cache.py", line 129, in set self._db[cni] = data File "/usr/lib/python3.7/shelve.py", line 125, in __setitem__ self.dict[key.encode(self.keyencoding)] = f.getvalue() _dbm.error: cannot add item to database
2023121421000557 Traceback (most recent call last): File "/usr/lib/python3/dist-packages/tornado/web.py", line 1595, in _execute result = yield result File "/usr/lib/python3/dist-packages/tornado/gen.py", line 1133, in run value = future.result() File "/usr/lib/python3/dist-packages/univention/management/console/saml.py", line 232, in get await acs(binding, message, relay_state) File "/usr/lib/python3/dist-packages/univention/management/console/saml.py", line 237, in attribute_consuming_service response = self.parse_authn_response(message, binding) File "/usr/lib/python3/dist-packages/univention/management/console/saml.py", line 324, in parse_authn_response response = self.sp.parse_authn_request_response(message, binding, self.outstanding_queries) File "/usr/lib/python3/dist-packages/saml2/client_base.py", line 717, in parse_authn_request_response self.users.add_information_about_person(resp.session_info()) File "/usr/lib/python3/dist-packages/saml2/population.py", line 27, in add_information_about_person session_info["not_on_or_after"]) File "/usr/lib/python3/dist-packages/saml2/cache.py", line 129, in set self._db[cni] = data File "/usr/lib/python3.7/shelve.py", line 125, in __setitem__ self.dict[key.encode(self.keyencoding)] = f.getvalue() _dbm.error: cannot add item to database
2023121521000162 Traceback (most recent call last): File "/usr/lib/python3/dist-packages/tornado/web.py", line 1595, in _execute result = yield result File "/usr/lib/python3/dist-packages/tornado/gen.py", line 1133, in run value = future.result() File "/usr/lib/python3/dist-packages/univention/management/console/saml.py", line 232, in get await acs(binding, message, relay_state) File "/usr/lib/python3/dist-packages/univention/management/console/saml.py", line 237, in attribute_consuming_service response = self.parse_authn_response(message, binding) File "/usr/lib/python3/dist-packages/univention/management/console/saml.py", line 324, in parse_authn_response response = self.sp.parse_authn_request_response(message, binding, self.outstanding_queries) File "/usr/lib/python3/dist-packages/saml2/client_base.py", line 717, in parse_authn_request_response self.users.add_information_about_person(resp.session_info()) File "/usr/lib/python3/dist-packages/saml2/population.py", line 27, in add_information_about_person session_info["not_on_or_after"]) File "/usr/lib/python3/dist-packages/saml2/cache.py", line 129, in set self._db[cni] = data File "/usr/lib/python3.7/shelve.py", line 125, in __setitem__ self.dict[key.encode(self.keyencoding)] = f.getvalue() _dbm.error: cannot add item to database
2023121421000441 _dbm.error: cannot add item to database self.dict[key.encode(self.keyencoding)] = f.getvalue() File "/usr/lib/python3.7/shelve.py", line 125, in __setitem__ self._db[cni] = data File "/usr/lib/python3/dist-packages/saml2/cache.py", line 129, in set session_info["not_on_or_after"]) File "/usr/lib/python3/dist-packages/saml2/population.py", line 27, in add_information_about_person self.users.add_information_about_person(resp.session_info()) File "/usr/lib/python3/dist-packages/saml2/client_base.py", line 717, in parse_authn_request_response response = self.sp.parse_authn_request_response(message, binding, self.outstanding_queries) File "/usr/lib/python3/dist-packages/univention/management/console/saml.py", line 324, in parse_authn_response response = self.parse_authn_response(message, binding) File "/usr/lib/python3/dist-packages/univention/management/console/saml.py", line 237, in attribute_consuming_service await acs(binding, message, relay_state) File "/usr/lib/python3/dist-packages/univention/management/console/saml.py", line 232, in get value = future.result() File "/usr/lib/python3/dist-packages/tornado/gen.py", line 1133, in run result = yield result File "/usr/lib/python3/dist-packages/tornado/web.py", line 1595, in _execute Traceback (most recent call last):
2023121421000324 Traceback (most recent call last): File "/usr/lib/python3/dist-packages/tornado/web.py", line 1595, in _execute result = yield result File "/usr/lib/python3/dist-packages/tornado/gen.py", line 1133, in run value = future.result() File "/usr/lib/python3/dist-packages/univention/management/console/saml.py", line 232, in get await acs(binding, message, relay_state) File "/usr/lib/python3/dist-packages/univention/management/console/saml.py", line 237, in attribute_consuming_service response = self.parse_authn_response(message, binding) File "/usr/lib/python3/dist-packages/univention/management/console/saml.py", line 324, in parse_authn_response response = self.sp.parse_authn_request_response(message, binding, self.outstanding_queries) File "/usr/lib/python3/dist-packages/saml2/client_base.py", line 717, in parse_authn_request_response self.users.add_information_about_person(resp.session_info()) File "/usr/lib/python3/dist-packages/saml2/population.py", line 27, in add_information_about_person session_info["not_on_or_after"]) File "/usr/lib/python3/dist-packages/saml2/cache.py", line 129, in set self._db[cni] = data File "/usr/lib/python3.7/shelve.py", line 125, in __setitem__ self.dict[key.encode(self.keyencoding)] = f.getvalue() _dbm.error: cannot add item to database
2023121421000217 Traceback (most recent call last): File "/usr/lib/python3/dist-packages/tornado/web.py", line 1595, in _execute result = yield result File "/usr/lib/python3/dist-packages/tornado/gen.py", line 1133, in run value = future.result() File "/usr/lib/python3/dist-packages/univention/management/console/saml.py", line 232, in get await acs(binding, message, relay_state) File "/usr/lib/python3/dist-packages/univention/management/console/saml.py", line 237, in attribute_consuming_service response = self.parse_authn_response(message, binding) File "/usr/lib/python3/dist-packages/univention/management/console/saml.py", line 324, in parse_authn_response response = self.sp.parse_authn_request_response(message, binding, self.outstanding_queries) File "/usr/lib/python3/dist-packages/saml2/client_base.py", line 717, in parse_authn_request_response self.users.add_information_about_person(resp.session_info()) File "/usr/lib/python3/dist-packages/saml2/population.py", line 27, in add_information_about_person session_info["not_on_or_after"]) File "/usr/lib/python3/dist-packages/saml2/cache.py", line 129, in set self._db[cni] = data File "/usr/lib/python3.7/shelve.py", line 125, in __setitem__ self.dict[key.encode(self.keyencoding)] = f.getvalue() _dbm.error: cannot add item to database
2023121421000119 Traceback (most recent call last): File "/usr/lib/python3/dist-packages/tornado/web.py", line 1595, in _execute result = yield result File "/usr/lib/python3/dist-packages/tornado/gen.py", line 1133, in run value = future.result() File "/usr/lib/python3/dist-packages/univention/management/console/saml.py", line 232, in get await acs(binding, message, relay_state) File "/usr/lib/python3/dist-packages/univention/management/console/saml.py", line 237, in attribute_consuming_service response = self.parse_authn_response(message, binding) File "/usr/lib/python3/dist-packages/univention/management/console/saml.py", line 324, in parse_authn_response response = self.sp.parse_authn_request_response(message, binding, self.outstanding_queries) File "/usr/lib/python3/dist-packages/saml2/client_base.py", line 717, in parse_authn_request_response self.users.add_information_about_person(resp.session_info()) File "/usr/lib/python3/dist-packages/saml2/population.py", line 27, in add_information_about_person session_info["not_on_or_after"]) File "/usr/lib/python3/dist-packages/saml2/cache.py", line 129, in set self._db[cni] = data File "/usr/lib/python3.7/shelve.py", line 125, in __setitem__ self.dict[key.encode(self.keyencoding)] = f.getvalue() _dbm.error: cannot add item to database
(In reply to Mika Westphal from comment #40) Same error 2023121821000148 2023121721000113 2023121621000044 2023121521000261 2023121321000415 2023121221000426 2023120421000316 2023112921000334 2023112721000392 2023111121000073 2023110821000249 2023110321000276 2023110221000287 2023110221000134 2023103021000548 2023101221000082 2023101021000193 2023100921000231 2023091921000456 2023090621000319 2023040221000353
Upstream bug report: https://github.com/IdentityPython/pysaml2/issues/946
MR: https://git.knut.univention.de/univention/ucs/-/issues/1354
univention-management-console.yaml 4d094f28c242 | fix(umc): remove on disk SAML identity cache univention-management-console (12.0.32-4) 4d094f28c242 | fix(umc): remove on disk SAML identity cache the on disk DBM SAML identity cache database had to be removed and switched back to an in-memory cache. We ruled out an sequential write corruption error due to every UMC multiprocessing server having it's own DB. Therefore it's also not an issue to use the in-memory cache now. The respective UCR Variable `umc/saml/in-memory-identity-cache` to switch from the on-disk to the in-memory one has been removed all well
OK: multiprocessing is realized via apache loadbalancing with sticky sessions, therefore no on-disk database required OK: code review OK: YAML
https://errata.software-univention.de/#/?erratum=5.0x914
Applying the upgrade is equal to setting: "ucr set umc/saml/in-memory-identity-cache=true". This is perfectly fine even in multiprocessing mode - because we have sticky session there.
Customer affected 2024031921000143 UCS: 5.0-5