Univention Bugzilla – Bug 53792
Keep last ID of UserImportJob in DB when updating from 4.4 to 5.0
Last modified: 2021-11-29 17:20:09 CET
+++ This bug was initially created as a clone of Bug #53451 +++ In UCS 5.0 univention-celery has been removed. The ucs-school-import uses a univention-celery configuration. We need to implement a systemd replacement. --------------------------------------------------------------------- The replacement library for dj_celery (django_celery_results) uses a different table to store the result data of Celery. Thus the primary key of the UserImportJob database objects will start at 1. That PK is used as the job ID in the directory path for storage of import artifacts in /var/lib/ucs-school-import/jobs/<YYYY>/<PK>. Restore the auto-increment ID of the SQL table storing UserImportJob data after an update.
ALTER SEQUENCE django_celery_results_taskresult_id_seq RESTART WITH 5; SELECT * FROM information_schema.sequences where sequence_name = 'django_celery_results_taskresult_id_seq';
Are the following also relevant?: import_api_textartifact_id_seq import_api_userimportjob_id_seq
SELECT nextval('celery_taskmeta_id_seq');
Added to the joinscript: printf "SELECT CONCAT('ALTER SEQUENCE django_celery_results_taskresult_id_seq RESTART WITH ', (SELECT nextval('celery_taskmeta_id_seq')), ';'); \\gexec" | su - postgres -c "psql -d '$POSTGRES_DB'" ucs-school-import (18.0.2) f868c6244791 | Bug #53792: keep last ID of UserImportJob in DB when updating from 4.4 to 5.0
I think "import_api_userimportjob_id_seq" is the relevant one, not "django_celery_results_taskresult_id_seq". In ucsschool.http_api.import_api.serializers.UserImportJobSerializer.create() "basedir" of an UserImportJob instance is set to "/var/.../{instance.pk}", with "pk" being the primary key column of the SQL table for UserImportJob objects. Please test the following: in 4.4 run a few imports, it doesn't matter if the dry-run fails. They will create directories "/var/lib/ucs-school-import/jobs/2021/{pk}" and they will create objects in "https://{IP}/api/v1/imports/users/". You can login (top right) with "Administrator". Then update to 5.0 and execute an import. It should create a new directory below "/var/lib/ucs-school-import/jobs/2021/" and a new entry in "https://{IP}/api/v1/imports/users/" with a higher number than the last import (from 4.4).
(In reply to Daniel Tröder from comment #6) > I think "import_api_userimportjob_id_seq" is the relevant one, not > "django_celery_results_taskresult_id_seq". Then we don't need any migration code at all because the table is not touched.
*** Bug 53928 has been marked as a duplicate of this bug. ***
REOPEN: (In reply to Tobias Wenzel from comment #0) > when upgrading a multi-server environment from 4.4 -> 5.0 with users created > in 4.4 the import joinscript failed with the following error: > > Upgrading schema from UCS@school 4.4 to UCS@school 5.0 > DELETE 0 > Operations to perform: > Apply all migrations: django_celery_results > Running migrations: > No migrations to apply. > FEHLER: Constraint > »import_api_userimportj_result_id_7383b884_fk_celery_taskmeta_id« von > Relation »import_api_userimportjob« existiert nicht > 40ucs-school-import-http-api.inst: Failed to drop djcelery constraint
Upgrading from 4.4 to 5.0 I get a different error (in /var/log/univention/join.log): RUNNING 40ucs-school-import-http-api.inst [..] Upgrading schema from UCS@school 4.4 to UCS@school 5.0 DELETE 3 Operations to perform: Apply all migrations: django_celery_results Running migrations: Applying django_celery_results.0001_initial... OK Applying django_celery_results.0002_add_task_name_args_kwargs... OK Applying django_celery_results.0003_auto_20181106_1101... OK ALTER TABLE FEHLER: Einfügen oder Aktualisieren in Tabelle »import_api_userimportjob« verletzt Fremdschlüssel-Constraint »impor_result_id_7383b884_fk_django_celery_results_taskresult_id« DETAIL: Schlüssel (result_id)=(1) ist nicht in Tabelle »django_celery_results_taskresult« vorhanden. 40ucs-school-import-http-api.inst: Failed to create django_celery_results constraint EXITCODE=1
When upgrading from 4.4 to 5.0, then complete import job DB will now be dropped and recreated. The last job ID can be retrieved in 5.0, so no action is required in 4.4. Please QA the "ucs-school-import" package in git branch "dtroeder/53792_drop_db": [4b18261a7] Bug #53792: drop and recreate complete import job DB when upgrading from 4.4 to 5.0
The code was merged and the package built. [5.0] 8df115379 Bug #53792: drop and recreate complete import job DB when upgrading from 4.4 to 5.0 [5.0] 2baba68c6 Bug #53792: changelog ucs-school-import (18.0.11)
I tested in the following manner: 4.4 UCS@school system Run some imports Activate testappcenter and update to 5.0 activate ucsschool 5.0 scope update to new import script make sure joinscript has run start another import -- On three test runs the importer will hang without any trace in the logs. A currently prepared system can be observed at *.6.110. Daniel, you might have a look at the system and see what might be wrong or if I made any mistake.
The join script updated the wrong database table sequence. It should work now. [5.0] eba40ec28 Bug #53792: update correct sequence number ucs-school-import (18.0.12) Note: The join script will not execute the migration code when it is force-executed. This is intentional, to allow users to run the join scripts as often as they like, without loosing their data. To execute the migration code edit /var/univention-join/status and set ucs-school-import-http-api to "v12".
It was tested with the updated UCS@school 5.0v1 version in the Test AppCenter. 4.4 UCS@school system Run some imports Activate testappcenter and update to 5.0 start another import It works properly and the correct next import number was used. We wait for the tests and then this will be VERIFIED.
Jenkins looks fine, no import tests fail.
UCS@school 5.0 v1 has been released. https://docs.software-univention.de/release-notes-ucsschool-5.0v1-de.html If this error occurs again, please clone this bug.