Bug 37720 - UMC-Caching
UMC-Caching
Status: CLOSED FIXED
Product: UCS
Classification: Unclassified
Component: UMC (Generic)
UCS 4.4
Other Linux
: P5 normal (vote)
: UCS 4.4-8-errata
Assigned To: Florian Best
Felix Botner
:
Depends on:
Blocks: 53680 53465
  Show dependency treegraph
 
Reported: 2015-02-06 14:25 CET by Florian Best
Modified: 2021-08-25 17:46 CEST (History)
7 users (show)

See Also:
What kind of report is it?: Development Internal
What type of bug is this?: ---
Who will be affected by this bug?: ---
How will those affected feel about the bug?: ---
User Pain:
Enterprise Customer affected?:
School Customer affected?:
ISV affected?:
Waiting Support:
Flags outvoted (downgraded) after PO Review:
Ticket number:
Bug group (optional): Cleanup, External feedback, Troubleshooting, Usability
Max CVSS v3 score:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Florian Best univentionstaff 2015-02-06 14:25:42 CET
When we build e.g. the UMC-frontend package the files in the deb archive will have mtime=$(build-time).
When dpkg installs the package the mtime from these files are taken instead of the extraction time.
When apache serves the files now it uses the mtime value with the build time.

Especially internally it is strange behavior as we are often working with VM's with set time in the past. When the file-mtime is in the future apache takes the current system time as Last-Modified value. → This prevents firefox from invalidating its cache.

Maybe we should trigger a touch over all UMC-apache-files in the postinst of UMC.
We probably can remove the $js-hash$ then, too and let apache handle caching completely (I suspect apache to do it correctly, our caching-workarounds probably exists due to this described problem. Chromium+Firefox can handle Last-Modified, and in UCS 4.0 we are supporting only IE>=9 → that should also work).

curl -i http://$host/univention-management-console/login.html | grep -i -e Etag -e Last-Modified
date --set '2014-12-13 14:15:16+0200'
Comment 1 Alexander Kläser univentionstaff 2015-02-10 15:52:55 CET
I observed problems with CSS background images that would not be updated after a page reload. Usually the time on my VMs is being synced upon login, just to exclude this as a possible cause for the misbehaviour.
Comment 2 Stefan Gohmann univentionstaff 2019-01-03 07:17:33 CET
This issue has been filled against UCS 4.0. The maintenance with bug and security fixes for UCS 4.0 has ended on 31st of May 2016.

Customers still on UCS 4.0 are encouraged to update to UCS 4.3. Please contact
your partner or Univention for any questions.

If this issue still occurs in newer UCS versions, please use "Clone this bug" or simply reopen the issue. In this case please provide detailed information on how this issue is affecting you.
Comment 3 Florian Best univentionstaff 2021-03-01 14:05:53 CET
The Last-Modified and E-Tag value still uses the file's modify-type from the date in the debian/changelog file (debhelper for reproducible build set it).
The solution we are going to implement is to touch all relevant files in the package postinst.

Further to this, we currently have caching settings of:
* 1 day for Javascript files
* 1 week for HTML/CSS files
* 1 month for images

(See ucs/management/univention-web/conffiles/etc/apache2/conf-available/univention-web.conf).

For the upgrade from UCS 4.4 to UCS 5.0 the images are not such a big problem, as we changed the path's of e.g. the icons.

To workaround the 1 week cache of HTML files, we should release an errata update for UCS 4.4 soon, which removes the time-based caching.

We change `Cache-Control: max-age=604800` for HTML/CSS/JS files to either
* `Cache-Control: must-revalidate, max-age=3600` → cache the file for one hour, after the hour always use conditional requests which compare the E-Tag
* `Cache-Control: no-cache` → always use conditional requests which compare the E-Tag
Comment 4 Florian Best univentionstaff 2021-03-26 00:10:50 CET
WIP patch: fbest/37720-umc-caching
Comment 5 Johannes Keiser univentionstaff 2021-04-27 16:07:08 CEST
/univention/management/modules/udm.js (and other files too)
are not covered by the Location directives (no Cache-Control headers are set for this file)

-----

Not sure if max-age=3600 is too long (e.g. update scenarios)

--------
I think the cache validation has to be adjusted
(https://developer.mozilla.org/en-US/docs/Web/HTTP/Caching#cache_validation)

In my test dojo.js was always downloaded regardless if e.g. ETag didn't change

// Changed file between last page visit
GET /univention/js/dojo/dojo.js HTTP/1.1
Host: 10.200.28.60
Connection: keep-alive
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.72 Safari/537.36
Accept: */*
Referer: http://10.200.28.60/univention/management/
Accept-Encoding: gzip, deflate
Accept-Language: de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7
Cookie: UMCLang=de-DE; UMCSessionId=570807bb-5a12-4022-a839-e63f2ad623c9; _pk_id.14.1e13=84acee4f4c68df23.1619504751.; _pk_ses.14.1e13=1
If-None-Match: "10b86c-5c0f07b11c740-gzip"
If-Modified-Since: Tue, 27 Apr 2021 08:59:33 GMT

HTTP/1.1 200 OK
Date: Tue, 27 Apr 2021 13:52:10 GMT
Server: Apache/2.4.38 (Univention)
X-Permitted-Cross-Domain-Policies: master-only
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Content-Security-Policy: frame-ancestors 'none';
Last-Modified: Tue, 27 Apr 2021 13:52:01 GMT
ETag: "10b883-5c0f4910a976b-gzip"
Accept-Ranges: bytes
Vary: Accept-Encoding
Content-Encoding: gzip
Cache-Control: max-age=0, must-revalidate
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Transfer-Encoding: chunked
Content-Type: application/javascript
	--------------------

// no file changes between last page visit
GET /univention/js/dojo/dojo.js HTTP/1.1
Host: 10.200.28.60
Connection: keep-alive
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.72 Safari/537.36
Accept: */*
Referer: http://10.200.28.60/univention/management/
Accept-Encoding: gzip, deflate
Accept-Language: de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7
Cookie: UMCLang=de-DE; UMCSessionId=570807bb-5a12-4022-a839-e63f2ad623c9; _pk_id.14.1e13=84acee4f4c68df23.1619504751.; _pk_ses.14.1e13=1
If-None-Match: "10b883-5c0f4910a976b-gzip"
If-Modified-Since: Tue, 27 Apr 2021 13:52:01 GMT

HTTP/1.1 200 OK
Date: Tue, 27 Apr 2021 13:54:48 GMT
Server: Apache/2.4.38 (Univention)
X-Permitted-Cross-Domain-Policies: master-only
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Content-Security-Policy: frame-ancestors 'none';
Last-Modified: Tue, 27 Apr 2021 13:52:01 GMT
ETag: "10b883-5c0f4910a976b-gzip"
Accept-Ranges: bytes
Vary: Accept-Encoding
Content-Encoding: gzip
Cache-Control: max-age=0, must-revalidate
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Transfer-Encoding: chunked
Content-Type: application/javascript
Comment 6 Florian Best univentionstaff 2021-05-25 17:55:46 CEST
(In reply to Johannes Keiser from comment #5)
> /univention/management/modules/udm.js (and other files too)
> are not covered by the Location directives (no Cache-Control headers are set
> for this file)
hm? They are!?!:
The correct path is 'management/js/umc/modules/udm.js'.
Comment 7 Florian Best univentionstaff 2021-06-18 12:26:12 CEST
We should additionally make an ajax-request with cache:no flag during the upgrade in the maintenance mode.
The maintenance mode should strictly set `Cache-Control: no-store` to all the "faked" pages.
Comment 8 Florian Best univentionstaff 2021-06-18 17:30:52 CEST
The following things have been done:

1. fix timestamp of *.js *.html *.css files, so that apache serves accurate Last-Modified infos.

2. during maintenance mode, the caching is disabled and new browsers are forced to reset the cache for the whole domain. This works brilliantly in Firefox, not really in Chromium somehow.
Via `Clear-Site-Data: "cache"` HTTP response header.
https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Clear-Site-Data
We can use this more intentionally in the future and enhance the whole upgrade process.

3. The important index.html umc.css and dojo.js now have "Cache-Control: no-cache" (aka. "must-revalidate, max-age=0" setting). Somehow Chromium always fetches the resources again without making conditional request.

4. Our caching defaults of 1 week for HTML/CSS and 1 day for Javascript files have been replaced with max-age = 1 hour. I don't like this change and am thinking about reverting it. We only loose cacheability with it and the upgrade is not affected due to 3).

univention-web.yaml
25ac9811b134 | Bug #37720: replace Expire-caching with Cache-Control "must-revalidate, max-age=..."

univention-web (3.0.6-3)
25ac9811b134 | Bug #37720: replace Expire-caching with Cache-Control "must-revalidate, max-age=..."

univention-management-console.yaml
4ba2ec304cf8 | Bug #37720: set explicit caching control in maintenance mode

univention-management-console (11.0.6-11)
4ba2ec304cf8 | Bug #37720: set explicit caching control in maintenance mode

univention-lib.yaml
04bd6e68287e | Bug #37720: touch all HTML/JS/CSS files in postinst via umc_frontend_new_hash

univention-lib (8.0.1-42)
04bd6e68287e | Bug #37720: touch all HTML/JS/CSS files in postinst via umc_frontend_new_hash
Comment 9 Daniel Tröder univentionstaff 2021-06-21 16:03:22 CEST
The change in umc.sh → umc_frontend_new_hash() breaks the join.
All Jenkins jobs fail.
----------------------------------------------------------
/var/log/apt/term.log

Setting up univention-management-console-module-apps (8.0.11-146A~4.4.0.202103041132) ...
File: /usr/share/univention-management-console/modules/apps.xml
File: /usr/share/univention-management-console/i18n/de/apps.mo
Calling joinscript 36univention-management-console-module-apps.inst ...
2021-06-21 15:26:29.906301108+02:00 (in joinscript_init)
The server has not joined yet
Joinscript 36univention-management-console-module-apps.inst finished with exitcode 1
Calling joinscript 36univention-management-console-module-apps.inst ...
2021-06-21 15:26:30.002103894+02:00 (in joinscript_init)
The server has not joined yet
Joinscript 36univention-management-console-module-apps.inst finished with exitcode 1
find: ‘locally’: No such file or directory
find: ‘diverted’: No such file or directory
find: ‘to:’: No such file or directory
find: ‘locally’: No such file or directory
find: ‘diverted’: No such file or directory
find: ‘to:’: No such file or directory
dpkg: error processing package univention-management-console-module-apps (--configure):
 subprocess installed post-installati
Comment 10 Florian Best univentionstaff 2021-06-21 17:59:22 CEST
(In reply to Daniel Tröder from comment #9)
> The change in umc.sh → umc_frontend_new_hash() breaks the join.
> All Jenkins jobs fail.
Where do you know from, that this change is the issue?
I can reproduce the error but it still exists with exit-code 0:

# DPKG_MAINTSCRIPT_PACKAGE=univention-management-console-module-apps umc_frontend_new_hash
find: ‘lokal’: Datei oder Verzeichnis nicht gefunden
find: ‘umgeleitet’: Datei oder Verzeichnis nicht gefunden
find: ‘zu:’: Datei oder Verzeichnis nicht gefunden
find: ‘lokal’: Datei oder Verzeichnis nicht gefunden
find: ‘umgeleitet’: Datei oder Verzeichnis nicht gefunden
find: ‘zu:’: Datei oder Verzeichnis nicht gefunden
Setting umc/web/cache_bust
File: /var/www/univention/meta.json
# echo $?
0
Comment 11 Florian Best univentionstaff 2021-06-21 18:04:49 CEST
(In reply to Florian Best from comment #10)
> (In reply to Daniel Tröder from comment #9)
> > The change in umc.sh → umc_frontend_new_hash() breaks the join.
> > All Jenkins jobs fail.
> Where do you know from, that this change is the issue?
> I can reproduce the error but it still exists with exit-code 0:
> 
> # DPKG_MAINTSCRIPT_PACKAGE=univention-management-console-module-apps
> umc_frontend_new_hash
> find: ‘lokal’: Datei oder Verzeichnis nicht gefunden
> find: ‘umgeleitet’: Datei oder Verzeichnis nicht gefunden
> find: ‘zu:’: Datei oder Verzeichnis nicht gefunden
> find: ‘lokal’: Datei oder Verzeichnis nicht gefunden
> find: ‘umgeleitet’: Datei oder Verzeichnis nicht gefunden
> find: ‘zu:’: Datei oder Verzeichnis nicht gefunden
> Setting umc/web/cache_bust
> File: /var/www/univention/meta.json
> # echo $?
> 0

Nevertheless, fixed that one with:

univention-lib 8.0.1-42A~4.4.0.202106211803

univention-lib (8.0.1-42)
ceaa90785268 | fixup! Bug #37720: touch all HTML/JS/CSS files in postinst via umc_frontend_new_hash
Comment 12 Daniel Tröder univentionstaff 2021-06-23 07:19:31 CEST
(In reply to Florian Best from comment #10)
> (In reply to Daniel Tröder from comment #9)
> > The change in umc.sh → umc_frontend_new_hash() breaks the join.
> > All Jenkins jobs fail.
> Where do you know from, that this change is the issue?
I reverted the change in a test VM and then the blocked package installed successfully.
Comment 13 Florian Best univentionstaff 2021-06-28 17:59:13 CEST
I added a call to umc_frontend_new_hash in the univention-web-{js,style} postinst:

commit f9aa4acd997c526f8a00c1354d141dfb4bc13565
    Bug #37720: call umc_frontend_new_hash also in univention-web postinst
Comment 14 Florian Best univentionstaff 2021-07-08 17:38:19 CEST
We have another problem with Apache http when we change the caching behavior from caching to conditional requests:

$ curl -i 'http://fqdn/univention/management/' -H 'Accept-Encoding: gzip'
HTTP/1.1 200 OK
ETag: "854-5c5828f690c80-gzip"

$ curl -i 'http://10.200.27.80/univention/management/' -H 'Accept-Encoding: gzip' -H 'If-None-Match: "854-5c5828f690c80-gzip"'
HTTP/1.1 200 OK
ETag: "854-5c5828f690c80-gzip"

→ Apache is unable to evaluate the etag if the browser/client allows gzip compression. So we receive a "200 OK" instead of a "304 Not Modified".

https://bz.apache.org/bugzilla/show_bug.cgi?id=39727
https://bz.apache.org/bugzilla/show_bug.cgi?id=45023

With Apache httpd 2.5 there is a solution, by setting "DeflateAlterETag NoChange". For Apache 2.4 there are only workarounds.
I will pick the best one.
The alternative would be to backport https://github.com/apache/httpd/commit/20274bdd20ebc66286c5c3f3be334ad91043ae25.
Comment 15 Florian Best univentionstaff 2021-07-08 17:51:38 CEST
Took workaround:
RequestHeader edit "If-None-Match" '^"((.*)-(gzip|br))"$' '"$1", "$2"'

univention-web (3.0.6-6)
51de12fb4301 | Bug #37720: enable conditional requests for gzip compressed requests
Comment 16 Felix Botner univentionstaff 2021-08-05 16:17:10 CEST
ok cache seems to work with these changes
but i am not sure about the update

with firefox (90.0) i am redirected to /univention/management after the maintenance mode, login screen is ok and login works, but then  i get an empty page, reload does not help, after 1h or so, reload works and i can see the UMC

dev tools:
http://10.200.7.90/univention/js/umc/dialog/NotificationDropDownButton.js
Status404
Not Found

Die Ressource von "http://10.200.7.90/univention/js/umc/dialog/NotificationDropDownButton.js" wurde wegen eines MIME-Typ-Konfliktes ("text/html") blockiert (X-Content-Type-Options: nosniff).
management
Laden fehlgeschlagen für das <script> mit der Quelle "http://10.200.7.90/univention/js/umc/dialog/NotificationDropDownButton.js". management:1:1
Error: scriptError: /univention/js/umc/dialog/NotificationDropDownButton.js
    Dojo 23
    callback (index):23
    callback config.js:150
    Dojo 2
    callback config.js:146
    Dojo 17

chromium works
Comment 17 Florian Best univentionstaff 2021-08-19 13:46:33 CEST
(In reply to Felix Botner from comment #16)
> chromium works
We can release the current state but have to improve the firefox handling, via Bug #53680.
Comment 18 Felix Botner univentionstaff 2021-08-19 14:07:51 CEST
OK univention-lib.yaml:bug: [37720, 53362]
OK univention-management-console.yaml:bug: [37720, 52353, 53511, 53638]
OK univention-web.yaml:bug: [37720]

OK Bug #53680