Bug 36989 - Quota nicht bei jeder Anmeldung auswerten
Quota nicht bei jeder Anmeldung auswerten
Status: CLOSED FIXED
Product: UCS
Classification: Unclassified
Component: Quota
UCS 4.0
Other Linux
: P2 enhancement (vote)
: UCS 4.0-1-errata
Assigned To: Stefan Gohmann
Felix Botner
:
: 36104 (view as bug list)
Depends on: 28729
Blocks: 36104
  Show dependency treegraph
 
Reported: 2014-11-25 09:02 CET by Stefan Gohmann
Modified: 2015-05-07 17:37 CEST (History)
9 users (show)

See Also:
What kind of report is it?: ---
What type of bug is this?: ---
Who will be affected by this bug?: ---
How will those affected feel about the bug?: ---
User Pain:
Enterprise Customer affected?:
School Customer affected?:
ISV affected?:
Waiting Support:
Ticket number:
Bug group (optional): UCS Performance
Max CVSS v3 score:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Stefan Gohmann univentionstaff 2014-11-25 09:02:14 CET
A backport to UCS 4 should be checked.

+++ This bug was initially created as a clone of Bug #28729 +++

An Ticket 2012072421004332 berichtete ein Kunde performance Probleme bei der Windows Anmeldung (sowie Fehlermeldungen beim "net rpc join") durch /usr/sbin/univention-user-quota.

Ggf. könnte man hier, ähnlich wie bei Mail-Quota, definierbar machen dass die Quota z.B. nur einmal am Tag, einmal in 4 Stunden o.ä. ausgewertet werden (steuerbar per UCR).

Als Workaround kann die Auswertung derzeit mit "ucr set quota/userdefault=no" deaktiviert werden.
Comment 1 Stefan Gohmann univentionstaff 2015-04-03 22:05:01 CEST
YAML file: 2015-04-03-univention-quota.yaml (r59593)

Fix: r59592
As described in Bug #28729. The listener module dumps the settings per share into a cache directory and the PAM scripts uses this cache directory.

Test cases: r59594
First test cases have been added.

Todo:
- Performance tests
- More test cases
Comment 2 Stefan Gohmann univentionstaff 2015-04-05 22:04:52 CEST
I've added one more test case: r59596

The ldap filter in the new listener module has been fixed: r59595
Comment 3 Stefan Gohmann univentionstaff 2015-04-07 06:21:30 CEST
More ucs-test updates: r59597 + r59598

The performance is quite good.

I've added 400 shares and it took about 5 seconds to re-create the cache for all shares after adding a policy to the LDAP base. That is done in the postrun of the listener thus it doesn't bother anyone.

The Samba Login is much better:

OLD:
--------------------------------------------------------------------------------
root@master401:~# time smbclient -U stefan%univention //master401/share198 -c ls
Domain=[DEADLOCK40] OS=[Windows 6.1] Server=[Samba 4.2.0rc2-Debian]
  .                                   D        0  Wed Jan 14 02:18:00 2015
  ..                                  D        0  Wed Jan 14 02:21:48 2015

                18982780 blocks of size 1024. 15801852 blocks available

real    0m5.681s
user    0m0.012s
sys     0m0.004s
--------------------------------------------------------------------------------

NEW:
--------------------------------------------------------------------------------
root@master401:~# time smbclient -U stefan%univention //master401/share198 -c ls
Domain=[DEADLOCK40] OS=[Windows 6.1] Server=[Samba 4.2.0rc2-Debian]
  .                                   D        0  Wed Jan 14 02:18:00 2015
  ..                                  D        0  Wed Jan 14 02:21:48 2015

                18982780 blocks of size 1024. 15741160 blocks available

real    0m0.402s
user    0m0.016s
sys     0m0.008s
root@master401:~#
--------------------------------------------------------------------------------
Comment 4 Stefan Gohmann univentionstaff 2015-04-07 06:47:17 CEST
*** Bug 36104 has been marked as a duplicate of this bug. ***
Comment 5 Stefan Gohmann univentionstaff 2015-04-10 05:30:59 CEST
The Jenkins test 01_base.99check_log_files.test fails on a member server:

[  1.097]Errors found in '/var/log/univention/join.log':
[  1.097]
[  1.097] E: join.log:118, Traceback (most recent call last):
[  1.097]  File "/usr/lib/univention-directory-listener/system/quota.py", line 225, in handler
[  1.097]    if _is_container_change_relevant(new, old):
[  1.097]  File "/usr/lib/univention-directory-listener/system/quota.py", line 147, in _is_container_change_relevant
[  1.097]    lo = _get_ldap_connection()
[  1.097]  File "/usr/lib/univention-directory-listener/system/quota.py", line 122, in _get_ldap_connection
[  1.097]    connection = univention.uldap.getMachineConnection(ldap_master=False)
[  1.097]  File "/usr/lib/pymodules/python2.7/univention/uldap.py", line 110, in getMachineConnection
[  1.097]    raise ldap.SERVER_DOWN, e
[  1.097]ldap.SERVER_DOWN: {'desc': "Can't contact LDAP server"}

http://jenkins.knut.univention.de:8080/job/UCS-4.0/job/UCS-4.0-1/job/Autotest%20MultiEnv/SambaVersion=s3,Systemrolle=member/64/testReport/01_base/99check_log_files/test/
Comment 6 Stefan Gohmann univentionstaff 2015-04-13 06:13:51 CEST
(In reply to Stefan Gohmann from comment #5)
> http://jenkins.knut.univention.de:8080/job/UCS-4.0/job/UCS-4.0-1/job/
> Autotest%20MultiEnv/SambaVersion=s3,Systemrolle=member/64/testReport/01_base/
> 99check_log_files/test/

This has been fixed: r59713 + r59716 + r59720 + r59722

A new directory /var/cache/univention-quota/todo has been added. The listener module now uses this directory to transfer the DNs from the handler to the postrun. The listener also uses the connection to the ldap/master if other ldap servers (ldap/server/*) are not reachable.
Comment 7 Felix Botner univentionstaff 2015-05-04 15:54:42 CEST
I got 

UNIVENTION_DEBUG_END    : uldap.__open host=master.four.test port=7389 base=dc=four,dc=test
Traceback (most recent call last):
  File "/usr/lib/univention-directory-listener/system/quota.py", line 210, in handler
    _add_all_shares_below_this_container_to_dn_list(dn)
  File "/usr/lib/univention-directory-listener/system/quota.py", line 167, in _add_all_shares_below_this_container_to_dn_list
    _add_share_to_dn_list(dn)
  File "/usr/lib/univention-directory-listener/system/quota.py", line 175, in _add_share_to_dn_list
    open(filename, 'w').close()
IOError: [Errno 2] No such file or directory: '/var/cache/univention-quota/todo/cn=opt1,cn=shares,dc=four,dc=test'
04.05.15 15:38:44.059  LISTENER    ( WARN    ) : handler: quota (failed)

in /var/log/univention/listener.log.

Problem seems to be, that during update (after unpacking univention-quota) other packages also try to restart the listener (e.g. univention-pam). But at this point, the postinst of univention-quota has not been executed and therefor the /var/cache/univention-quota/todo directory is missing. 

Maybe the SHARE_CACHE_TODO_DIR has to be created in _add_share_to_dn_list()?
Comment 8 Stefan Gohmann univentionstaff 2015-05-04 16:44:31 CEST
I've moved the directory creation to the preinst: r60359
Comment 9 Felix Botner univentionstaff 2015-05-05 10:23:28 CEST
FAIL - removing policy (at least, if the policy is linked to a container)
 _is_quota_policy - ok
  _get_all_quota_references - ok
   handler() - ok
    _is_container - ok
      _is_container_change_relevant -
        new_reference - ok  (cn=aaaa,cn=policies,dc=four,dc=test)
          _is_quota_policy - FAIL (policy already deleted)
Comment 10 Stefan Gohmann univentionstaff 2015-05-05 12:30:20 CEST
(In reply to Felix Botner from comment #9)
> FAIL - removing policy (at least, if the policy is linked to a container)
>  _is_quota_policy - ok
>   _get_all_quota_references - ok
>    handler() - ok
>     _is_container - ok
>       _is_container_change_relevant -
>         new_reference - ok  (cn=aaaa,cn=policies,dc=four,dc=test)
>           _is_quota_policy - FAIL (policy already deleted)

OK, fixed with r60403.
Comment 11 Felix Botner univentionstaff 2015-05-05 14:15:58 CEST
OK - tested on master and master/member

OK - share link policy/unlink policy/modify policy/remove policy
OK - base link policy/unlink policy/modify policy/remove policy
OK - container link policy/unlink policy/modify policy/remove policy 

OK - add 1000 shares to share list 0.3s
OK - postrun for 1000 shares 30s

OK - su Administrator with 1000 shares 0.5s (old 25s!)

OK - test

OK - YAML
Comment 12 Janek Walkenhorst univentionstaff 2015-05-07 17:37:34 CEST
<http://errata.univention.de/ucs/4.0/164.html>