Bug 45645 - [4.1r2] Teachers collect data from themselves, quickly multiplying the folder size
[4.1r2] Teachers collect data from themselves, quickly multiplying the folder...
Product: UCS@school
Classification: Unclassified
Component: UMC - Distribution
UCS@school 4.1 R2
Other Linux
: P5 normal (vote)
: UCS@school 4.1 R2 v16
Assigned To: Daniel Tröder
Florian Best
Depends on: 45572
  Show dependency treegraph
Reported: 2017-11-03 15:30 CET by Daniel Tröder
Modified: 2017-12-21 12:25 CET (History)
5 users (show)

See Also:
What kind of report is it?: Bug Report
What type of bug is this?: 2: Improvement: Would be a product improvement
Who will be affected by this bug?: 2: Will only affect a few installed domains
How will those affected feel about the bug?: 2: A Pain – users won’t like this once they notice it
User Pain: 0.046
Enterprise Customer affected?:
School Customer affected?:
ISV affected?:
Waiting Support:
Flags outvoted (downgraded) after PO Review:
Ticket number:
Bug group (optional):
Max CVSS v3 score:


Note You need to log in before you can comment on or make changes to this bug.
Description Daniel Tröder univentionstaff 2017-11-03 15:30:48 CET
Merge changes to 4.1r2.

+++ This bug was initially created as a clone of Bug #45572 +++

When a teacher collects a project with a class as member, files will be collected from every user inside that class.
This unfortunately includes his own project folder, which contains every previously collected version of this project.

If this teacher now collects a project multiple times, the size of the newly collected project versions will multiply, as they include all previously collected project versions inside the teachers project folder.

The collecting teacher shouldn't collect his own folder content.

Simple example, assuming no one touches any files:
1 Class containing
2 Students and
1 Teacher
1 Project with that Class as member and
1 Project File, Size: 1Mb

Teacher distributes project:
All class members (3) receive one 1Mb file

Teacher collects project:
Teacher collects two 1Mb files from every member excluding himself 
Teacher then collects his own folder including one 1Mb file and all folders from other members
:Total Teacher size: 1Mb + 2Mb + 3Mb = 6Mb

Teacher collects project a second time:
Teacher collects two 1Mb files from every member excluding himself 
Teacher then collects his own folder including 
 one 1Mb file, 
 his first collection of members (2Mb), 
 his second collection of members (2Mb) 
 his first collection of himself (3Mb) and 
 his new collection of himself (8Mb)
:Total Teacher size: 1Mb + 2Mb + 2Mb + 3Mb + 8Mb = 16Mb

The folder would have a size of 36Mb after a third collection.

Graphical Display after second collection:
├── [1.0M]  FILE
├── [3.0M]  teacher1
│   ├── [1.0M]  FILE
│   ├── [1.0M]  bschwartz
│   └── [1.0M]  testStudent
├── [8.0M]  teacher1 version2
│   ├── [1.0M]  FILE
│   ├── [3.0M]  teacher1
│   │   ├── [1.0M]  FILE
│   │   ├── [1.0M]  bschwartz
│   │   └── [1.0M]  testStudent
│   ├── [1.0M]  bschwartz
│   ├── [1.0M]  bschwartz version2
│   ├── [1.0M]  testStudent
│   └── [1.0M]  testStudent version2
├── [1.0M]  bschwartz
├── [1.0M]  bschwartz version2
├── [1.0M]  testStudent
└── [1.0M]  testStudent version2

Comment 1 Daniel Tröder univentionstaff 2017-11-03 15:41:03 CET
The teachers home directory is now skipped, when collecting distributed materials.

Cherry-picked from 4.2:

commit abae615691d927bf0dc1738f17d94ca81a668619
    Bug #45645: advisory
commit d3641c20febe7ba085c8c8ba190ce0cc11178b90
    Bug #45645: advisory
commit 3a0a40d74863cbf81e4d9a25f459e8e266198dff
    Bug #45572: advisory
commit f71d7a56bafeb3c063dd129053dda05dddef3cc8
    Bug #45645: don't collect teachers own data

ucs-school-umc-distribution 14.0.2-4.89.201711031537
Comment 2 Sönke Schwardt-Krummrich univentionstaff 2017-11-06 11:41:18 CET
If missing, please add a note in the UCS@school teacher manual, that clearly states, that files of users with role teacher are never collected.
Comment 3 Daniel Tröder univentionstaff 2017-11-06 15:56:53 CET
[4.1r2 39196da4] Bug #45645: mention that teachers file are not collected

Last sentence of chapter 3.7.3: http://jenkins.knut.univention.de:8080/job/UCSschool%204.1/job/UCSschool%204.1%20R2%20Manual/10/artifact/webroot/ucsschool-lehrer-handbuch-4.1r2.pdf
Comment 4 Daniel Tröder univentionstaff 2017-11-08 12:32:01 CET
Commit 724a09c1: store and compare school roles instead of DNs
Commit 35382053: update advisory
Comment 5 Sönke Schwardt-Krummrich univentionstaff 2017-11-09 15:09:10 CET
Copied comment from bug 45572:

(In reply to Florian Best from comment #7)
> So the behavior now is: "Teacher" and "Teachers and Staff" materials aren't
> collected. "Students" and "Staff" materials are collected.

Staff users are usually not replicated to the educational UCS@school servers (→ Edu DC Slave). I.e. only files of "student" users are collected.

(In reply to Florian Best from comment #8)
> The behavior is not consistent anymore. Why are the files distributed to
> teachers and teachers+staff but not collected from them? They should not be
> distributed at all then.

Good question. We initiated a new discussion with ProfServices and Development and we all think, that the current approach does not exactly hit the requirements. The current approach will prevent the collection of files distributed to other teachers.

Instead, a new solution has been discussed:

--- a/ucs-school-umc-distribution/umc/python/distribution/util.py
+++ b/ucs-school-umc-distribution/umc/python/distribution/util.py
@@ -54,6 +54,7 @@ _ = Translation('ucs-school-umc-distribution').translate
 DISTRIBUTION_CMD = '/usr/lib/ucs-school-umc-distribution/umc-distribution'
 DISTRIBUTION_DATA_PATH = ucr.get('ucsschool/datadistribution/cache', '/var/lib/ucs-school-umc-distribution')
+POSTFIX_DATADIR_SENDER_PROJECT_SUFFIX = ucr.get('ucsschool/datadistribution/datadir/sender/project/suffix', '-Ergebnisse')
 POSTFIX_DATADIR_SENDER = ucr.get('ucsschool/datadistribution/datadir/sender', 'Unterrichtsmaterial')
 POSTFIX_DATADIR_RECIPIENT = ucr.get('ucsschool/datadistribution/datadir/recipient', 'Unterrichtsmaterial')
@@ -264,7 +265,7 @@ class Project(_Dict):
 	def sender_projectdir(self):
 		'''The absolute path of the project directory in the senders home.'''
 		if self.sender and self.sender.homedir:
-			return os.path.join(self.sender.homedir, POSTFIX_DATADIR_SENDER, self.name)
+			return os.path.join(self.sender.homedir, POSTFIX_DATADIR_SENDER, '%s%s' % (self.name, POSTFIX_DATADIR_SENDER_PROJECT_SUFFIX))
 		return None

→ files are distributed to $HOME/Unterrichtsmaterial/${PROJECTNAME}/
→ files are collected to $HOME/Unterrichtsmaterial/${PROJECTNAME}-Ergebnisse/
→ this should prevent the exponential collection loop
→ please revert the modifications done so far and apply the patch above

A hotfix is currently also possible:
Use different directory names in "ucsschool/datadistribution/datadir/sender" and "ucsschool/datadistribution/datadir/recipient". Currently both default to "Unterrichtsmaterial".
Comment 6 Daniel Tröder univentionstaff 2017-11-22 11:55:31 CET
The previous commits were reverted:

commit 14c57a45f89e8555dab6a437dae91bf266dcb666
    Revert "Bug #45645: store and compare school roles instead of DNs"
    This reverts commit 724a09c181085c47e157ff6691a1c2a264297b5b.
commit 9f4ff232225c8beef686d77ac56140589c29f05c
    Revert "Bug #45645: mention that teachers file are not collected"
    This reverts commit 39196da4e78e21a837e056b52378c652dce9e410.
commit 10429905e71b3b5c51148a96bc3de1e47d332ebc
    Revert "Bug #45645: don't collect teachers own data"
    This reverts commit f71d7a56bafeb3c063dd129053dda05dddef3cc8.

And the new UCRV ucsschool/datadistribution/datadir/sender/project/suffix introduced (cherry-picked from 4.2).

commit 47104296ab90976f777cd29682a3caa7823df625
    Bug #45645: separate project distribution and collection directories

And explained in the manual (cherry-picked from 4.2).

commit 1670dfe35d4d29ba25673aefb070918e29a375f6
    Bug #45572: literal and anvar cannot be children of footnote
commit 3ab50683518efc47841a871c6ac6a87e4832a4ad
    Bug #45572: explain UCR in manual

Changelog and advisory:

commit 61ee9e655e5402df31d035f21d91ad34b765d0cd
    Bug #45645: advisory update
commit 08435b84b3a940110e4d4c5af7bcbd8be001b17b
    Bug #45645: changelog and advisory

ucs-school-umc-distribution 14.0.2-6.91.201711221148
Comment 8 Florian Best univentionstaff 2017-11-22 15:38:24 CET
OK: merge
OK: no UCR variable description (as other UCR variables aren't described either)
Comment 9 Sönke Schwardt-Krummrich univentionstaff 2017-12-21 12:25:13 CET
UCS@school 4.1 R2 v16 has been released.


If this error occurs again, please clone this bug.