Bug 49580 - verify structure and values of configuration file
verify structure and values of configuration file
Status: VERIFIED DUPLICATE of bug 49102
Product: UCS@school
Classification: Unclassified
Component: Import scripts
UCS@school 4.4
Other Linux
: P5 normal (vote)
: ---
Assigned To: Daniel Tröder
Jürn Brodersen
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2019-05-31 12:11 CEST by Christian Völker
Modified: 2020-02-19 16:57 CET (History)
2 users (show)

See Also:
What kind of report is it?: Bug Report
What type of bug is this?: 2: Improvement: Would be a product improvement
Who will be affected by this bug?: 2: Will only affect a few installed domains
How will those affected feel about the bug?: 2: A Pain – users won’t like this once they notice it
User Pain: 0.046
Enterprise Customer affected?:
School Customer affected?:
ISV affected?:
Waiting Support:
Flags outvoted (downgraded) after PO Review:
Ticket number:
Bug group (optional):
Max CVSS v3 score:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Christian Völker univentionstaff 2019-05-31 12:11:35 CEST
Trying to import.
=======================================
root@master:~# head -n2 schueler.csv.0
Schulen,Username,Vorname,Nachname,Klassen,Mailadresse,phone,id,src
SchuleLena,e.einverle,Einverstndnis,Einverständnis,SchuleLena-1c,e.einverle@schulen.ucs,"",0303,0
=======================================

=======================================
root@master:~# python -m json.tool<  /var/lib/ucs-school-import/configs/user_import.json
{
    "csv": {
        "allowed_missing_columns": [],
        "header_lines": 1,
        "incell-delimiter": {
            "default": ","
        },
        "mapping": {
            "Klassen": "school_classes",
            "Nachname": "lastname",
            "Schulen": "schools",
            "Vorname": "firstname",
            "id": "record_uid",
            "phone": "phone",
            "src": "source_uid"
        }
    },
    "maildomain": "schulen.ucs",
    "scheme": {
        "email": {
            "default": "<:umlauts><firstname>[0].<lastname><:lower>[COUNTER2]@<maildomain>"
        },
        "username": {
            "default": "<:umlauts><firstname>[0].<lastname><:lower>[COUNTER2]"
        }
    }
}
=======================================

=======================================
root@master:~# /usr/share/ucs-school-import/scripts/ucs-school-user-import -i schueler.csv.0  -u student  -n --source_uid 0
[...]
UsernameHandler(max_length=15, dry_run=True) storage_backend='MemoryStorageBackend'
expected string or buffer
Traceback (most recent call last):
  File "/usr/lib/pymodules/python2.7/ucsschool/importer/mass_import/mass_import.py", line 120, in import_users
    user_import.create_and_modify_users(imported_users)  # 90% - 100%
  File "/usr/lib/pymodules/python2.7/ucsschool/importer/mass_import/user_import.py", line 149, in create_and_modify_users
    user = self.determine_add_modify_action(imported_user)
  File "/usr/lib/pymodules/python2.7/ucsschool/importer/mass_import/user_import.py", line 270, in determine_add_modify_action
    return self.prepare_imported_user(imported_user, None)
  File "/usr/lib/pymodules/python2.7/ucsschool/importer/mass_import/user_import.py", line 249, in prepare_imported_user
    imported_user.prepare_all(new_user=not old_user)
  File "/usr/lib/pymodules/python2.7/ucsschool/importer/models/import_user.py", line 513, in prepare_all
    self.prepare_attributes(new_user)
  File "/usr/lib/pymodules/python2.7/ucsschool/importer/models/import_user.py", line 536, in prepare_attributes
    self.make_email()
  File "/usr/lib/pymodules/python2.7/ucsschool/importer/models/import_user.py", line 699, in make_email
    self.email = self.format_from_scheme("email", self.config["scheme"]["email"], maildomain=maildomain).lower()
  File "/usr/lib/pymodules/python2.7/ucsschool/importer/models/import_user.py", line 1259, in format_from_scheme
    self.solve_format_dependencies(prop_name, scheme, **kwargs)
  File "/usr/lib/pymodules/python2.7/ucsschool/importer/models/import_user.py", line 1198, in solve_format_dependencies
    props_used_in_scheme = [x[0] for x in self._prop_regex.findall(no_brackets) if x[0]]
TypeError: expected string or buffer
=======================================


Having modified .json to:
=======================================
root@master:~# python -m json.tool<  /var/lib/ucs-school-import/configs/user_import.json
{
    "csv": {
        "allowed_missing_columns": [],
        "header_lines": 1,
        "incell-delimiter": {
            "default": ","
        },
        "mapping": {
            "Klassen": "school_classes",
            "Nachname": "lastname",
            "Schulen": "schools",
            "Vorname": "firstname",
            "id": "record_uid",
            "phone": "phone",
            "src": "source_uid"
        }
    },
    "maildomain": "schulen.ucs",
    "scheme": {
        "email": "<:umlauts><firstname>[0].<lastname><:lower>[COUNTER2]@<maildomain>",
        "username": {
            "default": "<:umlauts><firstname>[0].<lastname><:lower>[COUNTER2]"
        }
    }
}
=======================================

Now it appears to be working
Comment 1 Daniel Tröder univentionstaff 2019-06-03 08:57:21 CEST
There is a user error: 

---------------------------------------------------
"scheme": {
        "email": {
            "default": …
        },
    },
---------------------------------------------------

should be

---------------------------------------------------
"scheme": {
        "email": …,
    },
---------------------------------------------------

A schema validation could find such configuration errors and display a helpful error message.
Comment 2 Daniel Tröder univentionstaff 2019-06-03 08:58:41 CEST
Not an error in the product or the documentation.
Comment 3 Daniel Tröder univentionstaff 2020-02-19 11:56:03 CET

*** This bug has been marked as a duplicate of bug 49102 ***