Bug 49838 - Wireless/Wired group GPO not replicated from Master Samba/AD to School Slave via OpenLDAP
Wireless/Wired group GPO not replicated from Master Samba/AD to School Slave ...
Status: CLOSED FIXED
Product: UCS
Classification: Unclassified
Component: S4 Connector
UCS 4.4
Other Linux
: P5 normal (vote)
: UCS 4.4-3-errata
Assigned To: Florian Best
Julia Bremer
https://docs.microsoft.com/de-de/wind...
:
Depends on:
Blocks: 50626 50641 50642
  Show dependency treegraph
 
Reported: 2019-07-11 17:55 CEST by Arvid Requate
Modified: 2019-12-18 13:33 CET (History)
7 users (show)

See Also:
What kind of report is it?: Bug Report
What type of bug is this?: 5: Major Usability: Impairs usability in key scenarios
Who will be affected by this bug?: 2: Will only affect a few installed domains
How will those affected feel about the bug?: 3: A User would likely not purchase the product
User Pain: 0.171
Enterprise Customer affected?:
School Customer affected?: Yes
ISV affected?:
Waiting Support:
Ticket number: 2019071021000966
Bug group (optional):
Max CVSS v3 score:
requate: Patch_Available+


Attachments
patch (git:fbest/hackathon-s4-connector-windows-policies) (25.76 KB, patch)
2019-10-09 16:38 CEST, Florian Best
Details | Diff
Screenshot how it looks in UMC (64.58 KB, image/png)
2019-10-09 16:39 CEST, Florian Best
Details
Screenshot Software-Policy Package (313.15 KB, image/png)
2019-12-04 18:15 CET, Florian Best
Details
Screenshot Software-Policy Category (318.26 KB, image/png)
2019-12-04 18:16 CET, Florian Best
Details
Screenshot IPsec policies (489.24 KB, image/png)
2019-12-04 18:16 CET, Florian Best
Details
Screenshot Wired Policy (360.39 KB, image/png)
2019-12-04 18:18 CET, Florian Best
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Arvid Requate univentionstaff 2019-07-11 17:55:28 CEST
When a UCS@school admin wants to configure a Wifi autologin profile via GPO (see URL abouve), this currently only works when it's done on each of the UCS@school Slaves separately. 

If the admin creates this GPO in the UCS Master Samba/AD instead, then the affected Microsoft Windows Clients fail to apply the GPO, because the S4-Connector ignores the essential GPO sub-object for the Wifi autologin profile:

==============================================================================
# record 5
dn:
CN=WiFi-Autologin-Radius-USC,CN=IEEE80211,CN=Windows,CN=Microsoft,CN=Machine,CN={6F5794D6-DFCA-48E5-89B3-4DEA576BE704},CN=Policies,CN=System,DC=school,DC=intranet
objectClass: top
objectClass: ms-net-ieee-80211-GroupPolicy
instanceType: 4
whenCreated: 20190707054723.0Z
uSNCreated: 4157
showInAdvancedViewOnly: TRUE
objectGUID: cfb69081-c6b1-417a-af40-ce4f85ea2501
objectCategory:
CN=ms-net-ieee-80211-GroupPolicy,CN=Schema,CN=Configuration,DC
 =school,DC=intranet
ms-net-ieee-80211-GP-PolicyGUID: {410c9b3b-eb20-4e62-82f6-93f7aa93b49a}
description: WiFi Autologin
cn: WiFi-Autologin-Radius-USC
name: WiFi-Autologin-Radius-USC
ms-net-ieee-80211-GP-PolicyData:: <Binary-blob-displayed-in-base64>
whenChanged: 20190711143114.0Z
uSNChanged: 4189
distinguishedName:
CN=WiFi-Autologin-Radius-USC,CN=IEEE80211,CN=Windows,CN=Mic
 rosoft,CN=Machine,CN={6F5794D6-DFCA-48E5-89B3-4DEA576BE704},CN=Policies,CN=Sy
 stem,DC=school,DC=intranet
==============================================================================

The technical UCS issue to solve here is the same as in Bug #40298.
Comment 1 Michel Smidt univentionstaff 2019-07-12 13:10:05 CEST
I'd say that's a key scenario for school customers (districts).
Comment 2 Arvid Requate univentionstaff 2019-07-15 12:35:29 CEST
Ok, if that is the case, then you need to put this issue onto the UCS@school board and flag it as "important" to address this issue. I did this now.
Comment 3 Arvid Requate univentionstaff 2019-07-18 19:12:45 CEST
This is specified in [MS-GPWL]: Group Policy: Wireless/Wired Protocol Extension

https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-gpwl/4ea37f20-c3b7-402b-ab25-bb82389198d1  says:

"
The following class names are used while constructing various LDAP messages:
    BLOB-based wireless Group Policy is stored as an Active Directory object that MUST be an instance of class msieee80211-Policy.
    [..]
    XML-based wireless Group Policy is stored as an Active Directory object that MUST be an instance of class ms-net-ieee-80211-GroupPolicy.
    XML-based wired Group Policy is stored as an Active Directory object that MUST be an instance of class ms-net-ieee-8023-GroupPolicy.
"

For details see: https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-gpwl/70cca0b1-92c2-40fa-80c8-abdeff9e7f9e
Comment 4 Florian Best univentionstaff 2019-10-09 16:38:24 CEST
Created attachment 10198 [details]
patch (git:fbest/hackathon-s4-connector-windows-policies)

(In reply to Michel Smidt from comment #1)
> I'd say that's a key scenario for school customers (districts).

(In reply to Arvid Requate from comment #2)
> Ok, if that is the case, then you need to put this issue onto the UCS@school
> board and flag it as "important" to address this issue. I did this now.

Or ask Florian to do this in a Hackathon.
Comment 5 Florian Best univentionstaff 2019-10-09 16:39:41 CEST
Created attachment 10199 [details]
Screenshot how it looks in UMC

The patch adds 3 new UDM modules (ms/gpwl-wired, ms/gpwl-wireless, ms/gpwl-wireless-blob) and adds them to the S4-Connector mapping.
Comment 6 Florian Best univentionstaff 2019-10-09 17:18:25 CEST
The git branch contains enhanced patches for some more policy types, adds translations, fixes some data types/UMC layout things.
Comment 7 Michel Smidt univentionstaff 2019-10-09 22:42:13 CEST
(In reply to Florian Best from comment #4)
> 
> Or ask Florian to do this in a Hackathon.

Great! :D
Comment 9 Florian Best univentionstaff 2019-12-04 15:22:36 CET
Information for the QA:

1. In the Samba4 LDAP there exists already some default IPsec policies. To let them be synced do:

diff --git services/univention-s4-connector/conffiles/etc/univention/s4connector/s4/mapping.py services/univention-s4-connector/conffiles/etc/univention/s4connector/s4/mapping.py
index f7a111e037..578af9e541 100644
--- services/univention-s4-connector/conffiles/etc/univention/s4connector/s4/mapping.py
+++ services/univention-s4-connector/conffiles/etc/univention/s4connector/s4/mapping.py
@@ -63,7 +63,7 @@ global_ignore_subtree = [
        'CN=Server,CN=System,@%@connector/s4/ldap/base@%@',
        'CN=ComPartitionSets,CN=System,@%@connector/s4/ldap/base@%@',
        'CN=ComPartitions,CN=System,@%@connector/s4/ldap/base@%@',
-       'CN=IP Security,CN=System,@%@connector/s4/ldap/base@%@',
+       #'CN=IP Security,CN=System,@%@connector/s4/ldap/base@%@',
        'CN=DFSR-GlobalSettings,CN=System,@%@connector/s4/ldap/base@%@',
        'CN=DomainUpdates,CN=System,@%@connector/s4/ldap/base@%@',
        'CN=Password Settings Container,CN=System,@%@connector/s4/ldap/base@%@',

ucr set connector/s4/mapping/msgpipsec=yes
service univention-s4-connector restart
/usr/share/univention-s4-connector/resync_object_from_s4.py --filter 'CN=IP Security'

2. In Samba4 LDAP is also already a Class Store for the Software installation policy:
We currently don't map/sync "CN=Default Domain Policy,CN=System,$ldap_base".

So either we can decide to sync this as well(?), or you can just create something like this:
udm containers/cn create --set name='Default Domain Policy' --position "CN=System,$ldap_base"

ucr set connector/s4/mapping/msgpsi=yes
service univention-s4-connector restart
/usr/share/univention-s4-connector/resync_object_from_s4.py --filter '(|(objectClass=classStore)(objectClass=categoryRegistration)(objectClass=packageRegistration))'

Then the class store gets synced. For the creation of the Package registration or Category registration objects the AD tools on a Windows must be used.
So far, I couldn't find any way to create a category registration object :-(

3. For the Wireless/Wired Group Policy objects the Windows tools must be used as well.
ucr set connector/s4/mapping/msgpwl=yes
service univention-s4-connector restart
Comment 10 Florian Best univentionstaff 2019-12-04 18:15:42 CET
Created attachment 10247 [details]
Screenshot Software-Policy Package
Comment 11 Florian Best univentionstaff 2019-12-04 18:16:10 CET
Created attachment 10248 [details]
Screenshot Software-Policy Category
Comment 12 Florian Best univentionstaff 2019-12-04 18:16:59 CET
Created attachment 10249 [details]
Screenshot IPsec policies
Comment 13 Florian Best univentionstaff 2019-12-04 18:18:01 CET
Created attachment 10250 [details]
Screenshot Wired Policy
Comment 14 Florian Best univentionstaff 2019-12-04 22:36:38 CET
Merged the branch:

univention-s4-connector (13.0.2-59)
4ec687c51a24 | Bug #49838: Merge branch 'fbest/hackathon-s4-connector-windows-policies' into 4.4-3
674870663584 | Bug #49838: Add UDM syntax classes
5540e1503bf3 | Bug #49838: add translations
5d8d4059206d | Bug #49838: fix stopping of s4-connector in debug mode
30856d80a4e5 | Bug #49838: execute compare_function for multivalue attributes
1b56eed93090 | Bug #49838: Add icons for UMC
2258e20c3f6a | Bug #49838: add binary attributes to DECODE_IGNORELIST
c033c808c8d7 | Bug #49838: Add MS Software installation policy
20d383c3c94c | Bug #49838: Add MS IPsec policy
ec50a4b1abf1 | Bug #49838: add MS Wireless/Wired Group Policy
53b1c3b5a973 | Bug #49838: use dh-python and univention-l10n

univention-s4-connector.yaml
4ec687c51a24 | Bug #49838: Merge branch 'fbest/hackathon-s4-connector-windows-policies' into 4.4-3


TBD: should we enable the new UCR variables? Or only for new installations?
TODO: Opening objects with binary data in UMC causes a module crash. Should we fix this now? Or let the Python 3 migration fix this automatically where we have explicit encoding in UDM.
Comment 15 Arvid Requate univentionstaff 2019-12-04 23:25:00 CET
> TBD: should we enable the new UCR variables? Or only for new installations?

We'll have to discuss this. Check out the current implementation of pushedprinterconnections, gpos and wmi filters.

One thing to keep in mind is that the S4-Connector always first synchronizes from OpenLDAP to Samba/AD.

IIRC we implemented one or two scripts called from joinscript to sync the objects from Samba/AD to OpenLDAP before starting the Connector with the new modules enabled. That way we try to avoid overwriting / removing stuff in Samba/AD, in particular in UCS@school scenarios where we have multiple connectors and new School slaves are joined and re-joined at any time.

> TODO: Opening objects with binary data in UMC causes a module crash.

Can we just hide those properties in UMC? That's useless/irritating information anyway.
Comment 16 Felix Botner univentionstaff 2019-12-05 09:02:21 CET
I see this in our Jenkins tests (s4 slave https://jenkins.knut.univention.de:8181/job/UCS-4.4/job/UCS-4.4-3/job/AutotestJoin/SambaVersion=s4,Systemrolle=slave/ws/test/join.log/*view*/)


Multifile: /etc/ldap/slapd.conf
Warning: slapd.service changed on disk. Run 'systemctl daemon-reload' to reload units.
SRV record _ldap._tcp for port 7389 not created because Samba4 DCs are present: master095
wait for named ?
Traceback (most recent call last):
  File "/usr/share/univention-directory-manager-tools/univention-dnsedit", line 455, in <module>
    main()
  File "/usr/share/univention-directory-manager-tools/univention-dnsedit", line 406, in main
    univention.admin.modules.update()
  File "/usr/lib/python2.7/dist-packages/univention/admin/modules.py", line 123, in update
    os.path.walk(dir, _walk, p)
  File "/usr/lib/python2.7/posixpath.py", line 239, in walk
    walk(name, func, arg)
  File "/usr/lib/python2.7/posixpath.py", line 231, in walk
    func(arg, top, names)
  File "/usr/lib/python2.7/dist-packages/univention/admin/modules.py", line 108, in _walk
    m = __import__(mod, globals(), locals(), name)
  File "/usr/lib/pymodules/python2.7/univention/admin/handlers/ms/gpsi-package-registration.py", line 101, in <module>
    syntax=univention.admin.syntax.octetstring,
AttributeError: 'module' object has no attribute 'octetstring'


And please keep in mind, these new UDM modules have to work with older UCS versions (4.3).
Comment 17 Felix Botner univentionstaff 2019-12-05 09:05:41 CET
And i see this in the s4connector tests

aster091c] 2019-12-04T23:23:12.741071	Object created: cn=Policies,cn=System,dc=AutoTest091c,dc=local
[master091c] 2019-12-04T23:23:12.860853	Object created: ou=Domain Controllers,dc=AutoTest091c,dc=local
[master091c] 2019-12-04T23:23:12.967560	Object created: cn=WMIPolicy,cn=System,dc=AutoTest091c,dc=local
[master091c] 2019-12-04T23:23:13.076967	Object created: cn=SOM,cn=WMIPolicy,cn=System,dc=AutoTest091c,dc=local
[master091c] 2019-12-04T23:23:13.613340	Usage: ucs_registerLDAPExtension [options]
[master091c] 2019-12-04T23:23:13.614290	ucs_registerLDAPExtension: error: --udm_syntax: file does not exist: usr/lib/python2.7/dist-packages/univention/admin/syntax.d/s4_connector.py
[master091c] 2019-12-04T23:23:14.202458	Object exists: cn=ldapschema,cn=univention,dc=AutoTest091c,dc=local
[master091c] 2019-12-04T23:23:14.848970	Object exists: cn=udm_module,cn=univention,dc=AutoTest091c,dc=local
Comment 18 Florian Best univentionstaff 2019-12-05 09:24:48 CET
(In reply to Felix Botner from comment #16)
> And please keep in mind, these new UDM modules have to work with older UCS
> versions (4.3).
No, I did set "--versionstart 4.4-0" when registering them as extensions.
Comment 19 Florian Best univentionstaff 2019-12-05 09:28:10 CET
(In reply to Felix Botner from comment #17)
> And i see this in the s4connector tests
> [master091c] 2019-12-04T23:23:13.613340	Usage: ucs_registerLDAPExtension
> [options]
> [master091c] 2019-12-04T23:23:13.614290	ucs_registerLDAPExtension: error:
> --udm_syntax: file does not exist:
> usr/lib/python2.7/dist-packages/univention/admin/syntax.d/s4_connector.py

Thanks, fixed the missing leading slash:

univention-s4-connector (13.0.2-60)
d056af32f139 | Bug #49838: fix typo registering syntax class
Comment 20 Florian Best univentionstaff 2019-12-05 11:25:05 CET
(In reply to Arvid Requate from comment #15)
> > TODO: Opening objects with binary data in UMC causes a module crash.
> 
> Can we just hide those properties in UMC? That's useless/irritating
> information anyway.
They are already hidden, but the problem is that UMC-UDM sends all values (obj.info) to the frontend.
When trying to json.dumps(obj.info) this fails because the values are bytestring (str, not unicode) and JSON is UTF-8 based so it tries to decode every value as UTF-8, which fails for some binary sequences:

MODULE      ( ERROR   ) : Traceback (most recent call last):
  File "/usr/sbin/univention-management-console-module", line 121, in <module>
    notifier.loop()
  File "/usr/lib/pymodules/python2.7/notifier/nf_generic.py", line 287, in loop
    step()
  File "/usr/lib/pymodules/python2.7/notifier/nf_generic.py", line 279, in step
    __min_timer = dispatch.dispatcher_run()
  File "/usr/lib/pymodules/python2.7/notifier/dispatch.py", line 72, in dispatcher_run
    if not disp():
  File "/usr/lib/pymodules/python2.7/notifier/threads.py", line 154, in _simple_threads_dispatcher
    task.announce()
  File "/usr/lib/pymodules/python2.7/notifier/threads.py", line 135, in announce
    self._callback( self, self._result )
  File "/usr/lib/pymodules/python2.7/notifier/__init__.py", line 104, in __call__
    return self._function( *tmp, **self._kwargs )
  File "/usr/lib/python2.7/dist-packages/univention/management/console/base.py", line 297, in thread_finished_callback
    self.finished(request.id, result)
  File "/usr/lib/python2.7/dist-packages/univention/management/console/base.py", line 478, in finished
    self.result(res)
  File "/usr/lib/python2.7/dist-packages/univention/management/console/base.py", line 482, in result
    self.signal_emit('success', response)
  File "/usr/lib/pymodules/python2.7/notifier/signals.py", line 75, in signal_emit
    self.__signals[ signal ].emit( *args )
  File "/usr/lib/pymodules/python2.7/notifier/signals.py", line 41, in emit
    cb( *args )
  File "/usr/lib/pymodules/python2.7/notifier/__init__.py", line 104, in __call__
    return self._function( *tmp, **self._kwargs )
  File "/usr/lib/python2.7/dist-packages/univention/management/console/protocol/modserver.py", line 130, in _reply
    self.response(msg)
  File "/usr/lib/python2.7/dist-packages/univention/management/console/protocol/modserver.py", line 361, in response
    self.__queue += str(msg)
  File "/usr/lib/python2.7/dist-packages/univention/management/console/protocol/message.py", line 305, in __str__
    return Message._formattedMessage(self._id, self._type, self.mimetype, self.command, body, self.arguments)
  File "/usr/lib/python2.7/dist-packages/univention/management/console/protocol/message.py", line 111, in _formattedMessage
    data = json.dumps(body)
  File "/usr/lib/python2.7/json/__init__.py", line 244, in dumps
    return _default_encoder.encode(obj)
  File "/usr/lib/python2.7/json/encoder.py", line 207, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib/python2.7/json/encoder.py", line 270, in iterencode
    return _iterencode(o, 0)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xac in position 1: invalid start byte

It's a conceptual problem we have since years, see Bug #33522 (Bug #40485, Bug #48061). The Python-3 Migration must fix it nevertheless.
Comment 21 Florian Best univentionstaff 2019-12-05 18:38:21 CET
The UMC module crashes have been fixed by decoding everything as base64 in the UDM properties - while keeping all attributes binary data.

univention-s4-connector (13.0.2-61)
06dbb0ba1d8c | Bug #49838: add base64 presentation in UMC
Comment 22 Florian Best univentionstaff 2019-12-10 11:28:25 CET
Question: Should we rename the UDM modules from ms/gp* to msgp/* ?
Comment 23 Jürn Brodersen univentionstaff 2019-12-11 22:15:08 CET
What I tested:
Wired and Wireless Policies are synced to ldap -> OK
Policies are shown in udm and umc -> OK
Policies, created on the master in a multi school env, are synced to a school -> OK
Changes in ldap are synced back to s4 -> OK
Test are green on my test machine -> OK, Thanks for adding them! :)

Note:
QA for IPsec Policies has been done by Julia
QA for Software Policies has been done by Johannes
Please add your results :) 

Waiting for jenkins results
Comment 24 Julia Bremer univentionstaff 2019-12-12 14:28:21 CET
What I tested:

GPIPSEC* Policies are synced to ldap -> OK
GPIPSEC* are shown in udm and umc -> OK
Policies, created on the master in a multi school env, are synced to a school -> OK
Changes in ldap are synced back to s4 -> OK
Changes in the Policies are synced only to Windows members of that school and apply -> OK

Yaml incomplete -> Fail 

Why are the CN=IP Security Objects in the ignore_subtree?
Comment 25 Arvid Requate univentionstaff 2019-12-12 21:18:50 CET
I just saw that the code in 97univention-s4-connector.inst uses resync_object_from_s4.py to trigger an initial sync from Samba/AD-> UDM. IIRC that tool just adds the DNs to the corresponding reject list, but the S4-Connector only considers that *after* the initial sync from OpenLDAP. Please check that again, I'm unsure. I know that Felix code for msprintconnectionpolicy used the same tool, but just in case...
Comment 26 Florian Best univentionstaff 2019-12-13 16:43:16 CET
The branch git:fbest/49838-domainpolicy contains a patch for synconizing also the "CN=Default Domain Policy,CN=System,$ldap_base" object / domainPolicy object class.
Comment 27 Florian Best univentionstaff 2019-12-16 14:50:15 CET
(In reply to Julia Bremer from comment #24)
> Why are the CN=IP Security Objects in the ignore_subtree?
I don't know. I created Bug #50642 for this.

(In reply to Florian Best from comment #26)
> The branch git:fbest/49838-domainpolicy contains a patch for synconizing
> also the "CN=Default Domain Policy,CN=System,$ldap_base" object /
> domainPolicy object class.

→ Moved to Bug #50641
Comment 28 Florian Best univentionstaff 2019-12-16 15:06:23 CET
(In reply to Arvid Requate from comment #25)
> I just saw that the code in 97univention-s4-connector.inst uses
> resync_object_from_s4.py to trigger an initial sync from Samba/AD-> UDM.
> IIRC that tool just adds the DNs to the corresponding reject list, but the
> S4-Connector only considers that *after* the initial sync from OpenLDAP.
> Please check that again, I'm unsure. I know that Felix code for
> msprintconnectionpolicy used the same tool, but just in case...

> Diese --write2ucs oder resync_object_from_s4.py Aufrufe in dem Joinscript sind ja dafür da, zu verhindern, dass der S4-Connector, der beim Start immer zuerst von sync_from_ucs durchläuft, Objekte im Samba/AD überbügelt. Die Skripte, die dort mit der Option --write2ucs aufgerufen werden, machen das alles direkt. Aber das resync_object_from_s4.py Skript sucht einfach nur die DNs raus und schreibt sie dem S4-Connecgtor in seine Reject-Tabellen. Ich befürchte, dass er die erst anschaut, wenn er mit dem Initialen sync_from_ucs fertig ist.

So, you would do it like it's done in the joinscript with the upgrade_msWMI-Som.py msGPOWQLFilter.py calls?

This adds the objects also to the reject table but prior it modifies it once (with same values).
I can add a tool which "touch"es every object once before adding it to the sqlite reject table.
Comment 29 Arvid Requate univentionstaff 2019-12-16 17:37:05 CET
> So, you would do it like it's done in the joinscript with the upgrade_msWMI-Som.py msGPOWQLFilter.py calls?

From reading the code, I see that msGPOWQLFilter with --write2ucs manually synchronizes a new attribute from Samba/AD to OpenLDAP. I don't know if that's required. Let's keep it the way your iplemented it, this worked also for msPrint-ConnectionPolicy.
Comment 30 Florian Best univentionstaff 2019-12-16 17:41:29 CET
Sounds good. Added version number to advisory:

univention-s4-connector.yaml
5dcc37c60de9 | YAML Bug #49838
Comment 31 Julia Bremer univentionstaff 2019-12-16 18:24:14 CET
Tested:
Software-Policy Packages Attributes are synced: OK
IPsec policies are synced if taken out of the ignore_subtree : OK
Wired policies are synced: OK
Wireless policies are synced: OK
UCR-Variables: OK
Works in ucs@chool environment: OK
Manual Windows Tests:  OK
Automated tests: OK
Default Domain Policy is not yet synced: OK
YAML: OK


-> Verified
Comment 32 Erik Damrose univentionstaff 2019-12-18 13:33:09 CET
<http://errata.software-univention.de/ucs/4.4/407.html>