Bug 56195 - samba VFS object 'full_audit' stopped working
samba VFS object 'full_audit' stopped working
Status: NEW
Product: UCS
Classification: Unclassified
Component: Samba
UCS 5.0
Other Linux
: P5 normal (vote)
: ---
Assigned To: Samba maintainers
Samba maintainers
https://lists.samba.org/archive/samba...
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2023-06-26 09:45 CEST by Moritz Bunkus
Modified: 2023-06-27 13:45 CEST (History)
3 users (show)

See Also:
What kind of report is it?: Bug Report
What type of bug is this?: 1: Cosmetic issue or missing function but workaround exists
Who will be affected by this bug?: 1: Will affect a very few installed domains
How will those affected feel about the bug?: 3: A User would likely not purchase the product
User Pain: 0.017
Enterprise Customer affected?:
School Customer affected?:
ISV affected?:
Waiting Support:
Flags outvoted (downgraded) after PO Review:
Ticket number:
Bug group (optional): Regression
Max CVSS v3 score:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Moritz Bunkus 2023-06-26 09:45:45 CEST
UCS 5.0-4 errata 716

Yesterday we upgraded all packages including Samba to the latest version. For Samba this meant upgrading from e.g. samba-vfs-modules:amd64 2:4.16.8-1A~5.0.0.202304271715 to 2:4.18.3-1A~5.0.0.202306091432.

Since the upgrade connects to any share that has the "full_audit" VFS object active will fail. smbclient will only say NT_STATUS_UNSUCCESSFULL. Only shares that have "full_audit" active are affected; the rest of the Samba server works normally (shares without "full_audit", printing, generally connecting to the server).

Just the act of removing the "full_audit" from the list of VFS objects will let connections succeed.

Bumping Sambas debug level to 5 yields the following entries in /var/log/samba/log.smb:

[2023/06/26 09:38:56.253411,  0] ../../source3/modules/vfs_full_audit.c:564(init_bitmap)
  init_bitmap: Could not find opname rename
[2023/06/26 09:38:56.253436,  0] ../../source3/modules/vfs_full_audit.c:753(smb_full_audit_connect)
  smb_full_audit_connect: Invalid success operations list. Failing connect

We've been using the "full_audit" module for years. It definitely worked last week before the update, it doesn't work today after the upgrade.
Comment 1 Florian Best univentionstaff 2023-06-26 10:03:14 CEST
In UCS 5.0-4 Bug #44997 (git:51a3676981dcfb9334b8a2603b9ec91d9382a50d) we overworked the shares module and changed VFS Objects into a combobox.

Can you have a look at (and attach) the representation of the UDM share object in UMC, UDM-CLI and ldapsearch.
And how it is then represented in the share configuration underneath of /etc/samba/shares.conf.d/?
This should not have changed.
Comment 2 Moritz Bunkus 2023-06-26 10:11:08 CEST
Before filing the bug I verified that this happens as described by creating a whole new share. That new share works if there's no VFS object configured. Connections to the share stop working as soon as I select "full_audit" from the drop-down box.

Your conversion to the combobox seems to be OK: the content of our shares configuration files contained the following before the update (copy-pasted from a file restored from backups):

vfs objects = acl_xattr full_audit

And here's the same line from the share newly created today, configured with "full_audit" from the combobox:

vfs objects = acl_xattr full_audit

Looks identical to me.
Comment 3 Arvid Requate univentionstaff 2023-06-26 10:58:55 CEST
Thanks for the detailed report.

https://lists.samba.org/archive/samba/2022-December/243281.html suggests that this can be fixed by adjusting the configuration of full_audit (man vfs_full_audit).
Comment 4 Moritz Bunkus 2023-06-26 11:06:44 CEST
Argh, right. I had the following in /etc/samba/local.conf (among other settings):

full_audit:success = pwrite write rename

Changing that to

full_audit:success = pwrite write renameat

and issuing "smbcontrol smbd reload-config" fixed the issue.

It's slightly unfortunate that such incompatible changes happen with seemingly innocent package updates.
Comment 5 Arvid Requate univentionstaff 2023-06-26 16:59:59 CEST
I agree, by default we check

* https://wiki.samba.org/index.php/Samba_4.17_Features_added/changed
* https://wiki.samba.org/index.php/Samba_4.18_Features_added/changed

but nothing there indicated anything in that direction. In this case
it even depends on the specifics of the vfs module config.

In this case to me it looks like a result of a combination of

* https://gitlab.com/samba-team/samba/-/commit/7aadbe5608e
  which is in since Samba 4.12
* and https://bugzilla.samba.org/show_bug.cgi?id=15098
  which isn't mentioned in the Samba release notes


What could we do to spot things like these in the future?
* git diff  samba-4.16.8 v4-18-stable -- docs-xml/manpage
Comment 6 Moritz Bunkus 2023-06-27 09:16:59 CEST
To be fair, I'm not sure if you as a downstream can really do something about it:

* upstream might not realize a change is a breaking change
* even if they realize it, they might decide not to mention it in the changes document
* even if they do, they might decide not to say so in the git commits
* even if do at least one of those, they might use verbiage that isn't easily greppable
* even if they do, their description might not be clear enough to really get across under which circumstances something changed

If you as Univention want to implement a check somewhere for this particular issue, go ahead. You'll probably have to try to parse the Samba config (or output of "testparm") for the corresponding settings & verify the list of function names against a list of currently known-good ones, I guess. Not sure what you would want to _do_ with that information, though; maybe fail the package upgrade (meaning do the check in a pre-inst hook?).

Only putting such a check into univention-run-diagnostic-check wouldn't have helped me personally as I didn't run that one when my colleagues notified me Monday morning that "the file server wasn't working". On the other hand, having a failed system upgrade would definitely have appeared on my radar.

Anyway, just brainstorming here. And as a background info why we're using full_audit in the first place: in order to have another layer of defense against ransomware attacks. We can combine this with remote syslogging for later analysis, and/or with fail2ban directly on the file server for immediately cutting off a machine doing too many suspicious operations.
Comment 7 Arvid Requate univentionstaff 2023-06-27 13:45:48 CEST
Ok, one could extend the UCS 5.0-4 preup.sh to add a check for something like

testparm -sv 2>/dev/null | sed -nE 's/^\s*full_audit:(success|failure) = //p'

and then parsing that output against the known-good list from man vfs_full_audit.