Univention Bugzilla – Bug 42279
docker inode exhaustion vs docker image layers
Last modified: 2019-01-03 07:23:43 CET
I tried to install a docker app. The AppCenter failed to install the app with: CalledProcessError: Command '['docker', 'create', '--hostname', 'xyz-1472723908245020', '-e', 'update_secure_apt=no', '-e', 'UPDATE_SECURE_APT=no', '-e', 'nameserver_external=false', '-e', 'NAMESERVER_EXTERNAL=false', '-e', 'repository_app_center_server=appcenter-test.software-univention.de', '-e', 'REPOSITORY_APP_CENTER_SERVER=appcenter-test.software-univention.de', '-e', 'server_role=memberserver', '-e', 'SERVER_ROLE=memberserver', '-e', 'repository_online_server=https://updates.software-univention.de/', '-e', 'REPOSITORY_ONLINE_SERVER=https://updates.software-univention.de/', '-e', 'nameserver2=172.16.0.23', '-e', 'NAMESERVER2=172.16.0.23', '-e', 'nameserver1=10.84.15.79', '-e', 'NAMESERVER1=10.84.15.79', '-e', 'domainname=ucs.demo', '-e', 'DOMAINNAME=ucs.demo', '-e', 'ldap_hostdn=cn=xyz-1472723908245020,cn=memberserver,cn=computers,dc=ucs,dc=demo', '-e', 'LDAP_HOSTDN=cn=xyz-1472723908245020,cn=memberserver,cn=computers,dc=ucs,dc=demo', '-e', 'locale=de_DE.UTF-8:UTF-8 en_US.UTF-8:UTF-8', '-e', 'LOCALE=de_DE.UTF-8:UTF-8 en_US.UTF-8:UTF-8', '-e', 'updater_identify=Docker App', '-e', 'UPDATER_IDENTIFY=Docker App', '-e', 'locale_default=de_DE.UTF-8:UTF-8', '-e', 'LOCALE_DEFAULT=de_DE.UTF-8:UTF-8', '-e', 'nameserver_option_timeout=2', '-e', 'NAMESERVER_OPTION_TIMEOUT=2', '-e', 'ldap_master_port=7389', '-e', 'LDAP_MASTER_PORT=7389', '-e', 'ldap_master=master.ucs.demo', '-e', 'LDAP_MASTER=master.ucs.demo', '-e', 'update_warning_releasenotes=no', '-e', 'UPDATE_WARNING_RELEASENOTES=no', '-e', 'appcenter_index_verify=false', '-e', 'APPCENTER_INDEX_VERIFY=false', '-p', '40000:80', '-p', '40001:443', '-v', '/var/lib/univention-appcenter/apps/xyz/data:/var/lib/univention-appcenter/apps/xyz/data', '-v', '/var/lib/univention-appcenter/apps/xyz/conf:/var/lib/univention-appcenter/apps/xyz/conf', 'xyz:3.0.12.6.9']' returned non-zero exit status 1 raise CalledProcessError(retcode, cmd, output=output) File "/usr/lib/python2.7/subprocess.py", line 544, in check_output return check_output(['docker', 'create'] + _args + [image] + command).strip() File "/usr/lib/pymodules/python2.7/univention/appcenter/docker.py", line 204, in create container = create(self.image, command, hostname, env, ports, volumes, env_file, args) File "/usr/lib/pymodules/python2.7/univention/appcenter/docker.py", line 302, in create container = docker.create(hostname, set_vars) File "/usr/lib/pymodules/python2.7/univention/appcenter/actions/docker_base.py", line 174, in _start_docker_image self._start_docker_image(app, hostdn, password, args) File "/usr/lib/pymodules/python2.7/univention/appcenter/actions/docker_install.py", line 59, in _install_app if self._install_app(app, args): File "/usr/lib/pymodules/python2.7/univention/appcenter/actions/install.py", line 79, in _do_it ret = super(Install, self)._do_it(app, args) File "/usr/lib/pymodules/python2.7/univention/appcenter/actions/docker_install.py", line 65, in _do_it self._do_it(app, args) File "/usr/lib/pymodules/python2.7/univention/appcenter/actions/install_base.py", line 107, in do_it return self.do_it(args) File "/usr/lib/pymodules/python2.7/univention/appcenter/actions/install.py", line 66, in main result = self.main(namespace) File "/usr/lib/pymodules/python2.7/univention/appcenter/actions/__init__.py", line 182, in call_with_namespace return obj.call_with_namespace(namespace) File "/usr/lib/pymodules/python2.7/univention/appcenter/actions/__init__.py", line 176, in call result['success'] = action.call(app=app, username=self.username, password=self.password, **kwargs) File "/usr/lib/pymodules/python2.7/univention/management/console/modules/appcenter/__init__.py", line 397, in invoke_docker yield function(self, *args) File "/usr/lib/pymodules/python2.7/univention/management/console/modules/decorators.py", line 284, in _fake_func return list(function(self, iterator, *nones)) File "/usr/lib/pymodules/python2.7/univention/management/console/modules/decorators.py", line 462, in _response result = _multi_response(self, request) File "/usr/lib/pymodules/python2.7/univention/management/console/modules/decorators.py", line 309, in _thread ret = progress_obj.poll() File "/usr/lib/pymodules/python2.7/univention/management/console/modules/mixins.py", line 149, in progress yield function(self, *args) File "/usr/lib/pymodules/python2.7/univention/management/console/modules/decorators.py", line 284, in _fake_func return list(function(self, iterator, *nones)) File "/usr/lib/pymodules/python2.7/univention/management/console/modules/decorators.py", line 462, in _response result = _multi_response(self, request) File "/usr/lib/pymodules/python2.7/univention/management/console/modules/decorators.py", line 318, in _response function(self, request) File "/usr/lib/pymodules/python2.7/univention/management/console/base.py", line 283, in execute Traceback (most recent call last): hmm, this is not very helpful. So i tried to start the container by hand: -> docker create docker-test.software-univention.de/xyz Pulling repository docker-test.software-univention.de/xyz FATA[0002] Error: image xyz:3.0.12.6.9 not found weired, the image is already installed. So, started docker in debug mode: -> docker -d -D -p /var/run/docker.pid --bip='172.17.42.1/16' --storage... ERRO[0006] Error from V2 registry: mkdir /var/lib/docker/overlay /515d178bd42aef51988d3bbd081c63d0eb5150cd9a0d068ef47fd60d86445b12 /tmproot901608538/opt/alfresco/postgresql/share/timezone/Arctic: no space left on device DEBU[0006] image does not exist on v2 registry, falling back to v1 ah, that is the problem. Turns out that the docker image alone exhausted all the available inodes of the system. -> df -i Dateisystem Inodes IBenutzt IFrei IUse% Eingehängt auf udev 477597 296 477301 1% /dev tmpfs 482297 264 482033 1% /run /dev/disk/by-uuid/...-8b3b72288eee 655360 654056 1304 100% / After removing the image, only 20% -> df -i Dateisystem Inodes IBenutzt IFrei IUse% Eingehängt auf udev 477597 296 477301 1% /dev tmpfs 482297 262 482035 1% /run /dev/disk/by-uuid/...-8b3b72288eee 655360 128109 527251 20% / Problem is that system partition is too small (only 10GB), but more important the docker image has 163 (!) layers (they are all linked together with hard links, which means inodes) What should/can we do? * Maybe we should have a limit for the number of layers for our docker images? * The AppCenter should check free disc space/inodes before (a) installing a image (b) starting a container. If more than X % are in use, abort the app installation a return an appropriate error message. * ...?
This issue has been filled against UCS 4.1. The maintenance with bug and security fixes for UCS 4.1 has ended on 5st of April 2018. Customers still on UCS 4.1 are encouraged to update to UCS 4.3. Please contact your partner or Univention for any questions. If this issue still occurs in newer UCS versions, please use "Clone this bug" or simply reopen the issue. In this case please provide detailed information on how this issue is affecting you.