Univention Bugzilla – Bug 38504
ec2-tools: Improve error handling
Last modified: 2021-03-15 11:58:29 CET
1. re-try throttled API calls: <http://docs.aws.amazon.com/general/latest/gr/api-retries.html> Traceback (most recent call last): File "./ucs-ec2-create", line 133, in <module> main() File "./ucs-ec2-create", line 93, in main vm.start() File "/var/lib/jenkins/workspace/UCS-4.0/UCS-4.0-1/Autotest MultiEnv/SambaVersion/s3/Systemrolle/backup/python/ec2/lib.py", line 698, in start self.instance.add_tag(var.lower(), os.getenv(var, '')) File "/usr/lib/python2.6/dist-packages/boto/ec2/ec2object.py", line 79, in add_tag status = self.connection.create_tags([self.id], {key : value}) File "/usr/lib/python2.6/dist-packages/boto/ec2/connection.py", line 2810, in create_tags return self.get_status('CreateTags', params, verb='POST') File "/usr/lib/python2.6/dist-packages/boto/connection.py", line 921, in get_status response = self.make_request(action, params, path, verb) File "/usr/lib/python2.6/dist-packages/boto/connection.py", line 868, in make_request return self._mexe(http_request) File "/usr/lib/python2.6/dist-packages/boto/connection.py", line 792, in _mexe raise BotoServerError(response.status, response.reason, body) boto.exception.BotoServerError: BotoServerError: 503 Service Unavailable <?xml version="1.0" encoding="UTF-8"?> <Response><Errors><Error><Code>RequestLimitExceeded</Code><Message>Request limit exceeded.</Message></Error></Errors><RequestID>92d6670b-4f83-4cc9-beb6-bf00363a053f</RequestID></Response> 2. re-try failed DNS queries 3. Reverse all instances at once or clean up if maximum number of allowed instances is exceeded. 4. Fix Traceback (most recent call last): File "./ucs-ec2-create", line 127, in <module> main() File "./ucs-ec2-create", line 88, in main vm.start() File "/var/lib/jenkins/workspace/Customers/01997_oxin8/OXIN8 - Generic AutoTest Env Cluster/default/python/ec2/lib.py", line 689, in start self._wait_instance() File "/var/lib/jenkins/workspace/Customers/01997_oxin8/OXIN8 - Generic AutoTest Env Cluster/default/python/ec2/lib.py", line 721, in _wait_instance raise TypeError: exceptions must be old-style classes or derived from BaseException, not NoneType
(In reply to Philipp Hahn from comment #0) > 4. Fix > > Traceback (most recent call last): > File "./ucs-ec2-create", line 127, in <module> > main() > File "./ucs-ec2-create", line 88, in main > vm.start() > File "/var/lib/jenkins/workspace/Customers/01997_oxin8/OXIN8 - Generic > AutoTest Env Cluster/default/python/ec2/lib.py", line 689, in start > self._wait_instance() > File "/var/lib/jenkins/workspace/Customers/01997_oxin8/OXIN8 - Generic > AutoTest Env Cluster/default/python/ec2/lib.py", line 721, in _wait_instance > raise > TypeError: exceptions must be old-style classes or derived from > BaseException, not NoneType This was a missing backport, i fixed it already in r60618
Again: <?xml version="1.0" encoding="UTF-8"?> <Response><Errors><Error><Code>RequestLimitExceeded</Code><Message>Request limit exceeded.</Message></Error></Errors><RequestID>d5ae17fc-445e-42af-bd3e-f34a1ac0b157</RequestID></Response> <http://jenkins.knut.univention.de:8080/job/UCS-3.2/job/UCS-3.2-6/job/Autotest%20MultiEnv/14/SambaVersion=s4,Systemrolle=backup/console> <http://jenkins.knut.univention.de:8080/job/UCS-4.0/job/UCS-4.0-2/job/Autotest%20MultiEnv/22/SambaVersion=s4,Systemrolle=slave/console>
Again: <http://jenkins.knut.univention.de:8080/job/UCS-4.0/job/UCS-4.0-2/job/App%20Autotest%20Update%20MultiEnv%20from%203.2/18/SambaVersion=s4,Systemrolle=slave/console> According to <http://boto.readthedocs.org/en/latest/boto_config_tut.html#boto>[Boto]num_retries calls are already re-tried 5 times by default.
Added some comments to several commits in gitlab. Besides that, the code review looks good. When do be switch to the new version? This should be communicated with all dev team members.
https://git.knut.univention.de/dist/repo-ng/-/issues/36 The new version of ucs-ec2-tools is already in production. We moved most of our tests back from AWS-EC2 to internal KVM, so we already do much less API calls. For all new error handling improvements create new issues in https://git.knut.univention.de/dist/ucs-ec2-tools/-/issues/