Univention Bugzilla – Bug 38601
Test IMAP server with several hundreds of connections
Last modified: 2016-10-05 20:19:48 CEST
We should have a test, that establishes several hundreds of IMAP connections (1100-1500) to the IMAP server to verify that - the server is able to open that much connections (>1024!) - the server does not consume too much memory for each connection (cyrus 2.2 uses about 1MiB per connection)
A new script with the name '30_imap_server_with_hundreds_of_connections' is created to test the mentioned cases: - Create 1500 imapd connection. - Login simultaneously to the open connections. - Logout. - Measure the consumed memory per connection (1.2 ~ 1.4)MB. The test script fails in case no enough memory is available or if it was not possible to do any of the above tasks.
Commit 61855 changed the test so that it uses SSL to connect to the IMAP server.
(In reply to Daniel Tröder from comment #2) > Commit 61855 changed the test so that it uses SSL to connect to the IMAP > server. With SSL the memory consumed per connection increases to be (~≃ 2MB)
You OOM-killed the S4-Master: 30_imap_server_with_hundreds_of_connections <http://jenkins.knut.univention.de:8080/job/UCS-4.0/job/UCS-4.0-2/job/Autotest%20MultiEnv/SambaVersion=s4,Systemrolle=master/69/> # grep -F "Jul 20 08:26:18" /var/log/kern.log Jul 20 08:26:18 master091 kernel: [ 8199.689108] CPU 0: hi: 186, btch: 31 usd: 30 Jul 20 08:26:18 master091 kernel: [ 8199.689111] active_anon:859441 inactive_anon:4401 isolated_anon:0 Jul 20 08:26:18 master091 kernel: [ 8199.689111] active_file:558 inactive_file:600 isolated_file:0 Jul 20 08:26:18 master091 kernel: [ 8199.689111] unevictable:0 dirty:0 writeback:0 unstable:0 Jul 20 08:26:18 master091 kernel: [ 8199.689111] free:15020 slab_reclaimable:6842 slab_unreclaimable:19454 Jul 20 08:26:18 master091 kernel: [ 8199.689111] mapped:7151 shmem:10350 pagetables:30282 bounce:0 Jul 20 08:26:18 master091 kernel: [ 8199.689111] free_cma:0 Jul 20 08:26:18 master091 kernel: [ 8199.689113] Node 0 DMA free:15212kB min:184kB low:228kB high:276kB active_anon:504kB inactive_anon:0kB active_file:0kB inactive_file:44kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15988kB managed:15904kB mlocked:0kB dirty:0kB writeback:0kB mapped:4kB shmem:0kB slab_reclaimable:4kB slab_unreclaimable:20kB kernel_stack:0kB pagetables:48kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:586 all_unreclaimable? yes Jul 20 08:26:18 master091 kernel: [ 8199.689116] lowmem_reserve[]: 0 3757 3757 3757 Jul 20 08:26:18 master091 kernel: [ 8199.689118] Node 0 DMA32 free:44868kB min:44868kB low:56084kB high:67300kB active_anon:3437260kB inactive_anon:17604kB active_file:2232kB inactive_file:2356kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3915776kB managed:3849416kB mlocked:0kB dirty:0kB writeback:0kB mapped:28600kB shmem:41400kB slab_reclaimable:27364kB slab_unreclaimable:77796kB kernel_stack:23952kB pagetables:121080kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:6911 all_unreclaimable? yes Jul 20 08:26:18 master091 kernel: [ 8199.689121] lowmem_reserve[]: 0 0 0 0 Jul 20 08:26:18 master091 kernel: [ 8199.689123] Node 0 DMA: 3*4kB (UEM) 2*8kB (EM) 5*16kB (UEM) 2*32kB (EM) 5*64kB (UE) 3*128kB (UEM) 2*256kB (EM) 1*512kB (E) 3*1024kB (UEM) 3*2048kB (EMR) 1*4096kB (M) = 15212kB Jul 20 08:26:18 master091 kernel: [ 8199.689130] Node 0 DMA32: 1229*4kB (EM) 736*8kB (UEM) 407*16kB (UEM) 149*32kB (UEM) 82*64kB (UEM) 51*128kB (UEM) 25*256kB (UEM) 1*512kB (E) 0*1024kB 0*2048kB 1*4096kB (R) = 44868kB Jul 20 08:26:18 master091 kernel: [ 8199.689137] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Jul 20 08:26:18 master091 kernel: [ 8199.689137] 11526 total pagecache pages Jul 20 08:26:18 master091 kernel: [ 8199.689138] 0 pages in swap cache Jul 20 08:26:18 master091 kernel: [ 8199.689139] Swap cache stats: add 0, delete 0, find 0/0 Jul 20 08:26:18 master091 kernel: [ 8199.689140] Free swap = 0kB Jul 20 08:26:18 master091 kernel: [ 8199.689141] Total swap = 0kB Jul 20 08:26:18 master091 kernel: [ 8199.689141] 982941 pages RAM Jul 20 08:26:18 master091 kernel: [ 8199.689142] 0 pages HighMem/MovableOnly Jul 20 08:26:18 master091 kernel: [ 8199.689143] 16590 pages reserved Jul 20 08:26:18 master091 kernel: [ 8199.689143] 0 pages hwpoisoned Jul 20 08:26:18 master091 kernel: [ 8199.689144] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name Jul 20 08:26:18 master091 kernel: [ 8199.689147] [ 578] 0 578 5546 322 16 0 -1000 udevd Jul 20 08:26:18 master091 kernel: [ 8199.689149] [ 950] 0 950 5545 323 15 0 -1000 udevd Jul 20 08:26:18 master091 kernel: [ 8199.689150] [ 951] 0 951 5545 323 15 0 -1000 udevd Jul 20 08:26:18 master091 kernel: [ 8199.689152] [ 2306] 0 2306 2494 575 8 0 0 dhclient Jul 20 08:26:18 master091 kernel: [ 8199.689154] [ 2551] 0 2551 4747 73 15 0 0 rpcbind Jul 20 08:26:18 master091 kernel: [ 8199.689155] [ 2583] 109 2583 5840 118 17 0 0 rpc.statd Jul 20 08:26:18 master091 kernel: [ 8199.689156] [ 2597] 0 2597 6327 52 18 0 0 rpc.idmapd Jul 20 08:26:18 master091 kernel: [ 8199.689158] [ 2602] 0 2602 7645 80 19 0 0 rpc.gssd Jul 20 08:26:18 master091 kernel: [ 8199.689159] [ 3006] 0 3006 49 4 3 0 0 runsvdir Jul 20 08:26:18 master091 kernel: [ 8199.689160] [ 3011] 0 3011 44 1 3 0 0 runsv Jul 20 08:26:18 master091 kernel: [ 8199.689162] [ 3012] 0 3012 44 1 3 0 0 runsv Jul 20 08:26:18 master091 kernel: [ 8199.689163] [ 3013] 0 3013 44 1 3 0 0 runsv Jul 20 08:26:18 master091 kernel: [ 8199.689164] [ 3014] 0 3014 44 1 3 0 0 runsv Jul 20 08:26:18 master091 kernel: [ 8199.689165] [ 3015] 0 3015 44 5 3 0 0 runsv Jul 20 08:26:18 master091 kernel: [ 8199.689167] [ 3016] 0 3016 44 1 3 0 0 runsv Jul 20 08:26:18 master091 kernel: [ 8199.689168] [ 3033] 0 3033 13214 106 22 0 0 rsyslogd Jul 20 08:26:18 master091 kernel: [ 8199.689169] [ 3063] 0 3063 1033 36 7 0 0 acpid Jul 20 08:26:18 master091 kernel: [ 8199.689171] [ 3093] 119 3093 7972 72 21 0 0 dbus-daemon Jul 20 08:26:18 master091 kernel: [ 8199.689172] [ 3265] 121 3265 23260 1465 38 0 -900 postgres Jul 20 08:26:18 master091 kernel: [ 8199.689174] [ 3267] 121 3267 23260 5456 44 0 0 postgres Jul 20 08:26:18 master091 kernel: [ 8199.689175] [ 3268] 121 3268 23260 508 31 0 0 postgres Jul 20 08:26:18 master091 kernel: [ 8199.689176] [ 3269] 121 3269 23471 481 33 0 0 postgres Jul 20 08:26:18 master091 kernel: [ 8199.689178] [ 3270] 121 3270 15397 341 30 0 0 postgres Jul 20 08:26:18 master091 kernel: [ 8199.689179] [ 3305] 0 3305 32931 12849 70 0 0 /usr/sbin/spamd Jul 20 08:26:18 master091 kernel: [ 8199.689180] [ 3307] 0 3307 32931 12849 67 0 0 spamd child Jul 20 08:26:18 master091 kernel: [ 8199.689182] [ 3308] 0 3308 32931 12849 67 0 0 spamd child Jul 20 08:26:18 master091 kernel: [ 8199.689183] [ 3401] 0 3401 31191 268 64 0 0 sshd Jul 20 08:26:18 master091 kernel: [ 8199.689184] [ 3444] 112 3444 101403 76253 195 0 0 clamd Jul 20 08:26:18 master091 kernel: [ 8199.689186] [ 3518] 112 3518 17352 828 32 0 0 freshclam Jul 20 08:26:18 master091 kernel: [ 8199.689187] [ 3591] 0 3591 1547 27 7 0 0 ifplugd Jul 20 08:26:18 master091 kernel: [ 8199.689189] [ 3717] 0 3717 51336 375 96 0 0 nmbd Jul 20 08:26:18 master091 kernel: [ 8199.689190] [ 3720] 0 3720 51195 359 96 0 0 nmbd Jul 20 08:26:18 master091 kernel: [ 8199.689191] [ 3785] 0 3785 2614 34 11 0 0 inetd Jul 20 08:26:18 master091 kernel: [ 8199.689192] [ 3982] 0 3982 12088 132 24 0 0 saslauthd Jul 20 08:26:18 master091 kernel: [ 8199.689194] [ 3984] 0 3984 12088 132 24 0 0 saslauthd Jul 20 08:26:18 master091 kernel: [ 8199.689195] [ 3985] 0 3985 12088 132 24 0 0 saslauthd Jul 20 08:26:18 master091 kernel: [ 8199.689196] [ 3986] 0 3986 12088 132 24 0 0 saslauthd Jul 20 08:26:18 master091 kernel: [ 8199.689198] [ 3987] 0 3987 12088 132 24 0 0 saslauthd Jul 20 08:26:18 master091 kernel: [ 8199.689199] [ 4065] 0 4065 2177 149 10 0 0 univention-dire Jul 20 08:26:18 master091 kernel: [ 8199.689200] [ 4169] 0 4169 103553 6303 193 0 0 univention-mana Jul 20 08:26:18 master091 kernel: [ 8199.689202] [ 4205] 0 4205 166478 5996 199 0 0 univention-mana Jul 20 08:26:18 master091 kernel: [ 8199.689203] [ 4285] 110 4285 9233 153 23 0 0 ntpd Jul 20 08:26:18 master091 kernel: [ 8199.689204] [ 4384] 0 4384 14625 419 30 0 0 squid3 Jul 20 08:26:18 master091 kernel: [ 8199.689205] [ 4386] 0 4386 24120 3255 39 0 0 squid3 Jul 20 08:26:18 master091 kernel: [ 8199.689207] [ 4438] 13 4438 5027 42 14 0 0 unlinkd Jul 20 08:26:18 master091 kernel: [ 8199.689208] [ 4455] 0 4455 24727 309 53 0 0 cupsd Jul 20 08:26:18 master091 kernel: [ 8199.689209] [ 4560] 0 4560 4170 36 12 0 0 atd Jul 20 08:26:18 master091 kernel: [ 8199.689211] [ 4651] 0 4651 2340 82 10 0 0 bash Jul 20 08:26:18 master091 kernel: [ 8199.689212] [ 4652] 0 4652 17712 2977 39 0 0 ucs-test Jul 20 08:26:18 master091 kernel: [ 8199.689213] [ 5241] 0 5241 2316 58 10 0 0 univention-welc Jul 20 08:26:18 master091 kernel: [ 8199.689214] [ 5256] 0 5256 4418 38 13 0 0 getty Jul 20 08:26:18 master091 kernel: [ 8199.689216] [ 5257] 0 5257 4418 38 13 0 0 getty Jul 20 08:26:18 master091 kernel: [ 8199.689217] [ 5258] 0 5258 4418 38 14 0 0 getty Jul 20 08:26:18 master091 kernel: [ 8199.689218] [ 5259] 0 5259 4418 40 14 0 0 getty Jul 20 08:26:18 master091 kernel: [ 8199.689220] [ 5260] 0 5260 4418 40 14 0 0 getty Jul 20 08:26:18 master091 kernel: [ 8199.689221] [ 5261] 0 5261 4418 40 14 0 0 getty Jul 20 08:26:18 master091 kernel: [ 8199.689222] [ 5281] 0 5281 3944 30 14 0 0 xinit Jul 20 08:26:18 master091 kernel: [ 8199.689224] [ 5282] 0 5282 30240 3311 63 0 0 Xorg Jul 20 08:26:18 master091 kernel: [ 8199.689225] [ 5563] 0 5563 25984 198 56 0 0 su Jul 20 08:26:18 master091 kernel: [ 8199.689226] [ 5571] 65534 5571 1048 19 8 0 0 sh Jul 20 08:26:18 master091 kernel: [ 8199.689228] [ 5572] 65534 5572 2321 62 10 0 0 univention-welc Jul 20 08:26:18 master091 kernel: [ 8199.689229] [ 5575] 65534 5575 27935 413 58 0 0 openbox Jul 20 08:26:18 master091 kernel: [ 8199.689230] [ 5576] 65534 5576 143530 26698 204 0 0 firefox Jul 20 08:26:18 master091 kernel: [ 8199.689232] [ 5579] 65534 5579 8956 1378 21 0 0 python Jul 20 08:26:18 master091 kernel: [ 8199.689233] [ 6326] 0 6326 9634 728 22 0 0 dhcpd Jul 20 08:26:18 master091 kernel: [ 8199.689234] [ 6384] 0 6384 34027 441 43 0 0 nscd Jul 20 08:26:18 master091 kernel: [ 8199.689236] [ 6452] 0 6452 127857 7743 240 0 0 samba Jul 20 08:26:18 master091 kernel: [ 8199.689238] [ 6459] 0 6459 127857 7743 220 0 0 samba Jul 20 08:26:18 master091 kernel: [ 8199.689239] [ 6460] 0 6460 117792 7665 224 0 0 smbd Jul 20 08:26:18 master091 kernel: [ 8199.689241] [ 6461] 0 6461 130566 7747 242 0 0 samba Jul 20 08:26:18 master091 kernel: [ 8199.689242] [ 6462] 0 6462 128485 7743 222 0 0 samba Jul 20 08:26:18 master091 kernel: [ 8199.689243] [ 6463] 0 6463 134665 7948 247 0 0 samba Jul 20 08:26:18 master091 kernel: [ 8199.689244] [ 6464] 0 6464 127857 7743 220 0 0 samba Jul 20 08:26:18 master091 kernel: [ 8199.689246] [ 6465] 0 6465 131080 7748 247 0 0 samba Jul 20 08:26:18 master091 kernel: [ 8199.689247] [ 6466] 0 6466 128442 7743 236 0 0 samba Jul 20 08:26:18 master091 kernel: [ 8199.689248] [ 6467] 0 6467 127857 7743 220 0 0 samba Jul 20 08:26:18 master091 kernel: [ 8199.689249] [ 6468] 0 6468 127857 7743 220 0 0 samba Jul 20 08:26:18 master091 kernel: [ 8199.689251] [ 6469] 0 6469 110260 7616 206 0 0 winbindd Jul 20 08:26:18 master091 kernel: [ 8199.689252] [ 6470] 0 6470 129592 7743 240 0 0 samba Jul 20 08:26:18 master091 kernel: [ 8199.689253] [ 6471] 0 6471 127857 7743 227 0 0 samba Jul 20 08:26:18 master091 kernel: [ 8199.689254] [ 6480] 0 6480 117792 7667 201 0 0 smbd Jul 20 08:26:18 master091 kernel: [ 8199.689256] [ 6482] 0 6482 112355 7619 197 0 0 winbindd Jul 20 08:26:18 master091 kernel: [ 8199.689257] [ 7408] 0 7408 120885 11131 219 0 0 named Jul 20 08:26:18 master091 kernel: [ 8199.689259] [11635] 0 11635 10911 138 26 0 -1000 sshd Jul 20 08:26:18 master091 kernel: [ 8199.689260] [15449] 0 15449 3352 55 11 0 0 cron Jul 20 08:26:18 master091 kernel: [ 8199.689261] [28336] 0 28336 123885 8621 230 0 0 python2.7 Jul 20 08:26:18 master091 kernel: [ 8199.689263] [ 1011] 0 1011 138687 8464 259 0 0 univention-dire Jul 20 08:26:18 master091 kernel: [ 8199.689264] [14607] 0 14607 603908 5245 111 0 0 slapd Jul 20 08:26:18 master091 kernel: [ 8199.689266] [15289] 0 15289 5425 146 16 0 0 rpc.mountd Jul 20 08:26:18 master091 kernel: [ 8199.689267] [19852] 108 19852 9948 272 20 0 0 nagios3 Jul 20 08:26:18 master091 kernel: [ 8199.689268] [20280] 0 20280 38632 1369 73 0 0 apache2 Jul 20 08:26:18 master091 kernel: [ 8199.689270] [20288] 108 20288 6880 133 17 0 0 nrpe Jul 20 08:26:18 master091 kernel: [ 8199.689271] [20292] 33 20292 39163 1374 71 0 0 apache2 Jul 20 08:26:18 master091 kernel: [ 8199.689272] [20293] 33 20293 39163 1374 71 0 0 apache2 Jul 20 08:26:18 master091 kernel: [ 8199.689274] [20294] 33 20294 39163 1374 71 0 0 apache2 Jul 20 08:26:18 master091 kernel: [ 8199.689275] [20295] 33 20295 39163 1374 71 0 0 apache2 Jul 20 08:26:18 master091 kernel: [ 8199.689276] [20296] 33 20296 39163 1374 71 0 0 apache2 Jul 20 08:26:18 master091 kernel: [ 8199.689278] [21406] 116 21406 58211 21736 111 0 0 /usr/sbin/amavi Jul 20 08:26:18 master091 kernel: [ 8199.689279] [21471] 116 21471 58547 21769 112 0 0 /usr/sbin/amavi Jul 20 08:26:18 master091 kernel: [ 8199.689281] [21472] 116 21472 58547 21769 112 0 0 /usr/sbin/amavi Jul 20 08:26:18 master091 kernel: [ 8199.689282] [21851] 0 21851 9470 141 21 0 0 master Jul 20 08:26:18 master091 kernel: [ 8199.689283] [21852] 103 21852 10504 137 23 0 0 pickup Jul 20 08:26:18 master091 kernel: [ 8199.689285] [21853] 103 21853 10547 153 23 0 0 qmgr Jul 20 08:26:18 master091 kernel: [ 8199.689286] [21857] 103 21857 11036 140 25 0 0 tlsmgr Jul 20 08:26:18 master091 kernel: [ 8199.689287] [23230] 0 23230 84133 5867 151 0 0 univention-cli- Jul 20 08:26:18 master091 kernel: [ 8199.689289] [23786] 0 23786 220938 190297 435 0 0 python Jul 20 08:26:18 master091 kernel: [ 8199.689290] [23856] 0 23856 5519 120 15 0 0 dovecot Jul 20 08:26:18 master091 kernel: [ 8199.689292] [23857] 113 23857 3929 634 13 0 0 anvil Jul 20 08:26:18 master091 kernel: [ 8199.689293] [23858] 0 23858 3402 90 12 0 0 log Jul 20 08:26:18 master091 kernel: [ 8199.689294] [23859] 0 23859 4884 88 14 0 0 ssl-params Jul 20 08:26:18 master091 kernel: [ 8199.689295] [23860] 114 23860 6233 277 19 0 0 imap-login Jul 20 08:26:18 master091 kernel: [ 8199.689297] [23861] 0 23861 7785 3467 22 0 0 config Jul 20 08:26:18 master091 kernel: [ 8199.689298] [23864] 113 23864 14083 4882 33 0 0 auth Jul 20 08:26:18 master091 kernel: [ 8199.689299] [23865] 114 23865 6233 276 19 0 0 imap-login [ ... 1226 more lines for imap-login ... ] Jul 20 08:26:18 master091 kernel: [ 8199.690915] [25162] 0 25162 6066 132 18 0 0 imap-login
The test creates 1500 concurrent IMAPS connections. It consumes around 4.2 GB memory with both Dovecot and Cyrus. --- Unrelated: The test currently fails for me with Cyrus: 1500 IMAP connections are OK (took 98.497419 seconds) Login Failed with exeption:error('Login failed: generic failure',) Unsetting mail/cyrus/imap/maxchilds File: /etc/horde/ingo/prefs.d/10-ucs.php Module: create-archivefolder Multifile: /etc/postfix/ldap.sharedfolderlocal Multifile: /etc/imapd/cyrus.conf Cleanup after exception: <class 'imaplib.error'> Login failed: generic failure Performing UCSTestUDM cleanup... Traceback (most recent call last): File "30_imap_server_with_hundreds_of_connections", line 92, in <module> main() File "30_imap_server_with_hundreds_of_connections", line 75, in main imap.log_in(usermail, password) File "/root/ucs-test/tests/40_mail/essential/mailclient.py", line 49, in log_in self.login(usermail, password) File "/usr/lib/python2.7/imaplib.py", line 507, in login raise self.error(dat[-1]) imaplib.error: Login failed: generic failure UCSTestUDM cleanup done Starting 1 ucs-test at 2015-07-21 07:58:01 to /dev/null Imap Server with hunderds of connections........................................................... Test failed
r62293 | Bug #38601 test: Disable 40/30 Tagged as producttest to disable test for now Package: ucs-test Version: 5.0.158-2.1097.201507211859 Branch: ucs_4.0-0 Scope: errata4.0-2
(In reply to Daniel Tröder from comment #5) > The test creates 1500 concurrent IMAPS connections. It consumes around 4.2 > GB memory with both Dovecot and Cyrus. $ grep -h ec2_instance_type examples/jenkins/autotest-09*.cfg | sort -u ec2_instance_type: m1.medium ec2_instance_type: m3.medium <http://aws.amazon.com/de/ec2/instance-types/#M3> | Modell vCPU Arbeitsspeicher (GiB) SSD-Speicherung (GB) | m3.medium 1 3,75 1 x 4 All UCS-EC2 instances have NO swap space configured, so the Kernel will kill any process in OOM situations! See Bug #34819 comment 5.
In my last test a VM with 4GB RAM went 730MB into the 1GB swap and passed the test. But once before it failed (maybe I forgot to stop OX and mysql, and it had less free mem), so 2 GB swap would be better.
(In reply to Daniel Tröder from comment #8) > In my last test a VM with 4GB RAM went 730MB into the 1GB swap And that's the difference: The EC2 instances have *NO* swap configured! When the kernel next needs memory, it will start killing processes, which may be slapd, UDL, sshd, or any other essential process.
r62391 | Bug #38601 test: Tags for producttest Same for tests/40_mail/44_dovecot_login_performance_test Was killed by timeout=1h (Bug #35153); all subsequent tests failed because cleanup did not happened and license limit was exceeded. According to <http://jenkins.knut.univention.de:8080/job/UCS-4.0/job/UCS-4.0-2/job/Autotest%20MultiEnv/SambaVersion=s3,Systemrolle=master/69/testReport/40_mail/> the test runs for 2-6h ! Package: ucs-test Version: 5.0.160-2.1100.201507240924 Branch: ucs_4.0-0 Scope: errata4.0-2
(In reply to Philipp Hahn from comment #10) > According to > <http://jenkins.knut.univention.de:8080/job/UCS-4.0/job/UCS-4.0-2/job/ > Autotest%20MultiEnv/SambaVersion=s3,Systemrolle=master/69/testReport/40_mail/ > > the test runs for 2-6h ! JFYI: During my tests on spoka, the test took 60 to 90 seconds. And the machine had 1 GB RAM with 2 GB SWAP. So I didn't see any problem with the jenkins environment at that moment.
(In reply to Sönke Schwardt-Krummrich from comment #11) > (In reply to Philipp Hahn from comment #10) > > According to > > <http://jenkins.knut.univention.de:8080/job/UCS-4.0/job/UCS-4.0-2/job/ > > Autotest%20MultiEnv/SambaVersion=s3,Systemrolle=master/69/testReport/40_mail/ > > > the test runs for 2-6h ! > > JFYI: > During my tests on spoka, the test took 60 to 90 seconds. And the machine > had > 1 GB RAM with 2 GB SWAP. So I didn't see any problem with the jenkins > environment at that moment. FYI: <http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSVolumeTypes.html> - We use "m3.medium"=(1CPU,3.75G) @ EC2 with 10 GB standard="magnetic disk". - Each IO operation (read/write) requires an IOP credit. - To allow a "good boot/installation experience", each instance starts with (30*60)s*3k IOPS credits to allow maximum performance in the first ½h. - Credits are replenished at 3 IOPS/GB, which is a ridiculous 30 for our 10 GB. - IOPS are capped at 3000 IOPS/s. If you need more, you have to pay more for "ios1" devices + EBS optimized instances. - Many other test have run before the section "40_mail" tests, which have consumed lots of IOPS. So the EC2 instance will be IO throttled, which our internal KVM environment isn't! You can see this running "vmstat": wa(it-for-IO) = last-column is close to 100, while us(er-space-time), sy(system-time) and id(le-time) are close to 0. Plus the EC2 instances don't have swap (but 3.75 GB RAM).
I suggest to make the number of concurrent connections configurable,. Then either adapt automatically to available memory or make it ~500 by default for aws. With a cmdline switch it could be set higher. With the test we want to check (please expand on this if I forget something): * UCR settings work to change connection limits * memory consumption per connection IMHO the test would do this with 500 simultaneous connections too: * the default is 400 connection, so the UCRVs must change for success * if 500 connection work in 3 GB, then it's OK. (1500 connections will very likely work in 6GB. Any other hardware on the UCS user side would not make sense anyway.)
(In reply to Daniel Tröder from comment #13) > I suggest to make the number of concurrent connections configurable,. Then > either adapt automatically to available memory or make it ~500 by default > for aws. With a cmdline switch it could be set higher. No. Main reason for the test script is to check, if more connections than the usual file descriptor limit of 1024 can be configured. > With the test we want to check (please expand on this if I forget something): > * UCR settings work to change connection limits > * memory consumption per connection * jump over the 1024 file descriptors limit and check if all components work correctly I'm ok with it, if the test is disabled in jenkins for now. But I'm a little bit puzzled that my machine with 3GB of available memory (RAM+SWAP) was able to handle that test easily while the jenkins machine with 3.75GB triggered the OOM. I would suggest, that we use the new feature "instance stores" for ucs-ec2-create and use the instance store with e.g. overlayfs to reduce IOPS to the magnetic disk and improve overall performance. If this has been implemented, the test may be reactivated.