QA » openQA Project

From Dmesg...

[776335.997190] Out of memory: Kill process 25008 (openqa) score 348 or sacrifice child
[776335.997201] Killed process 25008 (openqa) total-vm:3781972kB, anon-rss:3554776kB, file-rss:2136kB, shmem-rss:16kB
[776425.499945] openqa-websocke invoked oom-killer: gfp_mask=0x24201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD), nodemask=0, order=0, oom_score_adj=0
[776425.499958] openqa-websocke cpuset=/ mems_allowed=0
[776425.499961] CPU: 5 PID: 1989 Comm: openqa-websocke Not tainted 4.4.46-11-default #1
[776425.499962] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
[776425.499967] 0000000000000000 ffffffff81328657 ffff8802c7377c28 0000000000000000
[776425.499975] ffffffff8120149e ffffffff00000000 0000000000000000 0000000000000010
[776425.499977] 000000000000003a ffff8801042d1200 000000000000003d 0000000000000000
[776425.499977] Call Trace:
[776425.499996] [] dump_trace+0x59/0x320
[776425.500005] [] show_stack_log_lvl+0xfa/0x180
[776425.500008] [] show_stack+0x21/0x40
[776425.500013] [] dump_stack+0x5c/0x85
[776425.500019] [] dump_header+0x82/0x231
[776425.500024] [] oom_kill_process+0x222/0x410
[776425.500031] [] out_of_memory+0x4a8/0x4c0
[776425.500039] [] __alloc_pages_nodemask+0xb98/0xbc0
[776425.500044] [] alloc_pages_current+0x7f/0x100
[776425.500049] [] filemap_fault+0x1b7/0x450
[776425.500088] [] ext4_filemap_fault+0x31/0x50 [ext4]
[776425.500100] [] __do_fault+0x61/0xe0
[776425.500103] [] handle_pte_fault+0xb21/0x1570
[776425.500108] [] handle_mm_fault+0x2b7/0x570
[776425.500112] [] __do_page_fault+0x1af/0x450
[776425.500117] [] trace_do_page_fault+0x3c/0x130
[776425.500122] [] async_page_fault+0x28/0x30
[776425.501762] DWARF2 unwinder stuck at async_page_fault+0x28/0x30

[776425.501762] Leftover inexact backtrace:

[776425.502908] Mem-Info:
[776425.502914] active_anon:2056714 inactive_anon:324484 isolated_anon:0
active_file:18 inactive_file:42 isolated_file:0
unevictable:4206 dirty:1 writeback:0 unstable:0
slab_reclaimable:23050 slab_unreclaimable:50795
mapped:76973 shmem:91939 pagetables:22541 bounce:0
free:33869 free_pcp:211 free_cma:0
[776425.502919] Node 0 DMA free:15908kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
[776425.502921] lowmem_reserve[]: 0 2962 16015 16015 16015
[776425.502926] Node 0 DMA32 free:64604kB min:12488kB low:15608kB high:18728kB active_anon:1294488kB inactive_anon:431448kB active_file:0kB inactive_file:276kB unevictable:2408kB isolated(anon):0kB isolated(file):0kB present:3129336kB managed:1868364kB mlocked:2408kB dirty:4kB writeback:0kB mapped:59212kB shmem:66268kB slab_reclaimable:14368kB slab_unreclaimable:32952kB kernel_stack:1424kB pagetables:15312kB unstable:0kB bounce:0kB free_pcp:844kB local_pcp:120kB free_cma:0kB writeback_tmp:0kB pages_scanned:2296 all_unreclaimable? yes
[776425.502927] lowmem_reserve[]: 0 0 13053 13053 13053
[776425.502932] Node 0 Normal free:54964kB min:55028kB low:68784kB high:82540kB active_anon:6932368kB inactive_anon:866488kB active_file:112kB inactive_file:0kB unevictable:14416kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:8255364kB mlocked:14416kB dirty:0kB writeback:0kB mapped:248680kB shmem:301488kB slab_reclaimable:77832kB slab_unreclaimable:170228kB kernel_stack:5840kB pagetables:74852kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1240 all_unreclaimable? yes
[776425.502933] lowmem_reserve[]: 0 0 0 0 0
[776425.502940] Node 0 DMA: 1*4kB (U) 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15908kB
[776425.502947] Node 0 DMA32: 2554*4kB (UMEH) 2023*8kB (UMEH) 960*16kB (UMEH) 322*32kB (UMEH) 100*64kB (UMEH) 8*128kB (MEH) 2*256kB (H) 2*512kB (H) 0*1024kB 2*2048kB (H) 0*4096kB = 65120kB
[776425.502953] Node 0 Normal: 11973*4kB (UMEH) 7*8kB (H) 12*16kB (H) 0*32kB 7*64kB (H) 6*128kB (H) 4*256kB (H) 2*512kB (H) 0*1024kB 2*2048kB (H) 0*4096kB = 55500kB
[776425.502955] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[776425.502955] 123068 total pagecache pages
[776425.502957] 29325 pages in swap cache
[776425.502958] Swap cache stats: add 201653, delete 172328, find 54135695/54184068
[776425.502958] Free swap = 0kB
[776425.502959] Total swap = 256744kB
[776425.502959] 4194204 pages RAM
[776425.502960] 0 pages HighMem/MovableOnly
[776425.502960] 1659295 pages reserved
[776425.502960] 0 pages hwpoisoned
[776425.502961] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
[776425.502983] [ 421] 0 421 17316 9552 39 3 50 0 systemd-journal
[776425.502986] [ 432] 0 432 7273 496 19 4 44 -1000 dmeventd
[776425.502989] [ 476] 0 476 3044 437 12 3 1291 0 haveged
[776425.502992] [ 507] 0 507 11147 283 24 3 293 -1000 systemd-udevd
[776425.502995] [ 612] 0 612 48650 4122 39 3 0 -1000 multipathd
[776425.503002] [ 675] 0 675 23976 435 18 3 36 -1000 auditd
[776425.503005] [ 701] 0 701 5841 0 16 3 51 0 rpc.idmapd
[776425.503007] [ 712] 499 712 10113 739 24 3 58 -900 dbus-daemon
[776425.503010] [ 716] 482 716 201160 469 44 4 31 0 nscd
[776425.503013] [ 731] 0 731 7406 157 20 3 173 0 wickedd-dhcp6
[776425.503015] [ 734] 0 734 7406 539 20 3 252 0 wickedd-auto4
[776425.503017] [ 741] 0 741 7407 546 19 3 251 0 wickedd-dhcp4
[776425.503019] [ 749] 0 749 124896 900 46 3 163 0 syslog-ng
[776425.503021] [ 756] 0 756 7443 701 18 3 162 0 wickedd
[776425.503024] [ 759] 0 759 5583 191 16 3 39 0 systemd-logind
[776425.503026] [ 766] 0 766 7439 58 20 3 217 0 wickedd-nanny
[776425.503028] [ 768] 0 768 1187 338 8 3 26 0 agetty
[776425.503031] [ 1172] 0 1172 45944 967 95 3 5057 0 salt-minion
[776425.503033] [ 1177] 0 1177 14424 78 32 3 194 0 cupsd
[776425.503035] [ 1181] 0 1181 5462 272 16 3 82 0 vsftpd
[776425.503038] [ 1184] 0 1184 2193 354 10 3 30 0 xinetd
[776425.503040] [ 1194] 488 1194 67883 3822 135 4 6088 0 salt-master
[776425.503043] [ 1223] 497 1223 4562 359 13 3 112 0 nrpe
[776425.503045] [ 1228] 0 1228 11378 481 28 4 119 -1000 sshd
[776425.503047] [ 1234] 493 1234 13157 717 31 3 79 0 rpcbind
[776425.503049] [ 1246] 492 1246 9026 354 22 3 198 0 rpc.statd
[776425.503052] [ 1247] 0 1247 10456 706 26 3 121 0 rpc.mountd
[776425.503055] [ 1289] 26 1289 87418 4358 48 3 162 0 postgres
[776425.503059] [ 1327] 0 1327 53971 252 103 3 388 0 nmbd
[776425.503060] [ 1440] 26 1440 16630 283 35 3 161 0 postgres
[776425.503063] [ 1445] 74 1445 5025 49 14 3 97 0 ntpd
[776425.503065] [ 1446] 26 1446 87545 69430 174 3 140 0 postgres
[776425.503067] [ 1447] 26 1447 87451 68414 173 3 151 0 postgres
[776425.503069] [ 1448] 26 1448 87451 2149 42 3 175 0 postgres
[776425.503072] [ 1449] 26 1449 87563 4382 171 3 160 0 postgres
[776425.503074] [ 1450] 26 1450 16678 178 35 3 160 0 postgres
[776425.503076] [ 1451] 74 1451 5024 51 13 3 131 0 ntpd
[776425.503078] [ 1453] 0 1453 69124 468 132 3 478 0 smbd
[776425.503081] [ 1454] 0 1454 68599 215 127 3 489 0 smbd-notifyd
[776425.503084] [ 1455] 0 1455 68599 75 126 3 477 0 cleanupd
[776425.503086] [ 1480] 0 1480 69255 254 129 3 475 0 lpqd
[776425.503088] [ 1554] 0 1554 4901 310 14 3 40 0 master
[776425.503090] [ 1557] 51 1557 5460 370 16 3 82 0 qmgr
[776425.503092] [ 1559] 0 1559 145053 10686 148 3 1258 0 salt-minion
[776425.503094] [ 1575] 0 1575 4727 481 16 3 63 0 cron
[776425.503097] [ 1579] 1001 1579 66119 28739 131 3 2317 0 openqa-schedule
[776425.503099] [ 1580] 1001 1580 90888 49774 180 3 2439 0 openqa
[776425.503101] [ 1585] 0 1585 64826 630 92 3 5214 0 salt-minion
[776425.503103] [ 1586] 488 1586 46440 1655 90 4 4200 0 salt-master
[776425.503105] [ 1598] 488 1598 87965 3007 132 4 6200 0 salt-master
[776425.503107] [ 1601] 488 1601 71120 6616 135 4 5285 0 salt-master
[776425.503109] [ 1605] 488 1605 72035 12883 137 4 906 0 salt-master
[776425.503111] [ 1648] 488 1648 252659 9652 149 4 1531 0 salt-master
[776425.503113] [ 1649] 488 1649 67883 3414 128 4 6103 0 salt-master
[776425.503115] [ 1650] 488 1650 162097 3480 140 4 6137 0 salt-master
[776425.503117] [ 1652] 488 1652 91445 11180 141 4 1695 0 salt-master
[776425.503119] [ 1660] 488 1660 91148 11275 140 4 1693 0 salt-master
[776425.503121] [ 1669] 488 1669 91212 11952 141 4 1179 0 salt-master
[776425.503124] [ 1670] 488 1670 91148 11753 140 4 1201 0 salt-master
[776425.503126] [ 1671] 488 1671 91446 11974 141 4 1197 0 salt-master
[776425.503127] [ 1989] 1001 1989 69125 30530 139 3 3963 0 openqa-websocke
[776425.503129] [ 2127] 26 2127 100188 81572 198 3 435 0 postgres
[776425.503131] [ 2141] 0 2141 28892 1347 59 3 61 0 httpd-prefork
[776425.503133] [ 2152] 0 2152 1187 351 8 3 25 0 agetty
[776425.503136] [ 2288] 26 2288 90470 72420 179 3 250 0 postgres
[776425.503138] [ 2291] 26 2291 90460 72109 179 3 213 0 postgres
[776425.503140] [ 598] 1001 598 9181 500 22 3 96 0 systemd
[776425.503142] [ 602] 1001 602 21155 118 43 3 456 0 (sd-pam)
[776425.503145] [20801] 0 20801 9128 498 24 3 95 0 systemd
[776425.503147] [20805] 0 20805 21123 121 43 3 469 0 (sd-pam)
[776425.503150] [22753] 1001 22753 75288 13794 150 3 22183 0 openqa
[776425.503153] [22764] 26 22764 88109 3727 79 3 296 0 postgres
[776425.503155] [24331] 30 24331 28960 673 58 3 61 0 httpd-prefork
[776425.503177] [24792] 0 24792 71332 484 135 3 289 0 smbd
[776425.503179] [ 3524] 30 3524 29011 667 59 3 61 0 httpd-prefork
[776425.503182] [28542] 30 28542 29005 536 59 3 61 0 httpd-prefork
[776425.503184] [10089] 30 10089 28962 602 58 3 61 0 httpd-prefork
[776425.503186] [ 3568] 30 3568 28960 585 58 3 61 0 httpd-prefork
[776425.503188] [22685] 30 22685 29024 748 59 3 61 0 httpd-prefork
[776425.503191] [22687] 30 22687 29000 517 59 3 61 0 httpd-prefork
[776425.503194] [23887] 30 23887 28992 505 59 3 61 0 httpd-prefork
[776425.503196] [28736] 30 28736 28936 613 58 3 61 0 httpd-prefork
[776425.503198] [31627] 30 31627 28964 924 58 3 61 0 httpd-prefork
[776425.503201] [ 1492] 30 1492 28990 559 58 3 61 0 httpd-prefork
[776425.503203] [ 4959] 30 4959 28938 479 58 3 61 0 httpd-prefork
[776425.503206] [ 4961] 30 4961 28932 615 58 3 61 0 httpd-prefork
[776425.503208] [ 6533] 30 6533 29006 683 59 3 61 0 httpd-prefork
[776425.503210] [ 7200] 30 7200 28938 714 58 3 61 0 httpd-prefork
[776425.503213] [ 8956] 30 8956 29008 540 59 3 61 0 httpd-prefork
[776425.503214] [ 9125] 30 9125 29004 925 59 3 61 0 httpd-prefork
[776425.503217] [ 9287] 30 9287 28964 531 58 3 61 0 httpd-prefork
[776425.503219] [ 9291] 30 9291 28980 688 58 3 61 0 httpd-prefork
[776425.503222] [10777] 30 10777 28932 615 58 3 61 0 httpd-prefork
[776425.503224] [12535] 30 12535 29010 572 59 3 61 0 httpd-prefork
[776425.503239] [15810] 30 15810 28958 613 58 3 61 0 httpd-prefork
[776425.503241] [15840] 30 15840 28962 891 58 3 61 0 httpd-prefork
[776425.503244] [17587] 30 17587 29048 571 60 3 61 0 httpd-prefork
[776425.503246] [18165] 30 18165 28958 613 58 3 61 0 httpd-prefork
[776425.503248] [18170] 30 18170 28938 613 58 3 61 0 httpd-prefork
[776425.503250] [18325] 30 18325 28966 861 58 3 61 0 httpd-prefork
[776425.503253] [19696] 30 19696 28936 613 58 3 61 0 httpd-prefork
[776425.503263] [19701] 30 19701 28936 613 58 3 61 0 httpd-prefork
[776425.503266] [19889] 30 19889 28972 683 58 3 61 0 httpd-prefork
[776425.503268] [20123] 30 20123 28998 537 59 3 61 0 httpd-prefork
[776425.503270] [21457] 30 21457 28960 613 58 3 61 0 httpd-prefork
[776425.503272] [26460] 30 26460 28932 615 58 3 61 0 httpd-prefork
[776425.503274] [26554] 30 26554 28958 613 58 3 61 0 httpd-prefork
[776425.503276] [27043] 30 27043 28962 935 58 3 61 0 httpd-prefork
[776425.503288] [27969] 0 27969 25601 929 54 4 0 0 sshd
[776425.503291] [27973] 17256 27973 9181 612 22 3 0 0 systemd
[776425.503293] [27976] 17256 27976 22796 2141 46 3 52 0 (sd-pam)
[776425.503296] [27979] 17256 27979 25601 372 51 4 0 0 sshd
[776425.503298] [27980] 17256 27980 4139 1369 12 3 0 0 bash
[776425.503300] [28072] 0 28072 23664 900 49 3 0 0 sudo
[776425.503302] [28073] 0 28073 4588 1419 12 3 0 0 bash
[776425.503304] [ 3251] 30 3251 28936 847 58 3 61 0 httpd-prefork
[776425.503306] [30502] 30 30502 28960 613 58 3 61 0 httpd-prefork
[776425.503309] [ 2654] 30 2654 28964 612 58 3 61 0 httpd-prefork
[776425.503311] [ 6639] 30 6639 29004 958 59 3 61 0 httpd-prefork
[776425.503312] [ 6645] 30 6645 29028 736 59 3 61 0 httpd-prefork
[776425.503316] [ 6658] 30 6658 28960 604 58 3 61 0 httpd-prefork
[776425.503318] [ 6673] 30 6673 28962 924 58 3 61 0 httpd-prefork
[776425.503320] [ 6684] 30 6684 29044 716 60 3 61 0 httpd-prefork
[776425.503322] [ 6686] 30 6686 29016 719 59 3 61 0 httpd-prefork
[776425.503325] [ 8080] 30 8080 28958 539 58 3 61 0 httpd-prefork
[776425.503327] [ 8134] 30 8134 29000 783 59 3 61 0 httpd-prefork
[776425.503329] [ 8139] 30 8139 28962 894 58 3 61 0 httpd-prefork
[776425.503331] [ 8142] 30 8142 29030 561 59 3 61 0 httpd-prefork
[776425.503333] [ 8145] 30 8145 29004 681 59 3 61 0 httpd-prefork
[776425.503335] [ 8184] 30 8184 29012 662 59 3 61 0 httpd-prefork
[776425.503337] [ 8197] 30 8197 28966 523 58 3 61 0 httpd-prefork
[776425.503339] [ 9528] 30 9528 29004 944 59 3 61 0 httpd-prefork
[776425.503341] [ 9918] 30 9918 28936 685 58 3 61 0 httpd-prefork
[776425.503343] [ 9919] 30 9919 28956 617 58 3 61 0 httpd-prefork
[776425.503345] [ 9920] 30 9920 28980 707 58 3 61 0 httpd-prefork
[776425.503347] [ 9928] 30 9928 29070 606 60 3 61 0 httpd-prefork
[776425.503349] [ 9930] 30 9930 28962 520 58 3 61 0 httpd-prefork
[776425.503351] [10813] 30 10813 28970 883 58 3 61 0 httpd-prefork
[776425.503354] [11071] 30 11071 28964 894 58 3 61 0 httpd-prefork
[776425.503356] [14438] 30 14438 29010 697 59 3 61 0 httpd-prefork
[776425.503358] [14644] 30 14644 29026 714 59 3 61 0 httpd-prefork
[776425.503360] [14943] 30 14943 29020 552 59 3 61 0 httpd-prefork
[776425.503372] [15085] 30 15085 29028 702 59 3 61 0 httpd-prefork
[776425.503374] [15086] 30 15086 29020 882 59 3 61 0 httpd-prefork
[776425.503377] [15088] 30 15088 28964 533 58 3 61 0 httpd-prefork
[776425.503379] [15089] 30 15089 29014 698 59 3 61 0 httpd-prefork
[776425.503380] [15491] 30 15491 29016 943 59 3 61 0 httpd-prefork
[776425.503382] [17178] 30 17178 28958 617 58 3 61 0 httpd-prefork
[776425.503384] [17179] 30 17179 28946 481 58 3 61 0 httpd-prefork
[776425.503386] [17200] 30 17200 28964 617 58 3 61 0 httpd-prefork
[776425.503388] [17203] 30 17203 28960 617 58 3 61 0 httpd-prefork
[776425.503403] [17669] 30 17669 28950 617 58 3 61 0 httpd-prefork
[776425.503405] [17670] 30 17670 28932 619 58 3 61 0 httpd-prefork
[776425.503408] [17672] 30 17672 29002 516 59 3 61 0 httpd-prefork
[776425.503410] [17674] 30 17674 28970 853 58 3 61 0 httpd-prefork
[776425.503412] [18120] 30 18120 28966 841 58 3 61 0 httpd-prefork
[776425.503415] [19327] 30 19327 28960 530 58 3 61 0 httpd-prefork
[776425.503417] [19470] 30 19470 28960 576 58 3 61 0 httpd-prefork
[776425.503419] [19693] 30 19693 28936 617 58 3 61 0 httpd-prefork
[776425.503429] [19694] 30 19694 28936 617 58 3 61 0 httpd-prefork
[776425.503431] [19695] 30 19695 28936 757 58 3 61 0 httpd-prefork
[776425.503433] [20109] 30 20109 28964 617 58 3 61 0 httpd-prefork
[776425.503436] [20114] 30 20114 28952 617 58 3 61 0 httpd-prefork
[776425.503438] [20115] 30 20115 28962 923 58 3 61 0 httpd-prefork
[776425.503439] [20118] 30 20118 29000 529 59 3 61 0 httpd-prefork
[776425.503441] [20744] 30 20744 28962 554 58 3 61 0 httpd-prefork
[776425.503444] [20746] 30 20746 28960 862 58 3 61 0 httpd-prefork
[776425.503455] [20787] 30 20787 28962 535 58 3 61 0 httpd-prefork
[776425.503457] [21555] 30 21555 29016 846 59 3 61 0 httpd-prefork
[776425.503459] [21557] 30 21557 29002 526 59 3 61 0 httpd-prefork
[776425.503461] [21564] 30 21564 28948 619 58 3 61 0 httpd-prefork
[776425.503462] [21581] 30 21581 28960 617 58 3 61 0 httpd-prefork
[776425.503464] [22258] 30 22258 28960 617 58 3 61 0 httpd-prefork
[776425.503465] [22260] 30 22260 28960 617 58 3 61 0 httpd-prefork
[776425.503466] [22432] 30 22432 28958 617 58 3 61 0 httpd-prefork
[776425.503468] [22450] 30 22450 28960 617 58 3 61 0 httpd-prefork
[776425.503470] [22478] 30 22478 28960 617 58 3 61 0 httpd-prefork
[776425.503471] [27805] 30 27805 28960 617 58 3 61 0 httpd-prefork
[776425.503472] [28125] 30 28125 28932 480 58 3 61 0 httpd-prefork
[776425.503474] [31087] 30 31087 28954 714 58 3 61 0 httpd-prefork
[776425.503475] [31134] 30 31134 28998 625 59 3 61 0 httpd-prefork
[776425.503476] [31486] 30 31486 28998 951 59 3 61 0 httpd-prefork
[776425.503478] [31543] 30 31543 28954 617 58 3 61 0 httpd-prefork
[776425.503479] [31630] 30 31630 28932 480 58 3 61 0 httpd-prefork
[776425.503481] [23627] 30 23627 29000 759 59 3 61 0 httpd-prefork
[776425.503482] [24121] 30 24121 29014 714 59 3 61 0 httpd-prefork
[776425.503484] [26259] 30 26259 29010 693 59 3 61 0 httpd-prefork
[776425.503485] [26816] 30 26816 28954 838 58 3 61 0 httpd-prefork
[776425.503487] [26841] 30 26841 28998 572 59 3 61 0 httpd-prefork
[776425.503489] [28319] 30 28319 28958 617 58 3 61 0 httpd-prefork
[776425.503490] [30885] 30 30885 28956 604 58 3 61 0 httpd-prefork
[776425.503491] [31301] 30 31301 28956 701 58 3 61 0 httpd-prefork
[776425.503493] [ 3903] 30 3903 28936 617 58 3 61 0 httpd-prefork
[776425.503494] [ 5340] 30 5340 28984 829 59 3 61 0 httpd-prefork
[776425.503496] [ 9257] 30 9257 28958 573 58 3 61 0 httpd-prefork
[776425.503498] [31592] 30 31592 29010 705 59 3 61 0 httpd-prefork
[776425.503499] [16943] 30 16943 29016 598 59 3 61 0 httpd-prefork
[776425.504585] [17403] 1001 17403 101878 46852 202 3 15045 0 openqa
[776425.504588] [17410] 26 17410 90762 71513 180 3 130 0 postgres
[776425.504590] [17934] 1001 17934 102603 47347 203 3 15148 0 openqa
[776425.504593] [17935] 26 17935 89524 70874 177 3 130 0 postgres
[776425.504595] [17968] 1001 17968 89300 33887 177 3 15186 0 openqa
[776425.504597] [19315] 1001 19315 90372 35323 179 3 15135 0 openqa
[776425.504600] [19345] 1001 19345 100061 44496 198 3 15125 0 openqa
[776425.504612] [19365] 26 19365 89144 62533 176 3 130 0 postgres
[776425.504624] [19367] 26 19367 89258 62833 177 3 130 0 postgres
[776425.504635] [20170] 26 20170 89379 70502 177 3 130 0 postgres
[776425.504637] [21754] 51 21754 5365 410 15 3 0 0 pickup
[776425.504641] [24929] 1001 24929 97722 42600 194 3 15363 0 openqa
[776425.504654] [24932] 26 24932 88850 66901 176 3 130 0 postgres
[776425.504657] [24962] 30 24962 28996 928 59 3 61 0 httpd-prefork
[776425.504659] [24976] 30 24976 28994 558 59 3 61 0 httpd-prefork
[776425.504662] [24986] 30 24986 28994 594 59 3 61 0 httpd-prefork
[776425.504664] [24994] 30 24994 29010 961 59 3 61 0 httpd-prefork
[776425.504666] [25089] 30 25089 29010 958 59 3 61 0 httpd-prefork
[776425.504669] [27283] 1001 27283 1838027 1780317 3589 9 15375 0 openqa
[776425.504672] [28304] 26 28304 94361 72979 189 3 130 0 postgres
[776425.504697] [ 3672] 0 3672 25602 937 54 3 0 0 sshd
[776425.504699] [ 3674] 17258 3674 9181 595 23 3 0 0 systemd
[776425.504701] [ 3677] 17258 3677 23005 2380 46 3 52 0 (sd-pam)
[776425.504703] [ 3681] 17258 3681 26442 1130 57 3 0 0 sshd
[776425.504705] [ 3682] 17258 3682 1118 182 7 3 0 0 scp
[776425.504707] [ 3706] 1001 3706 76742 21750 153 3 15574 0 openqa
[776425.504711] [ 3781] 1001 3781 76842 21759 153 3 15557 0 openqa
[776425.504713] [ 3789] 26 3789 88399 4505 82 3 131 0 postgres
[776425.504715] [ 3929] 26 3929 88379 5067 90 3 131 0 postgres
[776425.504718] [ 3930] 30 3930 28936 874 58 3 61 0 httpd-prefork
[776425.504720] [ 3931] 30 3931 28936 874 58 3 61 0 httpd-prefork
[776425.504722] [ 3932] 30 3932 28936 874 58 3 61 0 httpd-prefork
[776425.504725] [ 3933] 30 3933 28982 861 59 3 61 0 httpd-prefork
[776425.504726] [ 3934] 30 3934 28936 874 58 3 61 0 httpd-prefork
[776425.504729] [ 3935] 30 3935 28936 874 58 3 61 0 httpd-prefork
[776425.504731] [ 3936] 30 3936 28936 874 58 3 61 0 httpd-prefork
[776425.504733] [ 3937] 30 3937 28936 874 58 3 61 0 httpd-prefork
[776425.504735] [ 3938] 30 3938 28938 874 58 3 61 0 httpd-prefork
[776425.504737] [ 3941] 30 3941 28936 874 58 3 61 0 httpd-prefork
[776425.504739] [ 3943] 30 3943 28938 874 58 3 61 0 httpd-prefork
[776425.504741] [ 3944] 30 3944 28936 874 58 3 61 0 httpd-prefork
[776425.504743] [ 3945] 30 3945 28936 874 58 3 61 0 httpd-prefork
[776425.504746] [ 3946] 30 3946 28936 874 58 3 61 0 httpd-prefork
[776425.504748] [ 3947] 30 3947 28936 874 58 3 61 0 httpd-prefork
[776425.504750] [ 3948] 30 3948 28936 874 58 3 61 0 httpd-prefork
[776425.504753] [ 3949] 30 3949 28936 874 58 3 61 0 httpd-prefork
[776425.504755] [ 3950] 30 3950 28936 874 58 3 61 0 httpd-prefork
[776425.504757] [ 3952] 30 3952 28936 874 58 3 61 0 httpd-prefork
[776425.504759] [ 3955] 30 3955 28936 874 58 3 61 0 httpd-prefork
[776425.504762] [ 3956] 30 3956 28936 874 58 3 61 0 httpd-prefork
[776425.504764] [ 3957] 30 3957 28936 874 58 3 61 0 httpd-prefork
[776425.504767] [ 3959] 30 3959 28936 874 58 3 61 0 httpd-prefork
[776425.504769] [ 3960] 30 3960 28936 874 58 3 61 0 httpd-prefork
[776425.504771] [ 3961] 30 3961 28936 874 58 3 61 0 httpd-prefork
[776425.504773] [ 3962] 30 3962 28936 874 58 3 61 0 httpd-prefork
[776425.504776] [ 3963] 30 3963 28936 874 58 3 61 0 httpd-prefork
[776425.504778] [ 3964] 30 3964 28936 874 58 3 61 0 httpd-prefork
[776425.504779] [ 3965] 30 3965 28936 874 58 3 61 0 httpd-prefork
[776425.504782] [ 3966] 30 3966 28936 874 58 3 61 0 httpd-prefork
[776425.504784] [ 3967] 30 3967 28936 872 58 3 61 0 httpd-prefork
[776425.504786] [ 3968] 30 3968 28936 870 58 3 61 0 httpd-prefork
[776425.504788] [ 3969] 30 3969 28936 874 58 3 61 0 httpd-prefork
[776425.504790] [ 3970] 30 3970 28936 874 58 3 61 0 httpd-prefork
[776425.504792] [ 3971] 30 3971 28936 874 58 3 61 0 httpd-prefork
[776425.504795] [ 3972] 30 3972 28936 874 58 3 61 0 httpd-prefork
[776425.504797] [ 3974] 30 3974 28936 874 58 3 61 0 httpd-prefork
[776425.504799] [ 3975] 30 3975 28936 874 58 3 61 0 httpd-prefork
[776425.504801] [ 3976] 30 3976 28936 874 58 3 61 0 httpd-prefork
[776425.504804] [ 3977] 30 3977 28936 874 58 3 61 0 httpd-prefork
[776425.504806] [ 3978] 30 3978 28936 874 58 3 61 0 httpd-prefork
[776425.504808] [ 3979] 30 3979 28936 874 58 3 61 0 httpd-prefork
[776425.504810] [ 3980] 30 3980 28936 874 58 3 61 0 httpd-prefork
[776425.504811] [ 3981] 30 3981 28936 874 58 3 61 0 httpd-prefork
[776425.504814] [ 3984] 30 3984 28936 874 58 3 61 0 httpd-prefork
[776425.504816] [ 3990] 30 3990 28936 874 58 3 61 0 httpd-prefork
[776425.504827] [ 3991] 30 3991 28936 874 58 3 61 0 httpd-prefork
[776425.504830] [ 3992] 1001 3992 76076 19806 149 3 16825 0 openqa
[776425.504832] Out of memory: Kill process 27283 (openqa) score 692 or sacrifice child
[776425.504839] Killed process 27283 (openqa) total-vm:7352108kB, anon-rss:7119236kB, file-rss:2016kB, shmem-rss:16kB
[901824.712472] hrtimer: interrupt took 463289 ns

But you realize that you justed pasted the Out of memory message? :)

but these are very old - and we crash way more often

The coredump is pretty pointless :(

(gdb) where
#0 0x00000000004ac934 in S_visit (my_perl=my_perl@entry=0x2263010, f=f@entry=0x4b5480 , flags=flags@entry=2048, mask=mask@entry=2048) at sv.c:417
#1 0x00000000004b5854 in Perl_sv_clean_objs (my_perl=my_perl@entry=0x2263010) at sv.c:577
#2 0x000000000043976e in perl_destruct (my_perl=0x2263010) at perl.c:766
#3 0x000000000041e8eb in main (argc=18, argv=0x7ffce2bf2768, env=0x7ffce2bf2800) at perlmain.c:125

I guess we'll need to enable profiling to get our bearings. There'll be a performance impact but simple installation and adding -d:NYTProf to the webui command line can provide some profiling data. See http://search.cpan.org/~timb/Devel-NYTProf-6.04/lib/Devel/NYTProf.pm#SYNOPSIS

It should be noted that we have not a single 500 today - so this is purely prefork workers shuting down