Actions
action #116722
closedopenqa.suse.de is not reachable 2022-09-18, no ping response, postgreSQL OOM and kernel panics size:M
Added by okurz over 2 years ago. Updated over 1 year ago.
Status:
Resolved
Priority:
Normal
Assignee:
Category:
-
Target version:
Start date:
2022-09-18
Due date:
% Done:
0%
Estimated time:
Tags:
Description
Observation¶
Multiple alerts on https://monitor.qa.suse.de/alerting/list?state=not_ok , all seem to point to openqa.suse.de itself being down. The first alert sending an email seems to be "Failed systemd services" https://monitor.qa.suse.de/d/KToPYLEWz/failed-systemd-services?orgId=1&from=1663476407099&to=1663478162658&viewPanel=2 from 2022-09-18 06:50 and following.
There seems to be a sudden increase in CPU load visible in
https://monitor.qa.suse.de/d/WebuiDb/webui-summary?orgId=1&from=1663473600000&to=1663480800000 going to 100 with spotty response from telegraf and no data soon afterwards. Also a significant increase in disk I/O requests in https://monitor.qa.suse.de/d/WebuiDb/webui-summary?orgId=1&from=1663473600000&to=1663480800000&viewPanel=119
Impact¶
openqa.suse.de is not reachable even over ping and not reachable for workers or users so completely unusable.
Suggestions¶
- DONE Increase memory of the VM
- DONE Find the root cause -> soft lockups in kernel since the latest kernel upgrade
- DONE Report product issue
- DONE Apply workaround, e.g. downgraded kernel version
- DONE Apply kernel.panic = 60
- Wait for product issue to be resolved and upgrade kernel again
- Optional: Review the changelog diff of the kernel
Files
| journal_osd_crash_2022-09-18.log.xz (300 KB) journal_osd_crash_2022-09-18.log.xz | system journal of crash period, multiple OOM situations | okurz, 2022-09-19 07:05 |
Updated by okurz over 2 years ago
- Status changed from New to In Progress
- Assignee set to okurz
Reported https://sd.suse.com/servicedesk/customer/portal/1/SD-98873 with request to Engineering Infrastructure to help debugging the VM.
EDIT: Also reported in https://suse.slack.com/archives/C029APBKLGK/p1663534410250609 and informed main users in https://suse.slack.com/archives/C02CANHLANP/p1663534467018299
Updated by openqa_review over 2 years ago
- Due date set to 2022-10-03
Setting due date based on mean cycle time of SUSE QE Tools
Updated by okurz over 2 years ago
- Priority changed from Immediate to Urgent
SD ticket was resolved, gschlotter could reboot the machine:
machine is back online.
it had a kernel panic.
We should check if we can find more data about the kernel panic and investigate further.
I checked all alerts and found multiple other hosts down or currently not reachable, created specific tickets for each.
Triggered:
for i in incomplete timeout_exceeded ; do host=openqa.suse.de failed_since=2022-09-18 result="result='$i'" openqa-advanced-retrigger-jobs ; done
Updated by okurz over 2 years ago
- Related to action #116740: [alert] openqaworker14: host up alert added
Updated by okurz over 2 years ago
- Related to action #116743: [alert] QA-Power8-5-kvm: host up alert added
Updated by okurz over 2 years ago
- Related to action #116746: [alert] openqaworker9: host up alert added
Updated by okurz over 2 years ago
- Related to action #116752: [alert] powerqaworker-qam-1: host up alert added
Updated by okurz over 2 years ago
Updated by okurz over 2 years ago
I looked into the system journal and found multiple OOM reports which might be the cause or just symptoms of the crash. Needs further investigation and crosschecking with data in grafana.
EDIT: Looked into the system journal in more detail
- What I find noteworthy observations is that "logrotate" prints a huge amount of messages like "Sep 18 07:00:00 openqa logrotate[18788]: Creating new state".
- Next problematic entry "Sep 18 07:00:06 openqa kernel: watchdog: BUG: soft lockup - CPU#2 stuck for 1050s! [kworker/2:2:24024]", then shortly afterwards
Sep 18 07:00:12 openqa openqa-scheduler-daemon[1487]: WARNING: terminating connection because of crash of another server process
Sep 18 07:00:12 openqa openqa-scheduler-daemon[1487]: DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
Sep 18 07:00:12 openqa openqa-scheduler-daemon[1487]: HINT: In a moment you should be able to reconnect to the database and repeat your command.
Sep 18 07:00:12 openqa openqa-scheduler-daemon[1487]: (in cleanup) DBIx::Class::Storage::DBI::_populate_dbh(): DBI Exception: DBD::Pg::st DESTROY failed: no connection to the server [for Statement "SELECT me.child_job_id, me.parent_job_id, me.dependency FROM job_dependencies me WHERE ( ( me.child_job_id = ? AND me.dependency = ? AND me.parent_job_id = ? ) )"] at /usr/share/openqa/script/../lib/OpenQA/Scheduler/Model/Jobs.pm line 29
did the database process terminate or did the database die? Next CPU stuck message at "Sep 18 07:00:46 openqa kernel: watchdog: BUG: soft lockup - CPU#2 stuck for 1088s! [kworker/2:2:24024]" and more later on
- "Sep 18 07:01:38 openqa kernel: postgres invoked oom-killer: gfp_mask=0x400dc0(GFP_KERNEL_ACCOUNT|__GFP_ZERO), order=0, oom_score_adj=0" so postgres exceeded memory usage? After that message multiple openQA processes understandably complain about the postgres server process being gone
Updated by mkittler over 2 years ago
I'm currently cleaning up the Minion dashboard. It contains many failures due to the issue, e.g. due to database issues like that:
DBI connect('dbname=openqa','geekotest',...) failed: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: FATAL: the database system is in recovery mode at /usr/lib/perl5/vendor_perl/5.26.1/Mojo/Pg.pm line 73.
Or because the Minion workers were crashing (or were killed) completely. Or because the needles dir was left in an unclean state (which we've already fixed).
Updated by okurz over 2 years ago
OSD is down again. Unfortunately this needs manual intervention by Engineering Infrastructure as they won't give us access and they don't monitor the machine on their own. Created https://sd.suse.com/servicedesk/customer/portal/1/SD-98988 . After the machine is up again we should ensure that the machine is configured to automatically reboot on kernel panics and also downgrade any package upgrades from last sunday and look into the memory related postgres configs, e.g. in https://gitlab.suse.de/openqa/salt-states-openqa/-/blob/master/openqa/server.sls#L331
Updated by okurz over 2 years ago
Up again, done by gschlotter in https://suse.slack.com/archives/C029APBKLGK/p1663651357301979 . Same problem source, out of memory:
Sep 20 01:16:02 openqa kernel: Out of memory: Killed process 1563 (postgres) total-vm:3349768kB, anon-rss:1420kB, file-rss:0kB, shmem-rss:2698412kB, UID:26 pgtables:5800kB oom_score_adj:0
Could that be the queries we conduct only once every 24h causing this problem?
EDIT: gschlotter also gave the VM +4GB RAM. I have enabled automatic reboots on panic with https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/733 . I wonder why we never considered that in the past or why that setting is not even default for servers on openSUSE. I have not yet downgraded any packages or changed postgres config. We should check if the problem really reproduces. Also we should configure kdump and enable notification if the problem happens again unless our monitoring would already alert us.
Also I am suggesting to reduce work_mem for postgres by factor 4, see https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/734
Updated by okurz over 2 years ago
- Related to coordination #112718: [alert][osd] openqa.suse.de is not reachable anymore, response times > 30s, multiple alerts over the weekend added
Updated by okurz over 2 years ago
- Related to action #115208: failed-systemd-services: logrotate-openqa alerting on and off size:M added
Updated by okurz over 2 years ago
- Related to action #116848: Ensure kdump is enabled and working on all OSD machines added
Updated by okurz over 2 years ago
- Subject changed from openqa.suse.de is not reachable, no ping response to openqa.suse.de is not reachable 2022-09-18, no ping response, postgreSQL OOM and kernel panics
Updated by okurz over 2 years ago
- Related to action #116911: [openQA][needle] Can not commit new needle for test suite on openqa.suse.de added
Updated by mkittler over 2 years ago
The OOM killer was invoked several times. We can try to make sense of logs to find the problematic processes:
Sep 21 00:02:25 openqa systemd-journald[25795]: Missed 142 kernel messages
Sep 21 00:02:25 openqa kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=openqa,pid=25595,uid=1001
Sep 21 00:02:25 openqa kernel: Out of memory: Killed process 25595 (openqa) total-vm:359592kB, anon-rss:153524kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:716kB oom_score_adj:0
Sep 21 00:02:25 openqa kernel: oom_reaper: reaped process 25595 (openqa), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Sep 21 00:02:26 openqa kernel: kthreadd invoked oom-killer: gfp_mask=0x40cc0(GFP_KERNEL|__GFP_COMP), order=2, oom_score_adj=0
Sep 21 00:02:26 openqa kernel: CPU: 6 PID: 2 Comm: kthreadd Tainted: G W L 5.14.21-150400.24.21-default #1 SLE15-SP4 7550826c4c7e8c258239e300508e0c8b2a69bad2
Sep 21 00:02:26 openqa kernel: Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
Sep 21 00:02:26 openqa kernel: Call Trace:
Sep 21 00:02:26 openqa kernel: <TASK>
Sep 21 00:02:26 openqa kernel: dump_stack_lvl+0x45/0x5b
Sep 21 00:02:26 openqa kernel: dump_header+0x4a/0x220
Sep 21 00:02:26 openqa kernel: oom_kill_process+0xe8/0x140
Sep 21 00:02:26 openqa kernel: out_of_memory+0x113/0x580
Sep 21 00:02:26 openqa kernel: __alloc_pages_slowpath.constprop.112+0x980/0xc70
Sep 21 00:02:26 openqa kernel: __alloc_pages+0x2d9/0x320
Sep 21 00:02:26 openqa kernel: new_slab+0x4c9/0x500
Sep 21 00:02:26 openqa kernel: ___slab_alloc+0x6ad/0x860
Sep 21 00:02:26 openqa kernel: ? copy_process+0x213/0x1cd0
Sep 21 00:02:26 openqa kernel: ? __slab_alloc.isra.92+0x1c/0x40
Sep 21 00:02:26 openqa kernel: __slab_alloc.isra.92+0x1c/0x40
Sep 21 00:02:26 openqa kernel: kmem_cache_alloc_node+0x114/0x310
Sep 21 00:02:26 openqa kernel: ? copy_process+0x213/0x1cd0
Sep 21 00:02:26 openqa kernel: copy_process+0x213/0x1cd0
Sep 21 00:02:26 openqa kernel: ? update_load_avg+0x1cc/0x620
Sep 21 00:02:26 openqa kernel: kernel_clone+0x9b/0x400
Sep 21 00:02:26 openqa kernel: kernel_thread+0x55/0x80
Sep 21 00:02:26 openqa kernel: ? set_kthread_struct+0x50/0x50
Sep 21 00:02:26 openqa kernel: kthreadd+0x2b4/0x300
Sep 21 00:02:26 openqa kernel: ? kthread_is_per_cpu+0x30/0x30
Sep 21 00:02:26 openqa kernel: ret_from_fork+0x22/0x30
Sep 21 00:02:26 openqa kernel: </TASK>
Sep 21 00:02:26 openqa kernel: Mem-Info:
Sep 21 00:02:26 openqa kernel: active_anon:370017 inactive_anon:123196 isolated_anon:109
active_file:2863 inactive_file:4608 isolated_file:687
unevictable:664 dirty:0 writeback:0
slab_reclaimable:4416751 slab_unreclaimable:1028155
mapped:10899 shmem:93386 pagetables:70799 bounce:0
free:93830 free_pcp:4003 free_cma:0
Sep 21 00:02:26 openqa kernel: Node 0 active_anon:1481108kB inactive_anon:492480kB active_file:8700kB inactive_file:21860kB unevictable:2656kB isolated(anon):436kB isolated(file):2760kB mapped:43304kB dirty:0kB writeback:0kB shmem:373544kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB kernel_>
Sep 21 00:02:26 openqa kernel: Node 0 DMA free:11248kB boost:0kB min:40kB low:52kB high:64kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Sep 21 00:02:26 openqa kernel: lowmem_reserve[]: 0 2991 24013 24013 24013
Sep 21 00:02:26 openqa kernel: Node 0 DMA32 free:129024kB boost:21640kB min:29996kB low:33032kB high:36068kB reserved_highatomic:18432KB active_anon:122628kB inactive_anon:27132kB active_file:1032kB inactive_file:1236kB unevictable:0kB writepending:0kB present:3129312kB managed:3063776kB mlocked:0kB bounce:0kB fre>
Sep 21 00:02:26 openqa kernel: lowmem_reserve[]: 0 0 21021 21021 21021
Sep 21 00:02:26 openqa kernel: Node 0 Normal free:235304kB boost:153324kB min:212508kB low:234024kB high:255540kB reserved_highatomic:22528KB active_anon:1360828kB inactive_anon:467376kB active_file:11608kB inactive_file:14584kB unevictable:2656kB writepending:0kB present:22020096kB managed:21525736kB mlocked:80kB>
Sep 21 00:02:26 openqa kernel: lowmem_reserve[]: 0 0 0 0 0
Sep 21 00:02:26 openqa kernel: Node 0 DMA: 0*4kB 0*8kB 1*16kB (U) 1*32kB (U) 1*64kB (U) 1*128kB (U) 1*256kB (U) 1*512kB (U) 0*1024kB 1*2048kB (M) 2*4096kB (M) = 11248kB
Sep 21 00:02:26 openqa kernel: Node 0 DMA32: 6447*4kB (UME) 12993*8kB (UME) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 129732kB
Sep 21 00:02:26 openqa kernel: Node 0 Normal: 13746*4kB (UME) 22568*8kB (UME) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 235528kB
Sep 21 00:02:26 openqa kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Sep 21 00:02:26 openqa kernel: 101977 total pagecache pages
Sep 21 00:02:26 openqa kernel: 442 pages in swap cache
Sep 21 00:02:26 openqa kernel: Swap cache stats: add 1390281, delete 1389759, find 96349271/96842395
Sep 21 00:02:26 openqa kernel: Free swap = 0kB
Sep 21 00:02:26 openqa kernel: Total swap = 499708kB
Sep 21 00:02:26 openqa kernel: 6291350 pages RAM
Sep 21 00:02:26 openqa kernel: 0 pages HighMem/MovableOnly
Sep 21 00:02:26 openqa kernel: 140132 pages reserved
Sep 21 00:02:26 openqa kernel: 0 pages cma reserved
Sep 21 00:02:26 openqa kernel: 0 pages hwpoisoned
…
Sep 21 00:02:26 openqa kernel: Tasks state (memory values in pages):
Sep 21 00:02:26 openqa kernel: [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
Sep 21 00:02:26 openqa kernel: [ 236] 0 236 3140 430 69632 1474 0 haveged
Sep 21 00:02:26 openqa kernel: [ 523] 0 523 27415 100 237568 549 -1000 systemd-udevd
Sep 21 00:02:26 openqa kernel: [ 676] 0 676 13017 0 135168 163 0 rpc.idmapd
Sep 21 00:02:26 openqa kernel: [ 677] 0 677 15090 28 147456 156 -1000 auditd
Sep 21 00:02:26 openqa kernel: [ 682] 499 682 19833 208 184320 190 -900 dbus-daemon
Sep 21 00:02:26 openqa kernel: [ 693] 0 693 24068 244 229376 223 0 systemd-logind
Sep 21 00:02:26 openqa kernel: [ 696] 0 696 10708 4 122880 308 0 wickedd-auto4
Sep 21 00:02:26 openqa kernel: [ 701] 0 701 10709 4 131072 310 0 wickedd-dhcp4
Sep 21 00:02:26 openqa kernel: [ 705] 0 705 10709 46 126976 269 0 wickedd-dhcp6
Sep 21 00:02:26 openqa kernel: [ 740] 0 740 10781 109 135168 270 0 wickedd
Sep 21 00:02:26 openqa kernel: [ 767] 0 767 10744 53 126976 276 0 wickedd-nanny
Sep 21 00:02:26 openqa kernel: [ 788] 0 788 18806 90 184320 337 0 systemd
Sep 21 00:02:26 openqa kernel: [ 789] 0 789 30693 104 270336 829 0 (sd-pam)
Sep 21 00:02:26 openqa kernel: [ 1307] 0 1307 28974 33 262144 290 0 rebootmgrd
Sep 21 00:02:26 openqa kernel: [ 1309] 0 1309 7763 8 102400 125 0 rsyncd
Sep 21 00:02:26 openqa kernel: [ 1311] 488 1311 74604 8038 626688 3148 0 salt-master
Sep 21 00:02:26 openqa kernel: [ 1312] 0 1312 49939 124 434176 5000 0 salt-minion
Sep 21 00:02:26 openqa kernel: [ 1319] 0 1319 6390 15 94208 109 0 vsftpd
Sep 21 00:02:26 openqa kernel: [ 1321] 0 1321 13501 2 139264 169 0 xinetd
Sep 21 00:02:26 openqa kernel: [ 1326] 493 1326 15982 44 176128 180 0 rpcbind
Sep 21 00:02:26 openqa kernel: [ 1327] 1001 1327 18840 145 188416 339 0 systemd
Sep 21 00:02:26 openqa kernel: [ 1329] 1001 1329 30736 105 270336 858 0 (sd-pam)
Sep 21 00:02:26 openqa kernel: [ 1334] 0 1334 18890 41 196608 221 -1000 sshd
Sep 21 00:02:26 openqa kernel: [ 1335] 492 1335 25274 505 249856 12024 0 rpc.statd
Sep 21 00:02:26 openqa kernel: [ 1337] 475 1337 39077 40 208896 256 0 chronyd
Sep 21 00:02:26 openqa kernel: [ 1345] 0 1345 18541 400 172032 6318 0 rpc.mountd
Sep 21 00:02:26 openqa kernel: [ 1360] 0 1360 82552 70 671744 690 0 smbd
Sep 21 00:02:26 openqa kernel: [ 1380] 497 1380 12618 12 143360 205 -500 nrpe
Sep 21 00:02:26 openqa kernel: [ 1411] 0 1411 81044 29 626688 698 0 smbd-notifyd
Sep 21 00:02:26 openqa kernel: [ 1412] 0 1412 81046 29 622592 698 0 cleanupd
Sep 21 00:02:26 openqa kernel: [ 1420] 0 1420 82373 50 659456 673 0 samba-bgqd
Sep 21 00:02:26 openqa kernel: [ 1428] 0 1428 36828 269 319488 406 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 1465] 30 1465 36952 322 319488 390 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 1467] 30 1467 36974 336 319488 387 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 1478] 30 1478 36954 262 315392 436 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 1479] 30 1479 36928 271 315392 414 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 1482] 30 1482 36950 312 319488 392 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 1485] 30 1485 36928 281 315392 407 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 1487] 30 1487 36928 253 307200 416 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 1488] 30 1488 36973 253 319488 469 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 1489] 30 1489 36948 206 319488 499 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 1490] 30 1490 36952 265 315392 433 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 1491] 30 1491 36963 244 319488 470 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 1496] 30 1496 36895 184 307200 474 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 1497] 30 1497 36952 235 319488 469 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 1502] 0 1502 11481 24 126976 143 0 agetty
Sep 21 00:02:26 openqa kernel: [ 1504] 30 1504 36928 294 315392 398 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 1507] 0 1507 11481 25 131072 139 0 agetty
Sep 21 00:02:26 openqa kernel: [ 1508] 30 1508 36976 336 319488 385 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 1514] 30 1514 36895 185 307200 473 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 1515] 30 1515 36895 194 307200 464 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 1517] 30 1517 36954 308 319488 405 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 1520] 30 1520 36951 324 319488 385 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 1525] 30 1525 36895 242 307200 432 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 1526] 30 1526 36953 264 315392 433 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 1529] 30 1529 36954 275 315392 437 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 1531] 30 1531 38546 1258 331776 1033 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 1535] 30 1535 36895 195 307200 463 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 1538] 30 1538 36928 202 311296 474 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 1541] 30 1541 36928 196 311296 478 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 1543] 30 1543 36895 186 307200 472 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 1546] 30 1546 36955 280 319488 429 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 1547] 30 1547 36961 316 315392 390 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 1549] 30 1549 36988 286 319488 449 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 1550] 30 1550 36962 251 319488 482 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 1553] 30 1553 36929 250 315392 423 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 1554] 30 1554 36958 317 319488 406 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 1555] 30 1555 36950 308 319488 403 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 1556] 30 1556 36947 282 315392 412 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 1557] 30 1557 36952 271 319488 438 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 1562] 30 1562 36954 290 315392 406 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 1563] 30 1563 36961 320 319488 391 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 1565] 30 1565 36895 237 307200 436 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 1568] 30 1568 36895 189 307200 468 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 1601] 0 1601 303126 7977 724992 4763 0 salt-minion
Sep 21 00:02:26 openqa kernel: [ 1617] 0 1617 24392 22 176128 215 0 master
Sep 21 00:02:26 openqa kernel: [ 1619] 51 1619 29041 64 192512 203 0 qmgr
Sep 21 00:02:26 openqa kernel: [ 1632] 0 1632 12792 28 147456 179 0 cron
Sep 21 00:02:26 openqa kernel: [ 1644] 488 1644 52386 743 425984 5067 0 salt-master
Sep 21 00:02:26 openqa kernel: [ 1645] 0 1645 70820 124 442368 5695 0 salt-minion
Sep 21 00:02:26 openqa kernel: [ 1674] 488 1674 93817 2285 606208 7603 0 salt-master
Sep 21 00:02:26 openqa kernel: [ 1677] 488 1677 79330 4994 626688 8218 0 salt-master
Sep 21 00:02:26 openqa kernel: [ 1682] 488 1682 267723 10237 712704 4900 0 salt-master
Sep 21 00:02:26 openqa kernel: [ 1684] 488 1684 73223 2351 577536 7475 0 salt-master
Sep 21 00:02:26 openqa kernel: [ 1686] 488 1686 167412 2924 626688 8289 0 salt-master
Sep 21 00:02:26 openqa kernel: [ 1688] 488 1688 267923 8753 696320 5344 0 salt-master
Sep 21 00:02:26 openqa kernel: [ 1695] 488 1695 268733 10653 724992 4598 0 salt-master
Sep 21 00:02:26 openqa kernel: [ 1697] 488 1697 268504 10665 716800 4505 0 salt-master
Sep 21 00:02:26 openqa kernel: [ 1698] 488 1698 268621 8968 708608 5230 0 salt-master
Sep 21 00:02:26 openqa kernel: [ 1699] 488 1699 268578 10007 712704 4529 0 salt-master
Sep 21 00:02:26 openqa kernel: [ 2192] 30 2192 36952 281 315392 411 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 2333] 30 2333 36949 296 319488 403 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 2438] 30 2438 36947 260 315392 431 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 2452] 30 2452 36953 319 319488 400 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 2458] 30 2458 36959 253 319488 451 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 2535] 30 2535 36929 260 315392 405 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 2537] 30 2537 36976 323 319488 399 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 2539] 30 2539 36964 355 319488 365 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 2540] 30 2540 36895 260 307200 385 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 2542] 30 2542 36895 269 307200 376 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 2543] 30 2543 36895 242 307200 403 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 2544] 30 2544 36895 223 307200 420 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 2546] 30 2546 36929 271 315392 413 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 2547] 30 2547 36929 292 315392 386 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 2816] 30 2816 36959 286 319488 433 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 2828] 30 2828 36978 274 319488 448 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 2831] 30 2831 36951 236 319488 470 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 2834] 30 2834 36929 302 315392 389 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 2835] 30 2835 36965 325 319488 380 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 2838] 30 2838 36949 293 315392 400 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 2848] 30 2848 36953 323 315392 373 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 2902] 30 2902 36949 326 319488 364 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 2904] 30 2904 36929 241 315392 445 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 2906] 30 2906 36952 310 319488 406 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 2909] 30 2909 36949 310 319488 382 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 2910] 30 2910 36960 277 319488 431 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 2911] 30 2911 36949 299 315392 393 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 2914] 30 2914 36980 268 319488 452 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 2915] 30 2915 36972 265 319488 453 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 2920] 30 2920 36955 246 315392 451 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 2921] 30 2921 36963 311 319488 407 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 2963] 30 2963 36947 295 315392 394 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 2964] 30 2964 36962 233 315392 480 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 2972] 30 2972 36957 244 319488 478 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 2973] 30 2973 36960 270 315392 434 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 2974] 30 2974 36949 228 319488 479 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 2978] 30 2978 36959 238 319488 461 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 2980] 30 2980 36949 241 315392 449 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 2984] 30 2984 36957 303 319488 418 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 2985] 30 2985 36960 288 319488 424 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 2987] 30 2987 36972 308 319488 407 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 3043] 30 3043 36949 244 315392 458 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 3046] 30 3046 36966 311 315392 399 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 3055] 30 3055 36975 333 319488 385 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 3058] 30 3058 38159 390 327680 1517 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 3059] 30 3059 36895 257 307200 389 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 3060] 30 3060 36959 240 319488 464 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 3061] 30 3061 36951 283 319488 421 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 3062] 30 3062 36963 328 319488 388 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 3066] 30 3066 37768 326 323584 1194 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 3069] 30 3069 36895 223 307200 427 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 3070] 30 3070 36895 219 307200 427 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 3072] 30 3072 36959 314 319488 403 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 3097] 30 3097 36959 294 319488 417 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 3100] 30 3100 36895 223 307200 426 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 3101] 30 3101 36895 245 307200 404 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 3114] 30 3114 36895 265 307200 380 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 3115] 30 3115 36895 271 307200 374 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 3116] 30 3116 36895 259 307200 386 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 3117] 30 3117 36962 319 319488 391 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 3121] 30 3121 36895 241 307200 404 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 3125] 30 3125 36929 249 315392 409 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 3127] 30 3127 36929 256 315392 405 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 3128] 30 3128 36895 192 307200 450 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 3135] 30 3135 36980 312 319488 411 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 3138] 30 3138 36895 186 307200 456 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 3139] 30 3139 36895 192 307200 450 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 3142] 30 3142 36895 245 307200 404 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 3143] 30 3143 36957 286 319488 417 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 3145] 30 3145 36929 290 315392 388 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 3147] 30 3147 36895 241 307200 408 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 3149] 30 3149 36949 257 315392 466 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 3159] 30 3159 38342 1238 327680 841 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 3161] 30 3161 36947 309 319488 388 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 3198] 30 3198 36977 286 319488 435 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 3199] 30 3199 36929 264 315392 426 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 3206] 30 3206 36986 253 319488 471 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 3211] 30 3211 36949 257 315392 444 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 3216] 30 3216 36947 241 319488 466 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 3219] 30 3219 36959 300 319488 407 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 3220] 30 3220 36961 259 319488 450 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 3226] 30 3226 36949 244 315392 453 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 3314] 30 3314 36949 244 319488 457 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 3319] 30 3319 36974 306 319488 408 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 3320] 30 3320 36927 242 315392 427 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 3325] 30 3325 36962 255 319488 452 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 3328] 30 3328 36951 272 319488 439 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 3362] 30 3362 36949 243 319488 464 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 3369] 30 3369 36961 246 319488 457 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 3380] 30 3380 36949 280 319488 418 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 3382] 30 3382 36949 255 319488 444 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 3385] 30 3385 36949 254 315392 442 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 3387] 30 3387 36949 278 319488 432 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 4762] 30 4762 36959 272 319488 436 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 5490] 30 5490 36949 265 315392 427 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 5491] 30 5491 36961 264 315392 449 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 5503] 30 5503 36949 313 319488 393 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 5507] 30 5507 36953 275 319488 432 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 5512] 30 5512 36947 257 319488 448 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 5519] 30 5519 36961 320 319488 384 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 5523] 30 5523 36981 293 319488 429 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 5524] 30 5524 36951 295 319488 419 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 5527] 30 5527 37228 297 319488 683 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 5531] 30 5531 36977 294 319488 429 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 5536] 30 5536 36971 313 319488 403 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 5539] 30 5539 36949 257 319488 450 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 5540] 30 5540 36951 282 319488 426 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 5543] 30 5543 38267 399 327680 1605 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 5545] 30 5545 36977 278 319488 442 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 5547] 30 5547 36972 318 319488 398 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 5550] 30 5550 36974 311 319488 406 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 5552] 30 5552 36949 318 319488 392 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 5556] 30 5556 36953 210 319488 496 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 5558] 30 5558 36952 225 315392 465 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 5562] 30 5562 36953 273 319488 440 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 5563] 30 5563 36971 323 319488 399 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 5571] 30 5571 36959 284 319488 437 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 5573] 30 5573 36948 317 319488 394 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 5574] 30 5574 36972 291 319488 428 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 5579] 30 5579 36987 265 319488 468 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 5609] 30 5609 36957 235 319488 469 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 5611] 30 5611 36947 187 315392 501 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 5612] 30 5612 36977 313 319488 413 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 5615] 30 5615 36953 304 319488 397 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 5617] 30 5617 36953 269 315392 429 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 5621] 30 5621 36951 306 315392 389 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 5624] 30 5624 36962 324 319488 398 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 5632] 30 5632 36977 318 319488 403 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 5634] 30 5634 36952 261 315392 437 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 5640] 30 5640 36949 298 319488 415 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 7590] 30 7590 36947 318 315392 377 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 7591] 30 7591 36953 291 315392 400 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 7594] 30 7594 36963 324 315392 383 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 7596] 30 7596 36949 310 319488 401 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 7597] 30 7597 36951 284 319488 427 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 7667] 30 7667 36949 278 319488 433 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 7670] 30 7670 36947 281 319488 427 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 7675] 30 7675 36962 315 319488 398 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 7676] 30 7676 36981 259 319488 459 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 7688] 30 7688 36949 264 319488 438 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 7693] 30 7693 36953 298 315392 408 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 7694] 30 7694 36963 320 319488 396 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 7695] 30 7695 36949 270 319488 440 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 10316] 30 10316 36949 241 319488 460 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 10319] 30 10319 36974 287 319488 431 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 10324] 30 10324 36961 321 319488 399 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 10326] 30 10326 36949 319 319488 395 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 26935] 26 26935 837409 179 434176 247 0 postgres
Sep 21 00:02:26 openqa kernel: [ 26939] 26 26939 26592 126 221184 267 0 postgres
Sep 21 00:02:26 openqa kernel: [ 20641] 30 20641 36962 314 319488 395 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 20668] 30 20668 36959 336 319488 384 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 20711] 30 20711 36963 335 319488 374 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 20724] 30 20724 36953 328 319488 381 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 20725] 30 20725 36963 333 319488 376 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 6443] 0 6443 1962560 23553 1011712 3103 0 telegraf
Sep 21 00:02:26 openqa kernel: [ 26445] 30 26445 36947 329 319488 378 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 26464] 30 26464 36963 338 319488 377 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 26466] 30 26466 36963 360 319488 351 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 13087] 30 13087 36959 318 319488 390 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 4473] 30 4473 36959 336 319488 369 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 4476] 30 4476 36959 332 319488 371 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 4478] 30 4478 36973 334 319488 378 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 4479] 30 4479 36963 350 319488 370 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 4480] 30 4480 36929 275 319488 415 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 4481] 30 4481 36947 322 315392 370 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 4498] 30 4498 36949 318 319488 393 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 4499] 30 4499 36959 329 319488 372 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 4500] 30 4500 36949 336 319488 367 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 4501] 30 4501 36951 332 315392 379 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 4502] 30 4502 36962 334 319488 372 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 4504] 30 4504 36951 342 319488 364 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 4505] 30 4505 36961 321 319488 389 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 5417] 30 5417 36963 344 319488 370 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 5425] 30 5425 36963 356 319488 365 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 5426] 30 5426 36959 329 319488 375 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 5427] 30 5427 36963 344 319488 365 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 5430] 30 5430 36963 329 319488 377 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 5431] 30 5431 36959 342 319488 376 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 5433] 30 5433 36951 308 319488 390 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 5435] 30 5435 36963 325 319488 382 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 5436] 30 5436 36959 326 319488 382 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 5438] 30 5438 36987 320 319488 410 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 30636] 30 30636 36949 309 315392 383 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 30682] 30 30682 36984 307 319488 418 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 30683] 30 30683 36959 297 319488 410 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 30700] 30 30700 36984 329 319488 398 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 30701] 30 30701 36947 301 319488 391 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 30702] 30 30702 36947 326 319488 374 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 30703] 30 30703 36958 320 319488 390 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 30754] 30 30754 36958 322 319488 386 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 30755] 30 30755 36947 278 315392 415 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 30756] 30 30756 36962 308 315392 398 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 30757] 30 30757 36963 318 319488 398 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 30758] 30 30758 36949 327 319488 374 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 30759] 30 30759 36977 335 319488 385 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 30760] 30 30760 36959 330 319488 377 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 30761] 30 30761 36946 298 319488 409 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 30799] 30 30799 36959 329 319488 384 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 30800] 30 30800 36949 316 319488 385 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 30801] 30 30801 36975 315 319488 398 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 30802] 30 30802 36962 340 319488 366 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 30803] 30 30803 36959 345 319488 364 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 30804] 30 30804 36961 343 319488 370 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 30805] 30 30805 36962 277 319488 428 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 30806] 30 30806 36949 312 315392 379 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 30807] 30 30807 36947 315 319488 386 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 30809] 30 30809 36959 326 319488 377 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 30810] 30 30810 36947 319 315392 380 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 30811] 30 30811 36962 338 315392 370 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 30812] 30 30812 36959 344 319488 364 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 30814] 30 30814 36972 319 319488 391 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 30821] 30 30821 36949 328 315392 364 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 30822] 30 30822 36959 334 319488 375 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 30823] 30 30823 36949 308 319488 396 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 30824] 30 30824 36953 323 319488 371 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 30825] 30 30825 36947 318 319488 384 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 30826] 30 30826 36949 328 319488 374 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 30827] 30 30827 36962 323 319488 382 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 30828] 30 30828 36949 303 319488 401 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 30829] 30 30829 36928 310 315392 379 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 30830] 30 30830 36956 313 315392 378 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 30831] 30 30831 36963 280 315392 413 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 30832] 30 30832 36947 318 315392 371 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 30833] 30 30833 36947 297 315392 393 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 30834] 30 30834 36959 322 319488 385 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 30835] 30 30835 36951 313 315392 382 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 30836] 30 30836 36959 339 319488 366 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 30837] 30 30837 36954 320 319488 383 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 30838] 30 30838 36947 296 315392 393 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 30839] 30 30839 36933 285 315392 391 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 30840] 30 30840 36959 282 319488 417 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 30841] 30 30841 36947 313 319488 386 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 30842] 30 30842 36949 313 319488 381 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 30843] 30 30843 36958 333 319488 374 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 30844] 30 30844 36959 311 319488 393 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 30845] 30 30845 36947 323 319488 385 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 30846] 30 30846 36933 332 319488 365 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 30847] 30 30847 36947 313 319488 385 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 30849] 30 30849 36962 327 315392 376 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 30850] 30 30850 36961 312 319488 385 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 30851] 30 30851 36959 326 319488 394 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 30852] 30 30852 36933 328 319488 371 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 30853] 30 30853 36933 317 315392 380 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 13412] 30 13412 36949 339 319488 363 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 13459] 30 13459 36947 338 319488 366 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 13460] 30 13460 36947 327 319488 364 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 13462] 30 13462 36949 308 315392 388 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 13463] 30 13463 36949 319 319488 391 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 13464] 30 13464 36951 316 315392 380 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 13465] 30 13465 36947 322 315392 370 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 13466] 30 13466 36933 297 315392 397 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 13467] 30 13467 36949 326 319488 379 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 13468] 30 13468 36950 299 315392 393 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 13469] 30 13469 36947 330 319488 377 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 13470] 30 13470 36949 339 319488 363 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 13471] 30 13471 36947 330 319488 367 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 13472] 30 13472 36933 329 315392 367 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 13473] 30 13473 36953 300 319488 401 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 13474] 30 13474 36947 326 315392 369 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 13475] 30 13475 36960 293 319488 410 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 13476] 30 13476 36950 320 319488 386 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 13477] 30 13477 36959 313 319488 385 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 13478] 30 13478 36928 268 315392 422 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 13479] 30 13479 36959 277 319488 430 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 13480] 30 13480 36933 279 315392 414 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 13481] 30 13481 36959 299 319488 409 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 13482] 30 13482 36959 331 319488 388 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 13483] 30 13483 36933 260 315392 434 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 13484] 30 13484 36949 340 319488 370 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 13485] 30 13485 36952 306 319488 393 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 13486] 30 13486 36952 307 315392 385 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 13487] 30 13487 36963 317 319488 387 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 13488] 30 13488 36949 326 319488 374 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 13489] 30 13489 36962 301 319488 404 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 13492] 30 13492 36963 319 319488 393 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 13493] 30 13493 36959 315 319488 389 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 13494] 30 13494 36927 282 315392 405 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 13495] 30 13495 36963 304 319488 403 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 13496] 30 13496 36947 298 319488 405 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 13497] 30 13497 36950 290 315392 403 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 13498] 30 13498 36929 243 307200 422 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 13499] 30 13499 36947 275 315392 416 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 13500] 30 13500 36959 295 319488 408 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 13501] 30 13501 36949 276 315392 414 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 13502] 30 13502 36947 267 319488 433 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 13503] 30 13503 36933 284 315392 408 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 13504] 30 13504 36959 340 319488 362 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 13505] 30 13505 36959 328 319488 376 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 13506] 30 13506 36929 247 315392 441 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 13507] 30 13507 36960 308 319488 396 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 13508] 30 13508 36948 287 315392 408 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 13509] 30 13509 36949 304 315392 386 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 13510] 30 13510 36951 307 319488 390 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 13511] 30 13511 36949 304 315392 401 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 13512] 30 13512 36962 320 319488 393 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 13513] 30 13513 36949 326 315392 371 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 13514] 30 13514 36960 309 319488 398 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 13515] 30 13515 36956 306 315392 390 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 13516] 30 13516 36953 312 319488 388 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 13517] 30 13517 36957 271 315392 433 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 13518] 30 13518 36948 294 319488 409 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 13519] 30 13519 36949 332 319488 369 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 13520] 30 13520 36949 286 319488 414 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 13521] 30 13521 36963 344 319488 367 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 13522] 30 13522 36947 317 319488 378 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 13523] 30 13523 36957 287 319488 412 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 26508] 30 26508 36949 292 315392 398 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 26555] 30 26555 36947 272 315392 417 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 26556] 30 26556 36959 312 319488 392 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 26568] 30 26568 36950 299 315392 400 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 26569] 30 26569 36951 286 319488 410 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 26570] 30 26570 36949 260 315392 431 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 26571] 30 26571 36959 308 319488 393 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 26611] 30 26611 36962 316 319488 390 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 26612] 30 26612 36949 296 319488 397 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 26613] 30 26613 36929 293 315392 399 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 26614] 30 26614 36950 281 315392 414 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 26615] 30 26615 36956 309 319488 398 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 26616] 30 26616 36949 292 315392 398 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 26617] 30 26617 36947 259 315392 431 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 26618] 30 26618 36933 274 315392 403 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 8596] 30 8596 36928 276 315392 428 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 8597] 30 8597 36931 262 315392 427 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 11284] 30 11284 36951 294 319488 406 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 8880] 30 8880 36950 315 319488 384 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 8881] 30 8881 36949 314 315392 380 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 8882] 30 8882 36959 248 319488 450 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 8921] 30 8921 36949 296 319488 397 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 8922] 30 8922 36947 306 315392 386 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 8923] 30 8923 36959 220 319488 483 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 8924] 30 8924 36953 238 315392 457 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 8954] 30 8954 36957 303 319488 397 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 8955] 30 8955 36929 302 319488 392 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 8956] 30 8956 36974 221 315392 493 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 8957] 30 8957 36949 311 319488 383 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 8958] 30 8958 36949 277 315392 416 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 8959] 30 8959 36963 298 319488 419 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 8960] 30 8960 36949 282 315392 414 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 8961] 30 8961 36946 245 315392 446 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 8962] 30 8962 36959 237 319488 462 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 8963] 30 8963 36949 241 319488 451 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 8964] 30 8964 36928 225 315392 462 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 8965] 30 8965 36949 289 319488 412 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 8966] 30 8966 36959 264 319488 442 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 8967] 30 8967 36929 241 315392 452 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 8968] 30 8968 36959 313 319488 394 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 8969] 30 8969 36954 208 315392 481 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 8970] 30 8970 36947 289 319488 416 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 8971] 30 8971 36958 226 319488 486 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 8972] 30 8972 36949 315 319488 381 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 8973] 30 8973 36949 314 319488 393 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 8974] 30 8974 36963 317 319488 392 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 8975] 30 8975 36949 209 315392 482 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 8976] 30 8976 36947 263 315392 432 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 8977] 30 8977 36949 211 315392 482 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 8981] 30 8981 36949 271 319488 426 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 8982] 30 8982 36928 244 315392 441 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 8983] 30 8983 36959 245 319488 454 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 8985] 30 8985 36947 302 319488 394 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 8986] 30 8986 36947 241 315392 456 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 8987] 30 8987 36959 253 319488 451 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 8988] 30 8988 36949 212 319488 490 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 8989] 30 8989 36951 254 315392 440 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 8990] 30 8990 36949 217 315392 474 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 8991] 30 8991 36949 306 319488 390 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 8992] 30 8992 36959 269 319488 447 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 8993] 30 8993 36949 294 315392 401 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 8994] 30 8994 36949 289 319488 413 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 8995] 30 8995 36933 261 315392 432 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 8996] 30 8996 36949 307 315392 385 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 8997] 30 8997 36962 232 319488 471 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 8998] 30 8998 36959 238 319488 462 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 8999] 30 8999 36963 280 319488 428 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 9000] 30 9000 36951 234 315392 459 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 9001] 30 9001 36957 286 319488 431 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 9002] 30 9002 36929 204 315392 478 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 9004] 30 9004 36929 289 319488 402 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 9005] 30 9005 36947 293 319488 403 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 9006] 30 9006 36959 217 319488 484 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 9007] 30 9007 36933 277 319488 416 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 9008] 30 9008 36947 301 319488 391 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 9009] 30 9009 36928 292 315392 397 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 9010] 30 9010 36947 247 315392 449 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 9011] 30 9011 36949 264 315392 430 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 9012] 30 9012 36947 205 315392 483 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 9013] 30 9013 36929 274 315392 414 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 9017] 30 9017 36929 241 315392 450 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 18384] 30 18384 36951 290 319488 411 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 18388] 30 18388 36949 298 315392 395 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 18389] 30 18389 36949 238 319488 461 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 15511] 30 15511 36951 303 315392 391 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 15513] 30 15513 36947 327 315392 367 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 15514] 30 15514 36949 318 319488 378 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 15540] 30 15540 36950 314 319488 380 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 15542] 30 15542 36947 333 319488 367 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 15543] 30 15543 36949 336 319488 361 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 15544] 30 15544 36947 306 315392 383 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 15569] 30 15569 36947 316 315392 373 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 15570] 30 15570 36951 321 315392 372 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 15571] 30 15571 36931 318 315392 375 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 15572] 30 15572 36954 321 315392 371 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 15573] 30 15573 36947 324 315392 367 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 15574] 30 15574 36949 321 319488 378 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 15575] 30 15575 36949 311 315392 380 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 15576] 30 15576 36933 318 319488 378 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 15596] 30 15596 36929 290 315392 398 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 15597] 30 15597 36949 325 315392 365 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 15598] 30 15598 36929 329 315392 363 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 15599] 30 15599 36947 322 315392 370 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 15600] 30 15600 36949 320 315392 371 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 15601] 30 15601 36949 313 315392 377 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 15602] 30 15602 36949 328 315392 364 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 15603] 30 15603 36959 332 319488 371 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 15604] 30 15604 36929 320 315392 370 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 15605] 30 15605 36929 317 319488 371 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 15606] 30 15606 36947 312 315392 378 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 15607] 30 15607 36947 328 319488 367 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 15608] 30 15608 36947 324 315392 367 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 15609] 30 15609 36949 335 319488 362 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 15610] 30 15610 36947 326 319488 369 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 15611] 30 15611 36929 318 315392 372 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 15624] 30 15624 36961 303 315392 405 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 15625] 30 15625 36959 332 319488 369 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 15626] 30 15626 36951 327 319488 375 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 15627] 30 15627 36929 323 315392 370 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 15628] 30 15628 36928 320 315392 370 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 15629] 30 15629 36949 312 315392 378 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 15630] 30 15630 36929 304 315392 378 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 15631] 30 15631 36953 316 315392 379 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 15632] 30 15632 36951 313 315392 379 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 15633] 30 15633 36960 327 319488 376 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 15634] 30 15634 36949 316 315392 375 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 15635] 30 15635 36949 312 315392 378 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 15636] 30 15636 36954 310 315392 383 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 15639] 30 15639 36949 328 319488 366 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 15640] 30 15640 36949 320 315392 375 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 15641] 30 15641 36951 321 315392 371 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 15642] 30 15642 36929 320 315392 371 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 15643] 30 15643 36933 302 315392 389 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 15644] 30 15644 36949 315 315392 375 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 15645] 30 15645 36959 341 319488 363 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 15646] 30 15646 36947 305 315392 384 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 15647] 30 15647 36949 315 315392 375 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 15648] 30 15648 36928 304 315392 380 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 15649] 30 15649 36949 320 315392 372 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 15650] 30 15650 36954 325 315392 368 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 15651] 30 15651 36933 305 315392 387 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 15652] 30 15652 36947 311 319488 387 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 15653] 30 15653 36947 310 319488 391 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 15654] 30 15654 36949 313 315392 379 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 15655] 30 15655 36933 304 319488 391 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 15656] 30 15656 36947 314 315392 380 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 15657] 30 15657 36952 304 315392 388 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 32020] 30 32020 36959 311 319488 389 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 32069] 30 32069 36959 306 319488 395 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 32070] 30 32070 36929 301 315392 388 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 32104] 30 32104 36929 314 319488 378 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 32105] 30 32105 36949 302 315392 389 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 32106] 30 32106 36949 300 315392 392 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 32107] 30 32107 36929 300 315392 390 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 32139] 30 32139 36954 325 315392 366 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 32140] 30 32140 36929 313 315392 376 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 32141] 30 32141 36949 316 315392 376 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 32142] 30 32142 36929 308 315392 379 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 32143] 30 32143 36956 318 315392 376 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 32144] 30 32144 36933 303 315392 389 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 32145] 30 32145 36949 344 319488 363 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 32146] 30 32146 36951 315 315392 378 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 8024] 30 8024 36949 316 315392 374 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 8052] 30 8052 36949 314 319488 378 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 8053] 30 8053 36951 325 315392 369 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 8054] 30 8054 36929 313 315392 376 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 8055] 30 8055 36947 314 315392 379 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 8056] 30 8056 36951 328 319488 368 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 8057] 30 8057 36929 305 315392 384 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 8064] 30 8064 36949 293 315392 397 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 8065] 30 8065 36947 316 315392 378 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 8066] 30 8066 36933 338 315392 354 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 8067] 30 8067 36951 333 319488 365 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 8068] 30 8068 36933 312 315392 377 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 8069] 30 8069 36953 335 319488 366 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 8070] 30 8070 36949 334 319488 364 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 8071] 30 8071 36951 325 315392 367 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 8076] 30 8076 36947 290 315392 398 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 8077] 30 8077 36949 327 315392 367 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 8078] 30 8078 36933 313 315392 381 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 8079] 30 8079 36929 317 315392 372 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 8080] 30 8080 36929 306 315392 382 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 8081] 30 8081 36928 319 315392 370 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 8082] 30 8082 36947 342 319488 360 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 8083] 30 8083 36949 318 315392 375 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 8084] 30 8084 36947 308 315392 387 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 8085] 30 8085 36947 309 319488 383 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 8086] 30 8086 36949 299 315392 391 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 8087] 30 8087 36962 333 319488 368 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 8088] 30 8088 36929 328 319488 369 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 8089] 30 8089 36955 337 319488 368 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 8090] 30 8090 36928 313 315392 377 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 8091] 30 8091 36947 314 315392 379 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 30609] 30 30609 36949 330 315392 363 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 19013] 30 19013 36949 287 315392 404 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 19050] 30 19050 36951 310 319488 390 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 19051] 30 19051 36929 288 315392 402 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 19103] 30 19103 36949 328 315392 364 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 19104] 30 19104 36929 261 307200 407 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 19105] 30 19105 36929 253 307200 413 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 19106] 30 19106 36947 319 319488 378 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 19140] 30 19140 36929 313 315392 373 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 19141] 30 19141 36953 314 319488 383 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 19142] 30 19142 36947 307 315392 381 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 19143] 30 19143 36929 256 307200 411 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 19144] 30 19144 36947 317 319488 377 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 19145] 30 19145 36949 295 315392 395 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 19146] 30 19146 36947 317 319488 373 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 19148] 30 19148 36951 320 319488 375 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 19179] 30 19179 36949 331 319488 368 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 19180] 30 19180 36951 293 315392 401 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 19181] 30 19181 36929 307 315392 378 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 19182] 30 19182 36949 304 315392 387 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 19183] 30 19183 36951 291 315392 401 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 19184] 30 19184 36953 303 315392 393 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 19185] 30 19185 36959 322 315392 381 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 19186] 30 19186 36929 252 307200 416 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 19187] 30 19187 36951 331 319488 364 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 19188] 30 19188 36947 311 319488 393 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 19189] 30 19189 36951 296 315392 396 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 19190] 30 19190 36951 294 315392 397 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 19191] 30 19191 36949 319 315392 373 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 19192] 30 19192 36929 225 307200 441 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 19193] 30 19193 36947 317 319488 389 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 19194] 30 19194 36949 296 315392 396 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 19217] 30 19217 36929 298 315392 391 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 19218] 30 19218 36949 303 319488 393 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 19219] 30 19219 36929 246 315392 441 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 19220] 30 19220 36951 309 319488 393 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 19221] 30 19221 36929 282 315392 404 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 19222] 30 19222 36949 307 319488 402 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 19223] 30 19223 36949 300 315392 392 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 19224] 30 19224 36949 289 315392 401 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 19225] 30 19225 36929 273 315392 415 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 19226] 30 19226 36929 245 307200 421 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 19227] 30 19227 36951 315 319488 387 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 19228] 30 19228 36951 301 315392 394 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 19229] 30 19229 36949 301 315392 391 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 19230] 30 19230 36947 299 315392 391 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 19231] 30 19231 36951 306 315392 389 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 19232] 30 19232 36955 298 319488 403 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 19233] 30 19233 36951 307 319488 391 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 19234] 30 19234 36951 314 319488 387 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 19235] 30 19235 36927 194 307200 466 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 19236] 30 19236 36949 299 315392 393 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 19238] 30 19238 36951 303 315392 390 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 19239] 30 19239 36929 254 315392 436 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 19240] 30 19240 36929 181 307200 481 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 19242] 30 19242 36929 292 315392 398 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 19243] 30 19243 36929 249 307200 417 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 19245] 30 19245 36951 303 319488 396 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 19246] 30 19246 36949 303 315392 392 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 19247] 30 19247 36947 261 315392 428 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 19248] 30 19248 36949 309 319488 386 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 19249] 30 19249 36951 301 315392 396 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 19250] 30 19250 36929 267 315392 417 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 19251] 30 19251 36951 306 319488 390 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 19324] 30 19324 36951 317 319488 385 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 1640] 488 1640 12062 662 126976 181 0 lvm
Sep 21 00:02:26 openqa kernel: [ 4920] 0 4920 17730 136 172032 78 0 systemd-user-ru
Sep 21 00:02:26 openqa kernel: [ 6777] 51 6777 23862 39 180224 191 0 pickup
Sep 21 00:02:26 openqa kernel: [ 16877] 30 16877 36929 285 315392 403 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 16889] 30 16889 36933 296 319488 392 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 16892] 30 16892 36929 249 307200 417 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 16911] 30 16911 36955 290 315392 401 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 16915] 30 16915 36929 234 307200 429 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 16916] 30 16916 36928 282 315392 397 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 16917] 30 16917 36933 293 319488 398 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 16950] 30 16950 36933 289 315392 390 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 16951] 30 16951 36929 297 319488 391 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 16953] 30 16953 36895 198 307200 451 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 16954] 30 16954 36933 274 315392 405 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 16956] 30 16956 36929 240 307200 421 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 16957] 30 16957 36929 257 311296 418 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 16958] 30 16958 36929 301 319488 391 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 16959] 30 16959 36928 273 315392 413 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 16988] 30 16988 36933 278 315392 406 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 16989] 30 16989 36929 299 319488 393 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 16991] 30 16991 36929 292 315392 399 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 16992] 30 16992 36929 290 315392 395 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 16993] 30 16993 36928 291 315392 395 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 16994] 30 16994 36929 240 307200 421 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 16996] 30 16996 36961 294 319488 402 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 16998] 30 16998 36927 257 311296 414 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 16999] 30 16999 36929 258 311296 416 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 17000] 30 17000 36955 288 315392 406 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 17001] 30 17001 36933 288 315392 394 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 17002] 30 17002 36951 303 319488 394 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 17003] 30 17003 36929 300 319488 390 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 17004] 30 17004 36929 290 315392 400 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 17005] 30 17005 36929 243 307200 423 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 17006] 30 17006 36929 238 307200 424 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 17017] 30 17017 36927 242 307200 418 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 17018] 30 17018 36963 303 319488 390 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 17019] 30 17019 36927 271 311296 414 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 17020] 30 17020 36927 260 315392 421 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 17021] 30 17021 36926 278 315392 410 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 17022] 30 17022 36929 286 315392 398 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 17023] 30 17023 36961 303 319488 393 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 17024] 30 17024 36933 304 315392 386 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 17025] 30 17025 36929 272 311296 414 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 17026] 30 17026 36929 287 315392 401 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 17027] 30 17027 36949 293 315392 397 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 17028] 30 17028 36929 291 319488 402 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 17029] 30 17029 36929 238 307200 423 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 17030] 30 17030 36928 287 315392 396 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 17031] 30 17031 36929 239 307200 423 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 17032] 30 17032 36929 265 311296 418 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 17033] 30 17033 36929 244 307200 421 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 17034] 30 17034 36929 244 307200 425 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 17036] 30 17036 36928 294 315392 392 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 17039] 30 17039 36949 287 315392 405 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 17043] 30 17043 36959 227 319488 460 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 17045] 30 17045 36929 270 311296 411 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 17047] 30 17047 36955 290 315392 406 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 17048] 30 17048 36929 239 307200 426 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 17049] 30 17049 36929 299 319488 392 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 17050] 30 17050 36955 297 319488 402 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 17051] 30 17051 36929 241 307200 421 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 17052] 30 17052 36929 294 315392 398 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 17053] 30 17053 36928 291 315392 395 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 17054] 30 17054 36929 245 307200 418 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 17055] 30 17055 36929 294 315392 395 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 17056] 30 17056 36928 284 315392 401 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 20843] 0 20843 17730 2 188416 212 0 systemd-user-ru
Sep 21 00:02:26 openqa kernel: [ 24587] 30 24587 36895 281 307200 372 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 24594] 30 24594 36927 305 311296 375 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 24595] 30 24595 36895 280 307200 373 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 24596] 30 24596 36895 278 307200 375 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 24597] 30 24597 36895 278 307200 375 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 24598] 30 24598 36895 277 307200 376 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 24599] 30 24599 36895 277 307200 376 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 24600] 30 24600 36927 308 311296 374 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 24601] 30 24601 36895 282 307200 371 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 24602] 30 24602 36895 283 307200 370 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 24603] 30 24603 36895 279 307200 374 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 24604] 30 24604 36927 311 311296 371 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 24605] 30 24605 36895 277 307200 376 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 24606] 30 24606 36895 278 307200 375 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 24607] 30 24607 36895 279 307200 374 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 24608] 30 24608 36895 276 307200 377 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 24613] 30 24613 36927 303 311296 369 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 24616] 30 24616 36933 317 315392 359 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 24618] 30 24618 36927 302 311296 371 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 24620] 30 24620 36927 299 311296 373 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 24621] 30 24621 36927 304 311296 370 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 24622] 30 24622 36927 299 311296 372 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 24623] 30 24623 36927 303 311296 368 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 24624] 30 24624 36895 277 307200 380 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 24625] 30 24625 36895 278 307200 379 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 24626] 30 24626 36927 296 311296 374 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 24627] 30 24627 36895 279 307200 379 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 24628] 30 24628 36895 272 307200 385 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 24629] 30 24629 36895 285 307200 373 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 24630] 30 24630 36895 278 307200 379 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 24631] 30 24631 36895 279 307200 378 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 24632] 30 24632 36927 299 311296 372 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 24633] 30 24633 36895 284 307200 370 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 24634] 30 24634 36927 300 311296 371 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 24635] 30 24635 36895 275 307200 377 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 24636] 30 24636 36895 274 307200 378 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 24637] 30 24637 36895 278 307200 376 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 24638] 30 24638 36927 300 311296 370 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 24639] 30 24639 36895 279 307200 375 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 24640] 30 24640 36927 302 311296 369 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 24641] 30 24641 36895 277 307200 375 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 24642] 30 24642 36927 298 311296 373 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 24643] 30 24643 36895 290 307200 368 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 24644] 30 24644 36895 283 307200 375 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 24645] 30 24645 36895 280 307200 372 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 24646] 30 24646 36895 284 307200 374 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 24647] 30 24647 36895 278 307200 374 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 24648] 30 24648 36895 286 307200 372 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 24649] 30 24649 36895 283 307200 375 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 24650] 30 24650 36895 286 307200 372 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 24651] 30 24651 36927 299 311296 371 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 24652] 30 24652 36927 301 311296 370 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 24659] 30 24659 36927 299 311296 371 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 24660] 30 24660 36927 299 311296 372 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 24670] 30 24670 36895 287 307200 371 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 24680] 30 24680 36927 299 311296 372 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 24687] 30 24687 36927 301 311296 370 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 24688] 30 24688 36927 302 311296 369 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 24689] 30 24689 36927 298 311296 373 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 24690] 30 24690 36927 300 311296 371 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 24691] 30 24691 36927 298 311296 373 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 24692] 30 24692 36927 301 311296 370 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 24973] 1001 24973 86795 37721 724992 2448 0 openqa
Sep 21 00:02:26 openqa kernel: [ 24988] 1001 24988 61936 28015 544768 2248 0 openqa-livehand
Sep 21 00:02:26 openqa kernel: [ 24999] 1001 24999 89898 37537 741376 2661 0 openqa
Sep 21 00:02:26 openqa kernel: [ 25006] 26 25006 837435 650 671744 236 0 postgres
Sep 21 00:02:26 openqa kernel: [ 25007] 26 25007 837409 223 303104 249 0 postgres
Sep 21 00:02:26 openqa kernel: [ 25008] 26 25008 837409 190 270336 242 0 postgres
Sep 21 00:02:26 openqa kernel: [ 25009] 26 25009 837580 329 270336 264 0 postgres
Sep 21 00:02:26 openqa kernel: [ 25010] 26 25010 27159 189 229376 245 0 postgres
Sep 21 00:02:26 openqa kernel: [ 25011] 26 25011 837517 196 249856 271 0 postgres
Sep 21 00:02:26 openqa kernel: [ 25015] 26 25015 837736 440 331776 235 0 postgres
Sep 21 00:02:26 openqa kernel: [ 25016] 26 25016 837736 447 331776 228 0 postgres
Sep 21 00:02:26 openqa kernel: [ 25022] 26 25022 837899 544 741376 230 0 postgres
Sep 21 00:02:26 openqa kernel: [ 25030] 26 25030 837976 1132 487424 234 0 postgres
Sep 21 00:02:26 openqa kernel: [ 25083] 1001 25083 86795 37716 704512 2459 0 openqa
Sep 21 00:02:26 openqa kernel: [ 25084] 26 25084 837919 581 356352 228 0 postgres
Sep 21 00:02:26 openqa kernel: [ 25128] 26 25128 838005 575 339968 239 0 postgres
Sep 21 00:02:26 openqa kernel: [ 25154] 26 25154 837829 524 339968 231 0 postgres
Sep 21 00:02:26 openqa kernel: [ 25296] 0 25296 12781 189 139264 6 0 netconfig
Sep 21 00:02:26 openqa kernel: [ 25302] 30 25302 36895 266 307200 374 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 25304] 30 25304 36895 265 307200 375 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 25305] 30 25305 36895 267 307200 374 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 25306] 30 25306 36895 265 307200 375 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 25307] 30 25307 36895 265 307200 375 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 25308] 30 25308 36895 271 307200 373 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 25309] 30 25309 36895 263 307200 377 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 25310] 30 25310 36895 265 307200 375 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 25312] 30 25312 36895 265 307200 375 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 25313] 30 25313 36895 267 307200 374 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 25314] 30 25314 36895 267 307200 373 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 25315] 30 25315 36895 268 307200 372 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 25316] 30 25316 36895 262 307200 378 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 25317] 30 25317 36895 266 307200 376 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 25318] 30 25318 36895 266 307200 374 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 25319] 30 25319 36895 268 307200 372 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 25320] 30 25320 36895 266 307200 374 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 25321] 30 25321 36895 266 307200 374 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 25322] 30 25322 36895 268 307200 373 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 25325] 30 25325 36895 265 307200 376 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 25327] 30 25327 36895 267 307200 374 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 25328] 30 25328 36895 266 307200 374 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 25329] 30 25329 36895 264 307200 376 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 25330] 30 25330 36895 263 307200 378 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 25331] 30 25331 36895 260 307200 380 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 25332] 30 25332 36895 264 307200 377 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 25333] 30 25333 36895 268 307200 373 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 25335] 30 25335 36895 266 307200 374 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 25338] 30 25338 36927 304 311296 372 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 25341] 30 25341 36895 267 307200 373 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 25344] 30 25344 36895 267 307200 373 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 25347] 0 25347 13074 468 139264 17 0 netconfig
Sep 21 00:02:26 openqa kernel: [ 25365] 30 25365 36895 265 307200 376 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 25366] 30 25366 36895 267 307200 374 0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [ 25379] 0 25379 301136 8407 704512 4358 0 salt-minion
Sep 21 00:02:26 openqa kernel: [ 25385] 1001 25385 88874 38301 729088 2385 0 openqa
Sep 21 00:02:26 openqa kernel: [ 25399] 26 25399 838252 1124 647168 228 0 postgres
Sep 21 00:02:26 openqa kernel: [ 25403] 1001 25403 88874 38099 729088 2532 0 openqa
Sep 21 00:02:26 openqa kernel: [ 25419] 26 25419 854512 1697 741376 224 0 postgres
Sep 21 00:02:26 openqa kernel: [ 25449] 26 25449 854553 1761 634880 221 0 postgres
Sep 21 00:02:26 openqa kernel: [ 25453] 0 25453 22647 131 221184 161 0 cron
Sep 21 00:02:26 openqa kernel: [ 25468] 488 25468 93037 8182 598016 3052 0 salt-master
Sep 21 00:02:26 openqa kernel: [ 25476] 0 25476 13074 469 131072 16 0 netconfig
Sep 21 00:02:26 openqa kernel: [ 25477] 0 25477 12880 306 139264 2 0 dns-resolver
Sep 21 00:02:26 openqa kernel: [ 25502] 26 25502 837919 2459 823296 225 0 postgres
Sep 21 00:02:26 openqa kernel: [ 25523] 1001 25523 3185 87 65536 1 0 fetchneedles
Sep 21 00:02:26 openqa kernel: [ 25544] 1001 25544 3185 91 61440 0 0 fetchneedles
Sep 21 00:02:26 openqa kernel: [ 25545] 1001 25545 88874 38375 729088 2405 0 openqa
Sep 21 00:02:26 openqa kernel: [ 25554] 1001 25554 86828 37785 700416 2405 0 openqa
Sep 21 00:02:26 openqa kernel: [ 25559] 1001 25559 88874 38111 724992 2421 0 openqa
Sep 21 00:02:26 openqa kernel: [ 25563] 26 25563 837976 1541 557056 223 0 postgres
Sep 21 00:02:26 openqa kernel: [ 25567] 26 25567 838043 1713 638976 226 0 postgres
Sep 21 00:02:26 openqa kernel: [ 25568] 26 25568 838479 572 380928 225 0 postgres
Sep 21 00:02:26 openqa kernel: [ 25573] 1001 25573 89898 38032 729088 2411 0 openqa
Sep 21 00:02:26 openqa kernel: [ 25579] 1001 25579 89898 38040 729088 2389 0 openqa
Sep 21 00:02:26 openqa kernel: [ 25585] 0 25585 22647 130 221184 157 0 cron
Sep 21 00:02:26 openqa kernel: [ 25587] 1001 25587 6109 287 86016 2 0 git
Sep 21 00:02:26 openqa kernel: [ 25590] 26 25590 837943 2018 520192 222 0 postgres
Sep 21 00:02:26 openqa kernel: [ 25600] 1001 25600 89898 38058 729088 2389 0 openqa
Sep 21 00:02:26 openqa kernel: [ 25604] 26 25604 837934 1598 491520 222 0 postgres
Sep 21 00:02:26 openqa kernel: [ 25605] 1001 25605 89898 38085 729088 2371 0 openqa
Sep 21 00:02:26 openqa kernel: [ 25613] 1001 25613 89898 38017 729088 2386 0 openqa
Sep 21 00:02:26 openqa kernel: [ 25615] 1001 25615 89898 37968 729088 2412 0 openqa
Sep 21 00:02:26 openqa kernel: [ 25616] 0 25616 22647 130 221184 157 0 cron
Sep 21 00:02:26 openqa kernel: [ 25617] 26 25617 837934 1915 524288 223 0 postgres
Sep 21 00:02:26 openqa kernel: [ 25625] 26 25625 837965 2204 512000 221 0 postgres
Sep 21 00:02:26 openqa kernel: [ 25627] 1001 25627 4403 68 73728 0 0 git
Sep 21 00:02:26 openqa kernel: [ 25630] 26 25630 837935 1777 462848 222 0 postgres
Sep 21 00:02:26 openqa kernel: [ 25632] 1001 25632 89898 37957 729088 2420 0 openqa
Sep 21 00:02:26 openqa kernel: [ 25633] 1001 25633 18990 181 167936 0 0 git-remote-http
Sep 21 00:02:26 openqa kernel: [ 25636] 1001 25636 89898 37946 729088 2434 0 openqa
Sep 21 00:02:26 openqa kernel: [ 25640] 26 25640 837887 1554 434176 226 0 postgres
Sep 21 00:02:26 openqa kernel: [ 25643] 1001 25643 89898 37962 729088 2423 0 openqa
Sep 21 00:02:26 openqa kernel: [ 25646] 26 25646 837900 1758 450560 221 0 postgres
Sep 21 00:02:26 openqa kernel: [ 25648] 1001 25648 89898 38062 724992 2457 0 openqa
Sep 21 00:02:26 openqa kernel: [ 25650] 26 25650 837878 2058 417792 223 0 postgres
Sep 21 00:02:26 openqa kernel: [ 25657] 26 25657 837899 1816 450560 221 0 postgres
Sep 21 00:02:26 openqa kernel: [ 25658] 1001 25658 89898 37968 724992 2494 0 openqa
Sep 21 00:02:26 openqa kernel: [ 25665] 1001 25665 89898 37837 724992 2477 0 openqa
Sep 21 00:02:26 openqa kernel: [ 25669] 1001 25669 89898 38152 724992 2467 0 openqa
Sep 21 00:02:26 openqa kernel: [ 25670] 26 25670 837868 2047 327680 222 0 postgres
Sep 21 00:02:26 openqa kernel: [ 25672] 26 25672 837856 2187 352256 222 0 postgres
Sep 21 00:02:26 openqa kernel: [ 25675] 0 25675 12792 34 139264 173 0 cron
Sep 21 00:02:26 openqa kernel: [ 25676] 0 25676 22680 127 221184 158 0 cron
Sep 21 00:02:26 openqa kernel: [ 25678] 0 25678 22680 124 221184 160 0 cron
Sep 21 00:02:26 openqa kernel: [ 25679] 0 25679 3152 71 65536 0 0 sh
Sep 21 00:02:26 openqa kernel: [ 25680] 0 25680 22680 125 221184 159 0 cron
Sep 21 00:02:26 openqa kernel: [ 25681] 0 25681 22680 126 221184 158 0 cron
Sep 21 00:02:26 openqa kernel: [ 25682] 0 25682 22680 125 221184 159 0 cron
Sep 21 00:02:26 openqa kernel: [ 25683] 0 25683 22680 125 221184 159 0 cron
Sep 21 00:02:26 openqa kernel: [ 25684] 0 25684 22680 125 221184 159 0 cron
Sep 21 00:02:26 openqa kernel: [ 25685] 26 25685 837680 2243 339968 221 0 postgres
Sep 21 00:02:26 openqa kernel: [ 25687] 1001 25687 89898 37842 724992 2474 0 openqa
Sep 21 00:02:26 openqa kernel: [ 25689] 0 25689 3152 71 69632 0 0 run-crons
Sep 21 00:02:26 openqa kernel: [ 25691] 1001 25691 89898 37835 724992 2477 0 openqa
Sep 21 00:02:26 openqa kernel: [ 25694] 26 25694 837649 1649 307200 224 0 postgres
Sep 21 00:02:26 openqa kernel: [ 25695] 26 25695 837680 2083 331776 222 0 postgres
Sep 21 00:02:26 openqa kernel: [ 25696] 1001 25696 89898 37643 724992 2580 0 openqa
Sep 21 00:02:26 openqa kernel: [ 25698] 26 25698 837447 201 241664 236 0 postgres
Sep 21 00:02:26 openqa kernel: [ 25700] 1001 25700 86795 37721 700416 2448 0 openqa
Sep 21 00:02:26 openqa kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=openqa,pid=25385,uid=1001
Sep 21 00:02:26 openqa kernel: Out of memory: Killed process 25385 (openqa) total-vm:355496kB, anon-rss:153204kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:712kB oom_score_adj:0
Sep 21 00:02:26 openqa kernel: oom_reaper: reaped process 25385 (openqa), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Sep 21 00:02:26 openqa kernel: kthreadd invoked oom-killer: gfp_mask=0x40cc0(GFP_KERNEL|__GFP_COMP), order=2, oom_score_adj=0
Sep 21 00:02:26 openqa kernel: CPU: 6 PID: 2 Comm: kthreadd Tainted: G W L 5.14.21-150400.24.21-default #1 SLE15-SP4 7550826c4c7e8c258239e300508e0c8b2a69bad2
Sep 21 00:02:26 openqa kernel: Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
…
Sep 21 00:02:26 openqa kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=openqa,pid=25658,uid=1001
Sep 21 00:02:26 openqa kernel: Out of memory: Killed process 25658 (openqa) total-vm:359592kB, anon-rss:152516kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:708kB oom_score_adj:0
Sep 21 00:02:26 openqa kernel: dns-resolver invoked oom-killer: gfp_mask=0x40cc0(GFP_KERNEL|__GFP_COMP), order=2, oom_score_adj=0
Sep 21 00:02:26 openqa kernel: oom_reaper: reaped process 25658 (openqa), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
…
Sep 21 00:02:26 openqa kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=openqa,pid=25669,uid=1001
Sep 21 00:02:26 openqa kernel: Out of memory: Killed process 25669 (openqa) total-vm:359592kB, anon-rss:153048kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:712kB oom_score_adj:0
Sep 21 00:02:26 openqa kernel: oom_reaper: reaped process 25669 (openqa), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
…
Sep 21 01:23:19 openqa kernel: Tasks state (memory values in pages):
Sep 21 01:23:19 openqa kernel: [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
Sep 21 01:23:19 openqa kernel: [ 523] 0 523 27415 0 237568 679 -1000 systemd-udevd
Sep 21 01:23:19 openqa kernel: [ 676] 0 676 13017 0 135168 164 0 rpc.idmapd
Sep 21 01:23:19 openqa kernel: [ 677] 0 677 15090 75 147456 197 -1000 auditd
Sep 21 01:23:19 openqa kernel: [ 682] 499 682 19866 383 184320 345 -900 dbus-daemon
Sep 21 01:23:19 openqa kernel: [ 31945] 0 693 28166 0 241664 472 0 close
Sep 21 01:23:19 openqa kernel: [ 696] 0 696 10708 3 122880 309 0 wickedd-auto4
Sep 21 01:23:19 openqa kernel: [ 701] 0 701 10709 3 131072 311 0 wickedd-dhcp4
Sep 21 01:23:19 openqa kernel: [ 705] 0 705 10709 0 126976 337 0 wickedd-dhcp6
Sep 21 01:23:19 openqa kernel: [ 740] 0 740 10781 0 135168 397 0 wickedd
Sep 21 01:23:19 openqa kernel: [ 767] 0 767 10744 0 126976 339 0 wickedd-nanny
Sep 21 01:23:19 openqa kernel: [ 788] 0 788 18806 0 184320 436 0 systemd
Sep 21 01:23:19 openqa kernel: [ 1307] 0 1307 28974 357 262144 298 0 rebootmgrd
Sep 21 01:23:19 openqa kernel: [ 1309] 0 1309 7763 0 102400 134 0 rsyncd
Sep 21 01:23:19 openqa kernel: [ 1319] 0 1319 6390 0 94208 127 0 vsftpd
Sep 21 01:23:19 openqa kernel: [ 1321] 0 1321 13501 1 139264 170 0 xinetd
Sep 21 01:23:19 openqa kernel: [ 1326] 493 1326 15982 0 176128 233 0 rpcbind
Sep 21 01:23:19 openqa kernel: [ 1334] 0 1334 18890 0 196608 267 -1000 sshd
Sep 21 01:23:19 openqa kernel: [ 1337] 475 1337 39077 1 208896 296 0 chronyd
Sep 21 01:23:19 openqa kernel: [ 1380] 497 1380 12618 0 143360 224 -500 nrpe
Sep 21 01:23:19 openqa kernel: [ 1502] 0 1502 11481 0 126976 190 0 agetty
Sep 21 01:23:19 openqa kernel: [ 1507] 0 1507 11481 0 131072 166 0 agetty
Sep 21 01:23:19 openqa kernel: [ 1617] 0 1617 24392 0 176128 258 0 master
Sep 21 01:23:19 openqa kernel: [ 1619] 51 1619 29041 0 192512 277 0 qmgr
Sep 21 01:23:19 openqa kernel: [ 1632] 0 1632 12792 0 147456 208 0 cron
Sep 21 01:23:19 openqa kernel: [ 32142] 30 32142 36929 0 315392 3 0 httpd-prefork
Sep 21 01:23:19 openqa kernel: [ 1640] 488 1640 12062 0 126976 0 0 lvm
Sep 21 01:23:19 openqa kernel: [ 4920] 0 4920 17730 1 172032 213 0 systemd-user-ru
Sep 21 01:23:19 openqa kernel: [ 20843] 0 20843 17730 1 188416 213 0 systemd-user-ru
Sep 21 01:23:19 openqa kernel: [ 25379] 0 25379 303441 0 712704 5 0 salt-minion
Sep 21 01:23:19 openqa kernel: [ 25829] 0 25829 19755 0 180224 0 0 (ogrotate)
Sep 21 01:23:19 openqa kernel: [ 27896] 488 27896 12062 0 114688 0 0 lvm
Sep 21 01:23:19 openqa kernel: [ 28021] 488 28021 12062 0 114688 0 0 lvm
Sep 21 01:23:19 openqa kernel: [ 28189] 488 28189 12062 0 122880 0 0 lvm
Sep 21 01:23:19 openqa kernel: [ 28218] 488 28218 12062 0 114688 0 0 lvm
Sep 21 01:23:19 openqa kernel: [ 28244] 488 28244 12062 0 118784 0 0 lvm
Sep 21 01:23:19 openqa kernel: [ 28562] 0 28562 10013 0 118784 0 0 lvm
Sep 21 01:23:19 openqa kernel: [ 28947] 0 28947 10013 0 110592 0 0 lvm
Sep 21 01:23:19 openqa kernel: [ 30535] 0 30535 17705 0 176128 211 0 systemd-user-ru
Sep 21 01:23:19 openqa kernel: [ 31136] 0 31136 22647 0 221184 320 0 cron
Sep 21 01:23:19 openqa kernel: [ 31224] 0 31224 22647 0 221184 313 0 cron
Sep 21 01:23:19 openqa kernel: [ 31770] 51 31770 23862 0 180224 240 0 pickup
Sep 21 01:23:19 openqa kernel: [ 31785] 0 31785 22647 0 221184 295 0 cron
Sep 21 01:23:19 openqa kernel: [ 31786] 0 31786 22647 0 221184 307 0 cron
Sep 21 01:23:19 openqa kernel: [ 31798] 0 31798 22647 0 221184 295 0 cron
Sep 21 01:23:19 openqa kernel: [ 31800] 0 31800 22647 0 221184 295 0 cron
Sep 21 01:23:19 openqa kernel: [ 31802] 0 31802 22647 0 221184 295 0 cron
Sep 21 01:23:19 openqa kernel: [ 31808] 0 31808 22647 0 221184 295 0 cron
Sep 21 01:23:20 openqa kernel: [ 31809] 0 31809 22647 0 221184 311 0 cron
Sep 21 01:23:20 openqa kernel: [ 31810] 0 31810 22647 0 221184 300 0 cron
Sep 21 01:23:20 openqa kernel: [ 31820] 0 31820 22647 0 221184 295 0 cron
Sep 21 01:23:20 openqa kernel: [ 31821] 0 31821 22647 0 221184 322 0 cron
Sep 21 01:23:20 openqa kernel: [ 31822] 0 31822 22647 0 221184 295 0 cron
Sep 21 01:23:20 openqa kernel: [ 31823] 0 31823 22647 0 221184 296 0 cron
Sep 21 01:23:20 openqa kernel: [ 31824] 0 31824 22647 0 221184 293 0 cron
Sep 21 01:23:20 openqa kernel: [ 31825] 0 31825 22647 0 221184 310 0 cron
Sep 21 01:23:20 openqa kernel: [ 31826] 0 31826 22647 0 221184 301 0 cron
Sep 21 01:23:20 openqa kernel: [ 31829] 0 31829 22647 0 221184 293 0 cron
Sep 21 01:23:20 openqa kernel: [ 31832] 0 31832 22647 0 221184 293 0 cron
Sep 21 01:23:20 openqa kernel: [ 31839] 0 31839 22647 0 221184 293 0 cron
Sep 21 01:23:20 openqa kernel: [ 31840] 0 31840 22647 0 221184 295 0 cron
Sep 21 01:23:20 openqa kernel: [ 31841] 0 31841 22647 0 221184 293 0 cron
Sep 21 01:23:20 openqa kernel: [ 31843] 0 31843 12792 0 139264 236 0 cron
Sep 21 01:23:20 openqa kernel: [ 31844] 0 31844 22647 0 221184 294 0 cron
Sep 21 01:23:20 openqa kernel: [ 31845] 0 31845 22647 0 221184 293 0 cron
Sep 21 01:23:20 openqa kernel: [ 31846] 0 31846 22647 0 221184 293 0 cron
Sep 21 01:23:20 openqa kernel: [ 31847] 0 31847 22647 0 221184 293 0 cron
Sep 21 01:23:20 openqa kernel: [ 31851] 0 31851 22647 0 221184 293 0 cron
Sep 21 01:23:20 openqa kernel: [ 31852] 0 31852 22647 0 221184 293 0 cron
Sep 21 01:23:20 openqa kernel: [ 31854] 0 31854 22647 0 221184 293 0 cron
Sep 21 01:23:20 openqa kernel: [ 31856] 0 31856 22647 0 221184 293 0 cron
Sep 21 01:23:20 openqa kernel: [ 31863] 0 31863 22647 0 221184 294 0 cron
Sep 21 01:23:20 openqa kernel: [ 31864] 0 31864 22647 0 221184 293 0 cron
Sep 21 01:23:20 openqa kernel: [ 31869] 0 31869 22647 0 221184 295 0 cron
Sep 21 01:23:20 openqa kernel: [ 31870] 0 31870 22647 0 221184 293 0 cron
Sep 21 01:23:20 openqa kernel: [ 32343] 0 32343 22647 0 221184 295 0 cron
Sep 21 01:23:20 openqa kernel: [ 32591] 0 32591 3929 1 77824 47 0 find
Sep 21 01:23:20 openqa kernel: [ 32754] 0 32754 22647 0 221184 304 0 cron
Sep 21 01:23:20 openqa kernel: [ 32755] 0 32755 22647 0 221184 304 0 cron
Sep 21 01:23:20 openqa kernel: [ 32756] 0 32756 22647 0 221184 304 0 cron
Sep 21 01:23:20 openqa kernel: [ 32757] 0 32757 22647 0 221184 303 0 cron
Sep 21 01:23:20 openqa kernel: [ 420] 0 420 22647 0 221184 308 0 cron
Sep 21 01:23:20 openqa kernel: [ 421] 0 421 22647 0 221184 303 0 cron
Sep 21 01:23:20 openqa kernel: [ 531] 0 531 3152 0 73728 73 0 reload-webserve
Sep 21 01:23:20 openqa kernel: [ 532] 0 532 17854 0 167936 240 0 systemctl
Sep 21 01:23:20 openqa kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=systemd-logind,pid=693,uid=0
Sep 21 01:23:20 openqa kernel: Out of memory: Killed process 31945 (close) total-vm:112664kB, anon-rss:0kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:236kB oom_score_adj:0
Sep 21 01:23:20 openqa kernel: oom_reaper: reaped process 31945 (close), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Sep 21 01:23:20 openqa kernel: systemd invoked oom-killer: gfp_mask=0x40cc0(GFP_KERNEL|__GFP_COMP), order=2, oom_score_adj=0
Sep 21 01:23:20 openqa kernel: CPU: 11 PID: 1 Comm: systemd Tainted: G W L 5.14.21-150400.24.21-default #1 SLE15-SP4 7550826c4c7e8c258239e300508e0c8b2a69bad2
Sep 21 01:23:20 openqa kernel: Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
Sep 21 01:23:20 openqa kernel: Call Trace:
Sep 21 01:23:20 openqa kernel: <TASK>
Sep 21 01:23:20 openqa kernel: dump_stack_lvl+0x45/0x5b
Sep 21 01:23:20 openqa kernel: dump_header+0x4a/0x220
Sep 21 01:23:20 openqa kernel: oom_kill_process+0xe8/0x140
Sep 21 01:23:20 openqa kernel: out_of_memory+0x113/0x580
Sep 21 01:23:20 openqa kernel: __alloc_pages_slowpath.constprop.112+0x980/0xc70
Sep 21 01:23:20 openqa kernel: __alloc_pages+0x2d9/0x320
Sep 21 01:23:20 openqa kernel: new_slab+0x33e/0x500
Sep 21 01:23:20 openqa kernel: ___slab_alloc+0x6ad/0x860
Sep 21 01:23:20 openqa kernel: ? copy_process+0x213/0x1cd0
Sep 21 01:23:20 openqa kernel: ? __mod_memcg_lruvec_state+0x1d/0x100
Sep 21 01:23:20 openqa kernel: ? __mod_lruvec_page_state+0x6f/0xb0
Sep 21 01:23:20 openqa kernel: ? lock_page_memcg+0x1c/0xa0
Sep 21 01:23:20 openqa kernel: ? __slab_alloc.isra.92+0x1c/0x40
Sep 21 01:23:20 openqa kernel: __slab_alloc.isra.92+0x1c/0x40
Sep 21 01:23:20 openqa kernel: kmem_cache_alloc_node+0x114/0x310
Sep 21 01:23:20 openqa kernel: ? copy_process+0x213/0x1cd0
Sep 21 01:23:20 openqa kernel: copy_process+0x213/0x1cd0
Sep 21 01:23:20 openqa kernel: kernel_clone+0x9b/0x400
Sep 21 01:23:20 openqa kernel: __do_sys_clone+0x72/0xa0
Sep 21 01:23:20 openqa kernel: do_syscall_64+0x5b/0x80
Sep 21 01:23:20 openqa kernel: ? exc_page_fault+0x67/0x150
Sep 21 01:23:20 openqa kernel: entry_SYSCALL_64_after_hwframe+0x61/0xcb
Sep 21 01:23:20 openqa kernel: RIP: 0033:0x7f15c3718a68
Sep 21 01:23:20 openqa kernel: Code: Unable to access opcode bytes at RIP 0x7f15c3718a3e.
Sep 21 01:23:20 openqa kernel: RSP: 002b:00007ffc905fd240 EFLAGS: 00000246 ORIG_RAX: 0000000000000038
Sep 21 01:23:20 openqa kernel: RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f15c3718a68
Sep 21 01:23:20 openqa kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000001200011
Sep 21 01:23:20 openqa kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: 00007f15c5446b00
Sep 21 01:23:20 openqa kernel: R10: 00007f15c5446dd0 R11: 0000000000000246 R12: 0000556f45bd8720
Sep 21 01:23:20 openqa kernel: R13: 00007ffc905fd490 R14: 0000556f45cc9758 R15: 0000000000000000
Sep 21 01:23:20 openqa kernel: </TASK>
Sep 21 01:23:20 openqa kernel: Mem-Info:
Sep 21 01:23:20 openqa kernel: active_anon:2 inactive_anon:196 isolated_anon:0
active_file:3131 inactive_file:4105 isolated_file:0
unevictable:20 dirty:0 writeback:0
slab_reclaimable:4904595 slab_unreclaimable:1125985
mapped:580 shmem:3 pagetables:3856 bounce:0
free:77366 free_pcp:3099 free_cma:0
Sep 21 01:23:20 openqa kernel: Node 0 active_anon:8kB inactive_anon:784kB active_file:12524kB inactive_file:16420kB unevictable:80kB isolated(anon):0kB isolated(file):0kB mapped:2320kB dirty:0kB writeback:0kB shmem:12kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB kernel_stack:4864kB pagetab>
Sep 21 01:23:20 openqa kernel: Node 0 DMA free:11024kB boost:0kB min:40kB low:52kB high:64kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Sep 21 01:23:20 openqa kernel: lowmem_reserve[]: 0 2991 24013 24013 24013
Sep 21 01:23:20 openqa kernel: Node 0 DMA32 free:114856kB boost:0kB min:8356kB low:11392kB high:14428kB reserved_highatomic:14336KB active_anon:0kB inactive_anon:0kB active_file:4kB inactive_file:4kB unevictable:0kB writepending:0kB present:3129312kB managed:3063776kB mlocked:0kB bounce:0kB free_pcp:1976kB local_p>
Sep 21 01:23:20 openqa kernel: lowmem_reserve[]: 0 0 21021 21021 21021
Sep 21 01:23:20 openqa kernel: Node 0 Normal free:183584kB boost:0kB min:59184kB low:80700kB high:102216kB reserved_highatomic:20480KB active_anon:8kB inactive_anon:904kB active_file:12492kB inactive_file:16808kB unevictable:80kB writepending:0kB present:22020096kB managed:21525736kB mlocked:80kB bounce:0kB free_p>
Sep 21 01:23:20 openqa kernel: lowmem_reserve[]: 0 0 0 0 0
Sep 21 01:23:20 openqa kernel: Node 0 DMA: 0*4kB 0*8kB 1*16kB (U) 2*32kB (UE) 1*64kB (U) 1*128kB (U) 2*256kB (UE) 2*512kB (UE) 1*1024kB (E) 2*2048kB (ME) 1*4096kB (M) = 11024kB
Sep 21 01:23:20 openqa kernel: Node 0 DMA32: 7927*4kB (UM) 10421*8kB (UME) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 115076kB
Sep 21 01:23:20 openqa kernel: Node 0 Normal: 44275*4kB (UM) 836*8kB (UM) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 183788kB
Sep 21 01:23:20 openqa kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Sep 21 01:23:20 openqa kernel: 7577 total pagecache pages
Sep 21 01:23:20 openqa kernel: 180 pages in swap cache
Sep 21 01:23:20 openqa kernel: Swap cache stats: add 11843458, delete 11843174, find 99241998/104794634
Sep 21 01:23:20 openqa kernel: Free swap = 439684kB
Sep 21 01:23:20 openqa kernel: Total swap = 499708kB
Sep 21 01:23:20 openqa kernel: 6291350 pages RAM
Sep 21 01:23:20 openqa kernel: 0 pages HighMem/MovableOnly
Sep 21 01:23:20 openqa kernel: 140132 pages reserved
Sep 21 01:23:20 openqa kernel: 0 pages cma reserved
Sep 21 01:23:20 openqa kernel: 0 pages hwpoisoned
Sep 21 01:23:20 openqa kernel: Unreclaimable slab info:
Sep 21 01:23:20 openqa kernel: Name Used Total
Sep 21 01:23:20 openqa kernel: xfs_icr 47KB 47KB
Sep 21 01:23:20 openqa kernel: xfs_efi_item 392KB 410KB
Sep 21 01:23:20 openqa kernel: xfs_buf_item 447KB 447KB
Sep 21 01:23:20 openqa kernel: xfs_da_state 191KB 191KB
Sep 21 01:23:20 openqa kernel: xfs_btree_cur 94KB 94KB
Sep 21 01:23:20 openqa kernel: nfsd4_openowners 372KB 391KB
Sep 21 01:23:20 openqa kernel: nfsd4_clients 127KB 127KB
Sep 21 01:23:20 openqa kernel: scsi_sense_cache 4KB 4KB
Sep 21 01:23:20 openqa kernel: ip6-frags 47KB 47KB
Sep 21 01:23:20 openqa kernel: PINGv6 370KB 370KB
Sep 21 01:23:20 openqa kernel: RAWv6 8148KB 8148KB
Sep 21 01:23:20 openqa kernel: UDPv6 3374KB 3374KB
Sep 21 01:23:20 openqa kernel: tw_sock_TCPv6 2229KB 2229KB
Sep 21 01:23:20 openqa kernel: request_sock_TCPv6 926KB 926KB
Sep 21 01:23:20 openqa kernel: TCPv6 3206KB 4519KB
Sep 21 01:23:20 openqa kernel: bfq_io_cq 95KB 95KB
Sep 21 01:23:20 openqa kernel: mqueue_inode_cache 496KB 496KB
Sep 21 01:23:20 openqa kernel: dnotify_struct 4KB 4KB
Sep 21 01:23:20 openqa kernel: UNIX 1133KB 1173KB
Sep 21 01:23:20 openqa kernel: ip4-frags 3KB 3KB
Sep 21 01:23:20 openqa kernel: PING 384KB 384KB
Sep 21 01:23:20 openqa kernel: RAW 10160KB 10160KB
Sep 21 01:23:20 openqa kernel: tw_sock_TCP 95KB 95KB
Sep 21 01:23:20 openqa kernel: request_sock_TCP 686KB 717KB
Sep 21 01:23:20 openqa kernel: TCP 693KB 913KB
Sep 21 01:23:20 openqa kernel: hugetlbfs_inode_cache 126KB 126KB
Sep 21 01:23:20 openqa kernel: ep_head 48KB 48KB
Sep 21 01:23:20 openqa kernel: bio_crypt_ctx 394KB 394KB
Sep 21 01:23:20 openqa kernel: request_queue 220KB 220KB
Sep 21 01:23:20 openqa kernel: biovec-max 556KB 604KB
Sep 21 01:23:20 openqa kernel: biovec-128 612KB 612KB
Sep 21 01:23:20 openqa kernel: biovec-64 384KB 384KB
Sep 21 01:23:20 openqa kernel: khugepaged_mm_slot 78KB 78KB
Sep 21 01:23:20 openqa kernel: dmaengine-unmap-256 30KB 30KB
Sep 21 01:23:20 openqa kernel: dmaengine-unmap-128 31KB 31KB
Sep 21 01:23:20 openqa kernel: skbuff_ext_cache 21628KB 21628KB
Sep 21 01:23:20 openqa kernel: skbuff_fclone_cache 399KB 436KB
Sep 21 01:23:20 openqa kernel: skbuff_head_cache 1447KB 1516KB
Sep 21 01:23:20 openqa kernel: file_lock_cache 93KB 93KB
Sep 21 01:23:20 openqa kernel: file_lock_ctx 263KB 263KB
Sep 21 01:23:20 openqa kernel: fsnotify_mark_connector 48KB 48KB
Sep 21 01:23:20 openqa kernel: net_namespace 1750KB 1750KB
Sep 21 01:23:20 openqa kernel: task_delay_info 71KB 71KB
Sep 21 01:23:20 openqa kernel: taskstats 228KB 228KB
Sep 21 01:23:20 openqa kernel: proc_dir_entry 6855KB 6855KB
Sep 21 01:23:20 openqa kernel: pde_opener 47KB 47KB
Sep 21 01:23:20 openqa kernel: seq_file 47KB 47KB
Sep 21 01:23:20 openqa kernel: shmem_inode_cache 3537KB 3537KB
Sep 21 01:23:20 openqa kernel: kernfs_node_cache 9236KB 9236KB
Sep 21 01:23:20 openqa kernel: mnt_cache 826KB 826KB
Sep 21 01:23:20 openqa kernel: filp 2139592KB 2139592KB
Sep 21 01:23:20 openqa kernel: names_cache 424KB 440KB
Sep 21 01:23:20 openqa kernel: lsm_file_cache 525KB 549KB
Sep 21 01:23:20 openqa kernel: uts_namespace 93KB 93KB
Sep 21 01:23:20 openqa kernel: vm_area_struct 4155KB 5608KB
Sep 21 01:23:20 openqa kernel: mm_struct 1601KB 1816KB
Sep 21 01:23:20 openqa kernel: files_cache 1205KB 1205KB
Sep 21 01:23:20 openqa kernel: signal_cache 86060KB 86060KB
Sep 21 01:23:20 openqa kernel: sighand_cache 1423KB 1705KB
Sep 21 01:23:20 openqa kernel: task_struct 634721KB 634721KB
Sep 21 01:23:20 openqa kernel: cred_jar 73485KB 73485KB
Sep 21 01:23:20 openqa kernel: anon_vma_chain 1317KB 1916KB
Sep 21 01:23:20 openqa kernel: anon_vma 1188KB 1671KB
Sep 21 01:23:20 openqa kernel: pid 154604KB 154604KB
Sep 21 01:23:20 openqa kernel: Acpi-Operand 165KB 165KB
Sep 21 01:23:20 openqa kernel: Acpi-ParseExt 7KB 7KB
Sep 21 01:23:20 openqa kernel: Acpi-State 71KB 71KB
Sep 21 01:23:20 openqa kernel: numa_policy 7KB 7KB
Sep 21 01:23:20 openqa kernel: perf_event 31KB 31KB
Sep 21 01:23:20 openqa kernel: trace_event_file 430KB 430KB
Sep 21 01:23:20 openqa kernel: ftrace_event_field 290KB 290KB
Sep 21 01:23:20 openqa kernel: pool_workqueue 1370KB 1376KB
Sep 21 01:23:20 openqa kernel: task_group 189KB 189KB
Sep 21 01:23:20 openqa kernel: vmap_area 301KB 732KB
Sep 21 01:23:20 openqa kernel: kmalloc-cg-8k 96KB 96KB
Sep 21 01:23:20 openqa kernel: kmalloc-cg-4k 544KB 544KB
Sep 21 01:23:20 openqa kernel: kmalloc-cg-2k 624KB 652KB
Sep 21 01:23:20 openqa kernel: kmalloc-cg-1k 1468KB 1740KB
Sep 21 01:23:20 openqa kernel: kmalloc-cg-512 2348KB 2348KB
Sep 21 01:23:20 openqa kernel: kmalloc-cg-256 96KB 96KB
Sep 21 01:23:20 openqa kernel: kmalloc-cg-192 315KB 315KB
Sep 21 01:23:20 openqa kernel: kmalloc-cg-96 47KB 47KB
Sep 21 01:23:20 openqa kernel: kmalloc-cg-64 104KB 104KB
Sep 21 01:23:20 openqa kernel: kmalloc-cg-32 48KB 48KB
Sep 21 01:23:20 openqa kernel: kmalloc-cg-16 2144KB 2144KB
Sep 21 01:23:20 openqa kernel: kmalloc-cg-8 3112KB 3112KB
Sep 21 01:23:20 openqa kernel: kmalloc-8k 12984KB 12984KB
Sep 21 01:23:20 openqa kernel: kmalloc-4k 33284KB 33292KB
Sep 21 01:23:20 openqa kernel: kmalloc-2k 34808KB 34808KB
Sep 21 01:23:20 openqa kernel: kmalloc-1k 250490KB 250716KB
Sep 21 01:23:20 openqa kernel: kmalloc-512 122796KB 124608KB
Sep 21 01:23:20 openqa kernel: kmalloc-256 103948KB 103948KB
Sep 21 01:23:20 openqa kernel: kmalloc-192 247859KB 247877KB
Sep 21 01:23:20 openqa kernel: kmalloc-128 5032KB 5032KB
Sep 21 01:23:20 openqa kernel: kmalloc-96 7508KB 7516KB
Sep 21 01:23:20 openqa kernel: kmalloc-64 75792KB 75816KB
Sep 21 01:23:20 openqa kernel: kmalloc-32 20752KB 20752KB
Sep 21 01:23:20 openqa kernel: kmalloc-16 19604KB 19604KB
Sep 21 01:23:20 openqa kernel: kmalloc-8 416KB 416KB
Sep 21 01:23:20 openqa kernel: kmem_cache_node 32KB 32KB
Sep 21 01:23:20 openqa kernel: kmem_cache 80KB 80KB
Sep 21 01:23:36 openqa kernel: Tasks state (memory values in pages):
Sep 21 01:23:36 openqa kernel: [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
Sep 21 01:23:36 openqa kernel: [ 523] 0 523 27415 0 237568 679 -1000 systemd-udevd
Sep 21 01:23:36 openqa kernel: [ 676] 0 676 13017 0 135168 164 0 rpc.idmapd
Sep 21 01:23:36 openqa kernel: [ 677] 0 677 15090 75 147456 197 -1000 auditd
Sep 21 01:23:36 openqa kernel: [ 682] 499 682 19866 481 184320 247 -900 dbus-daemon
Sep 21 01:23:36 openqa kernel: [ 31945] 0 693 28166 0 241664 0 0 close
Sep 21 01:23:36 openqa kernel: [ 696] 0 696 10708 3 122880 309 0 wickedd-auto4
Sep 21 01:23:36 openqa kernel: [ 701] 0 701 10709 3 131072 311 0 wickedd-dhcp4
Sep 21 01:23:36 openqa kernel: [ 1309] 0 1309 7763 0 102400 134 0 rsyncd
Sep 21 01:23:36 openqa kernel: [ 1319] 0 1319 6390 0 94208 127 0 vsftpd
Sep 21 01:23:36 openqa kernel: [ 1321] 0 1321 13501 1 139264 170 0 xinetd
Sep 21 01:23:36 openqa kernel: [ 1326] 493 1326 15982 0 176128 233 0 rpcbind
Sep 21 01:23:36 openqa kernel: [ 1334] 0 1334 18890 0 196608 267 -1000 sshd
Sep 21 01:23:36 openqa kernel: [ 1337] 475 1337 39077 1 208896 296 0 chronyd
Sep 21 01:23:36 openqa kernel: [ 1380] 497 1380 12618 0 143360 224 -500 nrpe
Sep 21 01:23:36 openqa kernel: [ 1502] 0 1502 11481 0 126976 190 0 agetty
Sep 21 01:23:36 openqa kernel: [ 1507] 0 1507 11481 0 131072 166 0 agetty
Sep 21 01:23:36 openqa kernel: [ 1617] 0 1617 24392 0 176128 258 0 master
Sep 21 01:23:36 openqa kernel: [ 1619] 51 1619 29041 0 192512 277 0 qmgr
Sep 21 01:23:36 openqa kernel: [ 1632] 0 1632 12792 0 147456 208 0 cron
Sep 21 01:23:36 openqa kernel: [ 32142] 30 32142 36929 0 315392 3 0 httpd-prefork
Sep 21 01:23:36 openqa kernel: [ 1640] 488 1640 12062 0 126976 0 0 lvm
Sep 21 01:23:36 openqa kernel: [ 4920] 0 4920 17730 1 172032 213 0 systemd-user-ru
Sep 21 01:23:36 openqa kernel: [ 20843] 0 20843 17730 1 188416 213 0 systemd-user-ru
Sep 21 01:23:36 openqa kernel: [ 25379] 0 25379 303441 0 712704 5 0 salt-minion
Sep 21 01:23:36 openqa kernel: [ 25829] 0 25829 19755 0 180224 0 0 (ogrotate)
Sep 21 01:23:36 openqa kernel: [ 27896] 488 27896 12062 0 114688 0 0 lvm
Sep 21 01:23:36 openqa kernel: [ 28021] 488 28021 12062 0 114688 0 0 lvm
Sep 21 01:23:36 openqa kernel: [ 28189] 488 28189 12062 0 122880 0 0 lvm
Sep 21 01:23:36 openqa kernel: [ 28218] 488 28218 12062 0 114688 0 0 lvm
Sep 21 01:23:36 openqa kernel: [ 28244] 488 28244 12062 0 118784 0 0 lvm
Sep 21 01:23:36 openqa kernel: [ 28562] 0 28562 10013 0 118784 0 0 lvm
Sep 21 01:23:36 openqa kernel: [ 28947] 0 28947 10013 0 110592 0 0 lvm
Sep 21 01:23:36 openqa kernel: [ 30535] 0 30535 17705 0 176128 211 0 systemd-user-ru
Sep 21 01:23:36 openqa kernel: [ 31770] 51 31770 23862 0 180224 240 0 pickup
Sep 21 01:23:37 openqa kernel: [ 31785] 0 31785 22647 0 221184 295 0 cron
Sep 21 01:23:37 openqa kernel: [ 31798] 0 31798 22647 0 221184 295 0 cron
Sep 21 01:23:37 openqa kernel: [ 31800] 0 31800 22647 0 221184 295 0 cron
Sep 21 01:23:37 openqa kernel: [ 31802] 0 31802 22647 0 221184 295 0 cron
Sep 21 01:23:37 openqa kernel: [ 31808] 0 31808 22647 0 221184 295 0 cron
Sep 21 01:23:37 openqa kernel: [ 31810] 0 31810 22647 0 221184 300 0 cron
Sep 21 01:23:37 openqa kernel: [ 31820] 0 31820 22647 0 221184 295 0 cron
Sep 21 01:23:37 openqa kernel: [ 31822] 0 31822 22647 0 221184 295 0 cron
Sep 21 01:23:37 openqa kernel: [ 31823] 0 31823 22647 0 221184 296 0 cron
Sep 21 01:23:37 openqa kernel: [ 31824] 0 31824 22647 0 221184 293 0 cron
Sep 21 01:23:37 openqa kernel: [ 31826] 0 31826 22647 0 221184 301 0 cron
Sep 21 01:23:37 openqa kernel: [ 31829] 0 31829 22647 0 221184 293 0 cron
Sep 21 01:23:37 openqa kernel: [ 31832] 0 31832 22647 0 221184 293 0 cron
Sep 21 01:23:37 openqa kernel: [ 31839] 0 31839 22647 0 221184 293 0 cron
Sep 21 01:23:37 openqa kernel: [ 31840] 0 31840 22647 0 221184 295 0 cron
Sep 21 01:23:37 openqa kernel: [ 31841] 0 31841 22647 0 221184 293 0 cron
Sep 21 01:23:37 openqa kernel: [ 31843] 0 31843 12792 0 139264 236 0 cron
Sep 21 01:23:37 openqa kernel: [ 31844] 0 31844 22647 0 221184 294 0 cron
Sep 21 01:23:37 openqa kernel: [ 31845] 0 31845 22647 0 221184 293 0 cron
Sep 21 01:23:37 openqa kernel: [ 31846] 0 31846 22647 0 221184 293 0 cron
Sep 21 01:23:37 openqa kernel: [ 31847] 0 31847 22647 0 221184 293 0 cron
Sep 21 01:23:37 openqa kernel: [ 31851] 0 31851 22647 0 221184 293 0 cron
Sep 21 01:23:37 openqa kernel: [ 31852] 0 31852 22647 0 221184 293 0 cron
Sep 21 01:23:37 openqa kernel: [ 31854] 0 31854 22647 0 221184 293 0 cron
Sep 21 01:23:37 openqa kernel: [ 31856] 0 31856 22647 0 221184 293 0 cron
Sep 21 01:23:37 openqa kernel: [ 31863] 0 31863 22647 0 221184 294 0 cron
Sep 21 01:23:37 openqa kernel: [ 31864] 0 31864 22647 0 221184 293 0 cron
Sep 21 01:23:37 openqa kernel: [ 31869] 0 31869 22647 0 221184 295 0 cron
Sep 21 01:23:37 openqa kernel: [ 31870] 0 31870 22647 0 221184 293 0 cron
Sep 21 01:23:37 openqa kernel: [ 32343] 0 32343 22647 0 221184 295 0 cron
Sep 21 01:23:37 openqa kernel: [ 32591] 0 32591 3929 1 77824 47 0 find
Sep 21 01:23:37 openqa kernel: [ 32754] 0 32754 22647 0 221184 304 0 cron
Sep 21 01:23:37 openqa kernel: [ 32757] 0 32757 22647 0 221184 303 0 cron
Sep 21 01:23:37 openqa kernel: [ 421] 0 421 22647 0 221184 303 0 cron
Sep 21 01:23:37 openqa kernel: [ 531] 0 531 3152 0 73728 73 0 reload-webserve
Sep 21 01:23:37 openqa kernel: [ 532] 0 532 17854 0 167936 240 0 systemctl
Sep 21 01:23:37 openqa kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=cron,pid=32754,uid=0
Sep 21 01:23:37 openqa kernel: Out of memory: Killed process 32754 (cron) total-vm:90588kB, anon-rss:0kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:216kB oom_score_adj:0
Sep 21 01:23:37 openqa kernel: oom_reaper: reaped process 32754 (cron), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Sep 21 01:23:37 openqa kernel: systemd invoked oom-killer: gfp_mask=0x40cc0(GFP_KERNEL|__GFP_COMP), order=2, oom_score_adj=0
Sep 21 01:23:37 openqa kernel: CPU: 11 PID: 1 Comm: systemd Tainted: G W L 5.14.21-150400.24.21-default #1 SLE15-SP4 7550826c4c7e8c258239e300508e0c8b2a69bad2
Sep 21 01:23:37 openqa kernel: Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
Sep 21 01:23:37 openqa kernel: Call Trace:
Sep 21 01:23:37 openqa kernel: <TASK>
Sep 21 01:23:37 openqa kernel: dump_stack_lvl+0x45/0x5b
Sep 21 01:23:37 openqa kernel: dump_header+0x4a/0x220
Sep 21 01:23:37 openqa kernel: oom_kill_process+0xe8/0x140
Sep 21 01:23:37 openqa kernel: out_of_memory+0x113/0x580
Sep 21 01:23:37 openqa kernel: __alloc_pages_slowpath.constprop.112+0x980/0xc70
Sep 21 01:23:37 openqa kernel: __alloc_pages+0x2d9/0x320
Sep 21 01:23:37 openqa kernel: new_slab+0x33e/0x500
Sep 21 01:23:37 openqa kernel: ___slab_alloc+0x6ad/0x860
Sep 21 01:23:37 openqa kernel: ? copy_process+0x213/0x1cd0
Sep 21 01:23:37 openqa kernel: ? __mod_memcg_lruvec_state+0x1d/0x100
Sep 21 01:23:37 openqa kernel: ? __mod_lruvec_page_state+0x6f/0xb0
Sep 21 01:23:37 openqa kernel: ? lock_page_memcg+0x1c/0xa0
Sep 21 01:23:37 openqa kernel: ? __slab_alloc.isra.92+0x1c/0x40
Sep 21 01:23:37 openqa kernel: __slab_alloc.isra.92+0x1c/0x40
Sep 21 01:23:37 openqa kernel: kmem_cache_alloc_node+0x114/0x310
Sep 21 01:23:37 openqa kernel: ? copy_process+0x213/0x1cd0
Sep 21 01:23:37 openqa kernel: copy_process+0x213/0x1cd0
Sep 21 01:23:37 openqa kernel: kernel_clone+0x9b/0x400
Sep 21 01:23:37 openqa kernel: __do_sys_clone+0x72/0xa0
Sep 21 01:23:37 openqa kernel: do_syscall_64+0x5b/0x80
Sep 21 01:23:37 openqa kernel: ? exc_page_fault+0x67/0x150
Sep 21 01:23:37 openqa kernel: entry_SYSCALL_64_after_hwframe+0x61/0xcb
Sep 21 01:23:37 openqa kernel: RIP: 0033:0x7f15c3718a68
Sep 21 01:23:37 openqa kernel: Code: Unable to access opcode bytes at RIP 0x7f15c3718a3e.
Sep 21 01:23:37 openqa kernel: RSP: 002b:00007ffc905fd240 EFLAGS: 00000246 ORIG_RAX: 0000000000000038
Sep 21 01:23:37 openqa kernel: RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f15c3718a68
Sep 21 01:23:37 openqa kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000001200011
Sep 21 01:23:37 openqa kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: 00007f15c5446b00
Sep 21 01:23:37 openqa kernel: R10: 00007f15c5446dd0 R11: 0000000000000246 R12: 0000556f45bd8720
Sep 21 01:23:37 openqa kernel: R13: 00007ffc905fd490 R14: 0000556f45cc9758 R15: 0000000000000000
Sep 21 01:23:37 openqa kernel: </TASK>
Sep 21 01:23:37 openqa kernel: Mem-Info:
Sep 21 01:23:37 openqa kernel: active_anon:2 inactive_anon:252 isolated_anon:0
active_file:2707 inactive_file:2658 isolated_file:0
unevictable:8 dirty:0 writeback:0
slab_reclaimable:4904595 slab_unreclaimable:1125807
mapped:422 shmem:3 pagetables:3197 bounce:0
free:79313 free_pcp:3745 free_cma:0
Sep 21 01:23:37 openqa kernel: Node 0 active_anon:8kB inactive_anon:1008kB active_file:10828kB inactive_file:10632kB unevictable:32kB isolated(anon):0kB isolated(file):0kB mapped:1688kB dirty:0kB writeback:0kB shmem:12kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB kernel_stack:4640kB pageta>
Sep 21 01:23:37 openqa kernel: Node 0 DMA free:11024kB boost:0kB min:40kB low:52kB high:64kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Sep 21 01:23:37 openqa kernel: lowmem_reserve[]: 0 2991 24013 24013 24013
Sep 21 01:23:37 openqa kernel: Node 0 DMA32 free:114856kB boost:0kB min:8356kB low:11392kB high:14428kB reserved_highatomic:14336KB active_anon:0kB inactive_anon:0kB active_file:4kB inactive_file:4kB unevictable:0kB writepending:0kB present:3129312kB managed:3063776kB mlocked:0kB bounce:0kB free_pcp:2404kB local_p>
Sep 21 01:23:37 openqa kernel: lowmem_reserve[]: 0 0 21021 21021 21021
Sep 21 01:23:37 openqa kernel: Node 0 Normal free:191372kB boost:0kB min:59184kB low:80700kB high:102216kB reserved_highatomic:20480KB active_anon:8kB inactive_anon:1128kB active_file:11088kB inactive_file:10728kB unevictable:32kB writepending:0kB present:22020096kB managed:21525736kB mlocked:32kB bounce:0kB free_>
Sep 21 01:23:37 openqa kernel: lowmem_reserve[]: 0 0 0 0 0
Sep 21 01:23:37 openqa kernel: Node 0 DMA: 0*4kB 0*8kB 1*16kB (U) 2*32kB (UE) 1*64kB (U) 1*128kB (U) 2*256kB (UE) 2*512kB (UE) 1*1024kB (E) 2*2048kB (ME) 1*4096kB (M) = 11024kB
Sep 21 01:23:37 openqa kernel: Node 0 DMA32: 7930*4kB (UM) 10420*8kB (UME) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 115080kB
Sep 21 01:23:37 openqa kernel: Node 0 Normal: 46158*4kB (UME) 902*8kB (UME) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 191848kB
Sep 21 01:23:37 openqa kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Sep 21 01:23:37 openqa kernel: 5555 total pagecache pages
Sep 21 01:23:37 openqa kernel: 113 pages in swap cache
Sep 21 01:23:37 openqa kernel: Swap cache stats: add 11843566, delete 11843351, find 99242001/104794745
Sep 21 01:23:37 openqa kernel: Free swap = 451972kB
Sep 21 01:23:37 openqa kernel: Total swap = 499708kB
Sep 21 01:23:37 openqa kernel: 6291350 pages RAM
Sep 21 01:23:37 openqa kernel: 0 pages HighMem/MovableOnly
Sep 21 01:23:37 openqa kernel: 140132 pages reserved
Sep 21 01:23:37 openqa kernel: 0 pages cma reserved
Sep 21 01:23:37 openqa kernel: 0 pages hwpoisoned
Sep 21 01:23:37 openqa kernel: Unreclaimable slab info:
Sep 21 01:23:37 openqa kernel: Name Used Total
Sep 21 01:23:37 openqa kernel: xfs_icr 47KB 47KB
Sep 21 01:23:37 openqa kernel: xfs_efi_item 392KB 410KB
Sep 21 01:23:37 openqa kernel: xfs_buf_item 447KB 447KB
Sep 21 01:23:37 openqa kernel: xfs_da_state 191KB 191KB
Sep 21 01:23:37 openqa kernel: xfs_btree_cur 94KB 94KB
Sep 21 01:23:37 openqa kernel: nfsd4_openowners 372KB 391KB
Sep 21 01:23:37 openqa kernel: nfsd4_clients 127KB 127KB
Sep 21 01:23:37 openqa kernel: scsi_sense_cache 4KB 4KB
Sep 21 01:23:37 openqa kernel: ip6-frags 47KB 47KB
Sep 21 01:23:37 openqa kernel: PINGv6 370KB 370KB
Sep 21 01:23:37 openqa kernel: RAWv6 8148KB 8148KB
Sep 21 01:23:37 openqa kernel: UDPv6 3374KB 3374KB
Sep 21 01:23:37 openqa kernel: tw_sock_TCPv6 2086KB 2157KB
Sep 21 01:23:37 openqa kernel: request_sock_TCPv6 972KB 972KB
Sep 21 01:23:37 openqa kernel: TCPv6 3686KB 4519KB
Sep 21 01:23:37 openqa kernel: bfq_io_cq 95KB 95KB
Sep 21 01:23:37 openqa kernel: mqueue_inode_cache 496KB 496KB
Sep 21 01:23:37 openqa systemd-journald[563]: Journal stopped
-- Boot f3b32b3cd2f04d3ca0f816236391c0c0 --
Sep 21 01:24:38 openqa kernel: Linux version 5.14.21-150400.24.21-default (geeko@buildhost) (gcc (SUSE Linux) 7.5.0, GNU ld (GNU Binutils; SUSE Linux Enterprise 15) 2.37.20211103-150100.7.37) #1 SMP PREEMPT_DYNAMIC Wed Sep 7 06:51:18 UTC 2022 (974d0aa)
Updated by okurz over 2 years ago
Both MRs merged to try to prevent further crashes.
As it's hard to investigate the system journal within OSD remotely I copied over the system journal based on nsinger's suggestion with running on OSD:
journalctl | grep -v 'openqa openqa' | nc -l -p 42423
and locally
nc openqa.suse.de 42423 | pv > osd_journal_$(date +%F).log
The journal starts at Sep 08 17:45:01 so 10 days before the problems started on Sep 18.
So I looked for soft lockups just to see if we are actually following red herrings or not.
The first occurence of the search term "soft lockup" is from Sep 18 06:41:46 so exactly when the problems started, not in before:
1202771 Sep 18 06:41:21 openqa kernel: BUG: workqueue leaked lock or atomic: kworker/u21:2/0x00000001/19034
1202772 last function: xs_error_handle [sunrpc]
1202773 Sep 18 06:41:21 openqa kernel: CPU: 2 PID: 19034 Comm: kworker/u21:2 Not tainted 5.14.21-150400.24.21-default #1 SLE15-SP4 7550826c4c7e8c258239e300508e0c8b2a69bad2
1202774 Sep 18 06:41:21 openqa kernel: Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
1202775 Sep 18 06:41:21 openqa kernel: Workqueue: xprtiod xs_error_handle [sunrpc]
1202776 Sep 18 06:41:21 openqa kernel: Call Trace:
1202777 Sep 18 06:41:21 openqa kernel: <TASK>
1202778 Sep 18 06:41:21 openqa kernel: dump_stack_lvl+0x45/0x5b
1202779 Sep 18 06:41:21 openqa kernel: process_one_work+0x390/0x440
1202780 Sep 18 06:41:21 openqa kernel: worker_thread+0x2d/0x3d0
1202781 Sep 18 06:41:21 openqa kernel: ? process_one_work+0x440/0x440
1202782 Sep 18 06:41:21 openqa kernel: kthread+0x156/0x180
1202783 Sep 18 06:41:21 openqa kernel: ? set_kthread_struct+0x50/0x50
1202784 Sep 18 06:41:21 openqa kernel: ret_from_fork+0x22/0x30
1202785 Sep 18 06:41:21 openqa kernel: </TASK>
1202786 Sep 18 06:41:21 openqa kernel: BUG: scheduling while atomic: kworker/u21:2/19034/0x00000002
1202787 Sep 18 06:41:21 openqa kernel: Modules linked in: dm_mod iscsi_ibft iscsi_boot_sysfs rfkill loop xfs libcrc32c kvm_amd ccp kvm virtio_net net_failover virtio_balloon failover irqbypass joydev i2c_piix4 pcspkr button nfsd auth_rpcgss nfs_acl lockd grace fuse sunrpc configfs ip _tables x_tables ext4 crc16 mbcache jbd2 drm_kms_helper ata_generic syscopyarea sysfillrect sysimgblt serio_raw fb_sys_fops cec ata_piix rc_core ahci uhci_hcd libahci ehci_hcd drm libata usbcore virtio_blk floppy qemu_fw_cfg sg scsi_mod
1202788 Sep 18 06:41:21 openqa kernel: Supported: Yes
1202789 Sep 18 06:41:21 openqa kernel: CPU: 2 PID: 19034 Comm: kworker/u21:2 Not tainted 5.14.21-150400.24.21-default #1 SLE15-SP4 7550826c4c7e8c258239e300508e0c8b2a69bad2
1202790 Sep 18 06:41:21 openqa kernel: Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
1202791 Sep 18 06:41:21 openqa kernel: Workqueue: 0x0 (xprtiod)
1202792 Sep 18 06:41:21 openqa kernel: Call Trace:
1202793 Sep 18 06:41:21 openqa kernel: <TASK>
1202794 Sep 18 06:41:21 openqa kernel: dump_stack_lvl+0x45/0x5b
1202795 Sep 18 06:41:21 openqa kernel: __schedule_bug+0x52/0x70
1202796 Sep 18 06:41:21 openqa kernel: __schedule+0xdc4/0x1140
1202797 Sep 18 06:41:21 openqa kernel: ? arch_local_irq_enable+0x7/0xc
1202798 Sep 18 06:41:21 openqa kernel: schedule+0x64/0xe0
1202799 Sep 18 06:41:21 openqa kernel: worker_thread+0xab/0x3d0
1202800 Sep 18 06:41:21 openqa kernel: ? process_one_work+0x440/0x440
1202801 Sep 18 06:41:21 openqa kernel: kthread+0x156/0x180
1202802 Sep 18 06:41:21 openqa kernel: ? set_kthread_struct+0x50/0x50
1202803 Sep 18 06:41:21 openqa kernel: ret_from_fork+0x22/0x30
1202804 Sep 18 06:41:21 openqa kernel: </TASK>
1202805 Sep 18 06:41:46 openqa kernel: watchdog: BUG: soft lockup - CPU#2 stuck for 26s! [kworker/2:2:24024]
the messages continue with an increasing lock time until postgres runs out of memory and other processes start to fail:
Sep 18 06:59:34 openqa kernel: Out of memory: Killed process 17018 (postgres) total-vm:3418572kB, anon-rss:7224kB, file-rss:1252kB, shmem-rss:2749876kB, UID:26 pgtables:6644kB oom_score_adj:0
The command grep '\<BUG: workqueue leaked lock or atomic' osd_journal_2022-09-21.log reveals
Sep 18 06:41:21 openqa kernel: BUG: workqueue leaked lock or atomic: kworker/u21:2/0x00000001/19034
Sep 20 00:50:17 openqa kernel: BUG: workqueue leaked lock or atomic: kworker/u21:2/0x00000001/19901
Sep 20 23:12:08 openqa kernel: BUG: workqueue leaked lock or atomic: kworker/u25:0/0x00000001/32698
so three times coinciding with "Ouf of memory" messages each time after that. grep 'Linux version' osd_journal_2022-09-21.log shows
Sep 11 03:32:06 openqa kernel: Linux version 5.14.21-150400.24.18-default (geeko@buildhost) (gcc (SUSE Linux) 7.5.0, GNU ld (GNU Binutils; SUSE Linux Enterprise 15) 2.37.20211103-150100.7.37) #1 SMP PREEMPT_DYNAMIC Thu Aug 4 14:17:48 UTC 2022 (e9f7bfc)
Sep 18 03:31:56 openqa kernel: Linux version 5.14.21-150400.24.21-default (geeko@buildhost) (gcc (SUSE Linux) 7.5.0, GNU ld (GNU Binutils; SUSE Linux Enterprise 15) 2.37.20211103-150100.7.37) #1 SMP PREEMPT_DYNAMIC Wed Sep 7 06:51:18 UTC 2022 (974d0aa)
The command grep '|install|' /var/log/zypp/history | grep '^2022-09' showing all packages that have been installed after the previous boot of the machine that would explain such problems:
2022-09-11 03:00:39|install|at-spi2-atk-common|2.38.0-150400.1.9|x86_64||repo-oss|e6a37e9b6b56e2b81fbd83ab3c7c024f05b7cc0b81671f5ac7a34bde077a8f5a|
2022-09-11 03:00:39|install|gtk2-data|2.24.33-150400.2.11|noarch||repo-oss|25a3a0ce770db21bf855fd865f162674f9c5b9c38791dc34298ae276ac8e0219|
2022-09-11 03:00:39|install|at-spi2-atk-gtk2|2.38.0-150400.1.9|x86_64||repo-oss|df797cd4235009b22045becf87537dc298c0a5a0f5d8972d8767ab426284ccd3|
2022-09-11 03:00:40|install|gtk2-tools|2.24.33-150400.2.11|x86_64||repo-oss|8ba113c7c0474b558617d53436be6c3b6a8061904cbc4cc8b4ffeb65a0801704|
2022-09-11 03:00:41|install|libgtk-2_0-0|2.24.33-150400.2.11|x86_64||repo-oss|445e7903b06c04ee62afedb5b44e988665b49e24caac5b8a89425600469ec0a2|
2022-09-11 03:00:41|install|gtk2-theming-engine-adwaita|3.22.3-4.3.1|x86_64||repo-oss|abccd6f46cee5525001278cdf869041d5a1a036aeee77919af1a3adeaf62ee95|
2022-09-11 03:00:41|install|gtk2-metatheme-adwaita|3.22.3-4.3.1|noarch||repo-oss|6b42319f73bea154f8513e659dc8b82808847a9161e2f0e34f8847aab6cc39d6|
2022-09-11 03:00:41|install|gtk2-branding-openSUSE|15.0-lp154.1.10|noarch||repo-oss|341b3e4437ec2310b0f53eb0b7a8d6ea3fd966465617639fbb89ba5b62d4f7b9|
2022-09-11 03:00:41|install|gtk2-immodule-amharic|2.24.33-150400.2.11|x86_64||repo-oss|5ce87e02c7da4f8c4e2c79a7e70c5933e5fe1d424d31582e6472e91ed9be4fd4|
2022-09-11 03:00:41|install|gtk2-immodule-inuktitut|2.24.33-150400.2.11|x86_64||repo-oss|da44b8d40307ee1c54d01575b5974d0686b18e4fc4a5883402be388ef373ba60|
2022-09-11 03:00:41|install|gtk2-immodule-thai|2.24.33-150400.2.11|x86_64||repo-oss|f8461a710ef69dade2b9630ababa60a33c0b4f5e5420d8a42f6bd048c9e44b4e|
2022-09-11 03:00:42|install|gtk2-immodule-tigrigna|2.24.33-150400.2.11|x86_64||repo-oss|44e39f874f3ccd4b8f7a6a2ca5e39fbb56c115501eb4adf12f236036ce70411f|
2022-09-11 03:00:42|install|gtk2-immodule-vietnamese|2.24.33-150400.2.11|x86_64||repo-oss|d0802b1bd69874b751860ff8300484d2f4256e1b6b8a13d9bc1bdcd2eae2eaca|
2022-09-11 03:00:42|install|openssh-askpass-gnome|8.4p1-3.9.1|x86_64||repo-oss|6125fb63447542287b2c22b2f68aff2ea49aae1c6d5f3597cc8b449a13ccb062|
2022-09-11 03:00:43|install|gtk2-lang|2.24.33-150400.2.11|noarch||repo-oss|e5bd7f648b90053bb74d44a199aa49c0698b604eec288fdbb0e20c5cc770ac9e|
2022-09-11 03:00:44|install|bash-completion|2.7-150000.4.9.1|noarch||repo-sle-update|25312061f297a8dd2bfef632cbf37847f13eb387576840506882aeedfa33b56e|
2022-09-11 03:00:44|install|bash-completion-devel|2.7-150000.4.9.1|noarch||repo-sle-update|068c7b566b6cfa4c90ac77b2ed34066797bcbcf014c78bbc100171bca294f797|
2022-09-11 03:00:45|install|bash-completion-doc|2.7-150000.4.9.1|noarch||repo-sle-update|0d90841a2d1fa0c4919eb7cbb689a981ad8ac5affb1e32051b03c6c71ef5bbc8|
2022-09-11 03:00:46|install|libgtk-3-0|3.24.34-150400.3.3.1|x86_64||repo-sle-update|52213422d40001d8d9ede45bdf20d2023939d54c6e05a2dbc58931a065c91301|
2022-09-11 03:00:46|install|gtk3-immodule-amharic|3.24.34-150400.3.3.1|x86_64||repo-sle-update|9ea6bec0d97979336d3dc84e9961d3cc1539029c42c774f5391d528a1ba54687|
2022-09-11 03:00:47|install|rpm|4.14.3-150300.49.1|x86_64||repo-sle-update|a4e8c65cb33a0bcd7bc9d33bcc052dde29ef85b0c57101284adba0c43d38771d|
2022-09-11 03:00:48|install|sysconfig|0.85.9-150200.12.1|x86_64||repo-sle-update|8d5ac96a9d6fd5a107156091ac572c31b2df4b620bf767c2b1c6109d19f3ff27|
2022-09-11 03:00:48|install|sysconfig-netconfig|0.85.9-150200.12.1|x86_64||repo-sle-update|47928d89fbef1cf3262c8d12600da08ed8424ea8854c8305e23da8fca481247d|
2022-09-11 03:00:48|install|vim-data-common|9.0.0313-150000.5.25.1|noarch||repo-sle-update|47aef9d6e8835e6acefc1a7bc4b731453207a264b27d7f7fd248cd526fd63cf1|
2022-09-11 03:00:51|install|vim-data|9.0.0313-150000.5.25.1|noarch||repo-sle-update|6b2ca45af22e5fa4f8ee63edf4ee4f097bc9ce72d43c600c2e274590d14ff496|
2022-09-11 03:00:52|install|vim|9.0.0313-150000.5.25.1|x86_64||repo-sle-update|7a5f228f5848cceddccb5896687c1591465b575ede1b7ae185273c9908785ba5|
2022-09-11 03:00:52|install|gtk3-immodule-inuktitut|3.24.34-150400.3.3.1|x86_64||repo-sle-update|f6068675daaf8ca33e203b73950332389d7990ed20b75b1ae8ce2f328e7e07a7|
2022-09-11 03:00:52|install|rpm-build|4.14.3-150300.49.1|x86_64||repo-sle-update|a1ebe5cb059cabdfff4d1cdca376a841db73bb07b6137566705c6707a44d5850|
2022-09-11 03:00:52|install|python3-rpm|4.14.3-150300.49.1|x86_64||repo-sle-update|e707d51a8c72556e8fe2fa5154cc9d325683113607cef5c98344bafa35166c85|
2022-09-11 03:00:53|install|gtk3-immodule-thai|3.24.34-150400.3.3.1|x86_64||repo-sle-update|f2d70c54bf994ec3747f27c846ac6c42d256048ba1aaeaacf34367f667fa2baa|
2022-09-11 03:00:53|install|gtk3-immodule-tigrigna|3.24.34-150400.3.3.1|x86_64||repo-sle-update|1fa97d985e49b259b40fb44190b2fc2ff97137ae121a9c7044c390cb835f99f3|
2022-09-11 03:00:54|install|gtk3-immodule-vietnamese|3.24.34-150400.3.3.1|x86_64||repo-sle-update|8a17bb6cd163a01d147c6147faac4e740d1c6ea6c9cf19369c6f3866c9d53c96|
2022-09-11 03:00:54|install|gtk3-branding-openSUSE|15.0-lp154.2.4.1|noarch||repo-update|5e0c8257d182fd0fe4d9deca5df9602964a7c2b4fb84c81462fde34da0aaad58|
2022-09-11 03:00:56|install|gtk3-lang|3.24.34-150400.3.3.1|noarch||repo-sle-update|bce829efe2a150e8d2eaf75878e55a8d5ec1c57365e1dfecb1d4d39681f73eea|
2022-09-12 13:02:03|install|perl-YAML-LibYAML|0.84-lp154.2.1|x86_64||devel_openQA_Modules|c35c6b0cca14e33e1b3b42d95059f9b738a0be364528dd4118a721fe908c4a31|
2022-09-13 03:02:39|install|cups-config|2.2.7-150000.3.35.1|x86_64||repo-sle-update|a92c1a6a82493c183d370a2edf82600bbef8d939139172cd6b1f87ce3416f53a|
2022-09-13 03:02:39|install|libbluetooth3|5.62-150400.4.5.1|x86_64||repo-sle-update|a6f0e8e753b38bbb17914c8ab1e8a1047dee04345e134d9ea3fc02bd6b90a22d|
2022-09-13 03:02:40|install|libfreetype6|2.10.4-150000.4.12.1|x86_64||repo-sle-update|d8f9f0d0ebcac901fec06c865924b6bd3ae002ba588e87a745c5228dc021844c|
2022-09-13 03:02:41|install|libcups2|2.2.7-150000.3.35.1|x86_64||repo-sle-update|3519dae9ea1b6a794d6113d9f26e8fd107b72dbf3ae9db8b4b7d2cb899668973|
2022-09-13 03:02:43|install|samba-client-libs|4.15.8+git.527.8d0c05d313e-150400.3.14.1|x86_64||repo-sle-update|5227bed1cfea85b53f16019b206ec28bcb96669bd606d7b601ab35ea0c184f90|
2022-09-13 03:02:45|install|samba-client|4.15.8+git.527.8d0c05d313e-150400.3.14.1|x86_64||repo-sle-update|105386882c31c15156a382604433e101be9426d5e716eaa3d8def205d26a3276|
2022-09-13 03:02:45|install|samba-libs-python3|4.15.8+git.527.8d0c05d313e-150400.3.14.1|x86_64||repo-sle-update|7f3d548ba903b0a2773ccebfea045ed422a6973cdfbc27f8eb23962e6a36813e|
2022-09-13 03:02:46|install|samba-libs|4.15.8+git.527.8d0c05d313e-150400.3.14.1|x86_64||repo-sle-update|da1b0710efb6f2ee611d9153a248ffac0e2791808fbf95279b3255c080f5fd10|
2022-09-13 03:02:52|install|samba|4.15.8+git.527.8d0c05d313e-150400.3.14.1|x86_64||repo-sle-update|ea800fd67b37a8f558a7e5f8267ffd2afae5c8de81e33100b0d2c93e7bfc7002|
2022-09-13 03:02:53|install|samba-ad-dc-libs|4.15.8+git.527.8d0c05d313e-150400.3.14.1|x86_64||repo-sle-update|544308cafee8a2c96fb81ddf3b71819556eb172656b3d42976786edfaa98070c|
2022-09-13 03:02:53|install|libsamba-policy0-python3|4.15.8+git.527.8d0c05d313e-150400.3.14.1|x86_64||repo-sle-update|3549cdf00edcfa5555beb4b9268f5d8ab675b5e1b3c9f6ffe2fdbd4fc33b957b|
2022-09-13 03:02:56|install|samba-python3|4.15.8+git.527.8d0c05d313e-150400.3.14.1|x86_64||repo-sle-update|178fafc76d634a1fa5f56c43f913ef8a7fc44c68d1b392fabe9d9d16a61cbbb5|
2022-09-13 06:46:08|install|p7zip|16.02-14.7.1|x86_64||repo-oss|f971403abaf6b85990da2e70c8d0ac823ce3afde6ea7ce6ad4cf4b5e0a8e4a11|
2022-09-13 06:46:08|install|p7zip-full|16.02-14.7.1|x86_64|root@openqa|repo-oss|2867f8f765219c16b585ddc3e88144f2267fbb360a44727ec7310d7f1f692e92|
2022-09-13 07:51:55|install|openQA-common|4.6.1662979895.bd3bc01-lp154.5235.1|noarch||devel_openQA|000b2a08cd5b374b51ec95cce7d8655ec887c188eaa3533d4323e3cf9c7d5c12|
2022-09-13 07:51:55|install|openQA-client|4.6.1662979895.bd3bc01-lp154.5235.1|noarch||devel_openQA|3e9f09708bc4cb4a76965a0090e270c9cc05529188ff96fdb63c99ee99006b58|
2022-09-13 07:52:08|install|openQA|4.6.1662979895.bd3bc01-lp154.5235.1|noarch||devel_openQA|fc467e51f652c2469f54c67b71275435d90cf79fb1f4eb34d3609d3801b553ed|
2022-09-13 07:52:08|install|openQA-doc|4.6.1662979895.bd3bc01-lp154.5235.1|noarch||devel_openQA|479391cc86fe18c565b4cfe8565c78b330eb643579e176f9dd26999252c6dc5d|
2022-09-14 03:01:03|install|libatomic1|11.3.0+git1637-150000.1.11.2|x86_64||repo-sle-update|78979c41e689c1436214e3aa41640cd5211c32b92beb5406be23909405851546|
2022-09-14 03:01:04|install|libgcc_s1|11.3.0+git1637-150000.1.11.2|x86_64||repo-sle-update|c7e826b9aa4264d613e70b52049a0698b3419369d31ea815872d3fc115b26a8c|
2022-09-14 03:01:04|install|libgomp1|11.3.0+git1637-150000.1.11.2|x86_64||repo-sle-update|fd2bea4a7ce9aa2a430e79cc7ed8c40fa311b2d87b0bbd52fa4de93784bf0749|
2022-09-14 03:01:04|install|libitm1|11.3.0+git1637-150000.1.11.2|x86_64||repo-sle-update|3ec0e3c6b82ddaaf04f3186d1b4e0b8988c037de3c23dcf2493f19ad3c5fb8c1|
2022-09-14 03:01:05|install|libquadmath0|11.3.0+git1637-150000.1.11.2|x86_64||repo-sle-update|77a59b95d89aba353181a67415a278afcdcc7ce77a2b8261d6130e27ba6da6f1|
2022-09-14 03:01:05|install|libstdc++6|11.3.0+git1637-150000.1.11.2|x86_64||repo-sle-update|ba0a6150183d657f41faf068fb52e2e2a0655b687e7a0441b08faab732082518|
2022-09-14 03:01:06|install|libgfortran5|11.3.0+git1637-150000.1.11.2|x86_64||repo-sle-update|9983e20ab28dd46eea658e58d3d2e6bd3f1c963a34acbddab6838ca002a80f85|
2022-09-14 03:01:06|install|libtsan0|11.3.0+git1637-150000.1.11.2|x86_64||repo-sle-update|ab31ae003ac457685327c115ef61ea8c6307acd077a8e14c2dcf47586b84794c|
2022-09-14 03:01:07|install|libstdc++6-pp-gcc11|11.3.0+git1637-150000.1.11.2|x86_64||repo-sle-update|bf88737ed6749921bb88f0d336abb017d230b88ca260912653739af54ac7c64c|
2022-09-14 03:01:07|install|liblsan0|11.3.0+git1637-150000.1.11.2|x86_64||repo-sle-update|96b1e5e4a953bbd2babc4f5f99576b50b6fe3b9be92a6f1cbb7e0bb75a56f183|
2022-09-15 03:02:31|install|perl-base|5.26.1-150300.17.11.1|x86_64||repo-sle-update|9035f8fb681ddde40e1524cc2adb6f386c5629bff44f6729fd9453cb8d142e9f|
2022-09-15 03:02:37|install|perl|5.26.1-150300.17.11.1|x86_64||repo-sle-update|e4f3ac9c1716596d0ee4dba3dd793bbe3c2202c33faf172d91b67ff1fcc83f5b|
2022-09-15 03:02:37|install|perl-core-DB_File|5.26.1-150300.17.11.1|x86_64||repo-sle-update|d8bf6b3304d36c3db452c3f7155bae15dbf4ff902db2983797e418432d29f924|
2022-09-15 07:17:36|install|openQA-common|4.6.1662979895.bd3bc01-lp154.5236.1|noarch||devel_openQA|0237f3d6779791dd1176c1875d4a017c42a10d5e35b337a050075a92bcbddc23|
2022-09-15 07:17:37|install|openQA-doc|4.6.1662979895.bd3bc01-lp154.5236.1|noarch||devel_openQA|703c3695c52465fba66d575c95e1cf9eeffc78acf0ad89e0ad1e7a3dcf6179ab|
2022-09-15 07:17:37|install|openQA-client|4.6.1662979895.bd3bc01-lp154.5236.1|noarch||devel_openQA|7262f62b724685c6066bfdb22b41984645b06162f934087a2d39096934f1f820|
2022-09-15 07:17:49|install|openQA|4.6.1662979895.bd3bc01-lp154.5236.1|noarch||devel_openQA|86acf4205ae255c7f5a2dbbd48144abce08c08b814b65f72a1da345ddb76828b|
2022-09-15 07:17:50|install|python3-typing_extensions|3.10.0.0-150400.3.2.1|noarch||repo-sle-update|20be425c3cd3596a7747bc92ff151dd0de5f9883fb26499808a69749239a9dc4|
2022-09-16 03:00:59|install|libgit2-1_3|1.3.0-150400.3.3.1|x86_64||repo-sle-update|676f509046a13f14b3e3606b469ea46b327be38109c32926f10bf3eb1bf18d2e|
2022-09-16 03:01:03|install|yast2-storage-ng|4.4.40-150400.3.3.1|x86_64||repo-sle-update|c7e0e8922c4212ccc6c62e4a406017c39707c63911f71278ecf326c28532ea6d|
2022-09-17 03:04:25|install|kernel-default|5.14.21-150400.24.21.2|x86_64||repo-sle-update|3b2bfeafc2944dc633ce2531b713395b40c1fce0aa018bc6736ec04a564f75ac|
2022-09-17 03:04:26|install|libruby2_5-2_5|2.5.9-150000.4.26.1|x86_64||repo-sle-update|2f3dc29b4ed67a3e0dab9848321fdede20f91bc0c8e16a96a9e42756fd92b9b6|
2022-09-17 03:04:35|install|ruby2.5-stdlib|2.5.9-150000.4.26.1|x86_64||repo-sle-update|c0af9bcc01a1b7116bb280c9bc6e97e4f39e3842a951742440ea802d65748c55|
2022-09-17 03:04:36|install|ruby2.5|2.5.9-150000.4.26.1|x86_64||repo-sle-update|1816b75a0ffd933ca11c8579d30029ded432c98c236713043eee9b1bdb66c8d2|
2022-09-17 03:05:07|install|kernel-default-extra|5.14.21-150400.24.21.2|x86_64||repo-sle-update|c91ba92cdf7e58ed5a2cb36d23a247389c34b403f2a241ccdf974ddd894dd913|
2022-09-17 03:05:24|install|kernel-default-optional|5.14.21-150400.24.21.2|x86_64||repo-sle-update|11f8f2f998cac335cc1ff102d6b26e2a9f1d81016721178c8568b1130897e382|
I consider it very likely that only the kernel update 5.14.21-150400.24.18-default -> 5.14.21-150400.24.21-default could explain this behaviour.
I suggest to report a product issue and downgrade the kernel version. I called
zypper rm kernel-default-5.14.21-150400.24.21 && zypper al kernel-default && reboot
now the old kernel is running. I reported https://bugzilla.opensuse.org/show_bug.cgi?id=1203630 . Let's await further results.
Updated by mkittler over 2 years ago
In the end dbux-daemon is the process with the highest virtual memory usage. However, it is only 481 pages, that should only be 1.87890625 MiB which is not very concerning.
The process with the highest resident set size (physical memory occupied by the process so not swapped out, not mmapped, …) is salt-minion. Its RSS is significantly higher at than the RSS of any other process with 712704. That should be 2.71875 GiB. On normal operation a PostgreSQL process leads that score (still below 1 GiB then). So that's quite a lot for salt. It is also strange that at this point no virtual memory is assigned to the process anymore (same counts also for many other processes).
Maybe the statistics at this point aren't that useful anymore. At an earlier time (Sep 21 00:02:26 but the OOM killer was already active and logging stats) the openqa web UI processes were leading the stats of virtual memory use (and were therefore also being killed by the OOM killer). However, they were still only at around 347.1640625 MiB use of VM each (that exact value is the highest value logged on Sep 21 00:02:26) which is quite normal. The RSS is quite big already at 2,78125 GiB (again the highest value at the time). I'm wondering whether I'm reading the stats correctly. It is still strange the the VM is less than the RSS. Most likely the VM stat is excluding the RSS in these tables. The process with the highest RSS at the time was telegraf with 3.859375 GiB. That's a lot although it is normally also leading that score (e.g. 875 MiB at the time I'm writing this). Some PostgreSQL processes were also quite high with up to 3.140625 GiB. The same counts for openQA process with up to 2.828125 GiB. When adding RSS and VM together than telegraf uses most with 3.94922256469727 GiB, then comes PostgreSQL with 3.15000534057617 GiB, then many openqa processes with up to 2.97131729125977 GiB and some more PosgreSQL processes and salt.
It is hard to pick a culprit. All salt process together use 22.038646697998 GiB of VM + RSS. For PostgreSQL it is 51.2796745300293 GiB and for openqa processes it is 78.4090690612793 GiB. Nothing should be so high obviously.
So maybe it is indeed the kernel misbehaving. It would be quite strange that many different services (using different programming languages) show the same symptom.
Updated by okurz over 2 years ago
No problem since yesterday, OSD is stable. I will revert the postgres work_mem changes. I doubt the have an impact.
https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/739
Updated by okurz about 2 years ago
- Subject changed from openqa.suse.de is not reachable 2022-09-18, no ping response, postgreSQL OOM and kernel panics to openqa.suse.de is not reachable 2022-09-18, no ping response, postgreSQL OOM and kernel panics size:M
- Description updated (diff)
Updated by okurz about 2 years ago
- Due date changed from 2022-10-03 to 2022-10-28
- Status changed from In Progress to Feedback
- Priority changed from Urgent to High
I took a look on o3 which has a journal going back to 2022-08-08 and journalctl | grep '\<BUG: workqueue leaked lock or atomic' returned no matches. https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/739 was merged and I checked also manually that we are back with work_mem = 64MB within postgres config. OSD is up since 1 day, 17:51. I will observe the situation over the next days/weeks and await either further problems or responses in the bug report. If there is no further crash nor response after weeks we can try a more recent kernel version if there is one in Leap or maybe the most recent vanilla version or Kernel:stable or Kernel:HEAD
Updated by jbaier_cz about 2 years ago
We have another episode, maybe the wrong kernel was re-installed and loaded during the auto-update/restart over the weekend?
This time, monitoring does not show the same symptoms like in the first case (i.e. no increased CPU or I/O activity).
Also it did not reboot itself, mkittler created SD-99619 to recover.
Updated by mkittler about 2 years ago
The VM has been recreated. Considering the journal it is indeed the same problem again:
Sep 26 15:47:53 openqa kernel: BUG: workqueue leaked lock or atomic: kworker/u25:1/0x00000001/12426
last function: xs_error_handle [sunrpc]
Sep 26 15:47:53 openqa kernel: CPU: 4 PID: 12426 Comm: kworker/u25:1 Not tainted 5.14.21-150400.24.21-default #1 SLE15-SP4 7550826c4c7e8c258239e300508e0c8b2a69bad2
Sep 26 15:47:53 openqa kernel: Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
Sep 26 15:47:53 openqa kernel: Workqueue: xprtiod xs_error_handle [sunrpc]
Sep 26 15:47:53 openqa kernel: Call Trace:
Sep 26 15:47:53 openqa kernel: <TASK>
Sep 26 15:47:53 openqa kernel: dump_stack_lvl+0x45/0x5b
Sep 26 15:47:53 openqa kernel: process_one_work+0x390/0x440
Sep 26 15:47:53 openqa kernel: worker_thread+0x2d/0x3d0
Sep 26 15:47:53 openqa kernel: ? process_one_work+0x440/0x440
Sep 26 15:47:53 openqa kernel: kthread+0x156/0x180
Sep 26 15:47:53 openqa kernel: ? set_kthread_struct+0x50/0x50
Sep 26 15:47:53 openqa kernel: ret_from_fork+0x22/0x30
Sep 26 15:47:53 openqa kernel: </TASK>
Sep 26 15:47:53 openqa kernel: BUG: scheduling while atomic: kworker/u25:1/12426/0x00000002
Sep 26 15:47:53 openqa kernel: Modules linked in: iscsi_ibft iscsi_boot_sysfs rfkill loop xfs libcrc32c virtio_net net_failover failover virtio_balloon button nfsd auth_rpcgss nfs_acl lockd grace fuse drm sunrpc configfs ip_tables x_tables ext4 crc16 mbcache jbd2 uhci_hcd ata_piix ehci_hcd ahci libahci usbcore lib>
Sep 26 15:47:53 openqa kernel: Supported: Yes
Sep 26 15:47:53 openqa kernel: CPU: 4 PID: 12426 Comm: kworker/u25:1 Not tainted 5.14.21-150400.24.21-default #1 SLE15-SP4 7550826c4c7e8c258239e300508e0c8b2a69bad2
Sep 26 15:47:53 openqa kernel: Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
Sep 26 15:47:53 openqa kernel: Workqueue: 0x0 (xprtiod)
Sep 26 15:47:53 openqa kernel: Call Trace:
Sep 26 15:47:53 openqa kernel: <TASK>
Sep 26 15:47:53 openqa kernel: dump_stack_lvl+0x45/0x5b
Sep 26 15:47:53 openqa kernel: __schedule_bug+0x52/0x70
Sep 26 15:47:53 openqa kernel: __schedule+0xdc4/0x1140
Sep 26 15:47:53 openqa kernel: ? arch_local_irq_enable+0x7/0xc
Sep 26 15:47:53 openqa kernel: schedule+0x64/0xe0
Sep 26 15:47:53 openqa kernel: worker_thread+0xab/0x3d0
Sep 26 15:47:53 openqa kernel: ? process_one_work+0x440/0x440
Sep 26 15:47:53 openqa kernel: kthread+0x156/0x180
Sep 26 15:47:53 openqa kernel: ? set_kthread_struct+0x50/0x50
Sep 26 15:47:53 openqa kernel: ret_from_fork+0x22/0x30
Sep 26 15:47:53 openqa kernel: </TASK>
…
Sep 26 16:21:14 openqa kernel: Out of memory: Killed process 1312 (postgres) total-vm:3349768kB, anon-rss:1200kB, file-rss:8kB, shmem-rss:2736620kB, UID:26 pgtables:5876kB oom_score_adj:0
Sep 26 16:21:14 openqa kernel: postgres invoked oom-killer: gfp_mask=0x1100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
Sep 26 16:21:14 openqa kernel: CPU: 7 PID: 6303 Comm: postgres Tainted: G W L 5.14.21-150400.24.21-default #1 SLE15-SP4 7550826c4c7e8c258239e300508e0c8b2a69bad2
Sep 26 16:21:14 openqa kernel: Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
Sep 26 16:21:14 openqa kernel: Call Trace:
Sep 26 16:21:14 openqa kernel: <TASK>
Sep 26 16:21:14 openqa kernel: dump_stack_lvl+0x45/0x5b
Sep 26 16:21:14 openqa kernel: dump_header+0x4a/0x220
Sep 26 16:21:14 openqa kernel: oom_kill_process+0xe8/0x140
Sep 26 16:21:14 openqa kernel: out_of_memory+0x113/0x580
Sep 26 16:21:14 openqa kernel: __alloc_pages_slowpath.constprop.112+0x980/0xc70
Sep 26 16:21:14 openqa kernel: __alloc_pages+0x2d9/0x320
Sep 26 16:21:14 openqa kernel: pagecache_get_page+0x1a8/0x480
Sep 26 16:21:14 openqa kernel: filemap_fault+0x4c1/0xa30
Sep 26 16:21:14 openqa kernel: ? filemap_map_pages+0x14e/0x580
Sep 26 16:21:14 openqa kernel: __do_fault+0x31/0xc0
Sep 26 16:21:14 openqa kernel: __handle_mm_fault+0xd64/0x1220
Sep 26 16:21:14 openqa kernel: handle_mm_fault+0xd7/0x290
Sep 26 16:21:14 openqa kernel: do_user_addr_fault+0x1eb/0x730
Sep 26 16:21:14 openqa kernel: exc_page_fault+0x67/0x150
Sep 26 16:21:14 openqa kernel: ? asm_exc_page_fault+0x8/0x30
Sep 26 16:21:14 openqa kernel: asm_exc_page_fault+0x1e/0x30
Sep 26 16:21:14 openqa kernel: RIP: 0033:0x55aef40e0a90
Sep 26 16:21:14 openqa kernel: Code: Unable to access opcode bytes at RIP 0x55aef40e0a66.
Sep 26 16:21:14 openqa kernel: RSP: 002b:00007ffca8429768 EFLAGS: 00010246
Sep 26 16:21:14 openqa kernel: RAX: 0000000000000001 RBX: 0000000000000001 RCX: 000055aef71bd788
Sep 26 16:21:14 openqa kernel: RDX: 000055aef6462900 RSI: 000055aef71b4e78 RDI: 000000000000066f
Sep 26 16:21:14 openqa kernel: RBP: 0000000000000011 R08: 0000000000000002 R09: 000055aef71bd658
Sep 26 16:21:14 openqa kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 000055aef64626b8
Sep 26 16:21:14 openqa kernel: R13: 0000000000000053 R14: 000055aef6462b98 R15: 00007faccb1e15f8
Sep 26 16:21:14 openqa kernel: </TASK>
Sep 26 16:21:14 openqa kernel: Mem-Info:
Sep 26 16:21:14 openqa kernel: active_anon:1238013 inactive_anon:560727 isolated_anon:0
active_file:2848 inactive_file:2875 isolated_file:32
unevictable:20 dirty:158 writeback:0
slab_reclaimable:3665090 slab_unreclaimable:481451
mapped:804445 shmem:804567 pagetables:99957 bounce:0
free:65873 free_pcp:4279 free_cma:0
Sep 26 16:21:14 openqa kernel: Node 0 active_anon:4952052kB inactive_anon:2242908kB active_file:11392kB inactive_file:11128kB unevictable:80kB isolated(anon):0kB isolated(file):128kB mapped:3211940kB dirty:632kB writeback:0kB shmem:3218268kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 430080kB writeback_tmp:0kB >
Sep 26 16:21:14 openqa kernel: Node 0 DMA free:11248kB boost:0kB min:40kB low:52kB high:64kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Sep 26 16:21:14 openqa kernel: lowmem_reserve[]: 0 2969 23982 23982 23982
Sep 26 16:21:14 openqa kernel: Node 0 DMA32 free:95108kB boost:2048kB min:10408kB low:13448kB high:16488kB reserved_highatomic:2048KB active_anon:718356kB inactive_anon:198232kB active_file:0kB inactive_file:1212kB unevictable:0kB writepending:0kB present:3129312kB managed:3063776kB mlocked:0kB bounce:0kB free_pcp>
Sep 26 16:21:14 openqa kernel: lowmem_reserve[]: 0 0 21013 21013 21013
Sep 26 16:21:14 openqa kernel: Node 0 Normal free:167224kB boost:114688kB min:173864kB low:195380kB high:216896kB reserved_highatomic:16384KB active_anon:4233988kB inactive_anon:2043588kB active_file:11116kB inactive_file:9840kB unevictable:80kB writepending:372kB present:22020096kB managed:21525736kB mlocked:80kB>
Sep 26 16:21:14 openqa kernel: lowmem_reserve[]: 0 0 0 0 0
Sep 26 16:21:14 openqa kernel: Node 0 DMA: 0*4kB 0*8kB 1*16kB (U) 1*32kB (U) 1*64kB (U) 1*128kB (U) 1*256kB (U) 1*512kB (U) 0*1024kB 1*2048kB (M) 2*4096kB (M) = 11248kB
Sep 26 16:21:14 openqa kernel: Node 0 DMA32: 3310*4kB (UME) 9535*8kB (UME) 33*16kB (UM) 6*32kB (UE) 6*64kB (UM) 1*128kB (M) 3*256kB (U) 0*512kB 0*1024kB 0*2048kB 1*4096kB (H) = 95616kB
Sep 26 16:21:14 openqa kernel: Node 0 Normal: 42623*4kB (UME) 121*8kB (UME) 22*16kB (UE) 4*32kB (U) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 171940kB
Sep 26 16:21:14 openqa kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Sep 26 16:21:14 openqa kernel: 831708 total pagecache pages
Sep 26 16:21:14 openqa kernel: 20027 pages in swap cache
Sep 26 16:21:14 openqa kernel: Swap cache stats: add 829916, delete 810190, find 231001956/231291553
Sep 26 16:21:14 openqa kernel: Free swap = 0kB
Sep 26 16:21:14 openqa kernel: Total swap = 499708kB
Sep 26 16:21:14 openqa kernel: 6291350 pages RAM
Sep 26 16:21:14 openqa kernel: 0 pages HighMem/MovableOnly
Sep 26 16:21:14 openqa kernel: 140132 pages reserved
Sep 26 16:21:14 openqa kernel: 0 pages cma reserved
Sep 26 16:21:14 openqa kernel: 0 pages hwpoisoned
Sep 26 16:21:14 openqa kernel: Tasks state (memory values in pages):
Sep 26 16:21:14 openqa kernel: [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
Sep 26 16:21:14 openqa kernel: [ 508] 0 508 3140 3 65536 1124 0 haveged
Sep 26 16:21:14 openqa kernel: [ 509] 0 509 177356 1156 1179648 179 -250 systemd-journal
Sep 26 16:21:14 openqa kernel: [ 527] 0 527 27090 133 225280 536 -1000 systemd-udevd
Sep 26 16:21:14 openqa kernel: [ 680] 0 680 13017 6 143360 157 0 rpc.idmapd
Sep 26 16:21:14 openqa kernel: [ 681] 0 681 15090 31 139264 150 -1000 auditd
Sep 26 16:21:14 openqa kernel: [ 686] 499 686 19832 269 192512 129 -900 dbus-daemon
Sep 26 16:21:14 openqa kernel: [ 690] 0 690 24207 265 229376 360 0 systemd-logind
Sep 26 16:21:14 openqa kernel: [ 691] 0 691 10708 73 126976 240 0 wickedd-auto4
Sep 26 16:21:14 openqa kernel: [ 694] 0 694 10709 71 126976 241 0 wickedd-dhcp4
Sep 26 16:21:14 openqa kernel: [ 697] 0 697 10709 103 131072 211 0 wickedd-dhcp6
Sep 26 16:21:14 openqa kernel: [ 753] 0 753 10752 216 135168 160 0 wickedd
Sep 26 16:21:14 openqa kernel: [ 779] 0 779 10745 227 131072 102 0 wickedd-nanny
…
Sep 26 16:25:30 openqa kernel: rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 4-... } 378193 jiffies s: 307889 root: 0x10/.
Sep 26 16:25:30 openqa kernel: rcu: blocking rcu_node structures (internal RCU debug):
Sep 26 16:25:30 openqa kernel: Task dump for CPU 4:
Sep 26 16:25:30 openqa kernel: task:kworker/4:1 state:R running task stack: 0 pid: 2187 ppid: 2 flags:0x00004008
Sep 26 16:25:30 openqa kernel: Workqueue: events xprt_destroy_cb [sunrpc]
Sep 26 16:25:30 openqa kernel: Call Trace:
Sep 26 16:25:30 openqa kernel: <TASK>
Sep 26 16:25:30 openqa kernel: ? __schedule+0x2d5/0x1140
Sep 26 16:25:30 openqa kernel: ? __inode_wait_for_writeback+0x7f/0xf0
Sep 26 16:25:30 openqa kernel: ? fsnotify_grab_connector+0x4b/0x90
Sep 26 16:25:30 openqa kernel: ? sysvec_apic_timer_interrupt+0xa/0x90
Sep 26 16:25:30 openqa kernel: ? asm_sysvec_apic_timer_interrupt+0x12/0x20
Sep 26 16:25:30 openqa kernel: ? native_queued_spin_lock_slowpath+0x21/0x1e0
Sep 26 16:25:30 openqa kernel: ? _raw_spin_lock+0x25/0x30
Sep 26 16:25:30 openqa kernel: ? xprt_disconnect_done+0x1b/0xb0 [sunrpc f37209057a2989e994f415cd3f1a1642d1001097]
Sep 26 16:25:30 openqa kernel: ? xs_close+0x1a/0x40 [sunrpc f37209057a2989e994f415cd3f1a1642d1001097]
Sep 26 16:25:30 openqa kernel: ? xs_destroy+0x26/0x70 [sunrpc f37209057a2989e994f415cd3f1a1642d1001097]
Sep 26 16:25:30 openqa kernel: ? process_one_work+0x267/0x440
Sep 26 16:25:30 openqa kernel: ? worker_thread+0x2d/0x3d0
Sep 26 16:25:30 openqa kernel: ? process_one_work+0x440/0x440
Sep 26 16:25:30 openqa kernel: ? kthread+0x156/0x180
Sep 26 16:25:30 openqa kernel: ? set_kthread_struct+0x50/0x50
Sep 26 16:25:30 openqa kernel: ? ret_from_fork+0x22/0x30
Sep 26 16:25:30 openqa kernel: </TASK>
…
Sep 26 16:26:14 openqa kernel: watchdog: BUG: soft lockup - CPU#4 stuck for 2146s! [kworker/4:1:2187]
Sep 26 16:26:14 openqa kernel: Modules linked in: iscsi_ibft iscsi_boot_sysfs rfkill loop xfs libcrc32c virtio_net net_failover failover virtio_balloon button nfsd auth_rpcgss nfs_acl lockd grace fuse drm sunrpc configfs ip_tables x_tables ext4 crc16 mbcache jbd2 uhci_hcd ata_piix ehci_hcd ahci libahci usbcore lib>
Sep 26 16:26:14 openqa kernel: Supported: Yes
Sep 26 16:26:14 openqa kernel: CPU: 4 PID: 2187 Comm: kworker/4:1 Tainted: G W L 5.14.21-150400.24.21-default #1 SLE15-SP4 7550826c4c7e8c258239e300508e0c8b2a69bad2
-- Boot 37b8d07bd19743f5b73de54f2d8baa4f --
Sep 26 16:27:27 openqa kernel: Linux version 5.14.21-150400.24.21-default (geeko@buildhost) (gcc (SUSE Linux) 7.5.0, GNU ld (GNU Binutils; SUSE Linux Enterprise 15) 2.37.20211103-150100.7.37) #1 SMP PREEMPT_DYNAMIC Wed Sep 7 06:51:18 UTC 2022 (974d0aa)
Sep 26 16:27:27 openqa kernel: Command line: BOOT_IMAGE=/boot/vmlinuz-5.14.21-150400.24.21-default root=UUID=7116dc72-ebc8-4b21-8847-b9f31dc95229 nomodeset console=tty0 console=ttyS0,115200 elevator=noop
Sep 26 16:27:27 openqa kernel: x86/fpu: x87 FPU will use FXSAVE
Sep 26 16:27:27 openqa kernel: signal: max sigframe size: 1440
Sep 26 16:27:27 openqa kernel: BIOS-provided physical RAM map:
Updated by mkittler about 2 years ago
Note that downgrading the kernel via zypper rm kernel-default-5.14.21-150400.24.21 && zypper al kernel-default && reboot (see end of #116722#note-20) did not work.
There's still only the old kernel version installed but the new version is nevertheless being loaded:
martchus@openqa:~> rpm -q kernel-default
kernel-default-5.14.21-150400.24.18.1.x86_64
martchus@openqa:~> uname -a
Linux openqa 5.14.21-150400.24.21-default #1 SMP PREEMPT_DYNAMIC Wed Sep 7 06:51:18 UTC 2022 (974d0aa) x86_64 x86_64 x86_64 GNU/Linux
Also see the logs in my previous comment.
Likely that's because under /boot still files for both kernel versions are present:
martchus@openqa:~> find /boot -iname '*5.14.21-150400.24.18*'
/boot/.vmlinuz-5.14.21-150400.24.18-default.hmac
/boot/sysctl.conf-5.14.21-150400.24.18-default
/boot/vmlinux-5.14.21-150400.24.18-default.gz
/boot/config-5.14.21-150400.24.18-default
/boot/System.map-5.14.21-150400.24.18-default
/boot/symvers-5.14.21-150400.24.18-default.gz
/boot/initrd-5.14.21-150400.24.18-default
/boot/vmlinuz-5.14.21-150400.24.18-default
martchus@openqa:~> find /boot -iname '*5.14.21-150400.24.21*'
/boot/.vmlinuz-5.14.21-150400.24.21-default.hmac
/boot/vmlinuz-5.14.21-150400.24.21-default
/boot/config-5.14.21-150400.24.21-default
/boot/sysctl.conf-5.14.21-150400.24.21-default
/boot/initrd-5.14.21-150400.24.21-default
/boot/symvers-5.14.21-150400.24.21-default.gz
/boot/vmlinux-5.14.21-150400.24.21-default.gz
/boot/System.map-5.14.21-150400.24.21-default
That's strange because those files actually belong to the corresponding kernel package and should normally be deleted on its removal:
martchus@openqa:~> rpm -qf /boot/vmlinux-5.14.21-150400.24.21-default.gz
kernel-default-base-5.14.21-150400.24.21.2.150400.24.7.2.x86_64
I now invoked sudo zypper rm kernel-default-base-5.14.21-150400.24.21.2.150400.24.7.2.x86_64 again and checked whether the files are gone. They are. Then I invoked zypper dup to see whether the package lock works. Apparently it doesn't:
martchus@openqa:~> sudo zypper dup
Metadaten von Repository 'devel_openQA' abrufen ....................................................................................................................................................................................................................................................................[fertig]
Cache für Repository 'devel_openQA' erzeugen .......................................................................................................................................................................................................................................................................[fertig]
Metadaten von Repository 'Update repository with updates from SUSE Linux Enterprise 15' abrufen ....................................................................................................................................................................................................................[fertig]
Cache für Repository 'Update repository with updates from SUSE Linux Enterprise 15' erzeugen .......................................................................................................................................................................................................................[fertig]
Repository-Daten werden geladen...
Installierte Pakete werden gelesen...
Warnung: Sie sind im Begriff, eine Distributionsaktualisierung mit allen aktivierten Repositorys durchzuführen. Vergewissern Sie sich, dass diese Repositorys kompatibel sind, bevor Sie fortfahren. Weitere Informationen zu diesem Kommando finden Sie unter 'man zypper'.
Distributions-Aktualisierungen werden verarbeitet...
Die folgenden 2 Objekte sind gesperrt und werden durch keine Aktivität geändert:
Verfügbar:
openQA-local-db
Installiert:
kernel-default
Die folgenden 3 NEUEN Pakete werden installiert:
kernel-default-base kernel-default-extra-5.14.21-150400.24.21.2 kernel-default-optional-5.14.21-150400.24.21.2
So that's our problem. I now invoked
martchus@openqa:~> sudo zypper al kernel-default-base
Die angegebene Sperre wurde erfolgreich hinzugefügt.
martchus@openqa:~> sudo zypper al kernel-default-extra
Die angegebene Sperre wurde erfolgreich hinzugefügt.
martchus@openqa:~> sudo zypper al kernel-default-optional
Die angegebene Sperre wurde erfolgreich hinzugefügt.
martchus@openqa:~> sudo zypper dup
Repository-Daten werden geladen...
Installierte Pakete werden gelesen...
Warnung: Sie sind im Begriff, eine Distributionsaktualisierung mit allen aktivierten Repositorys durchzuführen. Vergewissern Sie sich, dass diese Repositorys kompatibel sind, bevor Sie fortfahren. Weitere Informationen zu diesem Kommando finden Sie unter 'man zypper'.
Distributions-Aktualisierungen werden verarbeitet...
Die folgenden 5 Objekte sind gesperrt und werden durch keine Aktivität geändert:
Verfügbar:
kernel-default-base openQA-local-db
Installiert:
kernel-default kernel-default-extra kernel-default-optional
Keine auszuführenden Aktionen.
and now zypper dup wouldn't do a thing. So we only ran into the problem again due to a missing package lock. I've just invoked a reboot so we stay on the old kernel version.
ssh openqa.suse.de
Last login: Mon Sep 26 16:30:17 2022 from 2620:113:80c0:8340::1127
Have a lot of fun...
martchus@openqa:~> uname -a
Linux openqa 5.14.21-150400.24.18-default #1 SMP PREEMPT_DYNAMIC Thu Aug 4 14:17:48 UTC 2022 (e9f7bfc) x86_64 x86_64 x86_64 GNU/Linux
Updated by okurz about 2 years ago
Thank you. That's great! Do you have an idea why the kernel.panic and automatic reboot wasn't effective?
Updated by jbaier_cz about 2 years ago
okurz wrote:
Thank you. That's great! Do you have an idea why the kernel.panic and automatic reboot wasn't effective?
Maybe the machine wasn't panicking (it just lost connectivity due to killed services)?
Updated by mkittler about 2 years ago
There was no kernel panic, see #116722#note-26. We recovered the machine before it came to that. (Until it panics there's usually a long struggle, see timestamps in #116722#note-19 although it has even already started earlier.)
Updated by okurz about 2 years ago
- Status changed from Feedback to Blocked
Updated by okurz about 2 years ago
- Priority changed from High to Urgent
bug closed as duplicate of https://bugzilla.opensuse.org/show_bug.cgi?id=1203338, tracking that.
Updated by okurz about 2 years ago
- Due date changed from 2022-10-28 to 2022-11-25
- Priority changed from Urgent to Normal
IBS SRs were mentioned but no message yet about released update. We will wait for that and keep the older kernel running until then
Updated by okurz about 2 years ago
- Due date changed from 2022-11-25 to 2023-01-20
boo#1203338 still IN_PROGRESS and haven't see the full availability of bugfixes.
Updated by okurz almost 2 years ago
- Tags changed from infra to infra, reactive work
Updated by livdywan almost 2 years ago
- Due date deleted (
2023-01-20)
This is in fact still blocking on boo#1203338. Or at least I assume it is whilst I can't login to Bugzilla.
Updated by okurz almost 2 years ago
I asked with a comment in https://bugzilla.opensuse.org/show_bug.cgi?id=1203338#c70 about the status and plans.
Updated by okurz almost 2 years ago
- Related to action #126212: openqa.suse.de response times very slow. No alert fired size:M added
Updated by okurz over 1 year ago
- Status changed from Blocked to Workable
- Assignee deleted (
okurz)
https://bugzilla.opensuse.org/show_bug.cgi?id=1203338 is now RESOLVED FIXED. The updated kernel should be tried on osd+o3
Updated by mkittler over 1 year ago
- Status changed from Workable to In Progress
Updated by mkittler over 1 year ago
- Status changed from In Progress to Feedback
- Assignee set to mkittler
I've just enabled the normal kernel again on OSD:
$ sudo zypper removelock 'kernel-default*'
1 lock has been successfully removed.
$ sudo zypper dup --allow-vendor-change
…
The following 3 NEW packages are going to be installed:
kernel-default-5.14.21-150400.24.60.1 kernel-default-extra-5.14.21-150400.24.60.1 kernel-default-optional-5.14.21-150400.24.60.1
…
$ sudo reboot
The VM came back ok. Let's see how stable it'll run.
Updated by mkittler over 1 year ago
It looks still good on OSD and on o3 the kernel wasn't locked anyways (and a recent version is in use).
Updated by mkittler over 1 year ago
- Status changed from Feedback to Resolved
It looks still good so I'm resolving.
Actions