Project

General

Profile

Actions

action #116722

closed

openqa.suse.de is not reachable 2022-09-18, no ping response, postgreSQL OOM and kernel panics size:M

Added by okurz over 2 years ago. Updated over 1 year ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
-
Start date:
2022-09-18
Due date:
% Done:

0%

Estimated time:

Description

Observation

Multiple alerts on https://monitor.qa.suse.de/alerting/list?state=not_ok , all seem to point to openqa.suse.de itself being down. The first alert sending an email seems to be "Failed systemd services" https://monitor.qa.suse.de/d/KToPYLEWz/failed-systemd-services?orgId=1&from=1663476407099&to=1663478162658&viewPanel=2 from 2022-09-18 06:50 and following.
There seems to be a sudden increase in CPU load visible in
https://monitor.qa.suse.de/d/WebuiDb/webui-summary?orgId=1&from=1663473600000&to=1663480800000 going to 100 with spotty response from telegraf and no data soon afterwards. Also a significant increase in disk I/O requests in https://monitor.qa.suse.de/d/WebuiDb/webui-summary?orgId=1&from=1663473600000&to=1663480800000&viewPanel=119

Impact

openqa.suse.de is not reachable even over ping and not reachable for workers or users so completely unusable.

Suggestions

  • DONE Increase memory of the VM
  • DONE Find the root cause -> soft lockups in kernel since the latest kernel upgrade
  • DONE Report product issue
  • DONE Apply workaround, e.g. downgraded kernel version
  • DONE Apply kernel.panic = 60
  • Wait for product issue to be resolved and upgrade kernel again
  • Optional: Review the changelog diff of the kernel

Files

journal_osd_crash_2022-09-18.log.xz (300 KB) journal_osd_crash_2022-09-18.log.xz system journal of crash period, multiple OOM situations okurz, 2022-09-19 07:05

Related issues 9 (0 open9 closed)

Related to openQA Infrastructure (public) - action #116740: [alert] openqaworker14: host up alertResolvednicksinger2022-09-19

Actions
Related to openQA Infrastructure (public) - action #116743: [alert] QA-Power8-5-kvm: host up alertResolvednicksinger2022-09-192022-10-04

Actions
Related to openQA Infrastructure (public) - action #116746: [alert] openqaworker9: host up alertResolvednicksinger2022-09-19

Actions
Related to openQA Infrastructure (public) - action #116752: [alert] powerqaworker-qam-1: host up alertResolvednicksinger2022-09-19

Actions
Related to openQA Infrastructure (public) - coordination #112718: [alert][osd] openqa.suse.de is not reachable anymore, response times > 30s, multiple alerts over the weekendResolvedokurz2022-06-22

Actions
Related to openQA Infrastructure (public) - action #115208: failed-systemd-services: logrotate-openqa alerting on and off size:MResolvedlivdywan

Actions
Related to openQA Infrastructure (public) - action #116848: Ensure kdump is enabled and working on all OSD machinesResolvedmkittler2022-09-202022-10-25

Actions
Related to openQA Infrastructure (public) - action #116911: [openQA][needle] Can not commit new needle for test suite on openqa.suse.deResolvedmkittler2022-09-21

Actions
Related to openQA Infrastructure (public) - action #126212: openqa.suse.de response times very slow. No alert fired size:MResolvedmkittler2023-03-20

Actions
Actions #1

Updated by okurz over 2 years ago

  • Status changed from New to In Progress
  • Assignee set to okurz

Reported https://sd.suse.com/servicedesk/customer/portal/1/SD-98873 with request to Engineering Infrastructure to help debugging the VM.

EDIT: Also reported in https://suse.slack.com/archives/C029APBKLGK/p1663534410250609 and informed main users in https://suse.slack.com/archives/C02CANHLANP/p1663534467018299

Actions #2

Updated by openqa_review over 2 years ago

  • Due date set to 2022-10-03

Setting due date based on mean cycle time of SUSE QE Tools

Actions #3

Updated by okurz over 2 years ago

  • Priority changed from Immediate to Urgent

SD ticket was resolved, gschlotter could reboot the machine:

machine is back online.
it had a kernel panic.

We should check if we can find more data about the kernel panic and investigate further.

I checked all alerts and found multiple other hosts down or currently not reachable, created specific tickets for each.

Triggered:

for i in incomplete timeout_exceeded ; do host=openqa.suse.de failed_since=2022-09-18 result="result='$i'" openqa-advanced-retrigger-jobs ; done
Actions #4

Updated by okurz over 2 years ago

  • Related to action #116740: [alert] openqaworker14: host up alert added
Actions #5

Updated by okurz over 2 years ago

  • Related to action #116743: [alert] QA-Power8-5-kvm: host up alert added
Actions #6

Updated by okurz over 2 years ago

  • Related to action #116746: [alert] openqaworker9: host up alert added
Actions #7

Updated by okurz over 2 years ago

  • Related to action #116752: [alert] powerqaworker-qam-1: host up alert added
Actions #10

Updated by okurz over 2 years ago

I looked into the system journal and found multiple OOM reports which might be the cause or just symptoms of the crash. Needs further investigation and crosschecking with data in grafana.

EDIT: Looked into the system journal in more detail

  1. What I find noteworthy observations is that "logrotate" prints a huge amount of messages like "Sep 18 07:00:00 openqa logrotate[18788]: Creating new state".
  2. Next problematic entry "Sep 18 07:00:06 openqa kernel: watchdog: BUG: soft lockup - CPU#2 stuck for 1050s! [kworker/2:2:24024]", then shortly afterwards
Sep 18 07:00:12 openqa openqa-scheduler-daemon[1487]: WARNING:  terminating connection because of crash of another server process
Sep 18 07:00:12 openqa openqa-scheduler-daemon[1487]: DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
Sep 18 07:00:12 openqa openqa-scheduler-daemon[1487]: HINT:  In a moment you should be able to reconnect to the database and repeat your command.
Sep 18 07:00:12 openqa openqa-scheduler-daemon[1487]:         (in cleanup) DBIx::Class::Storage::DBI::_populate_dbh(): DBI Exception: DBD::Pg::st DESTROY failed: no connection to the server [for Statement "SELECT me.child_job_id, me.parent_job_id, me.dependency FROM job_dependencies me WHERE ( ( me.child_job_id = ? AND me.dependency = ? AND me.parent_job_id = ? ) )"] at /usr/share/openqa/script/../lib/OpenQA/Scheduler/Model/Jobs.pm line 29

did the database process terminate or did the database die? Next CPU stuck message at "Sep 18 07:00:46 openqa kernel: watchdog: BUG: soft lockup - CPU#2 stuck for 1088s! [kworker/2:2:24024]" and more later on

  1. "Sep 18 07:01:38 openqa kernel: postgres invoked oom-killer: gfp_mask=0x400dc0(GFP_KERNEL_ACCOUNT|__GFP_ZERO), order=0, oom_score_adj=0" so postgres exceeded memory usage? After that message multiple openQA processes understandably complain about the postgres server process being gone
Actions #11

Updated by mkittler over 2 years ago

I'm currently cleaning up the Minion dashboard. It contains many failures due to the issue, e.g. due to database issues like that:

      DBI connect('dbname=openqa','geekotest',...) failed: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: FATAL:  the database system is in recovery mode at /usr/lib/perl5/vendor_perl/5.26.1/Mojo/Pg.pm line 73.

Or because the Minion workers were crashing (or were killed) completely. Or because the needles dir was left in an unclean state (which we've already fixed).

Actions #12

Updated by okurz over 2 years ago

OSD is down again. Unfortunately this needs manual intervention by Engineering Infrastructure as they won't give us access and they don't monitor the machine on their own. Created https://sd.suse.com/servicedesk/customer/portal/1/SD-98988 . After the machine is up again we should ensure that the machine is configured to automatically reboot on kernel panics and also downgrade any package upgrades from last sunday and look into the memory related postgres configs, e.g. in https://gitlab.suse.de/openqa/salt-states-openqa/-/blob/master/openqa/server.sls#L331

Actions #13

Updated by okurz over 2 years ago

Up again, done by gschlotter in https://suse.slack.com/archives/C029APBKLGK/p1663651357301979 . Same problem source, out of memory:

Sep 20 01:16:02 openqa kernel: Out of memory: Killed process 1563 (postgres) total-vm:3349768kB, anon-rss:1420kB, file-rss:0kB, shmem-rss:2698412kB, UID:26 pgtables:5800kB oom_score_adj:0

Could that be the queries we conduct only once every 24h causing this problem?

EDIT: gschlotter also gave the VM +4GB RAM. I have enabled automatic reboots on panic with https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/733 . I wonder why we never considered that in the past or why that setting is not even default for servers on openSUSE. I have not yet downgraded any packages or changed postgres config. We should check if the problem really reproduces. Also we should configure kdump and enable notification if the problem happens again unless our monitoring would already alert us.

Also I am suggesting to reduce work_mem for postgres by factor 4, see https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/734

Actions #14

Updated by okurz over 2 years ago

  • Related to coordination #112718: [alert][osd] openqa.suse.de is not reachable anymore, response times > 30s, multiple alerts over the weekend added
Actions #15

Updated by okurz over 2 years ago

  • Related to action #115208: failed-systemd-services: logrotate-openqa alerting on and off size:M added
Actions #16

Updated by okurz over 2 years ago

  • Related to action #116848: Ensure kdump is enabled and working on all OSD machines added
Actions #17

Updated by okurz over 2 years ago

  • Subject changed from openqa.suse.de is not reachable, no ping response to openqa.suse.de is not reachable 2022-09-18, no ping response, postgreSQL OOM and kernel panics
Actions #18

Updated by okurz over 2 years ago

  • Related to action #116911: [openQA][needle] Can not commit new needle for test suite on openqa.suse.de added
Actions #19

Updated by mkittler over 2 years ago

The OOM killer was invoked several times. We can try to make sense of logs to find the problematic processes:

Sep 21 00:02:25 openqa systemd-journald[25795]: Missed 142 kernel messages
Sep 21 00:02:25 openqa kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=openqa,pid=25595,uid=1001
Sep 21 00:02:25 openqa kernel: Out of memory: Killed process 25595 (openqa) total-vm:359592kB, anon-rss:153524kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:716kB oom_score_adj:0
Sep 21 00:02:25 openqa kernel: oom_reaper: reaped process 25595 (openqa), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Sep 21 00:02:26 openqa kernel: kthreadd invoked oom-killer: gfp_mask=0x40cc0(GFP_KERNEL|__GFP_COMP), order=2, oom_score_adj=0
Sep 21 00:02:26 openqa kernel: CPU: 6 PID: 2 Comm: kthreadd Tainted: G        W    L      5.14.21-150400.24.21-default #1 SLE15-SP4 7550826c4c7e8c258239e300508e0c8b2a69bad2
Sep 21 00:02:26 openqa kernel: Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
Sep 21 00:02:26 openqa kernel: Call Trace:
Sep 21 00:02:26 openqa kernel:  <TASK>
Sep 21 00:02:26 openqa kernel:  dump_stack_lvl+0x45/0x5b
Sep 21 00:02:26 openqa kernel:  dump_header+0x4a/0x220
Sep 21 00:02:26 openqa kernel:  oom_kill_process+0xe8/0x140
Sep 21 00:02:26 openqa kernel:  out_of_memory+0x113/0x580
Sep 21 00:02:26 openqa kernel:  __alloc_pages_slowpath.constprop.112+0x980/0xc70
Sep 21 00:02:26 openqa kernel:  __alloc_pages+0x2d9/0x320
Sep 21 00:02:26 openqa kernel:  new_slab+0x4c9/0x500
Sep 21 00:02:26 openqa kernel:  ___slab_alloc+0x6ad/0x860
Sep 21 00:02:26 openqa kernel:  ? copy_process+0x213/0x1cd0
Sep 21 00:02:26 openqa kernel:  ? __slab_alloc.isra.92+0x1c/0x40
Sep 21 00:02:26 openqa kernel:  __slab_alloc.isra.92+0x1c/0x40
Sep 21 00:02:26 openqa kernel:  kmem_cache_alloc_node+0x114/0x310
Sep 21 00:02:26 openqa kernel:  ? copy_process+0x213/0x1cd0
Sep 21 00:02:26 openqa kernel:  copy_process+0x213/0x1cd0
Sep 21 00:02:26 openqa kernel:  ? update_load_avg+0x1cc/0x620
Sep 21 00:02:26 openqa kernel:  kernel_clone+0x9b/0x400
Sep 21 00:02:26 openqa kernel:  kernel_thread+0x55/0x80
Sep 21 00:02:26 openqa kernel:  ? set_kthread_struct+0x50/0x50
Sep 21 00:02:26 openqa kernel:  kthreadd+0x2b4/0x300
Sep 21 00:02:26 openqa kernel:  ? kthread_is_per_cpu+0x30/0x30
Sep 21 00:02:26 openqa kernel:  ret_from_fork+0x22/0x30
Sep 21 00:02:26 openqa kernel:  </TASK>
Sep 21 00:02:26 openqa kernel: Mem-Info:
Sep 21 00:02:26 openqa kernel: active_anon:370017 inactive_anon:123196 isolated_anon:109
                                active_file:2863 inactive_file:4608 isolated_file:687
                                unevictable:664 dirty:0 writeback:0
                                slab_reclaimable:4416751 slab_unreclaimable:1028155
                                mapped:10899 shmem:93386 pagetables:70799 bounce:0
                                free:93830 free_pcp:4003 free_cma:0
Sep 21 00:02:26 openqa kernel: Node 0 active_anon:1481108kB inactive_anon:492480kB active_file:8700kB inactive_file:21860kB unevictable:2656kB isolated(anon):436kB isolated(file):2760kB mapped:43304kB dirty:0kB writeback:0kB shmem:373544kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB kernel_>
Sep 21 00:02:26 openqa kernel: Node 0 DMA free:11248kB boost:0kB min:40kB low:52kB high:64kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Sep 21 00:02:26 openqa kernel: lowmem_reserve[]: 0 2991 24013 24013 24013
Sep 21 00:02:26 openqa kernel: Node 0 DMA32 free:129024kB boost:21640kB min:29996kB low:33032kB high:36068kB reserved_highatomic:18432KB active_anon:122628kB inactive_anon:27132kB active_file:1032kB inactive_file:1236kB unevictable:0kB writepending:0kB present:3129312kB managed:3063776kB mlocked:0kB bounce:0kB fre>
Sep 21 00:02:26 openqa kernel: lowmem_reserve[]: 0 0 21021 21021 21021
Sep 21 00:02:26 openqa kernel: Node 0 Normal free:235304kB boost:153324kB min:212508kB low:234024kB high:255540kB reserved_highatomic:22528KB active_anon:1360828kB inactive_anon:467376kB active_file:11608kB inactive_file:14584kB unevictable:2656kB writepending:0kB present:22020096kB managed:21525736kB mlocked:80kB>
Sep 21 00:02:26 openqa kernel: lowmem_reserve[]: 0 0 0 0 0
Sep 21 00:02:26 openqa kernel: Node 0 DMA: 0*4kB 0*8kB 1*16kB (U) 1*32kB (U) 1*64kB (U) 1*128kB (U) 1*256kB (U) 1*512kB (U) 0*1024kB 1*2048kB (M) 2*4096kB (M) = 11248kB
Sep 21 00:02:26 openqa kernel: Node 0 DMA32: 6447*4kB (UME) 12993*8kB (UME) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 129732kB
Sep 21 00:02:26 openqa kernel: Node 0 Normal: 13746*4kB (UME) 22568*8kB (UME) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 235528kB
Sep 21 00:02:26 openqa kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Sep 21 00:02:26 openqa kernel: 101977 total pagecache pages
Sep 21 00:02:26 openqa kernel: 442 pages in swap cache
Sep 21 00:02:26 openqa kernel: Swap cache stats: add 1390281, delete 1389759, find 96349271/96842395
Sep 21 00:02:26 openqa kernel: Free swap  = 0kB
Sep 21 00:02:26 openqa kernel: Total swap = 499708kB
Sep 21 00:02:26 openqa kernel: 6291350 pages RAM
Sep 21 00:02:26 openqa kernel: 0 pages HighMem/MovableOnly
Sep 21 00:02:26 openqa kernel: 140132 pages reserved
Sep 21 00:02:26 openqa kernel: 0 pages cma reserved
Sep 21 00:02:26 openqa kernel: 0 pages hwpoisoned
…
Sep 21 00:02:26 openqa kernel: Tasks state (memory values in pages):
Sep 21 00:02:26 openqa kernel: [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
Sep 21 00:02:26 openqa kernel: [    236]     0   236     3140      430    69632     1474             0 haveged
Sep 21 00:02:26 openqa kernel: [    523]     0   523    27415      100   237568      549         -1000 systemd-udevd
Sep 21 00:02:26 openqa kernel: [    676]     0   676    13017        0   135168      163             0 rpc.idmapd
Sep 21 00:02:26 openqa kernel: [    677]     0   677    15090       28   147456      156         -1000 auditd
Sep 21 00:02:26 openqa kernel: [    682]   499   682    19833      208   184320      190          -900 dbus-daemon
Sep 21 00:02:26 openqa kernel: [    693]     0   693    24068      244   229376      223             0 systemd-logind
Sep 21 00:02:26 openqa kernel: [    696]     0   696    10708        4   122880      308             0 wickedd-auto4
Sep 21 00:02:26 openqa kernel: [    701]     0   701    10709        4   131072      310             0 wickedd-dhcp4
Sep 21 00:02:26 openqa kernel: [    705]     0   705    10709       46   126976      269             0 wickedd-dhcp6
Sep 21 00:02:26 openqa kernel: [    740]     0   740    10781      109   135168      270             0 wickedd
Sep 21 00:02:26 openqa kernel: [    767]     0   767    10744       53   126976      276             0 wickedd-nanny
Sep 21 00:02:26 openqa kernel: [    788]     0   788    18806       90   184320      337             0 systemd
Sep 21 00:02:26 openqa kernel: [    789]     0   789    30693      104   270336      829             0 (sd-pam)
Sep 21 00:02:26 openqa kernel: [   1307]     0  1307    28974       33   262144      290             0 rebootmgrd
Sep 21 00:02:26 openqa kernel: [   1309]     0  1309     7763        8   102400      125             0 rsyncd
Sep 21 00:02:26 openqa kernel: [   1311]   488  1311    74604     8038   626688     3148             0 salt-master
Sep 21 00:02:26 openqa kernel: [   1312]     0  1312    49939      124   434176     5000             0 salt-minion
Sep 21 00:02:26 openqa kernel: [   1319]     0  1319     6390       15    94208      109             0 vsftpd
Sep 21 00:02:26 openqa kernel: [   1321]     0  1321    13501        2   139264      169             0 xinetd
Sep 21 00:02:26 openqa kernel: [   1326]   493  1326    15982       44   176128      180             0 rpcbind
Sep 21 00:02:26 openqa kernel: [   1327]  1001  1327    18840      145   188416      339             0 systemd
Sep 21 00:02:26 openqa kernel: [   1329]  1001  1329    30736      105   270336      858             0 (sd-pam)
Sep 21 00:02:26 openqa kernel: [   1334]     0  1334    18890       41   196608      221         -1000 sshd
Sep 21 00:02:26 openqa kernel: [   1335]   492  1335    25274      505   249856    12024             0 rpc.statd
Sep 21 00:02:26 openqa kernel: [   1337]   475  1337    39077       40   208896      256             0 chronyd
Sep 21 00:02:26 openqa kernel: [   1345]     0  1345    18541      400   172032     6318             0 rpc.mountd
Sep 21 00:02:26 openqa kernel: [   1360]     0  1360    82552       70   671744      690             0 smbd
Sep 21 00:02:26 openqa kernel: [   1380]   497  1380    12618       12   143360      205          -500 nrpe
Sep 21 00:02:26 openqa kernel: [   1411]     0  1411    81044       29   626688      698             0 smbd-notifyd
Sep 21 00:02:26 openqa kernel: [   1412]     0  1412    81046       29   622592      698             0 cleanupd
Sep 21 00:02:26 openqa kernel: [   1420]     0  1420    82373       50   659456      673             0 samba-bgqd
Sep 21 00:02:26 openqa kernel: [   1428]     0  1428    36828      269   319488      406             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   1465]    30  1465    36952      322   319488      390             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   1467]    30  1467    36974      336   319488      387             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   1478]    30  1478    36954      262   315392      436             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   1479]    30  1479    36928      271   315392      414             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   1482]    30  1482    36950      312   319488      392             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   1485]    30  1485    36928      281   315392      407             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   1487]    30  1487    36928      253   307200      416             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   1488]    30  1488    36973      253   319488      469             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   1489]    30  1489    36948      206   319488      499             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   1490]    30  1490    36952      265   315392      433             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   1491]    30  1491    36963      244   319488      470             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   1496]    30  1496    36895      184   307200      474             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   1497]    30  1497    36952      235   319488      469             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   1502]     0  1502    11481       24   126976      143             0 agetty
Sep 21 00:02:26 openqa kernel: [   1504]    30  1504    36928      294   315392      398             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   1507]     0  1507    11481       25   131072      139             0 agetty
Sep 21 00:02:26 openqa kernel: [   1508]    30  1508    36976      336   319488      385             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   1514]    30  1514    36895      185   307200      473             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   1515]    30  1515    36895      194   307200      464             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   1517]    30  1517    36954      308   319488      405             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   1520]    30  1520    36951      324   319488      385             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   1525]    30  1525    36895      242   307200      432             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   1526]    30  1526    36953      264   315392      433             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   1529]    30  1529    36954      275   315392      437             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   1531]    30  1531    38546     1258   331776     1033             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   1535]    30  1535    36895      195   307200      463             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   1538]    30  1538    36928      202   311296      474             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   1541]    30  1541    36928      196   311296      478             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   1543]    30  1543    36895      186   307200      472             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   1546]    30  1546    36955      280   319488      429             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   1547]    30  1547    36961      316   315392      390             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   1549]    30  1549    36988      286   319488      449             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   1550]    30  1550    36962      251   319488      482             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   1553]    30  1553    36929      250   315392      423             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   1554]    30  1554    36958      317   319488      406             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   1555]    30  1555    36950      308   319488      403             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   1556]    30  1556    36947      282   315392      412             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   1557]    30  1557    36952      271   319488      438             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   1562]    30  1562    36954      290   315392      406             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   1563]    30  1563    36961      320   319488      391             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   1565]    30  1565    36895      237   307200      436             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   1568]    30  1568    36895      189   307200      468             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   1601]     0  1601   303126     7977   724992     4763             0 salt-minion
Sep 21 00:02:26 openqa kernel: [   1617]     0  1617    24392       22   176128      215             0 master
Sep 21 00:02:26 openqa kernel: [   1619]    51  1619    29041       64   192512      203             0 qmgr
Sep 21 00:02:26 openqa kernel: [   1632]     0  1632    12792       28   147456      179             0 cron
Sep 21 00:02:26 openqa kernel: [   1644]   488  1644    52386      743   425984     5067             0 salt-master
Sep 21 00:02:26 openqa kernel: [   1645]     0  1645    70820      124   442368     5695             0 salt-minion
Sep 21 00:02:26 openqa kernel: [   1674]   488  1674    93817     2285   606208     7603             0 salt-master
Sep 21 00:02:26 openqa kernel: [   1677]   488  1677    79330     4994   626688     8218             0 salt-master
Sep 21 00:02:26 openqa kernel: [   1682]   488  1682   267723    10237   712704     4900             0 salt-master
Sep 21 00:02:26 openqa kernel: [   1684]   488  1684    73223     2351   577536     7475             0 salt-master
Sep 21 00:02:26 openqa kernel: [   1686]   488  1686   167412     2924   626688     8289             0 salt-master
Sep 21 00:02:26 openqa kernel: [   1688]   488  1688   267923     8753   696320     5344             0 salt-master
Sep 21 00:02:26 openqa kernel: [   1695]   488  1695   268733    10653   724992     4598             0 salt-master
Sep 21 00:02:26 openqa kernel: [   1697]   488  1697   268504    10665   716800     4505             0 salt-master
Sep 21 00:02:26 openqa kernel: [   1698]   488  1698   268621     8968   708608     5230             0 salt-master
Sep 21 00:02:26 openqa kernel: [   1699]   488  1699   268578    10007   712704     4529             0 salt-master
Sep 21 00:02:26 openqa kernel: [   2192]    30  2192    36952      281   315392      411             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   2333]    30  2333    36949      296   319488      403             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   2438]    30  2438    36947      260   315392      431             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   2452]    30  2452    36953      319   319488      400             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   2458]    30  2458    36959      253   319488      451             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   2535]    30  2535    36929      260   315392      405             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   2537]    30  2537    36976      323   319488      399             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   2539]    30  2539    36964      355   319488      365             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   2540]    30  2540    36895      260   307200      385             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   2542]    30  2542    36895      269   307200      376             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   2543]    30  2543    36895      242   307200      403             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   2544]    30  2544    36895      223   307200      420             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   2546]    30  2546    36929      271   315392      413             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   2547]    30  2547    36929      292   315392      386             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   2816]    30  2816    36959      286   319488      433             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   2828]    30  2828    36978      274   319488      448             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   2831]    30  2831    36951      236   319488      470             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   2834]    30  2834    36929      302   315392      389             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   2835]    30  2835    36965      325   319488      380             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   2838]    30  2838    36949      293   315392      400             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   2848]    30  2848    36953      323   315392      373             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   2902]    30  2902    36949      326   319488      364             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   2904]    30  2904    36929      241   315392      445             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   2906]    30  2906    36952      310   319488      406             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   2909]    30  2909    36949      310   319488      382             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   2910]    30  2910    36960      277   319488      431             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   2911]    30  2911    36949      299   315392      393             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   2914]    30  2914    36980      268   319488      452             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   2915]    30  2915    36972      265   319488      453             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   2920]    30  2920    36955      246   315392      451             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   2921]    30  2921    36963      311   319488      407             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   2963]    30  2963    36947      295   315392      394             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   2964]    30  2964    36962      233   315392      480             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   2972]    30  2972    36957      244   319488      478             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   2973]    30  2973    36960      270   315392      434             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   2974]    30  2974    36949      228   319488      479             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   2978]    30  2978    36959      238   319488      461             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   2980]    30  2980    36949      241   315392      449             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   2984]    30  2984    36957      303   319488      418             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   2985]    30  2985    36960      288   319488      424             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   2987]    30  2987    36972      308   319488      407             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   3043]    30  3043    36949      244   315392      458             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   3046]    30  3046    36966      311   315392      399             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   3055]    30  3055    36975      333   319488      385             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   3058]    30  3058    38159      390   327680     1517             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   3059]    30  3059    36895      257   307200      389             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   3060]    30  3060    36959      240   319488      464             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   3061]    30  3061    36951      283   319488      421             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   3062]    30  3062    36963      328   319488      388             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   3066]    30  3066    37768      326   323584     1194             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   3069]    30  3069    36895      223   307200      427             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   3070]    30  3070    36895      219   307200      427             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   3072]    30  3072    36959      314   319488      403             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   3097]    30  3097    36959      294   319488      417             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   3100]    30  3100    36895      223   307200      426             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   3101]    30  3101    36895      245   307200      404             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   3114]    30  3114    36895      265   307200      380             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   3115]    30  3115    36895      271   307200      374             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   3116]    30  3116    36895      259   307200      386             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   3117]    30  3117    36962      319   319488      391             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   3121]    30  3121    36895      241   307200      404             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   3125]    30  3125    36929      249   315392      409             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   3127]    30  3127    36929      256   315392      405             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   3128]    30  3128    36895      192   307200      450             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   3135]    30  3135    36980      312   319488      411             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   3138]    30  3138    36895      186   307200      456             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   3139]    30  3139    36895      192   307200      450             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   3142]    30  3142    36895      245   307200      404             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   3143]    30  3143    36957      286   319488      417             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   3145]    30  3145    36929      290   315392      388             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   3147]    30  3147    36895      241   307200      408             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   3149]    30  3149    36949      257   315392      466             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   3159]    30  3159    38342     1238   327680      841             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   3161]    30  3161    36947      309   319488      388             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   3198]    30  3198    36977      286   319488      435             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   3199]    30  3199    36929      264   315392      426             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   3206]    30  3206    36986      253   319488      471             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   3211]    30  3211    36949      257   315392      444             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   3216]    30  3216    36947      241   319488      466             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   3219]    30  3219    36959      300   319488      407             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   3220]    30  3220    36961      259   319488      450             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   3226]    30  3226    36949      244   315392      453             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   3314]    30  3314    36949      244   319488      457             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   3319]    30  3319    36974      306   319488      408             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   3320]    30  3320    36927      242   315392      427             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   3325]    30  3325    36962      255   319488      452             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   3328]    30  3328    36951      272   319488      439             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   3362]    30  3362    36949      243   319488      464             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   3369]    30  3369    36961      246   319488      457             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   3380]    30  3380    36949      280   319488      418             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   3382]    30  3382    36949      255   319488      444             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   3385]    30  3385    36949      254   315392      442             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   3387]    30  3387    36949      278   319488      432             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   4762]    30  4762    36959      272   319488      436             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   5490]    30  5490    36949      265   315392      427             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   5491]    30  5491    36961      264   315392      449             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   5503]    30  5503    36949      313   319488      393             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   5507]    30  5507    36953      275   319488      432             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   5512]    30  5512    36947      257   319488      448             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   5519]    30  5519    36961      320   319488      384             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   5523]    30  5523    36981      293   319488      429             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   5524]    30  5524    36951      295   319488      419             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   5527]    30  5527    37228      297   319488      683             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   5531]    30  5531    36977      294   319488      429             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   5536]    30  5536    36971      313   319488      403             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   5539]    30  5539    36949      257   319488      450             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   5540]    30  5540    36951      282   319488      426             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   5543]    30  5543    38267      399   327680     1605             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   5545]    30  5545    36977      278   319488      442             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   5547]    30  5547    36972      318   319488      398             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   5550]    30  5550    36974      311   319488      406             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   5552]    30  5552    36949      318   319488      392             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   5556]    30  5556    36953      210   319488      496             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   5558]    30  5558    36952      225   315392      465             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   5562]    30  5562    36953      273   319488      440             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   5563]    30  5563    36971      323   319488      399             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   5571]    30  5571    36959      284   319488      437             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   5573]    30  5573    36948      317   319488      394             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   5574]    30  5574    36972      291   319488      428             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   5579]    30  5579    36987      265   319488      468             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   5609]    30  5609    36957      235   319488      469             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   5611]    30  5611    36947      187   315392      501             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   5612]    30  5612    36977      313   319488      413             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   5615]    30  5615    36953      304   319488      397             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   5617]    30  5617    36953      269   315392      429             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   5621]    30  5621    36951      306   315392      389             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   5624]    30  5624    36962      324   319488      398             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   5632]    30  5632    36977      318   319488      403             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   5634]    30  5634    36952      261   315392      437             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   5640]    30  5640    36949      298   319488      415             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   7590]    30  7590    36947      318   315392      377             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   7591]    30  7591    36953      291   315392      400             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   7594]    30  7594    36963      324   315392      383             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   7596]    30  7596    36949      310   319488      401             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   7597]    30  7597    36951      284   319488      427             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   7667]    30  7667    36949      278   319488      433             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   7670]    30  7670    36947      281   319488      427             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   7675]    30  7675    36962      315   319488      398             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   7676]    30  7676    36981      259   319488      459             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   7688]    30  7688    36949      264   319488      438             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   7693]    30  7693    36953      298   315392      408             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   7694]    30  7694    36963      320   319488      396             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   7695]    30  7695    36949      270   319488      440             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  10316]    30 10316    36949      241   319488      460             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  10319]    30 10319    36974      287   319488      431             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  10324]    30 10324    36961      321   319488      399             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  10326]    30 10326    36949      319   319488      395             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  26935]    26 26935   837409      179   434176      247             0 postgres
Sep 21 00:02:26 openqa kernel: [  26939]    26 26939    26592      126   221184      267             0 postgres
Sep 21 00:02:26 openqa kernel: [  20641]    30 20641    36962      314   319488      395             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  20668]    30 20668    36959      336   319488      384             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  20711]    30 20711    36963      335   319488      374             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  20724]    30 20724    36953      328   319488      381             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  20725]    30 20725    36963      333   319488      376             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   6443]     0  6443  1962560    23553  1011712     3103             0 telegraf
Sep 21 00:02:26 openqa kernel: [  26445]    30 26445    36947      329   319488      378             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  26464]    30 26464    36963      338   319488      377             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  26466]    30 26466    36963      360   319488      351             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  13087]    30 13087    36959      318   319488      390             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   4473]    30  4473    36959      336   319488      369             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   4476]    30  4476    36959      332   319488      371             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   4478]    30  4478    36973      334   319488      378             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   4479]    30  4479    36963      350   319488      370             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   4480]    30  4480    36929      275   319488      415             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   4481]    30  4481    36947      322   315392      370             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   4498]    30  4498    36949      318   319488      393             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   4499]    30  4499    36959      329   319488      372             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   4500]    30  4500    36949      336   319488      367             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   4501]    30  4501    36951      332   315392      379             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   4502]    30  4502    36962      334   319488      372             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   4504]    30  4504    36951      342   319488      364             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   4505]    30  4505    36961      321   319488      389             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   5417]    30  5417    36963      344   319488      370             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   5425]    30  5425    36963      356   319488      365             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   5426]    30  5426    36959      329   319488      375             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   5427]    30  5427    36963      344   319488      365             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   5430]    30  5430    36963      329   319488      377             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   5431]    30  5431    36959      342   319488      376             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   5433]    30  5433    36951      308   319488      390             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   5435]    30  5435    36963      325   319488      382             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   5436]    30  5436    36959      326   319488      382             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   5438]    30  5438    36987      320   319488      410             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  30636]    30 30636    36949      309   315392      383             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  30682]    30 30682    36984      307   319488      418             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  30683]    30 30683    36959      297   319488      410             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  30700]    30 30700    36984      329   319488      398             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  30701]    30 30701    36947      301   319488      391             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  30702]    30 30702    36947      326   319488      374             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  30703]    30 30703    36958      320   319488      390             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  30754]    30 30754    36958      322   319488      386             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  30755]    30 30755    36947      278   315392      415             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  30756]    30 30756    36962      308   315392      398             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  30757]    30 30757    36963      318   319488      398             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  30758]    30 30758    36949      327   319488      374             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  30759]    30 30759    36977      335   319488      385             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  30760]    30 30760    36959      330   319488      377             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  30761]    30 30761    36946      298   319488      409             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  30799]    30 30799    36959      329   319488      384             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  30800]    30 30800    36949      316   319488      385             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  30801]    30 30801    36975      315   319488      398             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  30802]    30 30802    36962      340   319488      366             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  30803]    30 30803    36959      345   319488      364             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  30804]    30 30804    36961      343   319488      370             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  30805]    30 30805    36962      277   319488      428             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  30806]    30 30806    36949      312   315392      379             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  30807]    30 30807    36947      315   319488      386             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  30809]    30 30809    36959      326   319488      377             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  30810]    30 30810    36947      319   315392      380             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  30811]    30 30811    36962      338   315392      370             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  30812]    30 30812    36959      344   319488      364             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  30814]    30 30814    36972      319   319488      391             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  30821]    30 30821    36949      328   315392      364             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  30822]    30 30822    36959      334   319488      375             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  30823]    30 30823    36949      308   319488      396             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  30824]    30 30824    36953      323   319488      371             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  30825]    30 30825    36947      318   319488      384             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  30826]    30 30826    36949      328   319488      374             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  30827]    30 30827    36962      323   319488      382             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  30828]    30 30828    36949      303   319488      401             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  30829]    30 30829    36928      310   315392      379             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  30830]    30 30830    36956      313   315392      378             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  30831]    30 30831    36963      280   315392      413             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  30832]    30 30832    36947      318   315392      371             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  30833]    30 30833    36947      297   315392      393             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  30834]    30 30834    36959      322   319488      385             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  30835]    30 30835    36951      313   315392      382             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  30836]    30 30836    36959      339   319488      366             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  30837]    30 30837    36954      320   319488      383             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  30838]    30 30838    36947      296   315392      393             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  30839]    30 30839    36933      285   315392      391             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  30840]    30 30840    36959      282   319488      417             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  30841]    30 30841    36947      313   319488      386             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  30842]    30 30842    36949      313   319488      381             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  30843]    30 30843    36958      333   319488      374             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  30844]    30 30844    36959      311   319488      393             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  30845]    30 30845    36947      323   319488      385             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  30846]    30 30846    36933      332   319488      365             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  30847]    30 30847    36947      313   319488      385             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  30849]    30 30849    36962      327   315392      376             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  30850]    30 30850    36961      312   319488      385             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  30851]    30 30851    36959      326   319488      394             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  30852]    30 30852    36933      328   319488      371             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  30853]    30 30853    36933      317   315392      380             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  13412]    30 13412    36949      339   319488      363             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  13459]    30 13459    36947      338   319488      366             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  13460]    30 13460    36947      327   319488      364             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  13462]    30 13462    36949      308   315392      388             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  13463]    30 13463    36949      319   319488      391             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  13464]    30 13464    36951      316   315392      380             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  13465]    30 13465    36947      322   315392      370             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  13466]    30 13466    36933      297   315392      397             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  13467]    30 13467    36949      326   319488      379             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  13468]    30 13468    36950      299   315392      393             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  13469]    30 13469    36947      330   319488      377             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  13470]    30 13470    36949      339   319488      363             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  13471]    30 13471    36947      330   319488      367             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  13472]    30 13472    36933      329   315392      367             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  13473]    30 13473    36953      300   319488      401             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  13474]    30 13474    36947      326   315392      369             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  13475]    30 13475    36960      293   319488      410             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  13476]    30 13476    36950      320   319488      386             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  13477]    30 13477    36959      313   319488      385             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  13478]    30 13478    36928      268   315392      422             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  13479]    30 13479    36959      277   319488      430             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  13480]    30 13480    36933      279   315392      414             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  13481]    30 13481    36959      299   319488      409             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  13482]    30 13482    36959      331   319488      388             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  13483]    30 13483    36933      260   315392      434             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  13484]    30 13484    36949      340   319488      370             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  13485]    30 13485    36952      306   319488      393             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  13486]    30 13486    36952      307   315392      385             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  13487]    30 13487    36963      317   319488      387             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  13488]    30 13488    36949      326   319488      374             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  13489]    30 13489    36962      301   319488      404             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  13492]    30 13492    36963      319   319488      393             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  13493]    30 13493    36959      315   319488      389             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  13494]    30 13494    36927      282   315392      405             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  13495]    30 13495    36963      304   319488      403             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  13496]    30 13496    36947      298   319488      405             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  13497]    30 13497    36950      290   315392      403             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  13498]    30 13498    36929      243   307200      422             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  13499]    30 13499    36947      275   315392      416             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  13500]    30 13500    36959      295   319488      408             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  13501]    30 13501    36949      276   315392      414             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  13502]    30 13502    36947      267   319488      433             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  13503]    30 13503    36933      284   315392      408             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  13504]    30 13504    36959      340   319488      362             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  13505]    30 13505    36959      328   319488      376             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  13506]    30 13506    36929      247   315392      441             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  13507]    30 13507    36960      308   319488      396             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  13508]    30 13508    36948      287   315392      408             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  13509]    30 13509    36949      304   315392      386             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  13510]    30 13510    36951      307   319488      390             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  13511]    30 13511    36949      304   315392      401             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  13512]    30 13512    36962      320   319488      393             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  13513]    30 13513    36949      326   315392      371             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  13514]    30 13514    36960      309   319488      398             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  13515]    30 13515    36956      306   315392      390             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  13516]    30 13516    36953      312   319488      388             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  13517]    30 13517    36957      271   315392      433             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  13518]    30 13518    36948      294   319488      409             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  13519]    30 13519    36949      332   319488      369             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  13520]    30 13520    36949      286   319488      414             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  13521]    30 13521    36963      344   319488      367             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  13522]    30 13522    36947      317   319488      378             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  13523]    30 13523    36957      287   319488      412             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  26508]    30 26508    36949      292   315392      398             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  26555]    30 26555    36947      272   315392      417             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  26556]    30 26556    36959      312   319488      392             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  26568]    30 26568    36950      299   315392      400             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  26569]    30 26569    36951      286   319488      410             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  26570]    30 26570    36949      260   315392      431             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  26571]    30 26571    36959      308   319488      393             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  26611]    30 26611    36962      316   319488      390             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  26612]    30 26612    36949      296   319488      397             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  26613]    30 26613    36929      293   315392      399             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  26614]    30 26614    36950      281   315392      414             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  26615]    30 26615    36956      309   319488      398             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  26616]    30 26616    36949      292   315392      398             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  26617]    30 26617    36947      259   315392      431             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  26618]    30 26618    36933      274   315392      403             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   8596]    30  8596    36928      276   315392      428             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   8597]    30  8597    36931      262   315392      427             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  11284]    30 11284    36951      294   319488      406             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   8880]    30  8880    36950      315   319488      384             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   8881]    30  8881    36949      314   315392      380             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   8882]    30  8882    36959      248   319488      450             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   8921]    30  8921    36949      296   319488      397             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   8922]    30  8922    36947      306   315392      386             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   8923]    30  8923    36959      220   319488      483             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   8924]    30  8924    36953      238   315392      457             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   8954]    30  8954    36957      303   319488      397             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   8955]    30  8955    36929      302   319488      392             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   8956]    30  8956    36974      221   315392      493             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   8957]    30  8957    36949      311   319488      383             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   8958]    30  8958    36949      277   315392      416             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   8959]    30  8959    36963      298   319488      419             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   8960]    30  8960    36949      282   315392      414             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   8961]    30  8961    36946      245   315392      446             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   8962]    30  8962    36959      237   319488      462             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   8963]    30  8963    36949      241   319488      451             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   8964]    30  8964    36928      225   315392      462             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   8965]    30  8965    36949      289   319488      412             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   8966]    30  8966    36959      264   319488      442             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   8967]    30  8967    36929      241   315392      452             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   8968]    30  8968    36959      313   319488      394             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   8969]    30  8969    36954      208   315392      481             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   8970]    30  8970    36947      289   319488      416             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   8971]    30  8971    36958      226   319488      486             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   8972]    30  8972    36949      315   319488      381             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   8973]    30  8973    36949      314   319488      393             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   8974]    30  8974    36963      317   319488      392             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   8975]    30  8975    36949      209   315392      482             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   8976]    30  8976    36947      263   315392      432             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   8977]    30  8977    36949      211   315392      482             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   8981]    30  8981    36949      271   319488      426             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   8982]    30  8982    36928      244   315392      441             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   8983]    30  8983    36959      245   319488      454             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   8985]    30  8985    36947      302   319488      394             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   8986]    30  8986    36947      241   315392      456             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   8987]    30  8987    36959      253   319488      451             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   8988]    30  8988    36949      212   319488      490             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   8989]    30  8989    36951      254   315392      440             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   8990]    30  8990    36949      217   315392      474             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   8991]    30  8991    36949      306   319488      390             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   8992]    30  8992    36959      269   319488      447             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   8993]    30  8993    36949      294   315392      401             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   8994]    30  8994    36949      289   319488      413             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   8995]    30  8995    36933      261   315392      432             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   8996]    30  8996    36949      307   315392      385             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   8997]    30  8997    36962      232   319488      471             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   8998]    30  8998    36959      238   319488      462             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   8999]    30  8999    36963      280   319488      428             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   9000]    30  9000    36951      234   315392      459             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   9001]    30  9001    36957      286   319488      431             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   9002]    30  9002    36929      204   315392      478             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   9004]    30  9004    36929      289   319488      402             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   9005]    30  9005    36947      293   319488      403             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   9006]    30  9006    36959      217   319488      484             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   9007]    30  9007    36933      277   319488      416             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   9008]    30  9008    36947      301   319488      391             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   9009]    30  9009    36928      292   315392      397             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   9010]    30  9010    36947      247   315392      449             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   9011]    30  9011    36949      264   315392      430             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   9012]    30  9012    36947      205   315392      483             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   9013]    30  9013    36929      274   315392      414             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   9017]    30  9017    36929      241   315392      450             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  18384]    30 18384    36951      290   319488      411             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  18388]    30 18388    36949      298   315392      395             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  18389]    30 18389    36949      238   319488      461             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  15511]    30 15511    36951      303   315392      391             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  15513]    30 15513    36947      327   315392      367             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  15514]    30 15514    36949      318   319488      378             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  15540]    30 15540    36950      314   319488      380             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  15542]    30 15542    36947      333   319488      367             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  15543]    30 15543    36949      336   319488      361             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  15544]    30 15544    36947      306   315392      383             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  15569]    30 15569    36947      316   315392      373             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  15570]    30 15570    36951      321   315392      372             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  15571]    30 15571    36931      318   315392      375             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  15572]    30 15572    36954      321   315392      371             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  15573]    30 15573    36947      324   315392      367             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  15574]    30 15574    36949      321   319488      378             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  15575]    30 15575    36949      311   315392      380             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  15576]    30 15576    36933      318   319488      378             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  15596]    30 15596    36929      290   315392      398             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  15597]    30 15597    36949      325   315392      365             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  15598]    30 15598    36929      329   315392      363             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  15599]    30 15599    36947      322   315392      370             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  15600]    30 15600    36949      320   315392      371             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  15601]    30 15601    36949      313   315392      377             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  15602]    30 15602    36949      328   315392      364             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  15603]    30 15603    36959      332   319488      371             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  15604]    30 15604    36929      320   315392      370             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  15605]    30 15605    36929      317   319488      371             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  15606]    30 15606    36947      312   315392      378             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  15607]    30 15607    36947      328   319488      367             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  15608]    30 15608    36947      324   315392      367             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  15609]    30 15609    36949      335   319488      362             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  15610]    30 15610    36947      326   319488      369             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  15611]    30 15611    36929      318   315392      372             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  15624]    30 15624    36961      303   315392      405             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  15625]    30 15625    36959      332   319488      369             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  15626]    30 15626    36951      327   319488      375             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  15627]    30 15627    36929      323   315392      370             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  15628]    30 15628    36928      320   315392      370             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  15629]    30 15629    36949      312   315392      378             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  15630]    30 15630    36929      304   315392      378             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  15631]    30 15631    36953      316   315392      379             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  15632]    30 15632    36951      313   315392      379             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  15633]    30 15633    36960      327   319488      376             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  15634]    30 15634    36949      316   315392      375             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  15635]    30 15635    36949      312   315392      378             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  15636]    30 15636    36954      310   315392      383             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  15639]    30 15639    36949      328   319488      366             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  15640]    30 15640    36949      320   315392      375             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  15641]    30 15641    36951      321   315392      371             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  15642]    30 15642    36929      320   315392      371             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  15643]    30 15643    36933      302   315392      389             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  15644]    30 15644    36949      315   315392      375             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  15645]    30 15645    36959      341   319488      363             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  15646]    30 15646    36947      305   315392      384             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  15647]    30 15647    36949      315   315392      375             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  15648]    30 15648    36928      304   315392      380             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  15649]    30 15649    36949      320   315392      372             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  15650]    30 15650    36954      325   315392      368             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  15651]    30 15651    36933      305   315392      387             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  15652]    30 15652    36947      311   319488      387             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  15653]    30 15653    36947      310   319488      391             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  15654]    30 15654    36949      313   315392      379             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  15655]    30 15655    36933      304   319488      391             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  15656]    30 15656    36947      314   315392      380             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  15657]    30 15657    36952      304   315392      388             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  32020]    30 32020    36959      311   319488      389             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  32069]    30 32069    36959      306   319488      395             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  32070]    30 32070    36929      301   315392      388             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  32104]    30 32104    36929      314   319488      378             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  32105]    30 32105    36949      302   315392      389             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  32106]    30 32106    36949      300   315392      392             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  32107]    30 32107    36929      300   315392      390             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  32139]    30 32139    36954      325   315392      366             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  32140]    30 32140    36929      313   315392      376             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  32141]    30 32141    36949      316   315392      376             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  32142]    30 32142    36929      308   315392      379             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  32143]    30 32143    36956      318   315392      376             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  32144]    30 32144    36933      303   315392      389             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  32145]    30 32145    36949      344   319488      363             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  32146]    30 32146    36951      315   315392      378             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   8024]    30  8024    36949      316   315392      374             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   8052]    30  8052    36949      314   319488      378             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   8053]    30  8053    36951      325   315392      369             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   8054]    30  8054    36929      313   315392      376             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   8055]    30  8055    36947      314   315392      379             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   8056]    30  8056    36951      328   319488      368             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   8057]    30  8057    36929      305   315392      384             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   8064]    30  8064    36949      293   315392      397             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   8065]    30  8065    36947      316   315392      378             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   8066]    30  8066    36933      338   315392      354             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   8067]    30  8067    36951      333   319488      365             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   8068]    30  8068    36933      312   315392      377             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   8069]    30  8069    36953      335   319488      366             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   8070]    30  8070    36949      334   319488      364             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   8071]    30  8071    36951      325   315392      367             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   8076]    30  8076    36947      290   315392      398             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   8077]    30  8077    36949      327   315392      367             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   8078]    30  8078    36933      313   315392      381             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   8079]    30  8079    36929      317   315392      372             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   8080]    30  8080    36929      306   315392      382             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   8081]    30  8081    36928      319   315392      370             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   8082]    30  8082    36947      342   319488      360             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   8083]    30  8083    36949      318   315392      375             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   8084]    30  8084    36947      308   315392      387             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   8085]    30  8085    36947      309   319488      383             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   8086]    30  8086    36949      299   315392      391             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   8087]    30  8087    36962      333   319488      368             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   8088]    30  8088    36929      328   319488      369             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   8089]    30  8089    36955      337   319488      368             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   8090]    30  8090    36928      313   315392      377             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   8091]    30  8091    36947      314   315392      379             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  30609]    30 30609    36949      330   315392      363             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  19013]    30 19013    36949      287   315392      404             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  19050]    30 19050    36951      310   319488      390             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  19051]    30 19051    36929      288   315392      402             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  19103]    30 19103    36949      328   315392      364             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  19104]    30 19104    36929      261   307200      407             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  19105]    30 19105    36929      253   307200      413             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  19106]    30 19106    36947      319   319488      378             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  19140]    30 19140    36929      313   315392      373             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  19141]    30 19141    36953      314   319488      383             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  19142]    30 19142    36947      307   315392      381             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  19143]    30 19143    36929      256   307200      411             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  19144]    30 19144    36947      317   319488      377             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  19145]    30 19145    36949      295   315392      395             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  19146]    30 19146    36947      317   319488      373             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  19148]    30 19148    36951      320   319488      375             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  19179]    30 19179    36949      331   319488      368             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  19180]    30 19180    36951      293   315392      401             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  19181]    30 19181    36929      307   315392      378             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  19182]    30 19182    36949      304   315392      387             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  19183]    30 19183    36951      291   315392      401             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  19184]    30 19184    36953      303   315392      393             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  19185]    30 19185    36959      322   315392      381             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  19186]    30 19186    36929      252   307200      416             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  19187]    30 19187    36951      331   319488      364             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  19188]    30 19188    36947      311   319488      393             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  19189]    30 19189    36951      296   315392      396             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  19190]    30 19190    36951      294   315392      397             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  19191]    30 19191    36949      319   315392      373             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  19192]    30 19192    36929      225   307200      441             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  19193]    30 19193    36947      317   319488      389             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  19194]    30 19194    36949      296   315392      396             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  19217]    30 19217    36929      298   315392      391             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  19218]    30 19218    36949      303   319488      393             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  19219]    30 19219    36929      246   315392      441             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  19220]    30 19220    36951      309   319488      393             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  19221]    30 19221    36929      282   315392      404             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  19222]    30 19222    36949      307   319488      402             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  19223]    30 19223    36949      300   315392      392             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  19224]    30 19224    36949      289   315392      401             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  19225]    30 19225    36929      273   315392      415             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  19226]    30 19226    36929      245   307200      421             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  19227]    30 19227    36951      315   319488      387             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  19228]    30 19228    36951      301   315392      394             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  19229]    30 19229    36949      301   315392      391             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  19230]    30 19230    36947      299   315392      391             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  19231]    30 19231    36951      306   315392      389             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  19232]    30 19232    36955      298   319488      403             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  19233]    30 19233    36951      307   319488      391             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  19234]    30 19234    36951      314   319488      387             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  19235]    30 19235    36927      194   307200      466             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  19236]    30 19236    36949      299   315392      393             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  19238]    30 19238    36951      303   315392      390             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  19239]    30 19239    36929      254   315392      436             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  19240]    30 19240    36929      181   307200      481             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  19242]    30 19242    36929      292   315392      398             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  19243]    30 19243    36929      249   307200      417             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  19245]    30 19245    36951      303   319488      396             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  19246]    30 19246    36949      303   315392      392             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  19247]    30 19247    36947      261   315392      428             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  19248]    30 19248    36949      309   319488      386             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  19249]    30 19249    36951      301   315392      396             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  19250]    30 19250    36929      267   315392      417             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  19251]    30 19251    36951      306   319488      390             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  19324]    30 19324    36951      317   319488      385             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [   1640]   488  1640    12062      662   126976      181             0 lvm
Sep 21 00:02:26 openqa kernel: [   4920]     0  4920    17730      136   172032       78             0 systemd-user-ru
Sep 21 00:02:26 openqa kernel: [   6777]    51  6777    23862       39   180224      191             0 pickup
Sep 21 00:02:26 openqa kernel: [  16877]    30 16877    36929      285   315392      403             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  16889]    30 16889    36933      296   319488      392             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  16892]    30 16892    36929      249   307200      417             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  16911]    30 16911    36955      290   315392      401             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  16915]    30 16915    36929      234   307200      429             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  16916]    30 16916    36928      282   315392      397             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  16917]    30 16917    36933      293   319488      398             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  16950]    30 16950    36933      289   315392      390             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  16951]    30 16951    36929      297   319488      391             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  16953]    30 16953    36895      198   307200      451             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  16954]    30 16954    36933      274   315392      405             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  16956]    30 16956    36929      240   307200      421             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  16957]    30 16957    36929      257   311296      418             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  16958]    30 16958    36929      301   319488      391             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  16959]    30 16959    36928      273   315392      413             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  16988]    30 16988    36933      278   315392      406             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  16989]    30 16989    36929      299   319488      393             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  16991]    30 16991    36929      292   315392      399             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  16992]    30 16992    36929      290   315392      395             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  16993]    30 16993    36928      291   315392      395             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  16994]    30 16994    36929      240   307200      421             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  16996]    30 16996    36961      294   319488      402             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  16998]    30 16998    36927      257   311296      414             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  16999]    30 16999    36929      258   311296      416             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  17000]    30 17000    36955      288   315392      406             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  17001]    30 17001    36933      288   315392      394             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  17002]    30 17002    36951      303   319488      394             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  17003]    30 17003    36929      300   319488      390             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  17004]    30 17004    36929      290   315392      400             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  17005]    30 17005    36929      243   307200      423             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  17006]    30 17006    36929      238   307200      424             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  17017]    30 17017    36927      242   307200      418             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  17018]    30 17018    36963      303   319488      390             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  17019]    30 17019    36927      271   311296      414             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  17020]    30 17020    36927      260   315392      421             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  17021]    30 17021    36926      278   315392      410             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  17022]    30 17022    36929      286   315392      398             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  17023]    30 17023    36961      303   319488      393             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  17024]    30 17024    36933      304   315392      386             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  17025]    30 17025    36929      272   311296      414             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  17026]    30 17026    36929      287   315392      401             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  17027]    30 17027    36949      293   315392      397             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  17028]    30 17028    36929      291   319488      402             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  17029]    30 17029    36929      238   307200      423             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  17030]    30 17030    36928      287   315392      396             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  17031]    30 17031    36929      239   307200      423             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  17032]    30 17032    36929      265   311296      418             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  17033]    30 17033    36929      244   307200      421             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  17034]    30 17034    36929      244   307200      425             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  17036]    30 17036    36928      294   315392      392             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  17039]    30 17039    36949      287   315392      405             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  17043]    30 17043    36959      227   319488      460             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  17045]    30 17045    36929      270   311296      411             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  17047]    30 17047    36955      290   315392      406             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  17048]    30 17048    36929      239   307200      426             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  17049]    30 17049    36929      299   319488      392             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  17050]    30 17050    36955      297   319488      402             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  17051]    30 17051    36929      241   307200      421             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  17052]    30 17052    36929      294   315392      398             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  17053]    30 17053    36928      291   315392      395             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  17054]    30 17054    36929      245   307200      418             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  17055]    30 17055    36929      294   315392      395             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  17056]    30 17056    36928      284   315392      401             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  20843]     0 20843    17730        2   188416      212             0 systemd-user-ru
Sep 21 00:02:26 openqa kernel: [  24587]    30 24587    36895      281   307200      372             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  24594]    30 24594    36927      305   311296      375             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  24595]    30 24595    36895      280   307200      373             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  24596]    30 24596    36895      278   307200      375             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  24597]    30 24597    36895      278   307200      375             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  24598]    30 24598    36895      277   307200      376             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  24599]    30 24599    36895      277   307200      376             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  24600]    30 24600    36927      308   311296      374             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  24601]    30 24601    36895      282   307200      371             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  24602]    30 24602    36895      283   307200      370             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  24603]    30 24603    36895      279   307200      374             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  24604]    30 24604    36927      311   311296      371             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  24605]    30 24605    36895      277   307200      376             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  24606]    30 24606    36895      278   307200      375             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  24607]    30 24607    36895      279   307200      374             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  24608]    30 24608    36895      276   307200      377             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  24613]    30 24613    36927      303   311296      369             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  24616]    30 24616    36933      317   315392      359             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  24618]    30 24618    36927      302   311296      371             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  24620]    30 24620    36927      299   311296      373             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  24621]    30 24621    36927      304   311296      370             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  24622]    30 24622    36927      299   311296      372             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  24623]    30 24623    36927      303   311296      368             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  24624]    30 24624    36895      277   307200      380             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  24625]    30 24625    36895      278   307200      379             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  24626]    30 24626    36927      296   311296      374             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  24627]    30 24627    36895      279   307200      379             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  24628]    30 24628    36895      272   307200      385             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  24629]    30 24629    36895      285   307200      373             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  24630]    30 24630    36895      278   307200      379             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  24631]    30 24631    36895      279   307200      378             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  24632]    30 24632    36927      299   311296      372             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  24633]    30 24633    36895      284   307200      370             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  24634]    30 24634    36927      300   311296      371             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  24635]    30 24635    36895      275   307200      377             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  24636]    30 24636    36895      274   307200      378             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  24637]    30 24637    36895      278   307200      376             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  24638]    30 24638    36927      300   311296      370             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  24639]    30 24639    36895      279   307200      375             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  24640]    30 24640    36927      302   311296      369             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  24641]    30 24641    36895      277   307200      375             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  24642]    30 24642    36927      298   311296      373             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  24643]    30 24643    36895      290   307200      368             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  24644]    30 24644    36895      283   307200      375             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  24645]    30 24645    36895      280   307200      372             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  24646]    30 24646    36895      284   307200      374             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  24647]    30 24647    36895      278   307200      374             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  24648]    30 24648    36895      286   307200      372             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  24649]    30 24649    36895      283   307200      375             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  24650]    30 24650    36895      286   307200      372             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  24651]    30 24651    36927      299   311296      371             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  24652]    30 24652    36927      301   311296      370             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  24659]    30 24659    36927      299   311296      371             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  24660]    30 24660    36927      299   311296      372             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  24670]    30 24670    36895      287   307200      371             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  24680]    30 24680    36927      299   311296      372             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  24687]    30 24687    36927      301   311296      370             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  24688]    30 24688    36927      302   311296      369             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  24689]    30 24689    36927      298   311296      373             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  24690]    30 24690    36927      300   311296      371             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  24691]    30 24691    36927      298   311296      373             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  24692]    30 24692    36927      301   311296      370             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  24973]  1001 24973    86795    37721   724992     2448             0 openqa
Sep 21 00:02:26 openqa kernel: [  24988]  1001 24988    61936    28015   544768     2248             0 openqa-livehand
Sep 21 00:02:26 openqa kernel: [  24999]  1001 24999    89898    37537   741376     2661             0 openqa
Sep 21 00:02:26 openqa kernel: [  25006]    26 25006   837435      650   671744      236             0 postgres
Sep 21 00:02:26 openqa kernel: [  25007]    26 25007   837409      223   303104      249             0 postgres
Sep 21 00:02:26 openqa kernel: [  25008]    26 25008   837409      190   270336      242             0 postgres
Sep 21 00:02:26 openqa kernel: [  25009]    26 25009   837580      329   270336      264             0 postgres
Sep 21 00:02:26 openqa kernel: [  25010]    26 25010    27159      189   229376      245             0 postgres
Sep 21 00:02:26 openqa kernel: [  25011]    26 25011   837517      196   249856      271             0 postgres
Sep 21 00:02:26 openqa kernel: [  25015]    26 25015   837736      440   331776      235             0 postgres
Sep 21 00:02:26 openqa kernel: [  25016]    26 25016   837736      447   331776      228             0 postgres
Sep 21 00:02:26 openqa kernel: [  25022]    26 25022   837899      544   741376      230             0 postgres
Sep 21 00:02:26 openqa kernel: [  25030]    26 25030   837976     1132   487424      234             0 postgres
Sep 21 00:02:26 openqa kernel: [  25083]  1001 25083    86795    37716   704512     2459             0 openqa
Sep 21 00:02:26 openqa kernel: [  25084]    26 25084   837919      581   356352      228             0 postgres
Sep 21 00:02:26 openqa kernel: [  25128]    26 25128   838005      575   339968      239             0 postgres
Sep 21 00:02:26 openqa kernel: [  25154]    26 25154   837829      524   339968      231             0 postgres
Sep 21 00:02:26 openqa kernel: [  25296]     0 25296    12781      189   139264        6             0 netconfig
Sep 21 00:02:26 openqa kernel: [  25302]    30 25302    36895      266   307200      374             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  25304]    30 25304    36895      265   307200      375             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  25305]    30 25305    36895      267   307200      374             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  25306]    30 25306    36895      265   307200      375             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  25307]    30 25307    36895      265   307200      375             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  25308]    30 25308    36895      271   307200      373             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  25309]    30 25309    36895      263   307200      377             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  25310]    30 25310    36895      265   307200      375             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  25312]    30 25312    36895      265   307200      375             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  25313]    30 25313    36895      267   307200      374             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  25314]    30 25314    36895      267   307200      373             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  25315]    30 25315    36895      268   307200      372             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  25316]    30 25316    36895      262   307200      378             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  25317]    30 25317    36895      266   307200      376             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  25318]    30 25318    36895      266   307200      374             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  25319]    30 25319    36895      268   307200      372             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  25320]    30 25320    36895      266   307200      374             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  25321]    30 25321    36895      266   307200      374             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  25322]    30 25322    36895      268   307200      373             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  25325]    30 25325    36895      265   307200      376             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  25327]    30 25327    36895      267   307200      374             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  25328]    30 25328    36895      266   307200      374             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  25329]    30 25329    36895      264   307200      376             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  25330]    30 25330    36895      263   307200      378             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  25331]    30 25331    36895      260   307200      380             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  25332]    30 25332    36895      264   307200      377             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  25333]    30 25333    36895      268   307200      373             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  25335]    30 25335    36895      266   307200      374             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  25338]    30 25338    36927      304   311296      372             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  25341]    30 25341    36895      267   307200      373             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  25344]    30 25344    36895      267   307200      373             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  25347]     0 25347    13074      468   139264       17             0 netconfig
Sep 21 00:02:26 openqa kernel: [  25365]    30 25365    36895      265   307200      376             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  25366]    30 25366    36895      267   307200      374             0 httpd-prefork
Sep 21 00:02:26 openqa kernel: [  25379]     0 25379   301136     8407   704512     4358             0 salt-minion
Sep 21 00:02:26 openqa kernel: [  25385]  1001 25385    88874    38301   729088     2385             0 openqa
Sep 21 00:02:26 openqa kernel: [  25399]    26 25399   838252     1124   647168      228             0 postgres
Sep 21 00:02:26 openqa kernel: [  25403]  1001 25403    88874    38099   729088     2532             0 openqa
Sep 21 00:02:26 openqa kernel: [  25419]    26 25419   854512     1697   741376      224             0 postgres
Sep 21 00:02:26 openqa kernel: [  25449]    26 25449   854553     1761   634880      221             0 postgres
Sep 21 00:02:26 openqa kernel: [  25453]     0 25453    22647      131   221184      161             0 cron
Sep 21 00:02:26 openqa kernel: [  25468]   488 25468    93037     8182   598016     3052             0 salt-master
Sep 21 00:02:26 openqa kernel: [  25476]     0 25476    13074      469   131072       16             0 netconfig
Sep 21 00:02:26 openqa kernel: [  25477]     0 25477    12880      306   139264        2             0 dns-resolver
Sep 21 00:02:26 openqa kernel: [  25502]    26 25502   837919     2459   823296      225             0 postgres
Sep 21 00:02:26 openqa kernel: [  25523]  1001 25523     3185       87    65536        1             0 fetchneedles
Sep 21 00:02:26 openqa kernel: [  25544]  1001 25544     3185       91    61440        0             0 fetchneedles
Sep 21 00:02:26 openqa kernel: [  25545]  1001 25545    88874    38375   729088     2405             0 openqa
Sep 21 00:02:26 openqa kernel: [  25554]  1001 25554    86828    37785   700416     2405             0 openqa
Sep 21 00:02:26 openqa kernel: [  25559]  1001 25559    88874    38111   724992     2421             0 openqa
Sep 21 00:02:26 openqa kernel: [  25563]    26 25563   837976     1541   557056      223             0 postgres
Sep 21 00:02:26 openqa kernel: [  25567]    26 25567   838043     1713   638976      226             0 postgres
Sep 21 00:02:26 openqa kernel: [  25568]    26 25568   838479      572   380928      225             0 postgres
Sep 21 00:02:26 openqa kernel: [  25573]  1001 25573    89898    38032   729088     2411             0 openqa
Sep 21 00:02:26 openqa kernel: [  25579]  1001 25579    89898    38040   729088     2389             0 openqa
Sep 21 00:02:26 openqa kernel: [  25585]     0 25585    22647      130   221184      157             0 cron
Sep 21 00:02:26 openqa kernel: [  25587]  1001 25587     6109      287    86016        2             0 git
Sep 21 00:02:26 openqa kernel: [  25590]    26 25590   837943     2018   520192      222             0 postgres
Sep 21 00:02:26 openqa kernel: [  25600]  1001 25600    89898    38058   729088     2389             0 openqa
Sep 21 00:02:26 openqa kernel: [  25604]    26 25604   837934     1598   491520      222             0 postgres
Sep 21 00:02:26 openqa kernel: [  25605]  1001 25605    89898    38085   729088     2371             0 openqa
Sep 21 00:02:26 openqa kernel: [  25613]  1001 25613    89898    38017   729088     2386             0 openqa
Sep 21 00:02:26 openqa kernel: [  25615]  1001 25615    89898    37968   729088     2412             0 openqa
Sep 21 00:02:26 openqa kernel: [  25616]     0 25616    22647      130   221184      157             0 cron
Sep 21 00:02:26 openqa kernel: [  25617]    26 25617   837934     1915   524288      223             0 postgres
Sep 21 00:02:26 openqa kernel: [  25625]    26 25625   837965     2204   512000      221             0 postgres
Sep 21 00:02:26 openqa kernel: [  25627]  1001 25627     4403       68    73728        0             0 git
Sep 21 00:02:26 openqa kernel: [  25630]    26 25630   837935     1777   462848      222             0 postgres
Sep 21 00:02:26 openqa kernel: [  25632]  1001 25632    89898    37957   729088     2420             0 openqa
Sep 21 00:02:26 openqa kernel: [  25633]  1001 25633    18990      181   167936        0             0 git-remote-http
Sep 21 00:02:26 openqa kernel: [  25636]  1001 25636    89898    37946   729088     2434             0 openqa
Sep 21 00:02:26 openqa kernel: [  25640]    26 25640   837887     1554   434176      226             0 postgres
Sep 21 00:02:26 openqa kernel: [  25643]  1001 25643    89898    37962   729088     2423             0 openqa
Sep 21 00:02:26 openqa kernel: [  25646]    26 25646   837900     1758   450560      221             0 postgres
Sep 21 00:02:26 openqa kernel: [  25648]  1001 25648    89898    38062   724992     2457             0 openqa
Sep 21 00:02:26 openqa kernel: [  25650]    26 25650   837878     2058   417792      223             0 postgres
Sep 21 00:02:26 openqa kernel: [  25657]    26 25657   837899     1816   450560      221             0 postgres
Sep 21 00:02:26 openqa kernel: [  25658]  1001 25658    89898    37968   724992     2494             0 openqa
Sep 21 00:02:26 openqa kernel: [  25665]  1001 25665    89898    37837   724992     2477             0 openqa
Sep 21 00:02:26 openqa kernel: [  25669]  1001 25669    89898    38152   724992     2467             0 openqa
Sep 21 00:02:26 openqa kernel: [  25670]    26 25670   837868     2047   327680      222             0 postgres
Sep 21 00:02:26 openqa kernel: [  25672]    26 25672   837856     2187   352256      222             0 postgres
Sep 21 00:02:26 openqa kernel: [  25675]     0 25675    12792       34   139264      173             0 cron
Sep 21 00:02:26 openqa kernel: [  25676]     0 25676    22680      127   221184      158             0 cron
Sep 21 00:02:26 openqa kernel: [  25678]     0 25678    22680      124   221184      160             0 cron
Sep 21 00:02:26 openqa kernel: [  25679]     0 25679     3152       71    65536        0             0 sh
Sep 21 00:02:26 openqa kernel: [  25680]     0 25680    22680      125   221184      159             0 cron
Sep 21 00:02:26 openqa kernel: [  25681]     0 25681    22680      126   221184      158             0 cron
Sep 21 00:02:26 openqa kernel: [  25682]     0 25682    22680      125   221184      159             0 cron
Sep 21 00:02:26 openqa kernel: [  25683]     0 25683    22680      125   221184      159             0 cron
Sep 21 00:02:26 openqa kernel: [  25684]     0 25684    22680      125   221184      159             0 cron
Sep 21 00:02:26 openqa kernel: [  25685]    26 25685   837680     2243   339968      221             0 postgres
Sep 21 00:02:26 openqa kernel: [  25687]  1001 25687    89898    37842   724992     2474             0 openqa
Sep 21 00:02:26 openqa kernel: [  25689]     0 25689     3152       71    69632        0             0 run-crons
Sep 21 00:02:26 openqa kernel: [  25691]  1001 25691    89898    37835   724992     2477             0 openqa
Sep 21 00:02:26 openqa kernel: [  25694]    26 25694   837649     1649   307200      224             0 postgres
Sep 21 00:02:26 openqa kernel: [  25695]    26 25695   837680     2083   331776      222             0 postgres
Sep 21 00:02:26 openqa kernel: [  25696]  1001 25696    89898    37643   724992     2580             0 openqa
Sep 21 00:02:26 openqa kernel: [  25698]    26 25698   837447      201   241664      236             0 postgres
Sep 21 00:02:26 openqa kernel: [  25700]  1001 25700    86795    37721   700416     2448             0 openqa
Sep 21 00:02:26 openqa kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=openqa,pid=25385,uid=1001
Sep 21 00:02:26 openqa kernel: Out of memory: Killed process 25385 (openqa) total-vm:355496kB, anon-rss:153204kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:712kB oom_score_adj:0
Sep 21 00:02:26 openqa kernel: oom_reaper: reaped process 25385 (openqa), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Sep 21 00:02:26 openqa kernel: kthreadd invoked oom-killer: gfp_mask=0x40cc0(GFP_KERNEL|__GFP_COMP), order=2, oom_score_adj=0
Sep 21 00:02:26 openqa kernel: CPU: 6 PID: 2 Comm: kthreadd Tainted: G        W    L      5.14.21-150400.24.21-default #1 SLE15-SP4 7550826c4c7e8c258239e300508e0c8b2a69bad2
Sep 21 00:02:26 openqa kernel: Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
…
Sep 21 00:02:26 openqa kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=openqa,pid=25658,uid=1001
Sep 21 00:02:26 openqa kernel: Out of memory: Killed process 25658 (openqa) total-vm:359592kB, anon-rss:152516kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:708kB oom_score_adj:0
Sep 21 00:02:26 openqa kernel: dns-resolver invoked oom-killer: gfp_mask=0x40cc0(GFP_KERNEL|__GFP_COMP), order=2, oom_score_adj=0
Sep 21 00:02:26 openqa kernel: oom_reaper: reaped process 25658 (openqa), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
…
Sep 21 00:02:26 openqa kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=openqa,pid=25669,uid=1001
Sep 21 00:02:26 openqa kernel: Out of memory: Killed process 25669 (openqa) total-vm:359592kB, anon-rss:153048kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:712kB oom_score_adj:0
Sep 21 00:02:26 openqa kernel: oom_reaper: reaped process 25669 (openqa), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
…
Sep 21 01:23:19 openqa kernel: Tasks state (memory values in pages):
Sep 21 01:23:19 openqa kernel: [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
Sep 21 01:23:19 openqa kernel: [    523]     0   523    27415        0   237568      679         -1000 systemd-udevd
Sep 21 01:23:19 openqa kernel: [    676]     0   676    13017        0   135168      164             0 rpc.idmapd
Sep 21 01:23:19 openqa kernel: [    677]     0   677    15090       75   147456      197         -1000 auditd
Sep 21 01:23:19 openqa kernel: [    682]   499   682    19866      383   184320      345          -900 dbus-daemon
Sep 21 01:23:19 openqa kernel: [  31945]     0   693    28166        0   241664      472             0 close
Sep 21 01:23:19 openqa kernel: [    696]     0   696    10708        3   122880      309             0 wickedd-auto4
Sep 21 01:23:19 openqa kernel: [    701]     0   701    10709        3   131072      311             0 wickedd-dhcp4
Sep 21 01:23:19 openqa kernel: [    705]     0   705    10709        0   126976      337             0 wickedd-dhcp6
Sep 21 01:23:19 openqa kernel: [    740]     0   740    10781        0   135168      397             0 wickedd
Sep 21 01:23:19 openqa kernel: [    767]     0   767    10744        0   126976      339             0 wickedd-nanny
Sep 21 01:23:19 openqa kernel: [    788]     0   788    18806        0   184320      436             0 systemd
Sep 21 01:23:19 openqa kernel: [   1307]     0  1307    28974      357   262144      298             0 rebootmgrd
Sep 21 01:23:19 openqa kernel: [   1309]     0  1309     7763        0   102400      134             0 rsyncd
Sep 21 01:23:19 openqa kernel: [   1319]     0  1319     6390        0    94208      127             0 vsftpd
Sep 21 01:23:19 openqa kernel: [   1321]     0  1321    13501        1   139264      170             0 xinetd
Sep 21 01:23:19 openqa kernel: [   1326]   493  1326    15982        0   176128      233             0 rpcbind
Sep 21 01:23:19 openqa kernel: [   1334]     0  1334    18890        0   196608      267         -1000 sshd
Sep 21 01:23:19 openqa kernel: [   1337]   475  1337    39077        1   208896      296             0 chronyd
Sep 21 01:23:19 openqa kernel: [   1380]   497  1380    12618        0   143360      224          -500 nrpe
Sep 21 01:23:19 openqa kernel: [   1502]     0  1502    11481        0   126976      190             0 agetty
Sep 21 01:23:19 openqa kernel: [   1507]     0  1507    11481        0   131072      166             0 agetty
Sep 21 01:23:19 openqa kernel: [   1617]     0  1617    24392        0   176128      258             0 master
Sep 21 01:23:19 openqa kernel: [   1619]    51  1619    29041        0   192512      277             0 qmgr
Sep 21 01:23:19 openqa kernel: [   1632]     0  1632    12792        0   147456      208             0 cron
Sep 21 01:23:19 openqa kernel: [  32142]    30 32142    36929        0   315392        3             0 httpd-prefork
Sep 21 01:23:19 openqa kernel: [   1640]   488  1640    12062        0   126976        0             0 lvm
Sep 21 01:23:19 openqa kernel: [   4920]     0  4920    17730        1   172032      213             0 systemd-user-ru
Sep 21 01:23:19 openqa kernel: [  20843]     0 20843    17730        1   188416      213             0 systemd-user-ru
Sep 21 01:23:19 openqa kernel: [  25379]     0 25379   303441        0   712704        5             0 salt-minion
Sep 21 01:23:19 openqa kernel: [  25829]     0 25829    19755        0   180224        0             0 (ogrotate)
Sep 21 01:23:19 openqa kernel: [  27896]   488 27896    12062        0   114688        0             0 lvm
Sep 21 01:23:19 openqa kernel: [  28021]   488 28021    12062        0   114688        0             0 lvm
Sep 21 01:23:19 openqa kernel: [  28189]   488 28189    12062        0   122880        0             0 lvm
Sep 21 01:23:19 openqa kernel: [  28218]   488 28218    12062        0   114688        0             0 lvm
Sep 21 01:23:19 openqa kernel: [  28244]   488 28244    12062        0   118784        0             0 lvm
Sep 21 01:23:19 openqa kernel: [  28562]     0 28562    10013        0   118784        0             0 lvm
Sep 21 01:23:19 openqa kernel: [  28947]     0 28947    10013        0   110592        0             0 lvm
Sep 21 01:23:19 openqa kernel: [  30535]     0 30535    17705        0   176128      211             0 systemd-user-ru
Sep 21 01:23:19 openqa kernel: [  31136]     0 31136    22647        0   221184      320             0 cron
Sep 21 01:23:19 openqa kernel: [  31224]     0 31224    22647        0   221184      313             0 cron
Sep 21 01:23:19 openqa kernel: [  31770]    51 31770    23862        0   180224      240             0 pickup
Sep 21 01:23:19 openqa kernel: [  31785]     0 31785    22647        0   221184      295             0 cron
Sep 21 01:23:19 openqa kernel: [  31786]     0 31786    22647        0   221184      307             0 cron
Sep 21 01:23:19 openqa kernel: [  31798]     0 31798    22647        0   221184      295             0 cron
Sep 21 01:23:19 openqa kernel: [  31800]     0 31800    22647        0   221184      295             0 cron
Sep 21 01:23:19 openqa kernel: [  31802]     0 31802    22647        0   221184      295             0 cron
Sep 21 01:23:19 openqa kernel: [  31808]     0 31808    22647        0   221184      295             0 cron
Sep 21 01:23:20 openqa kernel: [  31809]     0 31809    22647        0   221184      311             0 cron
Sep 21 01:23:20 openqa kernel: [  31810]     0 31810    22647        0   221184      300             0 cron
Sep 21 01:23:20 openqa kernel: [  31820]     0 31820    22647        0   221184      295             0 cron
Sep 21 01:23:20 openqa kernel: [  31821]     0 31821    22647        0   221184      322             0 cron
Sep 21 01:23:20 openqa kernel: [  31822]     0 31822    22647        0   221184      295             0 cron
Sep 21 01:23:20 openqa kernel: [  31823]     0 31823    22647        0   221184      296             0 cron
Sep 21 01:23:20 openqa kernel: [  31824]     0 31824    22647        0   221184      293             0 cron
Sep 21 01:23:20 openqa kernel: [  31825]     0 31825    22647        0   221184      310             0 cron
Sep 21 01:23:20 openqa kernel: [  31826]     0 31826    22647        0   221184      301             0 cron
Sep 21 01:23:20 openqa kernel: [  31829]     0 31829    22647        0   221184      293             0 cron
Sep 21 01:23:20 openqa kernel: [  31832]     0 31832    22647        0   221184      293             0 cron
Sep 21 01:23:20 openqa kernel: [  31839]     0 31839    22647        0   221184      293             0 cron
Sep 21 01:23:20 openqa kernel: [  31840]     0 31840    22647        0   221184      295             0 cron
Sep 21 01:23:20 openqa kernel: [  31841]     0 31841    22647        0   221184      293             0 cron
Sep 21 01:23:20 openqa kernel: [  31843]     0 31843    12792        0   139264      236             0 cron
Sep 21 01:23:20 openqa kernel: [  31844]     0 31844    22647        0   221184      294             0 cron
Sep 21 01:23:20 openqa kernel: [  31845]     0 31845    22647        0   221184      293             0 cron
Sep 21 01:23:20 openqa kernel: [  31846]     0 31846    22647        0   221184      293             0 cron
Sep 21 01:23:20 openqa kernel: [  31847]     0 31847    22647        0   221184      293             0 cron
Sep 21 01:23:20 openqa kernel: [  31851]     0 31851    22647        0   221184      293             0 cron
Sep 21 01:23:20 openqa kernel: [  31852]     0 31852    22647        0   221184      293             0 cron
Sep 21 01:23:20 openqa kernel: [  31854]     0 31854    22647        0   221184      293             0 cron
Sep 21 01:23:20 openqa kernel: [  31856]     0 31856    22647        0   221184      293             0 cron
Sep 21 01:23:20 openqa kernel: [  31863]     0 31863    22647        0   221184      294             0 cron
Sep 21 01:23:20 openqa kernel: [  31864]     0 31864    22647        0   221184      293             0 cron
Sep 21 01:23:20 openqa kernel: [  31869]     0 31869    22647        0   221184      295             0 cron
Sep 21 01:23:20 openqa kernel: [  31870]     0 31870    22647        0   221184      293             0 cron
Sep 21 01:23:20 openqa kernel: [  32343]     0 32343    22647        0   221184      295             0 cron
Sep 21 01:23:20 openqa kernel: [  32591]     0 32591     3929        1    77824       47             0 find
Sep 21 01:23:20 openqa kernel: [  32754]     0 32754    22647        0   221184      304             0 cron
Sep 21 01:23:20 openqa kernel: [  32755]     0 32755    22647        0   221184      304             0 cron
Sep 21 01:23:20 openqa kernel: [  32756]     0 32756    22647        0   221184      304             0 cron
Sep 21 01:23:20 openqa kernel: [  32757]     0 32757    22647        0   221184      303             0 cron
Sep 21 01:23:20 openqa kernel: [    420]     0   420    22647        0   221184      308             0 cron
Sep 21 01:23:20 openqa kernel: [    421]     0   421    22647        0   221184      303             0 cron
Sep 21 01:23:20 openqa kernel: [    531]     0   531     3152        0    73728       73             0 reload-webserve
Sep 21 01:23:20 openqa kernel: [    532]     0   532    17854        0   167936      240             0 systemctl
Sep 21 01:23:20 openqa kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=systemd-logind,pid=693,uid=0
Sep 21 01:23:20 openqa kernel: Out of memory: Killed process 31945 (close) total-vm:112664kB, anon-rss:0kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:236kB oom_score_adj:0
Sep 21 01:23:20 openqa kernel: oom_reaper: reaped process 31945 (close), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Sep 21 01:23:20 openqa kernel: systemd invoked oom-killer: gfp_mask=0x40cc0(GFP_KERNEL|__GFP_COMP), order=2, oom_score_adj=0
Sep 21 01:23:20 openqa kernel: CPU: 11 PID: 1 Comm: systemd Tainted: G        W    L      5.14.21-150400.24.21-default #1 SLE15-SP4 7550826c4c7e8c258239e300508e0c8b2a69bad2
Sep 21 01:23:20 openqa kernel: Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
Sep 21 01:23:20 openqa kernel: Call Trace:
Sep 21 01:23:20 openqa kernel:  <TASK>
Sep 21 01:23:20 openqa kernel:  dump_stack_lvl+0x45/0x5b
Sep 21 01:23:20 openqa kernel:  dump_header+0x4a/0x220
Sep 21 01:23:20 openqa kernel:  oom_kill_process+0xe8/0x140
Sep 21 01:23:20 openqa kernel:  out_of_memory+0x113/0x580
Sep 21 01:23:20 openqa kernel:  __alloc_pages_slowpath.constprop.112+0x980/0xc70
Sep 21 01:23:20 openqa kernel:  __alloc_pages+0x2d9/0x320
Sep 21 01:23:20 openqa kernel:  new_slab+0x33e/0x500
Sep 21 01:23:20 openqa kernel:  ___slab_alloc+0x6ad/0x860
Sep 21 01:23:20 openqa kernel:  ? copy_process+0x213/0x1cd0
Sep 21 01:23:20 openqa kernel:  ? __mod_memcg_lruvec_state+0x1d/0x100
Sep 21 01:23:20 openqa kernel:  ? __mod_lruvec_page_state+0x6f/0xb0
Sep 21 01:23:20 openqa kernel:  ? lock_page_memcg+0x1c/0xa0
Sep 21 01:23:20 openqa kernel:  ? __slab_alloc.isra.92+0x1c/0x40
Sep 21 01:23:20 openqa kernel:  __slab_alloc.isra.92+0x1c/0x40
Sep 21 01:23:20 openqa kernel:  kmem_cache_alloc_node+0x114/0x310
Sep 21 01:23:20 openqa kernel:  ? copy_process+0x213/0x1cd0
Sep 21 01:23:20 openqa kernel:  copy_process+0x213/0x1cd0
Sep 21 01:23:20 openqa kernel:  kernel_clone+0x9b/0x400
Sep 21 01:23:20 openqa kernel:  __do_sys_clone+0x72/0xa0
Sep 21 01:23:20 openqa kernel:  do_syscall_64+0x5b/0x80
Sep 21 01:23:20 openqa kernel:  ? exc_page_fault+0x67/0x150
Sep 21 01:23:20 openqa kernel:  entry_SYSCALL_64_after_hwframe+0x61/0xcb
Sep 21 01:23:20 openqa kernel: RIP: 0033:0x7f15c3718a68
Sep 21 01:23:20 openqa kernel: Code: Unable to access opcode bytes at RIP 0x7f15c3718a3e.
Sep 21 01:23:20 openqa kernel: RSP: 002b:00007ffc905fd240 EFLAGS: 00000246 ORIG_RAX: 0000000000000038
Sep 21 01:23:20 openqa kernel: RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f15c3718a68
Sep 21 01:23:20 openqa kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000001200011
Sep 21 01:23:20 openqa kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: 00007f15c5446b00
Sep 21 01:23:20 openqa kernel: R10: 00007f15c5446dd0 R11: 0000000000000246 R12: 0000556f45bd8720
Sep 21 01:23:20 openqa kernel: R13: 00007ffc905fd490 R14: 0000556f45cc9758 R15: 0000000000000000
Sep 21 01:23:20 openqa kernel:  </TASK>
Sep 21 01:23:20 openqa kernel: Mem-Info:
Sep 21 01:23:20 openqa kernel: active_anon:2 inactive_anon:196 isolated_anon:0
                                active_file:3131 inactive_file:4105 isolated_file:0
                                unevictable:20 dirty:0 writeback:0
                                slab_reclaimable:4904595 slab_unreclaimable:1125985
                                mapped:580 shmem:3 pagetables:3856 bounce:0
                                free:77366 free_pcp:3099 free_cma:0
Sep 21 01:23:20 openqa kernel: Node 0 active_anon:8kB inactive_anon:784kB active_file:12524kB inactive_file:16420kB unevictable:80kB isolated(anon):0kB isolated(file):0kB mapped:2320kB dirty:0kB writeback:0kB shmem:12kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB kernel_stack:4864kB pagetab>
Sep 21 01:23:20 openqa kernel: Node 0 DMA free:11024kB boost:0kB min:40kB low:52kB high:64kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Sep 21 01:23:20 openqa kernel: lowmem_reserve[]: 0 2991 24013 24013 24013
Sep 21 01:23:20 openqa kernel: Node 0 DMA32 free:114856kB boost:0kB min:8356kB low:11392kB high:14428kB reserved_highatomic:14336KB active_anon:0kB inactive_anon:0kB active_file:4kB inactive_file:4kB unevictable:0kB writepending:0kB present:3129312kB managed:3063776kB mlocked:0kB bounce:0kB free_pcp:1976kB local_p>
Sep 21 01:23:20 openqa kernel: lowmem_reserve[]: 0 0 21021 21021 21021
Sep 21 01:23:20 openqa kernel: Node 0 Normal free:183584kB boost:0kB min:59184kB low:80700kB high:102216kB reserved_highatomic:20480KB active_anon:8kB inactive_anon:904kB active_file:12492kB inactive_file:16808kB unevictable:80kB writepending:0kB present:22020096kB managed:21525736kB mlocked:80kB bounce:0kB free_p>
Sep 21 01:23:20 openqa kernel: lowmem_reserve[]: 0 0 0 0 0
Sep 21 01:23:20 openqa kernel: Node 0 DMA: 0*4kB 0*8kB 1*16kB (U) 2*32kB (UE) 1*64kB (U) 1*128kB (U) 2*256kB (UE) 2*512kB (UE) 1*1024kB (E) 2*2048kB (ME) 1*4096kB (M) = 11024kB
Sep 21 01:23:20 openqa kernel: Node 0 DMA32: 7927*4kB (UM) 10421*8kB (UME) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 115076kB
Sep 21 01:23:20 openqa kernel: Node 0 Normal: 44275*4kB (UM) 836*8kB (UM) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 183788kB
Sep 21 01:23:20 openqa kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Sep 21 01:23:20 openqa kernel: 7577 total pagecache pages
Sep 21 01:23:20 openqa kernel: 180 pages in swap cache
Sep 21 01:23:20 openqa kernel: Swap cache stats: add 11843458, delete 11843174, find 99241998/104794634
Sep 21 01:23:20 openqa kernel: Free swap  = 439684kB
Sep 21 01:23:20 openqa kernel: Total swap = 499708kB
Sep 21 01:23:20 openqa kernel: 6291350 pages RAM
Sep 21 01:23:20 openqa kernel: 0 pages HighMem/MovableOnly
Sep 21 01:23:20 openqa kernel: 140132 pages reserved
Sep 21 01:23:20 openqa kernel: 0 pages cma reserved
Sep 21 01:23:20 openqa kernel: 0 pages hwpoisoned
Sep 21 01:23:20 openqa kernel: Unreclaimable slab info:
Sep 21 01:23:20 openqa kernel: Name                      Used          Total
Sep 21 01:23:20 openqa kernel: xfs_icr                   47KB         47KB
Sep 21 01:23:20 openqa kernel: xfs_efi_item             392KB        410KB
Sep 21 01:23:20 openqa kernel: xfs_buf_item             447KB        447KB
Sep 21 01:23:20 openqa kernel: xfs_da_state             191KB        191KB
Sep 21 01:23:20 openqa kernel: xfs_btree_cur             94KB         94KB
Sep 21 01:23:20 openqa kernel: nfsd4_openowners         372KB        391KB
Sep 21 01:23:20 openqa kernel: nfsd4_clients            127KB        127KB
Sep 21 01:23:20 openqa kernel: scsi_sense_cache           4KB          4KB
Sep 21 01:23:20 openqa kernel: ip6-frags                 47KB         47KB
Sep 21 01:23:20 openqa kernel: PINGv6                   370KB        370KB
Sep 21 01:23:20 openqa kernel: RAWv6                   8148KB       8148KB
Sep 21 01:23:20 openqa kernel: UDPv6                   3374KB       3374KB
Sep 21 01:23:20 openqa kernel: tw_sock_TCPv6           2229KB       2229KB
Sep 21 01:23:20 openqa kernel: request_sock_TCPv6        926KB        926KB
Sep 21 01:23:20 openqa kernel: TCPv6                   3206KB       4519KB
Sep 21 01:23:20 openqa kernel: bfq_io_cq                 95KB         95KB
Sep 21 01:23:20 openqa kernel: mqueue_inode_cache        496KB        496KB
Sep 21 01:23:20 openqa kernel: dnotify_struct             4KB          4KB
Sep 21 01:23:20 openqa kernel: UNIX                    1133KB       1173KB
Sep 21 01:23:20 openqa kernel: ip4-frags                  3KB          3KB
Sep 21 01:23:20 openqa kernel: PING                     384KB        384KB
Sep 21 01:23:20 openqa kernel: RAW                    10160KB      10160KB
Sep 21 01:23:20 openqa kernel: tw_sock_TCP               95KB         95KB
Sep 21 01:23:20 openqa kernel: request_sock_TCP         686KB        717KB
Sep 21 01:23:20 openqa kernel: TCP                      693KB        913KB
Sep 21 01:23:20 openqa kernel: hugetlbfs_inode_cache        126KB        126KB
Sep 21 01:23:20 openqa kernel: ep_head                   48KB         48KB
Sep 21 01:23:20 openqa kernel: bio_crypt_ctx            394KB        394KB
Sep 21 01:23:20 openqa kernel: request_queue            220KB        220KB
Sep 21 01:23:20 openqa kernel: biovec-max               556KB        604KB
Sep 21 01:23:20 openqa kernel: biovec-128               612KB        612KB
Sep 21 01:23:20 openqa kernel: biovec-64                384KB        384KB
Sep 21 01:23:20 openqa kernel: khugepaged_mm_slot         78KB         78KB
Sep 21 01:23:20 openqa kernel: dmaengine-unmap-256         30KB         30KB
Sep 21 01:23:20 openqa kernel: dmaengine-unmap-128         31KB         31KB
Sep 21 01:23:20 openqa kernel: skbuff_ext_cache       21628KB      21628KB
Sep 21 01:23:20 openqa kernel: skbuff_fclone_cache        399KB        436KB
Sep 21 01:23:20 openqa kernel: skbuff_head_cache       1447KB       1516KB
Sep 21 01:23:20 openqa kernel: file_lock_cache           93KB         93KB
Sep 21 01:23:20 openqa kernel: file_lock_ctx            263KB        263KB
Sep 21 01:23:20 openqa kernel: fsnotify_mark_connector         48KB         48KB
Sep 21 01:23:20 openqa kernel: net_namespace           1750KB       1750KB
Sep 21 01:23:20 openqa kernel: task_delay_info           71KB         71KB
Sep 21 01:23:20 openqa kernel: taskstats                228KB        228KB
Sep 21 01:23:20 openqa kernel: proc_dir_entry          6855KB       6855KB
Sep 21 01:23:20 openqa kernel: pde_opener                47KB         47KB
Sep 21 01:23:20 openqa kernel: seq_file                  47KB         47KB
Sep 21 01:23:20 openqa kernel: shmem_inode_cache       3537KB       3537KB
Sep 21 01:23:20 openqa kernel: kernfs_node_cache       9236KB       9236KB
Sep 21 01:23:20 openqa kernel: mnt_cache                826KB        826KB
Sep 21 01:23:20 openqa kernel: filp                 2139592KB    2139592KB
Sep 21 01:23:20 openqa kernel: names_cache              424KB        440KB
Sep 21 01:23:20 openqa kernel: lsm_file_cache           525KB        549KB
Sep 21 01:23:20 openqa kernel: uts_namespace             93KB         93KB
Sep 21 01:23:20 openqa kernel: vm_area_struct          4155KB       5608KB
Sep 21 01:23:20 openqa kernel: mm_struct               1601KB       1816KB
Sep 21 01:23:20 openqa kernel: files_cache             1205KB       1205KB
Sep 21 01:23:20 openqa kernel: signal_cache           86060KB      86060KB
Sep 21 01:23:20 openqa kernel: sighand_cache           1423KB       1705KB
Sep 21 01:23:20 openqa kernel: task_struct           634721KB     634721KB
Sep 21 01:23:20 openqa kernel: cred_jar               73485KB      73485KB
Sep 21 01:23:20 openqa kernel: anon_vma_chain          1317KB       1916KB
Sep 21 01:23:20 openqa kernel: anon_vma                1188KB       1671KB
Sep 21 01:23:20 openqa kernel: pid                   154604KB     154604KB
Sep 21 01:23:20 openqa kernel: Acpi-Operand             165KB        165KB
Sep 21 01:23:20 openqa kernel: Acpi-ParseExt              7KB          7KB
Sep 21 01:23:20 openqa kernel: Acpi-State                71KB         71KB
Sep 21 01:23:20 openqa kernel: numa_policy                7KB          7KB
Sep 21 01:23:20 openqa kernel: perf_event                31KB         31KB
Sep 21 01:23:20 openqa kernel: trace_event_file         430KB        430KB
Sep 21 01:23:20 openqa kernel: ftrace_event_field        290KB        290KB
Sep 21 01:23:20 openqa kernel: pool_workqueue          1370KB       1376KB
Sep 21 01:23:20 openqa kernel: task_group               189KB        189KB
Sep 21 01:23:20 openqa kernel: vmap_area                301KB        732KB
Sep 21 01:23:20 openqa kernel: kmalloc-cg-8k             96KB         96KB
Sep 21 01:23:20 openqa kernel: kmalloc-cg-4k            544KB        544KB
Sep 21 01:23:20 openqa kernel: kmalloc-cg-2k            624KB        652KB
Sep 21 01:23:20 openqa kernel: kmalloc-cg-1k           1468KB       1740KB
Sep 21 01:23:20 openqa kernel: kmalloc-cg-512          2348KB       2348KB
Sep 21 01:23:20 openqa kernel: kmalloc-cg-256            96KB         96KB
Sep 21 01:23:20 openqa kernel: kmalloc-cg-192           315KB        315KB
Sep 21 01:23:20 openqa kernel: kmalloc-cg-96             47KB         47KB
Sep 21 01:23:20 openqa kernel: kmalloc-cg-64            104KB        104KB
Sep 21 01:23:20 openqa kernel: kmalloc-cg-32             48KB         48KB
Sep 21 01:23:20 openqa kernel: kmalloc-cg-16           2144KB       2144KB
Sep 21 01:23:20 openqa kernel: kmalloc-cg-8            3112KB       3112KB
Sep 21 01:23:20 openqa kernel: kmalloc-8k             12984KB      12984KB
Sep 21 01:23:20 openqa kernel: kmalloc-4k             33284KB      33292KB
Sep 21 01:23:20 openqa kernel: kmalloc-2k             34808KB      34808KB
Sep 21 01:23:20 openqa kernel: kmalloc-1k            250490KB     250716KB
Sep 21 01:23:20 openqa kernel: kmalloc-512           122796KB     124608KB
Sep 21 01:23:20 openqa kernel: kmalloc-256           103948KB     103948KB
Sep 21 01:23:20 openqa kernel: kmalloc-192           247859KB     247877KB
Sep 21 01:23:20 openqa kernel: kmalloc-128             5032KB       5032KB
Sep 21 01:23:20 openqa kernel: kmalloc-96              7508KB       7516KB
Sep 21 01:23:20 openqa kernel: kmalloc-64             75792KB      75816KB
Sep 21 01:23:20 openqa kernel: kmalloc-32             20752KB      20752KB
Sep 21 01:23:20 openqa kernel: kmalloc-16             19604KB      19604KB
Sep 21 01:23:20 openqa kernel: kmalloc-8                416KB        416KB
Sep 21 01:23:20 openqa kernel: kmem_cache_node           32KB         32KB
Sep 21 01:23:20 openqa kernel: kmem_cache                80KB         80KB
Sep 21 01:23:36 openqa kernel: Tasks state (memory values in pages):
Sep 21 01:23:36 openqa kernel: [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
Sep 21 01:23:36 openqa kernel: [    523]     0   523    27415        0   237568      679         -1000 systemd-udevd
Sep 21 01:23:36 openqa kernel: [    676]     0   676    13017        0   135168      164             0 rpc.idmapd
Sep 21 01:23:36 openqa kernel: [    677]     0   677    15090       75   147456      197         -1000 auditd
Sep 21 01:23:36 openqa kernel: [    682]   499   682    19866      481   184320      247          -900 dbus-daemon
Sep 21 01:23:36 openqa kernel: [  31945]     0   693    28166        0   241664        0             0 close
Sep 21 01:23:36 openqa kernel: [    696]     0   696    10708        3   122880      309             0 wickedd-auto4
Sep 21 01:23:36 openqa kernel: [    701]     0   701    10709        3   131072      311             0 wickedd-dhcp4
Sep 21 01:23:36 openqa kernel: [   1309]     0  1309     7763        0   102400      134             0 rsyncd
Sep 21 01:23:36 openqa kernel: [   1319]     0  1319     6390        0    94208      127             0 vsftpd
Sep 21 01:23:36 openqa kernel: [   1321]     0  1321    13501        1   139264      170             0 xinetd
Sep 21 01:23:36 openqa kernel: [   1326]   493  1326    15982        0   176128      233             0 rpcbind
Sep 21 01:23:36 openqa kernel: [   1334]     0  1334    18890        0   196608      267         -1000 sshd
Sep 21 01:23:36 openqa kernel: [   1337]   475  1337    39077        1   208896      296             0 chronyd
Sep 21 01:23:36 openqa kernel: [   1380]   497  1380    12618        0   143360      224          -500 nrpe
Sep 21 01:23:36 openqa kernel: [   1502]     0  1502    11481        0   126976      190             0 agetty
Sep 21 01:23:36 openqa kernel: [   1507]     0  1507    11481        0   131072      166             0 agetty
Sep 21 01:23:36 openqa kernel: [   1617]     0  1617    24392        0   176128      258             0 master
Sep 21 01:23:36 openqa kernel: [   1619]    51  1619    29041        0   192512      277             0 qmgr
Sep 21 01:23:36 openqa kernel: [   1632]     0  1632    12792        0   147456      208             0 cron
Sep 21 01:23:36 openqa kernel: [  32142]    30 32142    36929        0   315392        3             0 httpd-prefork
Sep 21 01:23:36 openqa kernel: [   1640]   488  1640    12062        0   126976        0             0 lvm
Sep 21 01:23:36 openqa kernel: [   4920]     0  4920    17730        1   172032      213             0 systemd-user-ru
Sep 21 01:23:36 openqa kernel: [  20843]     0 20843    17730        1   188416      213             0 systemd-user-ru
Sep 21 01:23:36 openqa kernel: [  25379]     0 25379   303441        0   712704        5             0 salt-minion
Sep 21 01:23:36 openqa kernel: [  25829]     0 25829    19755        0   180224        0             0 (ogrotate)
Sep 21 01:23:36 openqa kernel: [  27896]   488 27896    12062        0   114688        0             0 lvm
Sep 21 01:23:36 openqa kernel: [  28021]   488 28021    12062        0   114688        0             0 lvm
Sep 21 01:23:36 openqa kernel: [  28189]   488 28189    12062        0   122880        0             0 lvm
Sep 21 01:23:36 openqa kernel: [  28218]   488 28218    12062        0   114688        0             0 lvm
Sep 21 01:23:36 openqa kernel: [  28244]   488 28244    12062        0   118784        0             0 lvm
Sep 21 01:23:36 openqa kernel: [  28562]     0 28562    10013        0   118784        0             0 lvm
Sep 21 01:23:36 openqa kernel: [  28947]     0 28947    10013        0   110592        0             0 lvm
Sep 21 01:23:36 openqa kernel: [  30535]     0 30535    17705        0   176128      211             0 systemd-user-ru
Sep 21 01:23:36 openqa kernel: [  31770]    51 31770    23862        0   180224      240             0 pickup
Sep 21 01:23:37 openqa kernel: [  31785]     0 31785    22647        0   221184      295             0 cron
Sep 21 01:23:37 openqa kernel: [  31798]     0 31798    22647        0   221184      295             0 cron
Sep 21 01:23:37 openqa kernel: [  31800]     0 31800    22647        0   221184      295             0 cron
Sep 21 01:23:37 openqa kernel: [  31802]     0 31802    22647        0   221184      295             0 cron
Sep 21 01:23:37 openqa kernel: [  31808]     0 31808    22647        0   221184      295             0 cron
Sep 21 01:23:37 openqa kernel: [  31810]     0 31810    22647        0   221184      300             0 cron
Sep 21 01:23:37 openqa kernel: [  31820]     0 31820    22647        0   221184      295             0 cron
Sep 21 01:23:37 openqa kernel: [  31822]     0 31822    22647        0   221184      295             0 cron
Sep 21 01:23:37 openqa kernel: [  31823]     0 31823    22647        0   221184      296             0 cron
Sep 21 01:23:37 openqa kernel: [  31824]     0 31824    22647        0   221184      293             0 cron
Sep 21 01:23:37 openqa kernel: [  31826]     0 31826    22647        0   221184      301             0 cron
Sep 21 01:23:37 openqa kernel: [  31829]     0 31829    22647        0   221184      293             0 cron
Sep 21 01:23:37 openqa kernel: [  31832]     0 31832    22647        0   221184      293             0 cron
Sep 21 01:23:37 openqa kernel: [  31839]     0 31839    22647        0   221184      293             0 cron
Sep 21 01:23:37 openqa kernel: [  31840]     0 31840    22647        0   221184      295             0 cron
Sep 21 01:23:37 openqa kernel: [  31841]     0 31841    22647        0   221184      293             0 cron
Sep 21 01:23:37 openqa kernel: [  31843]     0 31843    12792        0   139264      236             0 cron
Sep 21 01:23:37 openqa kernel: [  31844]     0 31844    22647        0   221184      294             0 cron
Sep 21 01:23:37 openqa kernel: [  31845]     0 31845    22647        0   221184      293             0 cron
Sep 21 01:23:37 openqa kernel: [  31846]     0 31846    22647        0   221184      293             0 cron
Sep 21 01:23:37 openqa kernel: [  31847]     0 31847    22647        0   221184      293             0 cron
Sep 21 01:23:37 openqa kernel: [  31851]     0 31851    22647        0   221184      293             0 cron
Sep 21 01:23:37 openqa kernel: [  31852]     0 31852    22647        0   221184      293             0 cron
Sep 21 01:23:37 openqa kernel: [  31854]     0 31854    22647        0   221184      293             0 cron
Sep 21 01:23:37 openqa kernel: [  31856]     0 31856    22647        0   221184      293             0 cron
Sep 21 01:23:37 openqa kernel: [  31863]     0 31863    22647        0   221184      294             0 cron
Sep 21 01:23:37 openqa kernel: [  31864]     0 31864    22647        0   221184      293             0 cron
Sep 21 01:23:37 openqa kernel: [  31869]     0 31869    22647        0   221184      295             0 cron
Sep 21 01:23:37 openqa kernel: [  31870]     0 31870    22647        0   221184      293             0 cron
Sep 21 01:23:37 openqa kernel: [  32343]     0 32343    22647        0   221184      295             0 cron
Sep 21 01:23:37 openqa kernel: [  32591]     0 32591     3929        1    77824       47             0 find
Sep 21 01:23:37 openqa kernel: [  32754]     0 32754    22647        0   221184      304             0 cron
Sep 21 01:23:37 openqa kernel: [  32757]     0 32757    22647        0   221184      303             0 cron
Sep 21 01:23:37 openqa kernel: [    421]     0   421    22647        0   221184      303             0 cron
Sep 21 01:23:37 openqa kernel: [    531]     0   531     3152        0    73728       73             0 reload-webserve
Sep 21 01:23:37 openqa kernel: [    532]     0   532    17854        0   167936      240             0 systemctl
Sep 21 01:23:37 openqa kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=cron,pid=32754,uid=0
Sep 21 01:23:37 openqa kernel: Out of memory: Killed process 32754 (cron) total-vm:90588kB, anon-rss:0kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:216kB oom_score_adj:0
Sep 21 01:23:37 openqa kernel: oom_reaper: reaped process 32754 (cron), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Sep 21 01:23:37 openqa kernel: systemd invoked oom-killer: gfp_mask=0x40cc0(GFP_KERNEL|__GFP_COMP), order=2, oom_score_adj=0
Sep 21 01:23:37 openqa kernel: CPU: 11 PID: 1 Comm: systemd Tainted: G        W    L      5.14.21-150400.24.21-default #1 SLE15-SP4 7550826c4c7e8c258239e300508e0c8b2a69bad2
Sep 21 01:23:37 openqa kernel: Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
Sep 21 01:23:37 openqa kernel: Call Trace:
Sep 21 01:23:37 openqa kernel:  <TASK>
Sep 21 01:23:37 openqa kernel:  dump_stack_lvl+0x45/0x5b
Sep 21 01:23:37 openqa kernel:  dump_header+0x4a/0x220
Sep 21 01:23:37 openqa kernel:  oom_kill_process+0xe8/0x140
Sep 21 01:23:37 openqa kernel:  out_of_memory+0x113/0x580
Sep 21 01:23:37 openqa kernel:  __alloc_pages_slowpath.constprop.112+0x980/0xc70
Sep 21 01:23:37 openqa kernel:  __alloc_pages+0x2d9/0x320
Sep 21 01:23:37 openqa kernel:  new_slab+0x33e/0x500
Sep 21 01:23:37 openqa kernel:  ___slab_alloc+0x6ad/0x860
Sep 21 01:23:37 openqa kernel:  ? copy_process+0x213/0x1cd0
Sep 21 01:23:37 openqa kernel:  ? __mod_memcg_lruvec_state+0x1d/0x100
Sep 21 01:23:37 openqa kernel:  ? __mod_lruvec_page_state+0x6f/0xb0
Sep 21 01:23:37 openqa kernel:  ? lock_page_memcg+0x1c/0xa0
Sep 21 01:23:37 openqa kernel:  ? __slab_alloc.isra.92+0x1c/0x40
Sep 21 01:23:37 openqa kernel:  __slab_alloc.isra.92+0x1c/0x40
Sep 21 01:23:37 openqa kernel:  kmem_cache_alloc_node+0x114/0x310
Sep 21 01:23:37 openqa kernel:  ? copy_process+0x213/0x1cd0
Sep 21 01:23:37 openqa kernel:  copy_process+0x213/0x1cd0
Sep 21 01:23:37 openqa kernel:  kernel_clone+0x9b/0x400
Sep 21 01:23:37 openqa kernel:  __do_sys_clone+0x72/0xa0
Sep 21 01:23:37 openqa kernel:  do_syscall_64+0x5b/0x80
Sep 21 01:23:37 openqa kernel:  ? exc_page_fault+0x67/0x150
Sep 21 01:23:37 openqa kernel:  entry_SYSCALL_64_after_hwframe+0x61/0xcb
Sep 21 01:23:37 openqa kernel: RIP: 0033:0x7f15c3718a68
Sep 21 01:23:37 openqa kernel: Code: Unable to access opcode bytes at RIP 0x7f15c3718a3e.
Sep 21 01:23:37 openqa kernel: RSP: 002b:00007ffc905fd240 EFLAGS: 00000246 ORIG_RAX: 0000000000000038
Sep 21 01:23:37 openqa kernel: RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f15c3718a68
Sep 21 01:23:37 openqa kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000001200011
Sep 21 01:23:37 openqa kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: 00007f15c5446b00
Sep 21 01:23:37 openqa kernel: R10: 00007f15c5446dd0 R11: 0000000000000246 R12: 0000556f45bd8720
Sep 21 01:23:37 openqa kernel: R13: 00007ffc905fd490 R14: 0000556f45cc9758 R15: 0000000000000000
Sep 21 01:23:37 openqa kernel:  </TASK>
Sep 21 01:23:37 openqa kernel: Mem-Info:
Sep 21 01:23:37 openqa kernel: active_anon:2 inactive_anon:252 isolated_anon:0
                                active_file:2707 inactive_file:2658 isolated_file:0
                                unevictable:8 dirty:0 writeback:0
                                slab_reclaimable:4904595 slab_unreclaimable:1125807
                                mapped:422 shmem:3 pagetables:3197 bounce:0
                                free:79313 free_pcp:3745 free_cma:0
Sep 21 01:23:37 openqa kernel: Node 0 active_anon:8kB inactive_anon:1008kB active_file:10828kB inactive_file:10632kB unevictable:32kB isolated(anon):0kB isolated(file):0kB mapped:1688kB dirty:0kB writeback:0kB shmem:12kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB kernel_stack:4640kB pageta>
Sep 21 01:23:37 openqa kernel: Node 0 DMA free:11024kB boost:0kB min:40kB low:52kB high:64kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Sep 21 01:23:37 openqa kernel: lowmem_reserve[]: 0 2991 24013 24013 24013
Sep 21 01:23:37 openqa kernel: Node 0 DMA32 free:114856kB boost:0kB min:8356kB low:11392kB high:14428kB reserved_highatomic:14336KB active_anon:0kB inactive_anon:0kB active_file:4kB inactive_file:4kB unevictable:0kB writepending:0kB present:3129312kB managed:3063776kB mlocked:0kB bounce:0kB free_pcp:2404kB local_p>
Sep 21 01:23:37 openqa kernel: lowmem_reserve[]: 0 0 21021 21021 21021
Sep 21 01:23:37 openqa kernel: Node 0 Normal free:191372kB boost:0kB min:59184kB low:80700kB high:102216kB reserved_highatomic:20480KB active_anon:8kB inactive_anon:1128kB active_file:11088kB inactive_file:10728kB unevictable:32kB writepending:0kB present:22020096kB managed:21525736kB mlocked:32kB bounce:0kB free_>
Sep 21 01:23:37 openqa kernel: lowmem_reserve[]: 0 0 0 0 0
Sep 21 01:23:37 openqa kernel: Node 0 DMA: 0*4kB 0*8kB 1*16kB (U) 2*32kB (UE) 1*64kB (U) 1*128kB (U) 2*256kB (UE) 2*512kB (UE) 1*1024kB (E) 2*2048kB (ME) 1*4096kB (M) = 11024kB
Sep 21 01:23:37 openqa kernel: Node 0 DMA32: 7930*4kB (UM) 10420*8kB (UME) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 115080kB
Sep 21 01:23:37 openqa kernel: Node 0 Normal: 46158*4kB (UME) 902*8kB (UME) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 191848kB
Sep 21 01:23:37 openqa kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Sep 21 01:23:37 openqa kernel: 5555 total pagecache pages
Sep 21 01:23:37 openqa kernel: 113 pages in swap cache
Sep 21 01:23:37 openqa kernel: Swap cache stats: add 11843566, delete 11843351, find 99242001/104794745
Sep 21 01:23:37 openqa kernel: Free swap  = 451972kB
Sep 21 01:23:37 openqa kernel: Total swap = 499708kB
Sep 21 01:23:37 openqa kernel: 6291350 pages RAM
Sep 21 01:23:37 openqa kernel: 0 pages HighMem/MovableOnly
Sep 21 01:23:37 openqa kernel: 140132 pages reserved
Sep 21 01:23:37 openqa kernel: 0 pages cma reserved
Sep 21 01:23:37 openqa kernel: 0 pages hwpoisoned
Sep 21 01:23:37 openqa kernel: Unreclaimable slab info:
Sep 21 01:23:37 openqa kernel: Name                      Used          Total
Sep 21 01:23:37 openqa kernel: xfs_icr                   47KB         47KB
Sep 21 01:23:37 openqa kernel: xfs_efi_item             392KB        410KB
Sep 21 01:23:37 openqa kernel: xfs_buf_item             447KB        447KB
Sep 21 01:23:37 openqa kernel: xfs_da_state             191KB        191KB
Sep 21 01:23:37 openqa kernel: xfs_btree_cur             94KB         94KB
Sep 21 01:23:37 openqa kernel: nfsd4_openowners         372KB        391KB
Sep 21 01:23:37 openqa kernel: nfsd4_clients            127KB        127KB
Sep 21 01:23:37 openqa kernel: scsi_sense_cache           4KB          4KB
Sep 21 01:23:37 openqa kernel: ip6-frags                 47KB         47KB
Sep 21 01:23:37 openqa kernel: PINGv6                   370KB        370KB
Sep 21 01:23:37 openqa kernel: RAWv6                   8148KB       8148KB
Sep 21 01:23:37 openqa kernel: UDPv6                   3374KB       3374KB
Sep 21 01:23:37 openqa kernel: tw_sock_TCPv6           2086KB       2157KB
Sep 21 01:23:37 openqa kernel: request_sock_TCPv6        972KB        972KB
Sep 21 01:23:37 openqa kernel: TCPv6                   3686KB       4519KB
Sep 21 01:23:37 openqa kernel: bfq_io_cq                 95KB         95KB
Sep 21 01:23:37 openqa kernel: mqueue_inode_cache        496KB        496KB
Sep 21 01:23:37 openqa systemd-journald[563]: Journal stopped
-- Boot f3b32b3cd2f04d3ca0f816236391c0c0 --
Sep 21 01:24:38 openqa kernel: Linux version 5.14.21-150400.24.21-default (geeko@buildhost) (gcc (SUSE Linux) 7.5.0, GNU ld (GNU Binutils; SUSE Linux Enterprise 15) 2.37.20211103-150100.7.37) #1 SMP PREEMPT_DYNAMIC Wed Sep 7 06:51:18 UTC 2022 (974d0aa)
Actions #20

Updated by okurz over 2 years ago

Both MRs merged to try to prevent further crashes.

As it's hard to investigate the system journal within OSD remotely I copied over the system journal based on nsinger's suggestion with running on OSD:

journalctl | grep -v 'openqa openqa' | nc -l -p 42423

and locally

nc openqa.suse.de 42423 | pv > osd_journal_$(date +%F).log

The journal starts at Sep 08 17:45:01 so 10 days before the problems started on Sep 18.
So I looked for soft lockups just to see if we are actually following red herrings or not.
The first occurence of the search term "soft lockup" is from Sep 18 06:41:46 so exactly when the problems started, not in before:

1202771 Sep 18 06:41:21 openqa kernel: BUG: workqueue leaked lock or atomic: kworker/u21:2/0x00000001/19034
1202772                                     last function: xs_error_handle [sunrpc]
1202773 Sep 18 06:41:21 openqa kernel: CPU: 2 PID: 19034 Comm: kworker/u21:2 Not tainted 5.14.21-150400.24.21-default #1 SLE15-SP4 7550826c4c7e8c258239e300508e0c8b2a69bad2
1202774 Sep 18 06:41:21 openqa kernel: Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
1202775 Sep 18 06:41:21 openqa kernel: Workqueue: xprtiod xs_error_handle [sunrpc]
1202776 Sep 18 06:41:21 openqa kernel: Call Trace:
1202777 Sep 18 06:41:21 openqa kernel:  <TASK>
1202778 Sep 18 06:41:21 openqa kernel:  dump_stack_lvl+0x45/0x5b
1202779 Sep 18 06:41:21 openqa kernel:  process_one_work+0x390/0x440
1202780 Sep 18 06:41:21 openqa kernel:  worker_thread+0x2d/0x3d0
1202781 Sep 18 06:41:21 openqa kernel:  ? process_one_work+0x440/0x440
1202782 Sep 18 06:41:21 openqa kernel:  kthread+0x156/0x180
1202783 Sep 18 06:41:21 openqa kernel:  ? set_kthread_struct+0x50/0x50
1202784 Sep 18 06:41:21 openqa kernel:  ret_from_fork+0x22/0x30
1202785 Sep 18 06:41:21 openqa kernel:  </TASK>
1202786 Sep 18 06:41:21 openqa kernel: BUG: scheduling while atomic: kworker/u21:2/19034/0x00000002
1202787 Sep 18 06:41:21 openqa kernel: Modules linked in: dm_mod iscsi_ibft iscsi_boot_sysfs rfkill loop xfs libcrc32c kvm_amd ccp kvm virtio_net net_failover virtio_balloon failover irqbypass joydev i2c_piix4 pcspkr button nfsd auth_rpcgss nfs_acl lockd grace fuse sunrpc configfs ip        _tables x_tables ext4 crc16 mbcache jbd2 drm_kms_helper ata_generic syscopyarea sysfillrect sysimgblt serio_raw fb_sys_fops cec ata_piix rc_core ahci uhci_hcd libahci ehci_hcd drm libata usbcore virtio_blk floppy qemu_fw_cfg sg scsi_mod
1202788 Sep 18 06:41:21 openqa kernel: Supported: Yes
1202789 Sep 18 06:41:21 openqa kernel: CPU: 2 PID: 19034 Comm: kworker/u21:2 Not tainted 5.14.21-150400.24.21-default #1 SLE15-SP4 7550826c4c7e8c258239e300508e0c8b2a69bad2
1202790 Sep 18 06:41:21 openqa kernel: Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
1202791 Sep 18 06:41:21 openqa kernel: Workqueue:  0x0 (xprtiod)
1202792 Sep 18 06:41:21 openqa kernel: Call Trace:
1202793 Sep 18 06:41:21 openqa kernel:  <TASK>
1202794 Sep 18 06:41:21 openqa kernel:  dump_stack_lvl+0x45/0x5b
1202795 Sep 18 06:41:21 openqa kernel:  __schedule_bug+0x52/0x70
1202796 Sep 18 06:41:21 openqa kernel:  __schedule+0xdc4/0x1140
1202797 Sep 18 06:41:21 openqa kernel:  ? arch_local_irq_enable+0x7/0xc
1202798 Sep 18 06:41:21 openqa kernel:  schedule+0x64/0xe0
1202799 Sep 18 06:41:21 openqa kernel:  worker_thread+0xab/0x3d0
1202800 Sep 18 06:41:21 openqa kernel:  ? process_one_work+0x440/0x440
1202801 Sep 18 06:41:21 openqa kernel:  kthread+0x156/0x180
1202802 Sep 18 06:41:21 openqa kernel:  ? set_kthread_struct+0x50/0x50
1202803 Sep 18 06:41:21 openqa kernel:  ret_from_fork+0x22/0x30
1202804 Sep 18 06:41:21 openqa kernel:  </TASK>
1202805 Sep 18 06:41:46 openqa kernel: watchdog: BUG: soft lockup - CPU#2 stuck for 26s! [kworker/2:2:24024]

the messages continue with an increasing lock time until postgres runs out of memory and other processes start to fail:

Sep 18 06:59:34 openqa kernel: Out of memory: Killed process 17018 (postgres) total-vm:3418572kB, anon-rss:7224kB, file-rss:1252kB, shmem-rss:2749876kB, UID:26 pgtables:6644kB oom_score_adj:0

The command grep '\<BUG: workqueue leaked lock or atomic' osd_journal_2022-09-21.log reveals

Sep 18 06:41:21 openqa kernel: BUG: workqueue leaked lock or atomic: kworker/u21:2/0x00000001/19034
Sep 20 00:50:17 openqa kernel: BUG: workqueue leaked lock or atomic: kworker/u21:2/0x00000001/19901
Sep 20 23:12:08 openqa kernel: BUG: workqueue leaked lock or atomic: kworker/u25:0/0x00000001/32698

so three times coinciding with "Ouf of memory" messages each time after that. grep 'Linux version' osd_journal_2022-09-21.log shows

Sep 11 03:32:06 openqa kernel: Linux version 5.14.21-150400.24.18-default (geeko@buildhost) (gcc (SUSE Linux) 7.5.0, GNU ld (GNU Binutils; SUSE Linux Enterprise 15) 2.37.20211103-150100.7.37) #1 SMP PREEMPT_DYNAMIC Thu Aug 4 14:17:48 UTC 2022 (e9f7bfc)
Sep 18 03:31:56 openqa kernel: Linux version 5.14.21-150400.24.21-default (geeko@buildhost) (gcc (SUSE Linux) 7.5.0, GNU ld (GNU Binutils; SUSE Linux Enterprise 15) 2.37.20211103-150100.7.37) #1 SMP PREEMPT_DYNAMIC Wed Sep 7 06:51:18 UTC 2022 (974d0aa)

The command grep '|install|' /var/log/zypp/history | grep '^2022-09' showing all packages that have been installed after the previous boot of the machine that would explain such problems:

2022-09-11 03:00:39|install|at-spi2-atk-common|2.38.0-150400.1.9|x86_64||repo-oss|e6a37e9b6b56e2b81fbd83ab3c7c024f05b7cc0b81671f5ac7a34bde077a8f5a|
2022-09-11 03:00:39|install|gtk2-data|2.24.33-150400.2.11|noarch||repo-oss|25a3a0ce770db21bf855fd865f162674f9c5b9c38791dc34298ae276ac8e0219|
2022-09-11 03:00:39|install|at-spi2-atk-gtk2|2.38.0-150400.1.9|x86_64||repo-oss|df797cd4235009b22045becf87537dc298c0a5a0f5d8972d8767ab426284ccd3|
2022-09-11 03:00:40|install|gtk2-tools|2.24.33-150400.2.11|x86_64||repo-oss|8ba113c7c0474b558617d53436be6c3b6a8061904cbc4cc8b4ffeb65a0801704|
2022-09-11 03:00:41|install|libgtk-2_0-0|2.24.33-150400.2.11|x86_64||repo-oss|445e7903b06c04ee62afedb5b44e988665b49e24caac5b8a89425600469ec0a2|
2022-09-11 03:00:41|install|gtk2-theming-engine-adwaita|3.22.3-4.3.1|x86_64||repo-oss|abccd6f46cee5525001278cdf869041d5a1a036aeee77919af1a3adeaf62ee95|
2022-09-11 03:00:41|install|gtk2-metatheme-adwaita|3.22.3-4.3.1|noarch||repo-oss|6b42319f73bea154f8513e659dc8b82808847a9161e2f0e34f8847aab6cc39d6|
2022-09-11 03:00:41|install|gtk2-branding-openSUSE|15.0-lp154.1.10|noarch||repo-oss|341b3e4437ec2310b0f53eb0b7a8d6ea3fd966465617639fbb89ba5b62d4f7b9|
2022-09-11 03:00:41|install|gtk2-immodule-amharic|2.24.33-150400.2.11|x86_64||repo-oss|5ce87e02c7da4f8c4e2c79a7e70c5933e5fe1d424d31582e6472e91ed9be4fd4|
2022-09-11 03:00:41|install|gtk2-immodule-inuktitut|2.24.33-150400.2.11|x86_64||repo-oss|da44b8d40307ee1c54d01575b5974d0686b18e4fc4a5883402be388ef373ba60|
2022-09-11 03:00:41|install|gtk2-immodule-thai|2.24.33-150400.2.11|x86_64||repo-oss|f8461a710ef69dade2b9630ababa60a33c0b4f5e5420d8a42f6bd048c9e44b4e|
2022-09-11 03:00:42|install|gtk2-immodule-tigrigna|2.24.33-150400.2.11|x86_64||repo-oss|44e39f874f3ccd4b8f7a6a2ca5e39fbb56c115501eb4adf12f236036ce70411f|
2022-09-11 03:00:42|install|gtk2-immodule-vietnamese|2.24.33-150400.2.11|x86_64||repo-oss|d0802b1bd69874b751860ff8300484d2f4256e1b6b8a13d9bc1bdcd2eae2eaca|
2022-09-11 03:00:42|install|openssh-askpass-gnome|8.4p1-3.9.1|x86_64||repo-oss|6125fb63447542287b2c22b2f68aff2ea49aae1c6d5f3597cc8b449a13ccb062|
2022-09-11 03:00:43|install|gtk2-lang|2.24.33-150400.2.11|noarch||repo-oss|e5bd7f648b90053bb74d44a199aa49c0698b604eec288fdbb0e20c5cc770ac9e|
2022-09-11 03:00:44|install|bash-completion|2.7-150000.4.9.1|noarch||repo-sle-update|25312061f297a8dd2bfef632cbf37847f13eb387576840506882aeedfa33b56e|
2022-09-11 03:00:44|install|bash-completion-devel|2.7-150000.4.9.1|noarch||repo-sle-update|068c7b566b6cfa4c90ac77b2ed34066797bcbcf014c78bbc100171bca294f797|
2022-09-11 03:00:45|install|bash-completion-doc|2.7-150000.4.9.1|noarch||repo-sle-update|0d90841a2d1fa0c4919eb7cbb689a981ad8ac5affb1e32051b03c6c71ef5bbc8|
2022-09-11 03:00:46|install|libgtk-3-0|3.24.34-150400.3.3.1|x86_64||repo-sle-update|52213422d40001d8d9ede45bdf20d2023939d54c6e05a2dbc58931a065c91301|
2022-09-11 03:00:46|install|gtk3-immodule-amharic|3.24.34-150400.3.3.1|x86_64||repo-sle-update|9ea6bec0d97979336d3dc84e9961d3cc1539029c42c774f5391d528a1ba54687|
2022-09-11 03:00:47|install|rpm|4.14.3-150300.49.1|x86_64||repo-sle-update|a4e8c65cb33a0bcd7bc9d33bcc052dde29ef85b0c57101284adba0c43d38771d|
2022-09-11 03:00:48|install|sysconfig|0.85.9-150200.12.1|x86_64||repo-sle-update|8d5ac96a9d6fd5a107156091ac572c31b2df4b620bf767c2b1c6109d19f3ff27|
2022-09-11 03:00:48|install|sysconfig-netconfig|0.85.9-150200.12.1|x86_64||repo-sle-update|47928d89fbef1cf3262c8d12600da08ed8424ea8854c8305e23da8fca481247d|
2022-09-11 03:00:48|install|vim-data-common|9.0.0313-150000.5.25.1|noarch||repo-sle-update|47aef9d6e8835e6acefc1a7bc4b731453207a264b27d7f7fd248cd526fd63cf1|
2022-09-11 03:00:51|install|vim-data|9.0.0313-150000.5.25.1|noarch||repo-sle-update|6b2ca45af22e5fa4f8ee63edf4ee4f097bc9ce72d43c600c2e274590d14ff496|
2022-09-11 03:00:52|install|vim|9.0.0313-150000.5.25.1|x86_64||repo-sle-update|7a5f228f5848cceddccb5896687c1591465b575ede1b7ae185273c9908785ba5|
2022-09-11 03:00:52|install|gtk3-immodule-inuktitut|3.24.34-150400.3.3.1|x86_64||repo-sle-update|f6068675daaf8ca33e203b73950332389d7990ed20b75b1ae8ce2f328e7e07a7|
2022-09-11 03:00:52|install|rpm-build|4.14.3-150300.49.1|x86_64||repo-sle-update|a1ebe5cb059cabdfff4d1cdca376a841db73bb07b6137566705c6707a44d5850|
2022-09-11 03:00:52|install|python3-rpm|4.14.3-150300.49.1|x86_64||repo-sle-update|e707d51a8c72556e8fe2fa5154cc9d325683113607cef5c98344bafa35166c85|
2022-09-11 03:00:53|install|gtk3-immodule-thai|3.24.34-150400.3.3.1|x86_64||repo-sle-update|f2d70c54bf994ec3747f27c846ac6c42d256048ba1aaeaacf34367f667fa2baa|
2022-09-11 03:00:53|install|gtk3-immodule-tigrigna|3.24.34-150400.3.3.1|x86_64||repo-sle-update|1fa97d985e49b259b40fb44190b2fc2ff97137ae121a9c7044c390cb835f99f3|
2022-09-11 03:00:54|install|gtk3-immodule-vietnamese|3.24.34-150400.3.3.1|x86_64||repo-sle-update|8a17bb6cd163a01d147c6147faac4e740d1c6ea6c9cf19369c6f3866c9d53c96|
2022-09-11 03:00:54|install|gtk3-branding-openSUSE|15.0-lp154.2.4.1|noarch||repo-update|5e0c8257d182fd0fe4d9deca5df9602964a7c2b4fb84c81462fde34da0aaad58|
2022-09-11 03:00:56|install|gtk3-lang|3.24.34-150400.3.3.1|noarch||repo-sle-update|bce829efe2a150e8d2eaf75878e55a8d5ec1c57365e1dfecb1d4d39681f73eea|
2022-09-12 13:02:03|install|perl-YAML-LibYAML|0.84-lp154.2.1|x86_64||devel_openQA_Modules|c35c6b0cca14e33e1b3b42d95059f9b738a0be364528dd4118a721fe908c4a31|
2022-09-13 03:02:39|install|cups-config|2.2.7-150000.3.35.1|x86_64||repo-sle-update|a92c1a6a82493c183d370a2edf82600bbef8d939139172cd6b1f87ce3416f53a|
2022-09-13 03:02:39|install|libbluetooth3|5.62-150400.4.5.1|x86_64||repo-sle-update|a6f0e8e753b38bbb17914c8ab1e8a1047dee04345e134d9ea3fc02bd6b90a22d|
2022-09-13 03:02:40|install|libfreetype6|2.10.4-150000.4.12.1|x86_64||repo-sle-update|d8f9f0d0ebcac901fec06c865924b6bd3ae002ba588e87a745c5228dc021844c|
2022-09-13 03:02:41|install|libcups2|2.2.7-150000.3.35.1|x86_64||repo-sle-update|3519dae9ea1b6a794d6113d9f26e8fd107b72dbf3ae9db8b4b7d2cb899668973|
2022-09-13 03:02:43|install|samba-client-libs|4.15.8+git.527.8d0c05d313e-150400.3.14.1|x86_64||repo-sle-update|5227bed1cfea85b53f16019b206ec28bcb96669bd606d7b601ab35ea0c184f90|
2022-09-13 03:02:45|install|samba-client|4.15.8+git.527.8d0c05d313e-150400.3.14.1|x86_64||repo-sle-update|105386882c31c15156a382604433e101be9426d5e716eaa3d8def205d26a3276|
2022-09-13 03:02:45|install|samba-libs-python3|4.15.8+git.527.8d0c05d313e-150400.3.14.1|x86_64||repo-sle-update|7f3d548ba903b0a2773ccebfea045ed422a6973cdfbc27f8eb23962e6a36813e|
2022-09-13 03:02:46|install|samba-libs|4.15.8+git.527.8d0c05d313e-150400.3.14.1|x86_64||repo-sle-update|da1b0710efb6f2ee611d9153a248ffac0e2791808fbf95279b3255c080f5fd10|
2022-09-13 03:02:52|install|samba|4.15.8+git.527.8d0c05d313e-150400.3.14.1|x86_64||repo-sle-update|ea800fd67b37a8f558a7e5f8267ffd2afae5c8de81e33100b0d2c93e7bfc7002|
2022-09-13 03:02:53|install|samba-ad-dc-libs|4.15.8+git.527.8d0c05d313e-150400.3.14.1|x86_64||repo-sle-update|544308cafee8a2c96fb81ddf3b71819556eb172656b3d42976786edfaa98070c|
2022-09-13 03:02:53|install|libsamba-policy0-python3|4.15.8+git.527.8d0c05d313e-150400.3.14.1|x86_64||repo-sle-update|3549cdf00edcfa5555beb4b9268f5d8ab675b5e1b3c9f6ffe2fdbd4fc33b957b|
2022-09-13 03:02:56|install|samba-python3|4.15.8+git.527.8d0c05d313e-150400.3.14.1|x86_64||repo-sle-update|178fafc76d634a1fa5f56c43f913ef8a7fc44c68d1b392fabe9d9d16a61cbbb5|
2022-09-13 06:46:08|install|p7zip|16.02-14.7.1|x86_64||repo-oss|f971403abaf6b85990da2e70c8d0ac823ce3afde6ea7ce6ad4cf4b5e0a8e4a11|
2022-09-13 06:46:08|install|p7zip-full|16.02-14.7.1|x86_64|root@openqa|repo-oss|2867f8f765219c16b585ddc3e88144f2267fbb360a44727ec7310d7f1f692e92|
2022-09-13 07:51:55|install|openQA-common|4.6.1662979895.bd3bc01-lp154.5235.1|noarch||devel_openQA|000b2a08cd5b374b51ec95cce7d8655ec887c188eaa3533d4323e3cf9c7d5c12|
2022-09-13 07:51:55|install|openQA-client|4.6.1662979895.bd3bc01-lp154.5235.1|noarch||devel_openQA|3e9f09708bc4cb4a76965a0090e270c9cc05529188ff96fdb63c99ee99006b58|
2022-09-13 07:52:08|install|openQA|4.6.1662979895.bd3bc01-lp154.5235.1|noarch||devel_openQA|fc467e51f652c2469f54c67b71275435d90cf79fb1f4eb34d3609d3801b553ed|
2022-09-13 07:52:08|install|openQA-doc|4.6.1662979895.bd3bc01-lp154.5235.1|noarch||devel_openQA|479391cc86fe18c565b4cfe8565c78b330eb643579e176f9dd26999252c6dc5d|
2022-09-14 03:01:03|install|libatomic1|11.3.0+git1637-150000.1.11.2|x86_64||repo-sle-update|78979c41e689c1436214e3aa41640cd5211c32b92beb5406be23909405851546|
2022-09-14 03:01:04|install|libgcc_s1|11.3.0+git1637-150000.1.11.2|x86_64||repo-sle-update|c7e826b9aa4264d613e70b52049a0698b3419369d31ea815872d3fc115b26a8c|
2022-09-14 03:01:04|install|libgomp1|11.3.0+git1637-150000.1.11.2|x86_64||repo-sle-update|fd2bea4a7ce9aa2a430e79cc7ed8c40fa311b2d87b0bbd52fa4de93784bf0749|
2022-09-14 03:01:04|install|libitm1|11.3.0+git1637-150000.1.11.2|x86_64||repo-sle-update|3ec0e3c6b82ddaaf04f3186d1b4e0b8988c037de3c23dcf2493f19ad3c5fb8c1|
2022-09-14 03:01:05|install|libquadmath0|11.3.0+git1637-150000.1.11.2|x86_64||repo-sle-update|77a59b95d89aba353181a67415a278afcdcc7ce77a2b8261d6130e27ba6da6f1|
2022-09-14 03:01:05|install|libstdc++6|11.3.0+git1637-150000.1.11.2|x86_64||repo-sle-update|ba0a6150183d657f41faf068fb52e2e2a0655b687e7a0441b08faab732082518|
2022-09-14 03:01:06|install|libgfortran5|11.3.0+git1637-150000.1.11.2|x86_64||repo-sle-update|9983e20ab28dd46eea658e58d3d2e6bd3f1c963a34acbddab6838ca002a80f85|
2022-09-14 03:01:06|install|libtsan0|11.3.0+git1637-150000.1.11.2|x86_64||repo-sle-update|ab31ae003ac457685327c115ef61ea8c6307acd077a8e14c2dcf47586b84794c|
2022-09-14 03:01:07|install|libstdc++6-pp-gcc11|11.3.0+git1637-150000.1.11.2|x86_64||repo-sle-update|bf88737ed6749921bb88f0d336abb017d230b88ca260912653739af54ac7c64c|
2022-09-14 03:01:07|install|liblsan0|11.3.0+git1637-150000.1.11.2|x86_64||repo-sle-update|96b1e5e4a953bbd2babc4f5f99576b50b6fe3b9be92a6f1cbb7e0bb75a56f183|
2022-09-15 03:02:31|install|perl-base|5.26.1-150300.17.11.1|x86_64||repo-sle-update|9035f8fb681ddde40e1524cc2adb6f386c5629bff44f6729fd9453cb8d142e9f|
2022-09-15 03:02:37|install|perl|5.26.1-150300.17.11.1|x86_64||repo-sle-update|e4f3ac9c1716596d0ee4dba3dd793bbe3c2202c33faf172d91b67ff1fcc83f5b|
2022-09-15 03:02:37|install|perl-core-DB_File|5.26.1-150300.17.11.1|x86_64||repo-sle-update|d8bf6b3304d36c3db452c3f7155bae15dbf4ff902db2983797e418432d29f924|
2022-09-15 07:17:36|install|openQA-common|4.6.1662979895.bd3bc01-lp154.5236.1|noarch||devel_openQA|0237f3d6779791dd1176c1875d4a017c42a10d5e35b337a050075a92bcbddc23|
2022-09-15 07:17:37|install|openQA-doc|4.6.1662979895.bd3bc01-lp154.5236.1|noarch||devel_openQA|703c3695c52465fba66d575c95e1cf9eeffc78acf0ad89e0ad1e7a3dcf6179ab|
2022-09-15 07:17:37|install|openQA-client|4.6.1662979895.bd3bc01-lp154.5236.1|noarch||devel_openQA|7262f62b724685c6066bfdb22b41984645b06162f934087a2d39096934f1f820|
2022-09-15 07:17:49|install|openQA|4.6.1662979895.bd3bc01-lp154.5236.1|noarch||devel_openQA|86acf4205ae255c7f5a2dbbd48144abce08c08b814b65f72a1da345ddb76828b|
2022-09-15 07:17:50|install|python3-typing_extensions|3.10.0.0-150400.3.2.1|noarch||repo-sle-update|20be425c3cd3596a7747bc92ff151dd0de5f9883fb26499808a69749239a9dc4|
2022-09-16 03:00:59|install|libgit2-1_3|1.3.0-150400.3.3.1|x86_64||repo-sle-update|676f509046a13f14b3e3606b469ea46b327be38109c32926f10bf3eb1bf18d2e|
2022-09-16 03:01:03|install|yast2-storage-ng|4.4.40-150400.3.3.1|x86_64||repo-sle-update|c7e0e8922c4212ccc6c62e4a406017c39707c63911f71278ecf326c28532ea6d|
2022-09-17 03:04:25|install|kernel-default|5.14.21-150400.24.21.2|x86_64||repo-sle-update|3b2bfeafc2944dc633ce2531b713395b40c1fce0aa018bc6736ec04a564f75ac|
2022-09-17 03:04:26|install|libruby2_5-2_5|2.5.9-150000.4.26.1|x86_64||repo-sle-update|2f3dc29b4ed67a3e0dab9848321fdede20f91bc0c8e16a96a9e42756fd92b9b6|
2022-09-17 03:04:35|install|ruby2.5-stdlib|2.5.9-150000.4.26.1|x86_64||repo-sle-update|c0af9bcc01a1b7116bb280c9bc6e97e4f39e3842a951742440ea802d65748c55|
2022-09-17 03:04:36|install|ruby2.5|2.5.9-150000.4.26.1|x86_64||repo-sle-update|1816b75a0ffd933ca11c8579d30029ded432c98c236713043eee9b1bdb66c8d2|
2022-09-17 03:05:07|install|kernel-default-extra|5.14.21-150400.24.21.2|x86_64||repo-sle-update|c91ba92cdf7e58ed5a2cb36d23a247389c34b403f2a241ccdf974ddd894dd913|
2022-09-17 03:05:24|install|kernel-default-optional|5.14.21-150400.24.21.2|x86_64||repo-sle-update|11f8f2f998cac335cc1ff102d6b26e2a9f1d81016721178c8568b1130897e382|

I consider it very likely that only the kernel update 5.14.21-150400.24.18-default -> 5.14.21-150400.24.21-default could explain this behaviour.

I suggest to report a product issue and downgrade the kernel version. I called

zypper rm kernel-default-5.14.21-150400.24.21 && zypper al kernel-default && reboot

now the old kernel is running. I reported https://bugzilla.opensuse.org/show_bug.cgi?id=1203630 . Let's await further results.

Actions #21

Updated by mkittler over 2 years ago

In the end dbux-daemon is the process with the highest virtual memory usage. However, it is only 481 pages, that should only be 1.87890625 MiB which is not very concerning.

The process with the highest resident set size (physical memory occupied by the process so not swapped out, not mmapped, …) is salt-minion. Its RSS is significantly higher at than the RSS of any other process with 712704. That should be 2.71875 GiB. On normal operation a PostgreSQL process leads that score (still below 1 GiB then). So that's quite a lot for salt. It is also strange that at this point no virtual memory is assigned to the process anymore (same counts also for many other processes).

Maybe the statistics at this point aren't that useful anymore. At an earlier time (Sep 21 00:02:26 but the OOM killer was already active and logging stats) the openqa web UI processes were leading the stats of virtual memory use (and were therefore also being killed by the OOM killer). However, they were still only at around 347.1640625 MiB use of VM each (that exact value is the highest value logged on Sep 21 00:02:26) which is quite normal. The RSS is quite big already at 2,78125 GiB (again the highest value at the time). I'm wondering whether I'm reading the stats correctly. It is still strange the the VM is less than the RSS. Most likely the VM stat is excluding the RSS in these tables. The process with the highest RSS at the time was telegraf with 3.859375 GiB. That's a lot although it is normally also leading that score (e.g. 875 MiB at the time I'm writing this). Some PostgreSQL processes were also quite high with up to 3.140625 GiB. The same counts for openQA process with up to 2.828125 GiB. When adding RSS and VM together than telegraf uses most with 3.94922256469727 GiB, then comes PostgreSQL with 3.15000534057617 GiB, then many openqa processes with up to 2.97131729125977 GiB and some more PosgreSQL processes and salt.

It is hard to pick a culprit. All salt process together use 22.038646697998 GiB of VM + RSS. For PostgreSQL it is 51.2796745300293 GiB and for openqa processes it is 78.4090690612793 GiB. Nothing should be so high obviously.

So maybe it is indeed the kernel misbehaving. It would be quite strange that many different services (using different programming languages) show the same symptom.

Actions #22

Updated by okurz about 2 years ago

No problem since yesterday, OSD is stable. I will revert the postgres work_mem changes. I doubt the have an impact.

https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/739

Actions #23

Updated by okurz about 2 years ago

  • Subject changed from openqa.suse.de is not reachable 2022-09-18, no ping response, postgreSQL OOM and kernel panics to openqa.suse.de is not reachable 2022-09-18, no ping response, postgreSQL OOM and kernel panics size:M
  • Description updated (diff)
Actions #24

Updated by okurz about 2 years ago

  • Due date changed from 2022-10-03 to 2022-10-28
  • Status changed from In Progress to Feedback
  • Priority changed from Urgent to High

I took a look on o3 which has a journal going back to 2022-08-08 and journalctl | grep '\<BUG: workqueue leaked lock or atomic' returned no matches. https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/739 was merged and I checked also manually that we are back with work_mem = 64MB within postgres config. OSD is up since 1 day, 17:51. I will observe the situation over the next days/weeks and await either further problems or responses in the bug report. If there is no further crash nor response after weeks we can try a more recent kernel version if there is one in Leap or maybe the most recent vanilla version or Kernel:stable or Kernel:HEAD

Actions #25

Updated by jbaier_cz about 2 years ago

We have another episode, maybe the wrong kernel was re-installed and loaded during the auto-update/restart over the weekend?

This time, monitoring does not show the same symptoms like in the first case (i.e. no increased CPU or I/O activity).

Also it did not reboot itself, mkittler created SD-99619 to recover.

Actions #26

Updated by mkittler about 2 years ago

The VM has been recreated. Considering the journal it is indeed the same problem again:

Sep 26 15:47:53 openqa kernel: BUG: workqueue leaked lock or atomic: kworker/u25:1/0x00000001/12426
                                    last function: xs_error_handle [sunrpc]
Sep 26 15:47:53 openqa kernel: CPU: 4 PID: 12426 Comm: kworker/u25:1 Not tainted 5.14.21-150400.24.21-default #1 SLE15-SP4 7550826c4c7e8c258239e300508e0c8b2a69bad2
Sep 26 15:47:53 openqa kernel: Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
Sep 26 15:47:53 openqa kernel: Workqueue: xprtiod xs_error_handle [sunrpc]
Sep 26 15:47:53 openqa kernel: Call Trace:
Sep 26 15:47:53 openqa kernel:  <TASK>
Sep 26 15:47:53 openqa kernel:  dump_stack_lvl+0x45/0x5b
Sep 26 15:47:53 openqa kernel:  process_one_work+0x390/0x440
Sep 26 15:47:53 openqa kernel:  worker_thread+0x2d/0x3d0
Sep 26 15:47:53 openqa kernel:  ? process_one_work+0x440/0x440
Sep 26 15:47:53 openqa kernel:  kthread+0x156/0x180
Sep 26 15:47:53 openqa kernel:  ? set_kthread_struct+0x50/0x50
Sep 26 15:47:53 openqa kernel:  ret_from_fork+0x22/0x30
Sep 26 15:47:53 openqa kernel:  </TASK>
Sep 26 15:47:53 openqa kernel: BUG: scheduling while atomic: kworker/u25:1/12426/0x00000002
Sep 26 15:47:53 openqa kernel: Modules linked in: iscsi_ibft iscsi_boot_sysfs rfkill loop xfs libcrc32c virtio_net net_failover failover virtio_balloon button nfsd auth_rpcgss nfs_acl lockd grace fuse drm sunrpc configfs ip_tables x_tables ext4 crc16 mbcache jbd2 uhci_hcd ata_piix ehci_hcd ahci libahci usbcore lib>
Sep 26 15:47:53 openqa kernel: Supported: Yes
Sep 26 15:47:53 openqa kernel: CPU: 4 PID: 12426 Comm: kworker/u25:1 Not tainted 5.14.21-150400.24.21-default #1 SLE15-SP4 7550826c4c7e8c258239e300508e0c8b2a69bad2
Sep 26 15:47:53 openqa kernel: Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
Sep 26 15:47:53 openqa kernel: Workqueue:  0x0 (xprtiod)
Sep 26 15:47:53 openqa kernel: Call Trace:
Sep 26 15:47:53 openqa kernel:  <TASK>
Sep 26 15:47:53 openqa kernel:  dump_stack_lvl+0x45/0x5b
Sep 26 15:47:53 openqa kernel:  __schedule_bug+0x52/0x70
Sep 26 15:47:53 openqa kernel:  __schedule+0xdc4/0x1140
Sep 26 15:47:53 openqa kernel:  ? arch_local_irq_enable+0x7/0xc
Sep 26 15:47:53 openqa kernel:  schedule+0x64/0xe0
Sep 26 15:47:53 openqa kernel:  worker_thread+0xab/0x3d0
Sep 26 15:47:53 openqa kernel:  ? process_one_work+0x440/0x440
Sep 26 15:47:53 openqa kernel:  kthread+0x156/0x180
Sep 26 15:47:53 openqa kernel:  ? set_kthread_struct+0x50/0x50
Sep 26 15:47:53 openqa kernel:  ret_from_fork+0x22/0x30
Sep 26 15:47:53 openqa kernel:  </TASK>
…
Sep 26 16:21:14 openqa kernel: Out of memory: Killed process 1312 (postgres) total-vm:3349768kB, anon-rss:1200kB, file-rss:8kB, shmem-rss:2736620kB, UID:26 pgtables:5876kB oom_score_adj:0
Sep 26 16:21:14 openqa kernel: postgres invoked oom-killer: gfp_mask=0x1100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
Sep 26 16:21:14 openqa kernel: CPU: 7 PID: 6303 Comm: postgres Tainted: G        W    L      5.14.21-150400.24.21-default #1 SLE15-SP4 7550826c4c7e8c258239e300508e0c8b2a69bad2
Sep 26 16:21:14 openqa kernel: Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
Sep 26 16:21:14 openqa kernel: Call Trace:
Sep 26 16:21:14 openqa kernel:  <TASK>
Sep 26 16:21:14 openqa kernel:  dump_stack_lvl+0x45/0x5b
Sep 26 16:21:14 openqa kernel:  dump_header+0x4a/0x220
Sep 26 16:21:14 openqa kernel:  oom_kill_process+0xe8/0x140
Sep 26 16:21:14 openqa kernel:  out_of_memory+0x113/0x580
Sep 26 16:21:14 openqa kernel:  __alloc_pages_slowpath.constprop.112+0x980/0xc70
Sep 26 16:21:14 openqa kernel:  __alloc_pages+0x2d9/0x320
Sep 26 16:21:14 openqa kernel:  pagecache_get_page+0x1a8/0x480
Sep 26 16:21:14 openqa kernel:  filemap_fault+0x4c1/0xa30
Sep 26 16:21:14 openqa kernel:  ? filemap_map_pages+0x14e/0x580
Sep 26 16:21:14 openqa kernel:  __do_fault+0x31/0xc0
Sep 26 16:21:14 openqa kernel:  __handle_mm_fault+0xd64/0x1220
Sep 26 16:21:14 openqa kernel:  handle_mm_fault+0xd7/0x290
Sep 26 16:21:14 openqa kernel:  do_user_addr_fault+0x1eb/0x730
Sep 26 16:21:14 openqa kernel:  exc_page_fault+0x67/0x150
Sep 26 16:21:14 openqa kernel:  ? asm_exc_page_fault+0x8/0x30
Sep 26 16:21:14 openqa kernel:  asm_exc_page_fault+0x1e/0x30
Sep 26 16:21:14 openqa kernel: RIP: 0033:0x55aef40e0a90
Sep 26 16:21:14 openqa kernel: Code: Unable to access opcode bytes at RIP 0x55aef40e0a66.
Sep 26 16:21:14 openqa kernel: RSP: 002b:00007ffca8429768 EFLAGS: 00010246
Sep 26 16:21:14 openqa kernel: RAX: 0000000000000001 RBX: 0000000000000001 RCX: 000055aef71bd788
Sep 26 16:21:14 openqa kernel: RDX: 000055aef6462900 RSI: 000055aef71b4e78 RDI: 000000000000066f
Sep 26 16:21:14 openqa kernel: RBP: 0000000000000011 R08: 0000000000000002 R09: 000055aef71bd658
Sep 26 16:21:14 openqa kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 000055aef64626b8
Sep 26 16:21:14 openqa kernel: R13: 0000000000000053 R14: 000055aef6462b98 R15: 00007faccb1e15f8
Sep 26 16:21:14 openqa kernel:  </TASK>
Sep 26 16:21:14 openqa kernel: Mem-Info:
Sep 26 16:21:14 openqa kernel: active_anon:1238013 inactive_anon:560727 isolated_anon:0
                                active_file:2848 inactive_file:2875 isolated_file:32
                                unevictable:20 dirty:158 writeback:0
                                slab_reclaimable:3665090 slab_unreclaimable:481451
                                mapped:804445 shmem:804567 pagetables:99957 bounce:0
                                free:65873 free_pcp:4279 free_cma:0
Sep 26 16:21:14 openqa kernel: Node 0 active_anon:4952052kB inactive_anon:2242908kB active_file:11392kB inactive_file:11128kB unevictable:80kB isolated(anon):0kB isolated(file):128kB mapped:3211940kB dirty:632kB writeback:0kB shmem:3218268kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 430080kB writeback_tmp:0kB >
Sep 26 16:21:14 openqa kernel: Node 0 DMA free:11248kB boost:0kB min:40kB low:52kB high:64kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Sep 26 16:21:14 openqa kernel: lowmem_reserve[]: 0 2969 23982 23982 23982
Sep 26 16:21:14 openqa kernel: Node 0 DMA32 free:95108kB boost:2048kB min:10408kB low:13448kB high:16488kB reserved_highatomic:2048KB active_anon:718356kB inactive_anon:198232kB active_file:0kB inactive_file:1212kB unevictable:0kB writepending:0kB present:3129312kB managed:3063776kB mlocked:0kB bounce:0kB free_pcp>
Sep 26 16:21:14 openqa kernel: lowmem_reserve[]: 0 0 21013 21013 21013
Sep 26 16:21:14 openqa kernel: Node 0 Normal free:167224kB boost:114688kB min:173864kB low:195380kB high:216896kB reserved_highatomic:16384KB active_anon:4233988kB inactive_anon:2043588kB active_file:11116kB inactive_file:9840kB unevictable:80kB writepending:372kB present:22020096kB managed:21525736kB mlocked:80kB>
Sep 26 16:21:14 openqa kernel: lowmem_reserve[]: 0 0 0 0 0
Sep 26 16:21:14 openqa kernel: Node 0 DMA: 0*4kB 0*8kB 1*16kB (U) 1*32kB (U) 1*64kB (U) 1*128kB (U) 1*256kB (U) 1*512kB (U) 0*1024kB 1*2048kB (M) 2*4096kB (M) = 11248kB
Sep 26 16:21:14 openqa kernel: Node 0 DMA32: 3310*4kB (UME) 9535*8kB (UME) 33*16kB (UM) 6*32kB (UE) 6*64kB (UM) 1*128kB (M) 3*256kB (U) 0*512kB 0*1024kB 0*2048kB 1*4096kB (H) = 95616kB
Sep 26 16:21:14 openqa kernel: Node 0 Normal: 42623*4kB (UME) 121*8kB (UME) 22*16kB (UE) 4*32kB (U) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 171940kB
Sep 26 16:21:14 openqa kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Sep 26 16:21:14 openqa kernel: 831708 total pagecache pages
Sep 26 16:21:14 openqa kernel: 20027 pages in swap cache
Sep 26 16:21:14 openqa kernel: Swap cache stats: add 829916, delete 810190, find 231001956/231291553
Sep 26 16:21:14 openqa kernel: Free swap  = 0kB
Sep 26 16:21:14 openqa kernel: Total swap = 499708kB
Sep 26 16:21:14 openqa kernel: 6291350 pages RAM
Sep 26 16:21:14 openqa kernel: 0 pages HighMem/MovableOnly
Sep 26 16:21:14 openqa kernel: 140132 pages reserved
Sep 26 16:21:14 openqa kernel: 0 pages cma reserved
Sep 26 16:21:14 openqa kernel: 0 pages hwpoisoned
Sep 26 16:21:14 openqa kernel: Tasks state (memory values in pages):
Sep 26 16:21:14 openqa kernel: [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
Sep 26 16:21:14 openqa kernel: [    508]     0   508     3140        3    65536     1124             0 haveged
Sep 26 16:21:14 openqa kernel: [    509]     0   509   177356     1156  1179648      179          -250 systemd-journal
Sep 26 16:21:14 openqa kernel: [    527]     0   527    27090      133   225280      536         -1000 systemd-udevd
Sep 26 16:21:14 openqa kernel: [    680]     0   680    13017        6   143360      157             0 rpc.idmapd
Sep 26 16:21:14 openqa kernel: [    681]     0   681    15090       31   139264      150         -1000 auditd
Sep 26 16:21:14 openqa kernel: [    686]   499   686    19832      269   192512      129          -900 dbus-daemon
Sep 26 16:21:14 openqa kernel: [    690]     0   690    24207      265   229376      360             0 systemd-logind
Sep 26 16:21:14 openqa kernel: [    691]     0   691    10708       73   126976      240             0 wickedd-auto4
Sep 26 16:21:14 openqa kernel: [    694]     0   694    10709       71   126976      241             0 wickedd-dhcp4
Sep 26 16:21:14 openqa kernel: [    697]     0   697    10709      103   131072      211             0 wickedd-dhcp6
Sep 26 16:21:14 openqa kernel: [    753]     0   753    10752      216   135168      160             0 wickedd
Sep 26 16:21:14 openqa kernel: [    779]     0   779    10745      227   131072      102             0 wickedd-nanny
…
Sep 26 16:25:30 openqa kernel: rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 4-... } 378193 jiffies s: 307889 root: 0x10/.
Sep 26 16:25:30 openqa kernel: rcu: blocking rcu_node structures (internal RCU debug):
Sep 26 16:25:30 openqa kernel: Task dump for CPU 4:
Sep 26 16:25:30 openqa kernel: task:kworker/4:1     state:R  running task     stack:    0 pid: 2187 ppid:     2 flags:0x00004008
Sep 26 16:25:30 openqa kernel: Workqueue: events xprt_destroy_cb [sunrpc]
Sep 26 16:25:30 openqa kernel: Call Trace:
Sep 26 16:25:30 openqa kernel:  <TASK>
Sep 26 16:25:30 openqa kernel:  ? __schedule+0x2d5/0x1140
Sep 26 16:25:30 openqa kernel:  ? __inode_wait_for_writeback+0x7f/0xf0
Sep 26 16:25:30 openqa kernel:  ? fsnotify_grab_connector+0x4b/0x90
Sep 26 16:25:30 openqa kernel:  ? sysvec_apic_timer_interrupt+0xa/0x90
Sep 26 16:25:30 openqa kernel:  ? asm_sysvec_apic_timer_interrupt+0x12/0x20
Sep 26 16:25:30 openqa kernel:  ? native_queued_spin_lock_slowpath+0x21/0x1e0
Sep 26 16:25:30 openqa kernel:  ? _raw_spin_lock+0x25/0x30
Sep 26 16:25:30 openqa kernel:  ? xprt_disconnect_done+0x1b/0xb0 [sunrpc f37209057a2989e994f415cd3f1a1642d1001097]
Sep 26 16:25:30 openqa kernel:  ? xs_close+0x1a/0x40 [sunrpc f37209057a2989e994f415cd3f1a1642d1001097]
Sep 26 16:25:30 openqa kernel:  ? xs_destroy+0x26/0x70 [sunrpc f37209057a2989e994f415cd3f1a1642d1001097]
Sep 26 16:25:30 openqa kernel:  ? process_one_work+0x267/0x440
Sep 26 16:25:30 openqa kernel:  ? worker_thread+0x2d/0x3d0
Sep 26 16:25:30 openqa kernel:  ? process_one_work+0x440/0x440
Sep 26 16:25:30 openqa kernel:  ? kthread+0x156/0x180
Sep 26 16:25:30 openqa kernel:  ? set_kthread_struct+0x50/0x50
Sep 26 16:25:30 openqa kernel:  ? ret_from_fork+0x22/0x30
Sep 26 16:25:30 openqa kernel:  </TASK>
…
Sep 26 16:26:14 openqa kernel: watchdog: BUG: soft lockup - CPU#4 stuck for 2146s! [kworker/4:1:2187]
Sep 26 16:26:14 openqa kernel: Modules linked in: iscsi_ibft iscsi_boot_sysfs rfkill loop xfs libcrc32c virtio_net net_failover failover virtio_balloon button nfsd auth_rpcgss nfs_acl lockd grace fuse drm sunrpc configfs ip_tables x_tables ext4 crc16 mbcache jbd2 uhci_hcd ata_piix ehci_hcd ahci libahci usbcore lib>
Sep 26 16:26:14 openqa kernel: Supported: Yes
Sep 26 16:26:14 openqa kernel: CPU: 4 PID: 2187 Comm: kworker/4:1 Tainted: G        W    L      5.14.21-150400.24.21-default #1 SLE15-SP4 7550826c4c7e8c258239e300508e0c8b2a69bad2
-- Boot 37b8d07bd19743f5b73de54f2d8baa4f --
Sep 26 16:27:27 openqa kernel: Linux version 5.14.21-150400.24.21-default (geeko@buildhost) (gcc (SUSE Linux) 7.5.0, GNU ld (GNU Binutils; SUSE Linux Enterprise 15) 2.37.20211103-150100.7.37) #1 SMP PREEMPT_DYNAMIC Wed Sep 7 06:51:18 UTC 2022 (974d0aa)
Sep 26 16:27:27 openqa kernel: Command line: BOOT_IMAGE=/boot/vmlinuz-5.14.21-150400.24.21-default root=UUID=7116dc72-ebc8-4b21-8847-b9f31dc95229 nomodeset console=tty0 console=ttyS0,115200 elevator=noop
Sep 26 16:27:27 openqa kernel: x86/fpu: x87 FPU will use FXSAVE
Sep 26 16:27:27 openqa kernel: signal: max sigframe size: 1440
Sep 26 16:27:27 openqa kernel: BIOS-provided physical RAM map:
Actions #27

Updated by mkittler about 2 years ago

Note that downgrading the kernel via zypper rm kernel-default-5.14.21-150400.24.21 && zypper al kernel-default && reboot (see end of #116722#note-20) did not work.

There's still only the old kernel version installed but the new version is nevertheless being loaded:

martchus@openqa:~> rpm -q kernel-default
kernel-default-5.14.21-150400.24.18.1.x86_64
martchus@openqa:~> uname -a
Linux openqa 5.14.21-150400.24.21-default #1 SMP PREEMPT_DYNAMIC Wed Sep 7 06:51:18 UTC 2022 (974d0aa) x86_64 x86_64 x86_64 GNU/Linux

Also see the logs in my previous comment.

Likely that's because under /boot still files for both kernel versions are present:

martchus@openqa:~> find /boot -iname '*5.14.21-150400.24.18*'
/boot/.vmlinuz-5.14.21-150400.24.18-default.hmac
/boot/sysctl.conf-5.14.21-150400.24.18-default
/boot/vmlinux-5.14.21-150400.24.18-default.gz
/boot/config-5.14.21-150400.24.18-default
/boot/System.map-5.14.21-150400.24.18-default
/boot/symvers-5.14.21-150400.24.18-default.gz
/boot/initrd-5.14.21-150400.24.18-default
/boot/vmlinuz-5.14.21-150400.24.18-default
martchus@openqa:~> find /boot -iname '*5.14.21-150400.24.21*'
/boot/.vmlinuz-5.14.21-150400.24.21-default.hmac
/boot/vmlinuz-5.14.21-150400.24.21-default
/boot/config-5.14.21-150400.24.21-default
/boot/sysctl.conf-5.14.21-150400.24.21-default
/boot/initrd-5.14.21-150400.24.21-default
/boot/symvers-5.14.21-150400.24.21-default.gz
/boot/vmlinux-5.14.21-150400.24.21-default.gz
/boot/System.map-5.14.21-150400.24.21-default

That's strange because those files actually belong to the corresponding kernel package and should normally be deleted on its removal:

martchus@openqa:~> rpm -qf /boot/vmlinux-5.14.21-150400.24.21-default.gz
kernel-default-base-5.14.21-150400.24.21.2.150400.24.7.2.x86_64

I now invoked sudo zypper rm kernel-default-base-5.14.21-150400.24.21.2.150400.24.7.2.x86_64 again and checked whether the files are gone. They are. Then I invoked zypper dup to see whether the package lock works. Apparently it doesn't:

martchus@openqa:~> sudo zypper dup
Metadaten von Repository 'devel_openQA' abrufen ....................................................................................................................................................................................................................................................................[fertig]
Cache für Repository 'devel_openQA' erzeugen .......................................................................................................................................................................................................................................................................[fertig]
Metadaten von Repository 'Update repository with updates from SUSE Linux Enterprise 15' abrufen ....................................................................................................................................................................................................................[fertig]
Cache für Repository 'Update repository with updates from SUSE Linux Enterprise 15' erzeugen .......................................................................................................................................................................................................................[fertig]
Repository-Daten werden geladen...
Installierte Pakete werden gelesen...
Warnung: Sie sind im Begriff, eine Distributionsaktualisierung mit allen aktivierten Repositorys durchzuführen. Vergewissern Sie sich, dass diese Repositorys kompatibel sind, bevor Sie fortfahren. Weitere Informationen zu diesem Kommando finden Sie unter 'man zypper'.
Distributions-Aktualisierungen werden verarbeitet...

Die folgenden 2 Objekte sind gesperrt und werden durch keine Aktivität geändert:
 Verfügbar:
  openQA-local-db
 Installiert:
  kernel-default

Die folgenden 3 NEUEN Pakete werden installiert:
  kernel-default-base kernel-default-extra-5.14.21-150400.24.21.2 kernel-default-optional-5.14.21-150400.24.21.2

So that's our problem. I now invoked

martchus@openqa:~> sudo zypper al kernel-default-base
Die angegebene Sperre wurde erfolgreich hinzugefügt.
martchus@openqa:~> sudo zypper al kernel-default-extra
Die angegebene Sperre wurde erfolgreich hinzugefügt.
martchus@openqa:~> sudo zypper al kernel-default-optional
Die angegebene Sperre wurde erfolgreich hinzugefügt.
martchus@openqa:~> sudo zypper dup
Repository-Daten werden geladen...
Installierte Pakete werden gelesen...
Warnung: Sie sind im Begriff, eine Distributionsaktualisierung mit allen aktivierten Repositorys durchzuführen. Vergewissern Sie sich, dass diese Repositorys kompatibel sind, bevor Sie fortfahren. Weitere Informationen zu diesem Kommando finden Sie unter 'man zypper'.
Distributions-Aktualisierungen werden verarbeitet...

Die folgenden 5 Objekte sind gesperrt und werden durch keine Aktivität geändert:
 Verfügbar:
  kernel-default-base openQA-local-db
 Installiert:
  kernel-default kernel-default-extra kernel-default-optional
Keine auszuführenden Aktionen.

and now zypper dup wouldn't do a thing. So we only ran into the problem again due to a missing package lock. I've just invoked a reboot so we stay on the old kernel version.

ssh openqa.suse.de
Last login: Mon Sep 26 16:30:17 2022 from 2620:113:80c0:8340::1127
Have a lot of fun...
martchus@openqa:~> uname -a
Linux openqa 5.14.21-150400.24.18-default #1 SMP PREEMPT_DYNAMIC Thu Aug 4 14:17:48 UTC 2022 (e9f7bfc) x86_64 x86_64 x86_64 GNU/Linux
Actions #28

Updated by okurz about 2 years ago

Thank you. That's great! Do you have an idea why the kernel.panic and automatic reboot wasn't effective?

Actions #29

Updated by jbaier_cz about 2 years ago

okurz wrote:

Thank you. That's great! Do you have an idea why the kernel.panic and automatic reboot wasn't effective?

Maybe the machine wasn't panicking (it just lost connectivity due to killed services)?

Actions #30

Updated by mkittler about 2 years ago

There was no kernel panic, see #116722#note-26. We recovered the machine before it came to that. (Until it panics there's usually a long struggle, see timestamps in #116722#note-19 although it has even already started earlier.)

Actions #31

Updated by okurz about 2 years ago

  • Status changed from Feedback to Blocked
Actions #32

Updated by okurz about 2 years ago

  • Priority changed from High to Urgent

bug closed as duplicate of https://bugzilla.opensuse.org/show_bug.cgi?id=1203338, tracking that.

Actions #33

Updated by okurz about 2 years ago

  • Due date changed from 2022-10-28 to 2022-11-25
  • Priority changed from Urgent to Normal

IBS SRs were mentioned but no message yet about released update. We will wait for that and keep the older kernel running until then

Actions #34

Updated by okurz about 2 years ago

  • Due date changed from 2022-11-25 to 2023-01-20

boo#1203338 still IN_PROGRESS and haven't see the full availability of bugfixes.

Actions #35

Updated by okurz about 2 years ago

  • Tags set to infra
Actions #36

Updated by okurz almost 2 years ago

  • Tags changed from infra to infra, reactive work
Actions #37

Updated by livdywan almost 2 years ago

  • Due date deleted (2023-01-20)

This is in fact still blocking on boo#1203338. Or at least I assume it is whilst I can't login to Bugzilla.

Actions #38

Updated by okurz almost 2 years ago

I asked with a comment in https://bugzilla.opensuse.org/show_bug.cgi?id=1203338#c70 about the status and plans.

Actions #39

Updated by okurz almost 2 years ago

  • Related to action #126212: openqa.suse.de response times very slow. No alert fired size:M added
Actions #40

Updated by okurz over 1 year ago

  • Status changed from Blocked to Workable
  • Assignee deleted (okurz)

https://bugzilla.opensuse.org/show_bug.cgi?id=1203338 is now RESOLVED FIXED. The updated kernel should be tried on osd+o3

Actions #41

Updated by mkittler over 1 year ago

  • Status changed from Workable to In Progress
Actions #42

Updated by mkittler over 1 year ago

  • Status changed from In Progress to Feedback
  • Assignee set to mkittler

I've just enabled the normal kernel again on OSD:

$ sudo zypper removelock 'kernel-default*'
1 lock has been successfully removed.
$ sudo zypper dup --allow-vendor-change
…
The following 3 NEW packages are going to be installed:
  kernel-default-5.14.21-150400.24.60.1 kernel-default-extra-5.14.21-150400.24.60.1 kernel-default-optional-5.14.21-150400.24.60.1
…
$ sudo reboot

The VM came back ok. Let's see how stable it'll run.

Actions #43

Updated by mkittler over 1 year ago

Everything looks still good.

Actions #44

Updated by mkittler over 1 year ago

It looks still good on OSD and on o3 the kernel wasn't locked anyways (and a recent version is in use).

Actions #45

Updated by mkittler over 1 year ago

  • Status changed from Feedback to Resolved

It looks still good so I'm resolving.

Actions

Also available in: Atom PDF