Project

General

Profile

Actions

action #105618

closed

[Alerting] CPU Load alert size:S

Added by okurz over 2 years ago. Updated over 2 years ago.

Status:
Rejected
Priority:
High
Assignee:
Category:
-
Target version:
Start date:
2022-01-27
Due date:
% Done:

0%

Estimated time:

Description

Observation

From https://stats.openqa-monitor.qa.suse.de/d/WebuiDb/webui-summary?orgId=1&from=1643204443403&to=1643210742310

5 Minutes Average has a value of 64.389 , that's a lot, right?

Suggestions

  • NOT postgres vaccuming
  • A surge of scheduled jobs
  • Consider increasing the threshold if we have a good hypothesis to accept what acceptable situation led to the alert
  • mkittler observed a gap in monitoring data for the same time period. Could point to the same underlying problem

Files

poo#105618.logs (4.03 MB) poo#105618.logs nicksinger, 2022-01-27 16:24

Related issues 2 (0 open2 closed)

Related to openQA Project - action #107701: [osd] Job detail page fails to loadResolvedtinita2022-02-28

Actions
Copied to openQA Infrastructure - action #105621: [Alerting] Failed systemd services alertResolvedmkittler2022-01-27

Actions
Actions #1

Updated by okurz over 2 years ago

  • Copied to action #105621: [Alerting] Failed systemd services alert added
Actions #2

Updated by nicksinger over 2 years ago

  • Description updated (diff)
Actions #3

Updated by nicksinger over 2 years ago

I've updated the link to include a more precise timestamp. What comes to my attention is how our metrics look in general at this point in time. We have missing data for a lot of metrics and the CPU graph shows that shortly after we receive data again we have >90% system utilization. "IO-wait" is quite low and storage graphs don't seem to be extra ordinarily high so this indicates we didn't see storage related problems here. However, shortly before we start missing metrics the RAM usage skyrocketed. After it cooled down, RAM was fine too again (higher but this could be explained by piled up minion jobs).

Actions #4

Updated by nicksinger over 2 years ago

I found something interesting in the journal with journalctl --since "2022-01-26 15:15:00" --until "2022-01-26 15:20:00":

-- Logs begin at Sat 2022-01-22 04:23:25 CET, end at Thu 2022-01-27 17:18:43 CET. --
Jan 26 15:15:01 openqa systemd[1]: Started Session 10654 of user geekotest.
Jan 26 15:15:01 openqa systemd[1]: Started Session 10657 of user geekotest.
Jan 26 15:15:01 openqa cron[27986]: pam_unix(crond:session): session opened for user geekotest by (uid=0)
Jan 26 15:15:01 openqa systemd[1]: Started Session 10656 of user geekotest.
Jan 26 15:15:01 openqa cron[27984]: pam_unix(crond:session): session opened for user geekotest by (uid=0)
Jan 26 15:15:01 openqa systemd[1]: Created slice User Slice of UID 17307.
Jan 26 15:15:01 openqa CRON[27996]: (geekotest) CMD (([ "$(df --output=pcent /results/testresults | sed '1d;s/[^0-9]//g')" -ge 88 ] && (! pgrep --full --list-full 'find /results/testresults' >/dev/null) && find /results/testresults -type f -iname '*.ogv' -mtime +28 -delete))
Jan 26 15:15:01 openqa systemd[1]: Starting User Runtime Directory /run/user/17307...
Jan 26 15:15:01 openqa systemd[1]: Started Session 10658 of user geekotest.
Jan 26 15:15:01 openqa systemd[1]: Started Session 10660 of user geekotest.
Jan 26 15:15:01 openqa systemd[1]: Started Session 10655 of user geekotest.
Jan 26 15:15:01 openqa cron[27990]: pam_unix(crond:session): session opened for user geekotest by (uid=0)
Jan 26 15:15:01 openqa CRON[27986]: pam_unix(crond:session): session closed for user geekotest
Jan 26 15:15:01 openqa cron[27987]: pam_unix(crond:session): session opened for user geekotest by (uid=0)
Jan 26 15:15:01 openqa systemd[1]: session-10654.scope: Succeeded.
Jan 26 15:15:01 openqa cron[27988]: pam_unix(crond:session): session opened for user geekotest by (uid=0)
Jan 26 15:15:01 openqa CRON[28019]: (geekotest) CMD ((cd /var/lib/openqa/share/factory/repo/ && for i in fixed/SLE-15-*-s390x-GM-*1/ ; do ln -sf $i ; done))
Jan 26 15:15:01 openqa cron[27989]: pam_unix(crond:session): session opened for user geekotest by (uid=0)
Jan 26 15:15:01 openqa systemd[1]: Finished User Runtime Directory /run/user/17307.
Jan 26 15:15:01 openqa systemd[1]: Starting User Manager for UID 17307...
Jan 26 15:15:01 openqa CRON[28027]: (geekotest) CMD ((cd /var/lib/openqa/share/factory/repo/ && for i in fixed/SLE-12-{,SP?-}Server-DVD-s390x-GM-DVD1/ ; do ln -sf $i ; done))
Jan 26 15:15:01 openqa CRON[28022]: (geekotest) CMD ((cd /var/lib/openqa/share/factory/repo/ && for i in fixed/SLE-12-SP?-SDK-POOL-{x86_64,aarch64,ppc64le,s390x}-BuildGM-Media1/ ; do ln -sf $i ; done))
Jan 26 15:15:01 openqa systemd[28030]: pam_unix(systemd-user:session): session opened for user cvdupdate by (uid=0)
Jan 26 15:15:02 openqa CRON[27988]: pam_unix(crond:session): session closed for user geekotest
Jan 26 15:15:02 openqa systemd[1]: session-10660.scope: Succeeded.
Jan 26 15:15:02 openqa CRON[27989]: pam_unix(crond:session): session closed for user geekotest
Jan 26 15:15:02 openqa systemd[1]: session-10655.scope: Succeeded.
Jan 26 15:15:02 openqa CRON[27987]: pam_unix(crond:session): session closed for user geekotest
Jan 26 15:15:02 openqa systemd[1]: session-10658.scope: Succeeded.
Jan 26 15:15:02 openqa systemd[28030]: Queued start job for default target Main User Target.
Jan 26 15:15:02 openqa systemd[28030]: Reached target Paths.
Jan 26 15:15:02 openqa systemd[28030]: Reached target Timers.
Jan 26 15:15:02 openqa systemd[28030]: Starting D-Bus User Message Bus Socket.
Jan 26 15:15:02 openqa systemd[28030]: Listening on D-Bus User Message Bus Socket.
Jan 26 15:15:02 openqa systemd[28030]: Reached target Sockets.
Jan 26 15:15:02 openqa systemd[28030]: Reached target Basic System.
Jan 26 15:15:02 openqa cron[27985]: pam_unix(crond:session): session opened for user cvdupdate by (uid=0)
Jan 26 15:15:02 openqa systemd[28030]: Reached target Main User Target.
Jan 26 15:15:02 openqa CRON[28114]: (cvdupdate) CMD (/home/cvdupdate/.local/bin/cvdupdate update)
Jan 26 15:15:02 openqa systemd[28030]: Startup finished in 815ms.
Jan 26 15:15:02 openqa CRON[27984]: pam_unix(crond:session): session closed for user geekotest
Jan 26 15:15:02 openqa systemd[1]: Started User Manager for UID 17307.
Jan 26 15:15:02 openqa systemd[1]: Started Session 10659 of user cvdupdate.
Jan 26 15:15:02 openqa systemd[1]: session-10657.scope: Succeeded.
Jan 26 15:15:05 openqa CRON[27985]: pam_unix(crond:session): session closed for user cvdupdate
Jan 26 15:15:05 openqa systemd[1]: session-10659.scope: Succeeded.
Jan 26 15:15:05 openqa postfix/pickup[22426]: C240A343F2: uid=17307 from=<cvdupdate>
Jan 26 15:15:05 openqa postfix/cleanup[28431]: C240A343F2: message-id=<20220126141505.C240A343F2@localhost>
Jan 26 15:15:05 openqa postfix/qmgr[1688]: C240A343F2: from=<cvdupdate@localhost>, size=1127, nrcpt=1 (queue active)
Jan 26 15:15:05 openqa postfix/local[28433]: C240A343F2: to=<cvdupdate@localhost>, orig_to=<cvdupdate>, relay=local, delay=0.26, delays=0.2/0.06/0/0.01, dsn=2.0.0, status=sent (delivered to mailbox)
Jan 26 15:15:05 openqa postfix/qmgr[1688]: C240A343F2: removed
Jan 26 15:15:10 openqa systemd[1]: session-10656.scope: Succeeded.
Jan 26 15:15:10 openqa CRON[27990]: pam_unix(crond:session): session closed for user geekotest
Jan 26 15:15:15 openqa systemd[1]: Stopping User Manager for UID 17307...
Jan 26 15:15:15 openqa systemd[28030]: Stopped target Main User Target.
Jan 26 15:15:15 openqa systemd[28030]: Stopped target Basic System.
Jan 26 15:15:15 openqa systemd[28030]: Stopped target Paths.
Jan 26 15:15:15 openqa systemd[28030]: Stopped target Sockets.
Jan 26 15:15:15 openqa systemd[28030]: Stopped target Timers.
Jan 26 15:15:15 openqa systemd[28030]: dbus.socket: Succeeded.
Jan 26 15:15:15 openqa systemd[28030]: Closed D-Bus User Message Bus Socket.
Jan 26 15:15:15 openqa systemd[28030]: Reached target Shutdown.
Jan 26 15:15:15 openqa systemd[28030]: systemd-exit.service: Succeeded.
Jan 26 15:15:15 openqa systemd[28030]: Finished Exit the Session.
Jan 26 15:15:15 openqa systemd[28030]: Reached target Exit the Session.
Jan 26 15:15:15 openqa systemd[1]: user@17307.service: Succeeded.
Jan 26 15:15:15 openqa systemd[1]: Stopped User Manager for UID 17307.
Jan 26 15:15:15 openqa systemd[1]: Stopping User Runtime Directory /run/user/17307...
Jan 26 15:15:15 openqa systemd[1298]: run-user-17307.mount: Succeeded.
Jan 26 15:15:15 openqa systemd[25535]: run-user-17307.mount: Succeeded.
Jan 26 15:15:15 openqa systemd[16764]: run-user-17307.mount: Succeeded.
Jan 26 15:15:15 openqa systemd[1297]: run-user-17307.mount: Succeeded.
Jan 26 15:15:15 openqa systemd[1]: run-user-17307.mount: Succeeded.
Jan 26 15:15:15 openqa systemd[1116]: run-user-17307.mount: Succeeded.
Jan 26 15:15:15 openqa systemd[1200]: run-user-17307.mount: Succeeded.
Jan 26 15:15:16 openqa systemd[1]: user-runtime-dir@17307.service: Succeeded.
Jan 26 15:15:16 openqa systemd[1]: Stopped User Runtime Directory /run/user/17307.
Jan 26 15:15:16 openqa systemd[1]: Removed slice User Slice of UID 17307.
Jan 26 15:19:01 openqa systemd-journald[29174]: Journal started
Jan 26 15:19:01 openqa systemd-journald[29174]: System Journal (/var/log/journal/84f4f4f356b525388b60f0ae547597e0) is 40.9G, max 80.0G, 39.0G free.
Jan 26 15:19:01 openqa kernel: sshd invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=-1000
Jan 26 15:19:01 openqa kernel: CPU: 8 PID: 1279 Comm: sshd Not tainted 5.3.18-59.40-default #1 SLE15-SP3
Jan 26 15:19:01 openqa kernel: Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
Jan 26 15:19:01 openqa kernel: Call Trace:
Jan 26 15:19:01 openqa kernel:  dump_stack+0x66/0x8b
Jan 26 15:19:01 openqa kernel:  dump_header+0x4a/0x200
Jan 26 15:19:01 openqa kernel:  oom_kill_process+0xe8/0x130
Jan 26 15:19:01 openqa kernel:  out_of_memory+0x105/0x510
Jan 26 15:19:01 openqa kernel:  __alloc_pages_slowpath+0x9f3/0xd20
Jan 26 15:19:01 openqa kernel:  __alloc_pages_nodemask+0x2cb/0x320
Jan 26 15:19:01 openqa kernel:  pagecache_get_page+0xa5/0x2c0
Jan 26 15:19:01 openqa kernel:  filemap_fault+0x381/0x9b0
Jan 26 15:19:01 openqa kernel:  ? xas_load+0x9/0x80
Jan 26 15:19:01 openqa kernel:  ? xas_find+0x177/0x1b0
Jan 26 15:19:01 openqa kernel:  ? filemap_map_pages+0xa3/0x3d0
Jan 26 15:19:01 openqa kernel:  ext4_filemap_fault+0x2c/0x40 [ext4]
Jan 26 15:19:01 openqa kernel:  __do_fault+0x31/0xb0
Jan 26 15:19:01 openqa kernel:  __handle_mm_fault+0xdfe/0x1260
Jan 26 15:19:01 openqa kernel:  handle_mm_fault+0xc4/0x200
Jan 26 15:19:01 openqa kernel:  __do_page_fault+0x2ce/0x500
Jan 26 15:19:01 openqa kernel:  do_page_fault+0x30/0x110
Jan 26 15:19:01 openqa kernel:  async_page_fault+0x3e/0x50
Jan 26 15:19:01 openqa kernel: RIP: 0033:0x557a912b2815
Jan 26 15:19:01 openqa kernel: Code: Bad RIP value.
Jan 26 15:19:01 openqa kernel: RSP: 002b:00007ffc4c702be0 EFLAGS: 00010203
Jan 26 15:19:01 openqa kernel: RAX: 0000000000000001 RBX: 0000000000000064 RCX: 00007fc165a922c6
Jan 26 15:19:01 openqa kernel: RDX: 0000000000000000 RSI: 0000557a921a6b50 RDI: 0000000000000009
Jan 26 15:19:01 openqa kernel: RBP: 0000557a9157c820 R08: 0000000000000000 R09: 0000000000000000
Jan 26 15:19:01 openqa kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 0000557a921a6b50
Jan 26 15:19:01 openqa kernel: R13: 0000000000000063 R14: 000000000000018c R15: 0000000000000001
Jan 26 15:19:01 openqa kernel: Mem-Info:
Jan 26 15:19:01 openqa kernel: active_anon:4366804 inactive_anon:336196 isolated_anon:0
                                active_file:8296 inactive_file:8624 isolated_file:332
                                unevictable:6872 dirty:17356 writeback:0 unstable:0
                                slab_reclaimable:110044 slab_unreclaimable:48237
                                mapped:782865 shmem:781828 pagetables:113940 bounce:0
                                free:64596 free_pcp:0 free_cma:0
Jan 26 15:19:01 openqa kernel: Node 0 active_anon:17467216kB inactive_anon:1344784kB active_file:43112kB inactive_file:24704kB unevictable:27488kB isolated(anon):0kB isolated(file):1072kB mapped:3131460kB dirty:69424kB writeback:0kB shmem:3127312kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 112640kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
Jan 26 15:19:01 openqa kernel: Node 0 DMA free:11812kB min:40kB low:52kB high:64kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15908kB mlocked:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Jan 26 15:19:01 openqa kernel: lowmem_reserve[]: 0 2976 24003 24003 24003
Jan 26 15:19:01 openqa kernel: Node 0 DMA32 free:109736kB min:26808kB low:29856kB high:32904kB active_anon:2446572kB inactive_anon:226320kB active_file:7100kB inactive_file:4596kB unevictable:0kB writepending:12820kB present:3129268kB managed:3063732kB mlocked:0kB kernel_stack:3016kB pagetables:57952kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Jan 26 15:19:01 openqa kernel: lowmem_reserve[]: 0 0 21026 21026 21026
Jan 26 15:19:01 openqa kernel: Node 0 Normal free:136836kB min:212484kB low:234012kB high:255540kB active_anon:15020644kB inactive_anon:1118464kB active_file:17260kB inactive_file:36568kB unevictable:27488kB writepending:56604kB present:22020096kB managed:17345388kB mlocked:27488kB kernel_stack:17800kB pagetables:397808kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Jan 26 15:19:01 openqa kernel: lowmem_reserve[]: 0 0 0 0 0
Jan 26 15:19:01 openqa kernel: Node 0 DMA: 1*4kB (U) 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 2*4096kB (M) = 11812kB
Jan 26 15:19:01 openqa kernel: Node 0 DMA32: 2436*4kB (UME) 2407*8kB (UME) 2024*16kB (UME) 1055*32kB (UME) 224*64kB (UME) 2*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 109736kB
Jan 26 15:19:01 openqa kernel: Node 0 Normal: 13361*4kB (UME) 5070*8kB (UME) 2677*16kB (UME) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 136836kB
Jan 26 15:19:01 openqa kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Jan 26 15:19:01 openqa kernel: 814777 total pagecache pages
Jan 26 15:19:01 openqa kernel: 13318 pages in swap cache
Jan 26 15:19:01 openqa kernel: Swap cache stats: add 1916152, delete 1902840, find 38055382/38633622
Jan 26 15:19:01 openqa kernel: Free swap  = 0kB
Jan 26 15:19:01 openqa kernel: Total swap = 499708kB
Jan 26 15:19:01 openqa kernel: 6291339 pages RAM
Jan 26 15:19:01 openqa kernel: 0 pages HighMem/MovableOnly
Jan 26 15:19:01 openqa kernel: 1185082 pages reserved
Jan 26 15:19:01 openqa kernel: 0 pages cma reserved
Jan 26 15:19:01 openqa kernel: 0 pages hwpoisoned
Jan 26 15:19:01 openqa kernel: Tasks state (memory values in pages):
Jan 26 15:19:01 openqa kernel: [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
Jan 26 15:19:01 openqa kernel: [    532]     0   532     3103      341    69632      802             0 haveged
Jan 26 15:19:01 openqa kernel: [    540]     0   540    30694      956   266240      578         -1000 systemd-udevd
Jan 26 15:19:01 openqa kernel: [    619]     0   619    85559     6663   221184        0         -1000 multipathd
Jan 26 15:19:01 openqa kernel: [    692]     0   692    13177      232   131072       90         -1000 auditd
Jan 26 15:19:01 openqa kernel: [    696]     0   696    11125        0   122880       91             0 rpc.idmapd
Jan 26 15:19:01 openqa kernel: [    712]     0   712    27239      533   245760      215             0 cupsd
Jan 26 15:19:01 openqa kernel: [    713]   499   713    16904     1004   176128      111          -900 dbus-daemon
Jan 26 15:19:01 openqa kernel: [    720]     0   720    27680     1246   249856      191             0 systemd-logind
Jan 26 15:19:01 openqa kernel: [    722]     0   722    11660      720   139264      299             0 wickedd-auto4
Jan 26 15:19:01 openqa kernel: [    724]     0   724    11661      733   135168      299             0 wickedd-dhcp4
Jan 26 15:19:01 openqa kernel: [    726]     0   726    11661      569   139264      257             0 wickedd-dhcp6
Jan 26 15:19:01 openqa kernel: [    767]     0   767    11697      867   135168      240             0 wickedd
Jan 26 15:19:01 openqa kernel: [    796]     0   796    11667      100   143360      258             0 wickedd-nanny
Jan 26 15:19:01 openqa kernel: [   1215]     0  1215    27550      254   249856      264             0 rebootmgrd
Jan 26 15:19:01 openqa kernel: [   1217]     0  1217     6270      443    94208      127             0 rsyncd
Jan 26 15:19:01 openqa kernel: [   1219]     0  1219    71029     1460   598016     9123             0 salt-master
Jan 26 15:19:01 openqa kernel: [   1221]     0  1221    50734      733   425984     4841             0 salt-minion
Jan 26 15:19:01 openqa kernel: [   1243]   475  1243    30914      381   155648       88             0 chronyd
Jan 26 15:19:01 openqa kernel: [   1244]     0  1244    11607      464   126976       96             0 xinetd
Jan 26 15:19:01 openqa kernel: [   1248]     0  1248   849306    13417   815104     8496             0 telegraf
Jan 26 15:19:01 openqa kernel: [   1260]   493  1260    13891      810   155648       94             0 rpcbind
Jan 26 15:19:01 openqa kernel: [   1272]   492  1272    23380      548   221184    12416             0 rpc.statd
Jan 26 15:19:01 openqa kernel: [   1279]     0  1279    18586      820   184320      194         -1000 sshd
Jan 26 15:19:01 openqa kernel: [   1297]     0  1297    22224      604   204800      265             0 systemd
Jan 26 15:19:01 openqa kernel: [   1298]  1001  1298    22224      949   217088      260             0 systemd
Jan 26 15:19:01 openqa kernel: [   1305]    26  1305   835822    16905   421888      256             0 postgres
Jan 26 15:19:01 openqa kernel: [   1316]     0  1316    16924      498   167936     6775             0 rpc.mountd
Jan 26 15:19:01 openqa kernel: [   1323]   497  1323    11969      469   139264      181          -500 nrpe
Jan 26 15:19:01 openqa kernel: [   1329]     0  1329    82871      763   679936      671             0 smbd
Jan 26 15:19:01 openqa kernel: [   1335]  1001  1335    33594       60   282624      853             0 (sd-pam)
Jan 26 15:19:01 openqa kernel: [   1336]     0  1336    33594       60   282624      853             0 (sd-pam)
Jan 26 15:19:01 openqa kernel: [   1438]    26  1438    25762      390   217088      301             0 postgres
Jan 26 15:19:01 openqa kernel: [   1462]     0  1462    81362       56   647168      675             0 smbd-notifyd
Jan 26 15:19:01 openqa kernel: [   1463]     0  1463    81364       48   634880      667             0 cleanupd
Jan 26 15:19:01 openqa kernel: [   1475]    26  1475   835855   693769  6184960      259             0 postgres
Jan 26 15:19:01 openqa kernel: [   1477]    26  1477   835855   303007  5476352      261             0 postgres
Jan 26 15:19:01 openqa kernel: [   1478]    26  1478   835822     4590   266240      270             0 postgres
Jan 26 15:19:01 openqa kernel: [   1479]    26  1479   835997      556   266240      322             0 postgres
Jan 26 15:19:01 openqa kernel: [   1480]    26  1480    26368      329   225280      271             0 postgres
Jan 26 15:19:01 openqa kernel: [   1481]    26  1481   835929      347   245760      311             0 postgres
Jan 26 15:19:01 openqa kernel: [   1500]     0  1500    82998      468   667648      643             0 lpqd
Jan 26 15:19:01 openqa kernel: [   1516]     0  1516    35826     1214   327680      321             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [   1587]     0  1587     9589      291   114688       67             0 agetty
Jan 26 15:19:01 openqa kernel: [   1591]     0  1591     9589      447   118784       68             0 agetty
Jan 26 15:19:01 openqa kernel: [   1676]     0  1676   303623     8625   729088     6812             0 salt-minion
Jan 26 15:19:01 openqa kernel: [   1686]     0  1686    22489      528   180224      142             0 master
Jan 26 15:19:01 openqa kernel: [   1688]    51  1688    27139      265   176128      127             0 qmgr
Jan 26 15:19:01 openqa kernel: [   1706]     0  1706    10895      516   126976      168             0 cron
Jan 26 15:19:01 openqa kernel: [   1720]     0  1720    53485      718   446464     6148             0 salt-master
Jan 26 15:19:01 openqa kernel: [   1721]     0  1721    71890      300   450560     6525             0 salt-minion
Jan 26 15:19:01 openqa kernel: [   1727]     0  1727    91888     1802   585728     8969             0 salt-master
Jan 26 15:19:01 openqa kernel: [   1730]     0  1730    77405     5694   626688     9620             0 salt-master
Jan 26 15:19:01 openqa kernel: [   1733]     0  1733   297356     6874   708608     5545             0 salt-master
Jan 26 15:19:01 openqa kernel: [   1735]     0  1735   269436    17090   741376     1831             0 salt-master
Jan 26 15:19:01 openqa kernel: [   1736]     0  1736    70985     1279   577536     9113             0 salt-master
Jan 26 15:19:01 openqa kernel: [   1738]     0  1738   165704     3077   634880     9570             0 salt-master
Jan 26 15:19:01 openqa kernel: [   1739]     0  1739   265103    12755   704512     1785             0 salt-master
Jan 26 15:19:01 openqa kernel: [   1748]     0  1748   265131    12531   712704     1872             0 salt-master
Jan 26 15:19:01 openqa kernel: [   1749]     0  1749   265103    12605   708608     1812             0 salt-master
Jan 26 15:19:01 openqa kernel: [   1750]     0  1750   265194    12480   708608     1899             0 salt-master
Jan 26 15:19:01 openqa kernel: [   1752]     0  1752    89910     2489   593920     8904             0 salt-master
Jan 26 15:19:01 openqa kernel: [   1753]     0  1753   265024    12675   704512     1686             0 salt-master
Jan 26 15:19:01 openqa kernel: [   3217]    26  3217   836302    75133  1130496      359             0 postgres
Jan 26 15:19:01 openqa kernel: [   3218]    26  3218   836304     3315   303104      443             0 postgres
Jan 26 15:19:01 openqa kernel: [   3237]    26  3237   836498     3493   315392      373             0 postgres
Jan 26 15:19:01 openqa kernel: [   5348]    26  5348   867579   243570  5758976     1426             0 postgres
Jan 26 15:19:01 openqa kernel: [  17168]    26 17168   853622   154280  2170880     2202             0 postgres
Jan 26 15:19:01 openqa kernel: [  29178]     0 29178   170234     1330  1191936      186          -250 systemd-journal
Jan 26 15:19:01 openqa kernel: [   1200] 17254  1200    22226      559   217088      253             0 systemd
Jan 26 15:19:01 openqa kernel: [   1201] 17254  1201    34092      166   290816      980             0 (sd-pam)
Jan 26 15:19:01 openqa kernel: [  16332] 17254 16332    13822      495   135168      824             0 screen
Jan 26 15:19:01 openqa kernel: [  16333] 17254 16333    12789      641   147456     1246             0 bash
Jan 26 15:19:01 openqa kernel: [  28183]    30 28183    35956      623   315392      278             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  28184]    30 28184    35958      856   315392      284             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  28187]    30 28187    40336     5110   348160      268             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  28188]    30 28188    35948      622   315392      275             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  28189]    30 28189    35958      654   315392      282             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  28191]    30 28191    35958      607   315392      279             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  28192]    30 28192    35960      895   315392      288             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  28193]    30 28193    35961      608   315392      273             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  28194]    30 28194    35961     1023   315392      288             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  28196]    30 28196    35958      859   315392      268             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  28197]    30 28197    35955      921   315392      259             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  28198]    30 28198    35950      983   315392      277             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  28199]    30 28199    35958      926   315392      274             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  28202]    30 28202    35957      906   315392      281             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  28203]    30 28203    35956      917   315392      277             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  28204]    30 28204    35956      741   315392      277             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  28206]    30 28206    35956      908   315392      281             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  28207]    30 28207    35956      836   315392      271             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  28208]    30 28208    35954      798   315392      281             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  28211]    30 28211    35954      875   315392      281             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  28212]    30 28212    35952      783   315392      282             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  28213]    30 28213    35959      916   315392      277             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  28215]    30 28215    35957      627   315392      285             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  28216]    30 28216    35957      911   315392      279             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  28217]    30 28217    35961      907   315392      278             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  28218]    30 28218    35955      914   315392      277             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  28220]    30 28220    35954      910   315392      283             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  28221]    30 28221    35960      633   315392      281             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  28222]    30 28222    37101     1889   323584      278             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  28223]    30 28223    35955      825   315392      291             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  28224]    30 28224    35960      912   315392      280             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  28225]    30 28225    35960      925   315392      276             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  28226]    30 28226    35961      910   315392      280             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  28227]    30 28227    35958      900   315392      277             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  28228]    30 28228    35957      907   315392      286             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  28229]    30 28229    35959      903   315392      284             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  28230]    30 28230    35959      725   315392      278             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  28232]    30 28232    35956      910   315392      279             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  28233]    30 28233    35958      725   315392      280             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  28234]    30 28234    35951      704   315392      284             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  28235]    30 28235    35954      898   315392      282             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  28236]    30 28236    35957      812   315392      282             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  28238]    30 28238    35963      617   315392      278             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  28239]    30 28239    35951      903   315392      283             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  28240]    30 28240    35959      912   315392      280             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  28241]    30 28241    35959      909   315392      282             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  28242]    30 28242    35961      914   315392      280             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  28243]    30 28243    35955      895   315392      283             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  28244]    30 28244    35959      635   315392      283             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  28245]    30 28245    35959      708   315392      282             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  28246]    30 28246    35957     1042   315392      275             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  28247]    30 28247    35957      744   315392      317             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  28248]    30 28248    35948      844   315392      280             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  28249]    30 28249    35960      619   315392      279             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  28250]    30 28250    35955      825   315392      277             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  28251]    30 28251    35963      911   315392      277             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  28253]    30 28253    35959      907   315392      279             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  28254]    30 28254    35957      615   315392      275             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  28255]    30 28255    35959      873   315392      273             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  28256]    30 28256    35960      915   315392      274             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  28257]    30 28257    35967      829   315392      275             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  28258]    30 28258    35978      922   315392      278             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  28259]    30 28259    35985     1056   315392      272             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  28261]    30 28261    35958      854   315392      279             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  28262]    30 28262    35971      831   315392      274             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  28263]    30 28263    35955      892   315392      279             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  28265]    30 28265    35955      911   315392      280             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  28266]    30 28266    35956     1002   315392      268             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  28267]    30 28267    35953      908   315392      283             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  28268]    30 28268    35956      913   315392      276             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  28269]    30 28269    35955      803   315392      277             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  28270]    30 28270    35955      892   315392      274             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  28271]    30 28271    35959      630   315392      281             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  28272]    30 28272    35955      634   315392      278             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  28273]    30 28273    35958      911   315392      279             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  28274]    30 28274    35954      900   315392      293             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  28275]    30 28275    35956      858   315392      267             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  28276]    30 28276    35953      717   315392      270             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  28277]    30 28277    35958      715   315392      280             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  28278]    30 28278    35967      848   315392      277             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  28279]    30 28279    35957      714   315392      278             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  28280]    30 28280    35953      867   315392      280             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  28282]    30 28282    35953      833   315392      282             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  28283]    30 28283    35956      788   315392      288             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  28285]    30 28285    35952      880   315392      279             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  28315]    30 28315    35960      696   315392      281             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  28318]    30 28318    35958      764   315392      283             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  28319]    30 28319    35959      605   315392      276             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  28320]    30 28320    35954      679   315392      282             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  28354]    30 28354    35960      702   315392      281             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  28355]    30 28355    35959      860   315392      278             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  28411]    30 28411    35959      705   315392      277             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  28583]    30 28583    35957      837   315392      280             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  28801]    30 28801    35957      851   315392      280             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30814]    30 30814    35977      904   315392      278             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30815]    30 30815    35959      793   315392      264             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30817]    30 30817    35954      908   315392      269             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30819]    30 30819    35947      890   315392      282             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30820]    30 30820    35959      823   315392      277             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30826]    30 30826    35956      588   315392      280             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30827]    30 30827    35959      624   315392      283             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30828]    30 30828    35961      820   315392      282             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30829]    30 30829    35957      587   315392      289             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30831]    30 30831    35956      958   315392      280             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30832]    30 30832    35953      875   315392      284             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30834]    30 30834    35956      826   315392      271             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30841]    30 30841    35958      898   315392      274             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30842]    30 30842    35956      891   315392      280             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30843]    30 30843    35976      905   315392      277             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30844]    30 30844    35956      588   315392      279             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30846]    30 30846    35974      609   315392      282             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30847]    30 30847    35955      915   315392      276             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30848]    30 30848    35959      853   315392      292             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30850]    30 30850    35953      890   315392      286             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30852]    30 30852    35957      894   315392      275             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30854]    30 30854    35954      909   315392      281             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30855]    30 30855    35977      920   315392      273             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30856]    30 30856    35954      911   315392      282             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30877]    30 30877    35964      917   315392      274             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30878]    30 30878    35955      614   315392      282             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30879]    30 30879    35958      613   315392      274             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30881]    30 30881    35962      907   315392      277             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30882]    30 30882    35952      932   315392      283             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30884]    30 30884    35958      833   315392      276             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30885]    30 30885    35958      710   315392      276             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30886]    30 30886    36148     1115   315392      268             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30887]    30 30887    35960      918   315392      279             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30888]    30 30888    35960      603   315392      282             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30889]    30 30889    35957      917   315392      270             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30890]    30 30890    35969      852   315392      287             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30891]    30 30891    35952      913   315392      276             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30892]    30 30892    40342     5258   348160      278             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30893]    30 30893    35969      909   315392      281             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30894]    30 30894    35961      888   315392      279             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30895]    30 30895    35946      800   315392      334             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30896]    30 30896    35960      861   315392      338             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30897]    30 30897    35953      667   315392      280             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30898]    30 30898    35957      915   315392      279             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30899]    30 30899    35956      894   315392      277             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30900]    30 30900    35952      640   315392      281             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30901]    30 30901    35946      891   315392      274             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30902]    30 30902    35958     1037   315392      276             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30903]    30 30903    35950      863   315392      297             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30904]    30 30904    35957      900   315392      273             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30905]    30 30905    35952     1030   315392      285             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30906]    30 30906    35960      873   315392      285             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30907]    30 30907    35960      985   315392      281             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30908]    30 30908    35972      911   315392      280             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30927]    30 30927    35956      887   315392      286             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30928]    30 30928    35959      908   315392      280             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30929]    30 30929    35960      597   315392      281             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30931]    30 30931    35958      914   315392      277             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30932]    30 30932    35953      646   315392      272             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30933]    30 30933    35960      921   315392      275             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30934]    30 30934    35965      911   315392      282             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30936]    30 30936    35967      876   315392      278             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30937]    30 30937    35958      847   315392      277             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30938]    30 30938    35951      911   315392      276             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30939]    30 30939    35960      798   315392      280             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30940]    30 30940    35958      914   315392      277             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30942]    30 30942    35952      904   315392      282             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30944]    30 30944    35981      924   315392      280             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30945]    30 30945    35956      911   315392      275             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30946]    30 30946    35962      926   315392      263             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30947]    30 30947    35946      907   315392      278             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30949]    30 30949    35972      910   315392      282             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30950]    30 30950    35959      841   315392      282             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30951]    30 30951    35959      822   315392      277             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30952]    30 30952    35947      907   315392      279             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30953]    30 30953    35968      852   315392      281             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30955]    30 30955    35955      902   315392      284             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30956]    30 30956    35972      704   315392      292             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30957]    30 30957    35978      949   315392      274             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30958]    30 30958    35956      992   315392      283             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30959]    30 30959    35948      843   315392      324             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30960]    30 30960    35957      890   315392      283             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30961]    30 30961    35973      918   315392      279             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30962]    30 30962    35954      906   315392      284             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30963]    30 30963    35960      880   315392      287             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30965]    30 30965    35970      716   315392      277             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30966]    30 30966    35952     1034   315392      280             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30967]    30 30967    35956      602   315392      291             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30969]    30 30969    35960      814   315392      281             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30970]    30 30970    35961      862   315392      283             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30971]    30 30971    35968      907   315392      284             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30973]    30 30973    35946      913   315392      276             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30974]    30 30974    35960      908   315392      280             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30975]    30 30975    35953      896   315392      281             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30976]    30 30976    35952      861   315392      283             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30977]    30 30977    35967      909   315392      282             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30978]    30 30978    35967      917   315392      278             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30982]    30 30982    35956     1048   315392      262             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30983]    30 30983    35961      763   315392      277             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30984]    30 30984    35948      924   315392      261             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30985]    30 30985    35957      913   315392      278             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30986]    30 30986    35977      902   315392      273             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30987]    30 30987    35965      725   315392      277             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30988]    30 30988    35951      848   315392      268             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30989]    30 30989    35967      788   315392      277             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  30990]    30 30990    35972      914   315392      281             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  20862]    30 20862    35957      860   315392      260             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  20864]    30 20864    35948      813   315392      265             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  20930]    30 20930    35956      645   315392      269             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21477]    30 21477    35951      813   315392      270             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21481]    30 21481    35958      821   315392      275             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21482]    30 21482    35954      911   315392      266             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21485]    30 21485    35957      771   315392      269             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21486]    30 21486    35956      908   315392      269             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21487]    30 21487    35953      890   315392      269             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21488]    30 21488    35956      914   315392      268             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21516]    30 21516    35970      770   315392      276             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21517]    30 21517    35953      900   315392      277             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21518]    30 21518    35955      901   315392      279             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21519]    30 21519    35955      632   315392      273             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21520]    30 21520    35955      716   315392      277             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21521]    30 21521    35956      821   315392      285             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21523]    30 21523    35954      883   315392      279             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21524]    30 21524    35953      889   315392      285             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21525]    30 21525    35959      635   315392      273             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21526]    30 21526    35956      903   315392      282             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21527]    30 21527    35958      911   315392      279             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21528]    30 21528    35959      914   315392      273             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21529]    30 21529    35953      911   315392      281             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21530]    30 21530    35954      906   315392      276             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21531]    30 21531    35953      608   315392      279             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21532]    30 21532    35980      881   315392      273             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21533]    30 21533    35957      904   315392      283             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21534]    30 21534    35956      905   315392      279             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21537]    30 21537    35961      939   315392      282             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21538]    30 21538    35953      820   315392      281             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21539]    30 21539    35977      618   315392      280             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21543]    30 21543    35959      829   315392      281             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21544]    30 21544    35957      903   315392      280             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21545]    30 21545    35956      915   315392      277             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21547]    30 21547    36015      708   315392      290             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21548]    30 21548    35954      639   315392      279             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21549]    30 21549    35952      920   315392      274             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21550]    30 21550    35953      895   315392      282             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21551]    30 21551    35958     1038   315392      273             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21552]    30 21552    35955      718   315392      281             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21554]    30 21554    35955      917   315392      282             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21555]    30 21555    35971      904   315392      277             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21556]    30 21556    35956      853   315392      277             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21557]    30 21557    35956      818   315392      279             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21558]    30 21558    35960      912   315392      278             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21559]    30 21559    35954      862   315392      280             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21560]    30 21560    35952      911   315392      280             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21561]    30 21561    35954      899   315392      285             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21562]    30 21562    35958      811   315392      282             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21563]    30 21563    35955      854   315392      282             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21564]    30 21564    35971      844   315392      270             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21565]    30 21565    35958      908   315392      279             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21566]    30 21566    35954      919   315392      275             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21567]    30 21567    35965      909   315392      280             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21568]    30 21568    35961      813   315392      277             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21569]    30 21569    35955      915   315392      275             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21571]    30 21571    35956      857   315392      286             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21572]    30 21572    35958      908   315392      280             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21573]    30 21573    35958      803   315392      280             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21574]    30 21574    35958      637   315392      279             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21575]    30 21575    35954      910   315392      280             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21579]    30 21579    35952      750   315392      256             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21580]    30 21580    35956      872   315392      264             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21581]    30 21581    35952      930   315392      262             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21582]    30 21582    35960      621   315392      261             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21584]    30 21584    35952      898   315392      281             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21585]    30 21585    35957      659   315392      266             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21586]    30 21586    35959      931   315392      262             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21587]    30 21587    35956      937   315392      261             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21588]    30 21588    35955      896   315392      276             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21589]    30 21589    35982      895   315392      266             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21590]    30 21590    35954      729   315392      263             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21591]    30 21591    35959      928   315392      258             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21592]    30 21592    35957      735   315392      260             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21593]    30 21593    35962      664   315392      260             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21594]    30 21594    35954      932   315392      263             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21595]    30 21595    35987      948   315392      260             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21596]    30 21596    35976      766   315392      260             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21597]    30 21597    35959      861   315392      259             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21599]    30 21599    35960      843   315392      261             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21600]    30 21600    35961      911   315392      261             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21601]    30 21601    35954      933   315392      259             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21602]    30 21602    35954      649   315392      270             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21603]    30 21603    35969      819   315392      261             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21604]    30 21604    35984      936   315392      267             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21605]    30 21605    35955      820   315392      298             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21606]    30 21606    35957      645   315392      268             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21607]    30 21607    35958      924   315392      268             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21608]    30 21608    35960      844   315392      273             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21609]    30 21609    35958      922   315392      268             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21610]    30 21610    36054      947   315392      299             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21613]    30 21613    35948      907   315392      258             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21614]    30 21614    35956      936   315392      259             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21615]    30 21615    35958      931   315392      256             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21616]    30 21616    35948      878   315392      261             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21617]    30 21617    35958     1051   315392      261             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21618]    30 21618    35954      934   315392      265             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21619]    30 21619    35956      661   315392      261             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21620]    30 21620    35954      938   315392      255             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21621]    30 21621    35954      917   315392      263             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21622]    30 21622    35955      934   315392      254             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21623]    30 21623    35976      937   315392      258             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21624]    30 21624    35955      742   315392      257             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21625]    30 21625    35954      888   315392      259             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21626]    30 21626    35955      941   315392      253             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21627]    30 21627    35970      700   315392      258             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21628]    30 21628    35948      639   315392      258             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21629]    30 21629    35953      855   315392      261             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21630]    30 21630    35959      924   315392      268             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21631]    30 21631    35954      934   315392      259             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21632]    30 21632    35954      892   315392      255             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21633]    30 21633    35954      936   315392      259             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21634]    30 21634    35954      798   315392      263             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21635]    30 21635    35957      741   315392      263             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21636]    30 21636    35967      936   315392      258             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21637]    30 21637    35961      883   315392      265             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21638]    30 21638    35958      785   315392      262             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21639]    30 21639    35956      928   315392      261             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21640]    30 21640    35957      611   315392      260             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21641]    30 21641    35962      941   315392      257             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21642]    30 21642    35955      644   315392      255             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21643]    30 21643    35955      937   315392      261             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21644]    30 21644    35976      921   315392      260             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21651]    30 21651    35956      756   315392      313             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21652]    30 21652    35982      939   315392      262             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21653]    30 21653    35956      924   315392      267             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21654]    30 21654    35958      914   315392      257             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21655]    30 21655    35962      920   315392      266             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21657]    30 21657    35981      939   315392      260             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21658]    30 21658    35960      666   315392      258             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21660]    30 21660    35962      862   315392      262             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21661]    30 21661    35970      936   315392      257             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21662]    30 21662    35954      918   315392      256             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21664]    30 21664    35972      899   315392      260             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21665]    30 21665    35954     1048   315392      262             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21666]    30 21666    35956      918   315392      259             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21667]    30 21667    35960      926   315392      263             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21668]    30 21668    35956      671   315392      256             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21669]    30 21669    35962      633   315392      258             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21670]    30 21670    35953      934   315392      261             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21671]    30 21671    35959      931   315392      258             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21673]    30 21673    35956      876   315392      260             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21674]    30 21674    35948      895   315392      264             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21675]    30 21675    35948     1052   315392      257             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21676]    30 21676    35954      922   315392      268             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21678]    30 21678    35957      927   315392      260             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21679]    30 21679    35956      879   315392      267             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21680]    30 21680    35954      907   315392      266             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21681]    30 21681    35958      664   315392      256             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21682]    30 21682    35955      658   315392      258             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21685]    30 21685    35955      786   315392      259             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21686]    30 21686    35970      824   315392      261             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21687]    30 21687    35950      930   315392      260             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21688]    30 21688    35958      934   315392      258             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21689]    30 21689    35948      921   315392      265             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21691]    30 21691    35956      935   315392      261             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21692]    30 21692    35946      907   315392      283             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21693]    30 21693    35954     1057   315392      258             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21694]    30 21694    35950      927   315392      254             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21695]    30 21695    35950      657   315392      263             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21696]    30 21696    35946      649   315392      265             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21697]    30 21697    35952      664   315392      253             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21698]    30 21698    35954      895   315392      258             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21699]    30 21699    35957      668   315392      255             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21701]    30 21701    35958      656   315392      261             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21702]    30 21702    35953      885   315392      265             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21703]    30 21703    35959      934   315392      255             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21704]    30 21704    35967      935   315392      257             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21705]    30 21705    35970      941   315392      260             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21706]    30 21706    35962      932   315392      259             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21708]    30 21708    35968      937   315392      257             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21709]    30 21709    35961      919   315392      265             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21710]    30 21710    35955      897   315392      255             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21711]    30 21711    35954      629   315392      266             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21714]    30 21714    35960      666   315392      258             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21715]    30 21715    35958      925   315392      258             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21716]    30 21716    35960      629   315392      268             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21717]    30 21717    35957      831   315392      264             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21721]    30 21721    35960      854   315392      279             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21722]    30 21722    35955      755   315392      258             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21723]    30 21723    35954      660   315392      264             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21724]    30 21724    35955     1057   315392      259             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21726]    30 21726    35958      822   315392      274             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21727]    30 21727    35958      859   315392      269             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21728]    30 21728    35955      867   315392      271             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21729]    30 21729    35958      932   315392      255             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21730]    30 21730    35957      842   315392      265             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21731]    30 21731    35954      935   315392      260             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21732]    30 21732    35957      928   315392      262             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21733]    30 21733    36344     1306   315392      261             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21734]    30 21734    35957     1059   315392      260             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21735]    30 21735    35959      748   315392      255             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21736]    30 21736    35958      646   315392      263             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21737]    30 21737    35966      851   315392      262             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21738]    30 21738    35960      905   315392      259             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21739]    30 21739    35954      655   315392      270             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21740]    30 21740    35963      821   315392      341             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21741]    30 21741    35972      935   315392      262             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21742]    30 21742    35972      873   315392      265             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21743]    30 21743    35948      876   315392      258             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21744]    30 21744    35961      862   315392      266             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21745]    30 21745    38461     3419   335872      259             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21746]    30 21746    35957      926   315392      261             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21747]    30 21747    35954      727   315392      267             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21748]    30 21748    35956      629   315392      263             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21749]    30 21749    35959      917   315392      267             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  21750]    30 21750    35971      871   315392      271             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  17972]    30 17972    35954      912   315392      263             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  17973]    30 17973    35954      703   315392      271             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  17974]    30 17974    35972      720   315392      259             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  17977]    30 17977    35983      920   315392      267             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  17979]    30 17979    35960      896   315392      255             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  17980]    30 17980    35960      866   315392      258             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  17984]    30 17984    35954      903   315392      273             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  17986]    30 17986    35954      829   315392      274             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  17987]    30 17987    35954      870   315392      276             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  17988]    30 17988    35958      908   315392      274             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  17989]    30 17989    35948      859   315392      274             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  17990]    30 17990    35975      900   315392      281             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  17991]    30 17991    35971      922   315392      271             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  17992]    30 17992    35973      918   315392      275             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  17994]    30 17994    35970      842   315392      277             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  17995]    30 17995    35955      913   315392      281             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  17996]    30 17996    35968      897   315392      277             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  17997]    30 17997    35958      919   315392      283             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  17998]    30 17998    35987      648   315392      286             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  17999]    30 17999    35954      902   315392      276             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  18000]    30 18000    35970      636   315392      282             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  18001]    30 18001    35954      721   315392      275             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  18002]    30 18002    35954      914   315392      277             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  18003]    30 18003    35958      918   315392      274             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  18004]    30 18004    35971      619   315392      279             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  18005]    30 18005    35960      902   315392      295             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  18006]    30 18006    35954      715   315392      271             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  18007]    30 18007    35952      621   315392      275             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  18009]    30 18009    35957     1030   315392      281             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  18014]    30 18014    35987      952   315392      256             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  18015]    30 18015    35974      932   315392      257             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  18016]    30 18016    35954      921   315392      273             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  18017]    30 18017    35961      897   315392      266             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  18018]    30 18018    35954      933   315392      261             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  18019]    30 18019    35961      865   315392      271             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  18020]    30 18020    35958      751   315392      255             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  18021]    30 18021    35955      916   315392      263             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  18022]    30 18022    35952      868   315392      257             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  18023]    30 18023    35948      915   315392      270             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  18024]    30 18024    35958      745   315392      260             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  18025]    30 18025    35948      626   315392      264             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  18026]    30 18026    35960      642   315392      258             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  18027]    30 18027    35960      938   315392      257             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  18028]    30 18028    35967      936   315392      258             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  18030]    30 18030    35955      761   315392      257             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  18031]    30 18031    35960      647   315392      261             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  18032]    30 18032    35957      932   315392      262             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  18033]    30 18033    35951      676   315392      266             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  18034]    30 18034    35952     1019   315392      258             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  18035]    30 18035    35960      935   315392      263             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  18036]    30 18036    35948      730   315392      257             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  18037]    30 18037    35973      922   315392      265             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  18038]    30 18038    35960     1061   315392      257             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  18040]    30 18040    35956      759   315392      261             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  18041]    30 18041    35960      933   315392      265             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  18042]    30 18042    35956      885   315392      266             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  18043]    30 18043    35952      933   315392      259             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  18044]    30 18044    35967      890   315392      257             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  18046]    30 18046    35961      923   315392      258             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  18047]    30 18047    35948      922   315392      264             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  18048]    30 18048    35954      934   315392      257             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  18062]    30 18062    35954     1048   315392      268             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  18063]    30 18063    35975      647   315392      259             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  18064]    30 18064    35981      883   315392      261             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  18065]    30 18065    35959      916   315392      256             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  18066]    30 18066    35960      938   315392      259             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  18067]    30 18067    35950      996   315392      258             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  18068]    30 18068    35960      659   315392      262             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  18069]    30 18069    35960      933   315392      261             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  18071]    30 18071    35956      638   315392      260             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  18072]    30 18072    35954      939   315392      255             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  18073]    30 18073    35960      941   315392      255             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  18074]    30 18074    35974      660   315392      263             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  18075]    30 18075    35954      935   315392      263             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  18076]    30 18076    35962      925   315392      263             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  18077]    30 18077    35958     1062   315392      259             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  18078]    30 18078    35973      805   315392      259             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  18079]    30 18079    35961      859   315392      258             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  18080]    30 18080    35954      937   315392      254             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  18081]    30 18081    35948      922   315392      261             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  18082]    30 18082    35952      663   315392      258             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  18083]    30 18083    35960      895   315392      263             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  18084]    30 18084    35950      824   315392      261             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  18085]    30 18085    35954      934   315392      259             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  18087]    30 18087    35970      929   315392      265             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  18088]    30 18088    35954      937   315392      260             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  18089]    30 18089    35960      946   315392      254             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  18090]    30 18090    35956      899   315392      272             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  18091]    30 18091    35985      774   315392      296             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  18092]    30 18092    35952      938   315392      257             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  18093]    30 18093    35946      769   315392      266             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [    451]    30   451    35960      892   315392      277             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [    455]    30   455    35958      895   315392      282             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [    456]    30   456    35958      833   315392      285             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [    475]    30   475    35955      844   315392      284             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [    476]    30   476    35954      771   315392      334             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [    478]    30   478    35960      907   315392      272             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [    479]    30   479    35955      620   315392      286             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [    503]    30   503    35952      881   315392      285             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [    504]    30   504    35953      885   315392      275             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [    506]    30   506    35954      612   315392      282             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [    507]    30   507    35956      912   315392      277             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [    508]    30   508    35971      644   315392      276             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [    509]    30   509    35957      913   315392      278             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [    510]    30   510    35953      908   315392      282             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [    519]    30   519    35971      624   315392      276             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [    520]    30   520    35973      638   315392      279             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [    521]    30   521    35952      642   315392      278             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [    522]    30   522    35986      918   315392      279             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [    523]    30   523    35951      909   315392      279             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [    524]    30   524    35955      791   315392      283             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [    525]    30   525    35954      919   315392      276             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [    526]    30   526    35954      918   315392      276             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [    527]    30   527    35955      619   315392      278             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [    528]    30   528    35957      632   315392      280             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [    529]    30   529    35967      599   315392      277             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [    530]    30   530    35943      853   315392      282             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [    531]    30   531    35975      917   315392      281             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [    534]    30   534    35956      865   315392      279             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [    541]    30   541    36011      918   315392      279             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [    542]    30   542    35956      893   315392      284             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [    543]    30   543    35955      918   315392      276             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [    544]    30   544    35954      720   315392      278             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [    545]    30   545    35950      899   315392      282             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [    546]    30   546    35955      911   315392      283             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [    547]    30   547    35960      764   315392      279             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [    548]    30   548    35973      869   315392      291             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [    549]    30   549    35956      912   315392      283             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [    550]    30   550    35954      912   315392      278             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [    551]    30   551    35973      913   315392      275             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [    552]    30   552    35971      891   315392      286             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [    553]    30   553    35960      921   315392      276             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [    554]    30   554    35954      638   315392      274             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [    555]    30   555    35954      894   315392      279             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [    556]    30   556    35974      728   315392      274             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [    557]    30   557    35952      633   315392      281             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [    558]    30   558    35954      857   315392      276             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [    559]    30   559    35958      905   315392      282             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [    560]    30   560    35985      891   315392      279             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [    561]    30   561    35954      917   315392      277             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [    562]    30   562    35981      744   315392      272             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [    564]    30   564    35984      931   315392      275             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [    565]    30   565    35952      912   315392      280             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [    566]    30   566    35952      924   315392      279             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [    567]    30   567    35954      916   315392      276             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [    568]    30   568    35961      911   315392      277             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [    569]    30   569    35956      841   315392      275             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [    570]    30   570    35973      655   315392      276             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [    571]    30   571    35961      643   315392      274             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [    572]    30   572    35960      923   315392      275             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [   1307]    30  1307    35960      881   315392      286             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [   1358]    30  1358    35960      822   315392      281             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [   1359]    30  1359    35948      629   315392      276             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [   1364]    30  1364    38537     3413   335872      279             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [   1365]    30  1365    35950      976   315392      281             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [   1367]    30  1367    35977      898   315392      289             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [   1368]    30  1368    35953      824   315392      278             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [   1372]    30  1372    35952      894   315392      282             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [   1373]    30  1373    35953      882   315392      278             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [   1374]    30  1374    35954      901   315392      280             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [   1375]    30  1375    35960      857   315392      278             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [   1376]    30  1376    35960      679   315392      286             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [   1377]    30  1377    35984      614   315392      281             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [   1378]    30  1378    35954      899   315392      279             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [   1379]    30  1379    35956      899   315392      280             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [   1386]    30  1386    35960      912   315392      281             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [   1387]    30  1387    35960      707   315392      280             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [   1388]    30  1388    35950      853   315392      288             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [   1390]    30  1390    35956      913   315392      280             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [   1391]    30  1391    35960      904   315392      281             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [   1392]    30  1392    35960      920   315392      277             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [   1393]    30  1393    35960      639   315392      279             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [   1394]    30  1394    35956      908   315392      282             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [   1395]    30  1395    35960      708   315392      289             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [   1396]    30  1396    35952      616   315392      276             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [   1397]    30  1397    35962      838   315392      282             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [   1399]    30  1399    35955      913   315392      276             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [   1400]    30  1400    38402     3353   331776      282             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [   1401]    30  1401    35954      901   315392      281             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [   1406]    30  1406    35954      909   315392      285             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [   1407]    30  1407    35962      922   315392      278             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [   1408]    30  1408    35956      796   315392      368             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [   1409]    30  1409    35954      853   315392      279             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [   1410]    30  1410    35960      918   315392      276             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [   1411]    30  1411    35956      725   315392      278             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [   1412]    30  1412    35973      863   315392      283             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [   1413]    30  1413    35958      819   315392      286             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [   1414]    30  1414    35954      911   315392      281             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [   1415]    30  1415    35950      907   315392      280             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [   1416]    30  1416    35950      875   315392      313             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [   1417]    30  1417    35960      912   315392      281             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [   1419]    30  1419    35960      847   315392      286             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [   1420]    30  1420    35987      924   315392      279             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [   1421]    30  1421    35974      901   315392      279             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [   1422]    30  1422    35960      912   315392      280             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [   1423]    30  1423    35958      800   315392      297             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [   1424]    30  1424    35961      911   315392      276             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [   1425]    30  1425    35950      915   315392      276             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [   1426]    30  1426    35956      829   315392      284             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [   1427]    30  1427    35960      912   315392      281             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [   1428]    30  1428    35960      899   315392      280             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [   1429]    30  1429    35961      893   315392      289             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [   1430]    30  1430    35960      916   315392      282             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [   1435]    30  1435    35973      728   315392      279             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [   1436]    30  1436    35952      626   315392      289             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [   1437]    30  1437    35960      922   315392      277             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [   1439]    30  1439    35948      863   315392      280             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [   1440]    30  1440    35958      916   315392      280             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [   1441]    30  1441    37518     2208   327680      279             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [   1442]    30  1442    35960      915   315392      282             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [   1443]    30  1443    35972      906   315392      282             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [   1449]    30  1449    35973      922   315392      277             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [   1450]    30  1450    35960      880   315392      283             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [   1451]    30  1451    35960      621   315392      277             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [   1452]    30  1452    35960      843   315392      280             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [   1453]    30  1453    35960      823   315392      294             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [   1454]    30  1454    35952      816   315392      281             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [   1455]    30  1455    35962      878   315392      280             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [   1456]    30  1456    35956      634   315392      283             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [   1457]    30  1457    35954      628   315392      285             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [   1458]    30  1458    35960      824   315392      284             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [   1459]    30  1459    35958      905   315392      287             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [   1460]    30  1460    35982      922   315392      281             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [   1461]    30  1461    35960      892   315392      285             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [   1464]    30  1464    35960      724   315392      281             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [   1466]    30  1466    35960      641   315392      282             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [   1469]    30  1469    35958      916   315392      280             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [   1470]    30  1470    35955      902   315392      282             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [   1471]    30  1471    36956     1876   323584      279             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [   1472]    30  1472    35960      897   315392      289             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [   1473]    30  1473    35960      624   315392      279             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [   1474]    30  1474    35954      630   315392      288             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [   1476]    30  1476    35958      906   315392      282             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [   1482]    30  1482    35950      859   315392      278             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [   1483]    30  1483    35960      859   315392      280             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [   1484]    30  1484    36503     1467   319488      279             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [   1485]    30  1485    35986      928   315392      277             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [   1486]    30  1486    35954      908   315392      282             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [   1487]    30  1487    35986      748   315392      277             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [   1488]    30  1488    35957      872   315392      281             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [   1489]    30  1489    35958      922   315392      277             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [   1490]    30  1490    35952      907   315392      284             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  11617]     0 11617     5707      418    90112       98             0 vsftpd
Jan 26 15:19:01 openqa kernel: [  16419]    30 16419    35961      922   315392      253             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  16423]    30 16423    35972      923   315392      254             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  16424]    30 16424    35956      858   315392      254             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  16426]    30 16426    35957      913   315392      254             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  16427]    30 16427    35960      620   315392      254             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  16428]    30 16428    35962      908   315392      262             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  16429]    30 16429    35955      928   315392      254             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  16431]    30 16431    35968      931   315392      254             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  16432]    30 16432    35961      881   315392      254             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  16433]    30 16433    35961      918   315392      254             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  16434]    30 16434    35950      832   315392      254             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  16435]    30 16435    35961      925   315392      254             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  16436]    30 16436    35952      712   315392      254             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  16437]    30 16437    35971      462   315392      254             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  16438]    30 16438    35960      916   315392      254             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  16473]    30 16473    35951      919   315392      255             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  16474]    30 16474    35967      890   315392      254             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  16475]    30 16475    35979      931   315392      254             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  16476]    30 16476    35952      903   315392      254             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  16478]    30 16478    35944      876   315392      254             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  16479]    30 16479    35961      723   315392      255             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  16480]    30 16480    35955     1057   315392      253             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  16483]    30 16483    35960      742   315392      254             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  16484]    30 16484    35981      946   315392      253             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  16485]    30 16485    35960      935   315392      254             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  16486]    30 16486    35948      656   315392      253             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  16487]    30 16487    35954      736   315392      254             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  16488]    30 16488    35960      932   315392      254             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  16489]    30 16489    35972      841   315392      254             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  16490]    30 16490    35960      907   315392      254             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  16491]    30 16491    35946      886   315392      254             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  16522]    30 16522    35961      933   315392      254             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  16523]    30 16523    35952      919   315392      254             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  16524]    30 16524    35976      942   315392      254             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  16525]    30 16525    35950      870   315392      253             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  16526]    30 16526    35954      848   315392      254             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  16527]    30 16527    35952      938   315392      254             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  16528]    30 16528    35959      962   315392      253             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  16529]    30 16529    35959      937   315392      262             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  16531]    30 16531    35950      903   315392      253             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  16532]    30 16532    35961      932   315392      254             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  16533]    30 16533    35960      832   315392      254             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  16534]    30 16534    35961      926   315392      254             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  16535]    30 16535    35969      653   315392      253             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  16537]    30 16537    35948      879   315392      253             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  16538]    30 16538    35956      932   315392      254             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  16539]    30 16539    35961      826   315392      254             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  16540]    30 16540    35952      651   315392      254             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  16541]    30 16541    35954      935   315392      254             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  16542]    30 16542    35959      852   315392      254             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  16543]    30 16543    35954      856   315392      254             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  16544]    30 16544    35959      636   315392      254             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  16546]    30 16546    35948      938   315392      253             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  16547]    30 16547    35985      929   315392      253             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  16550]    30 16550    35969      938   315392      254             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  16551]    30 16551    35960      890   315392      254             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  16552]    30 16552    35953      926   315392      262             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  16553]    30 16553    35962      934   315392      254             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  16554]    30 16554    35962      661   315392      254             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  16555]    30 16555    35960      663   315392      254             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  16556]    30 16556    35967      937   315392      253             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  16557]    30 16557    35976      930   315392      254             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  16580]    30 16580    35959      918   315392      254             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  16581]    30 16581    35962      812   315392      254             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  16582]    30 16582    35974     1062   315392      254             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  16583]    30 16583    35972      938   315392      253             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  16584]    30 16584    35970      651   315392      254             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  16585]    30 16585    35946      639   315392      254             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  16587]    30 16587    35975      940   315392      254             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  16588]    30 16588    35961      870   315392      254             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  16589]    30 16589    35948      925   315392      262             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  16591]    30 16591    35950      893   315392      254             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  16592]    30 16592    35951      759   315392      262             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  16593]    30 16593    35955      660   315392      254             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  16594]    30 16594    35960      752   315392      254             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  16595]    30 16595    35955      638   315392      254             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  16596]    30 16596    35971      752   315392      253             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  16597]    30 16597    36495     1481   319488      253             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  16598]    30 16598    35954      816   315392      254             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  16599]    30 16599    35948      933   315392      254             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  16600]    30 16600    35962      666   315392      254             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  16601]    30 16601    35960      730   315392      254             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  16602]    30 16602    35948      932   315392      254             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  16603]    30 16603    35948      664   315392      254             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  16604]    30 16604    35948      758   315392      254             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  16605]    30 16605    35970      900   315392      253             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  16606]    30 16606    35952      932   315392      254             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  16607]    30 16607    35978      875   315392      254             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  16608]    30 16608    35968      878   315392      253             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  16609]    30 16609    35948      910   315392      254             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  16610]    30 16610    35960      933   315392      253             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  16611]    30 16611    35948      931   315392      254             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  16612]    30 16612    35972      937   315392      254             0 httpd-prefork
Jan 26 15:19:01 openqa kernel: [  12194]    26 12194   856681   563705  6602752      202             0 postgres
Jan 26 15:19:01 openqa kernel: [  16390]  1001 16390    76927    37552   655360        0             0 openqa-livehand
Jan 26 15:19:01 openqa kernel: [  16392]  1001 16392    75112    35744   634880        0             0 openqa-schedule
Jan 26 15:19:01 openqa kernel: [  16393]  1001 16393    78719    39283   675840        0             0 openqa-websocke
Jan 26 15:19:01 openqa kernel: [  16394]  1001 16394    88013    40045   724992        0             0 openqa
Jan 26 15:19:01 openqa kernel: [  16441]  1001 16441    84954    40005   712704        0             0 openqa
Jan 26 15:19:01 openqa kernel: [  16444]    26 16444   838179   216394  5898240      213             0 postgres
Jan 26 15:19:01 openqa kernel: [  16464]    26 16464   836147     3723   368640      216             0 postgres
Jan 26 15:19:01 openqa kernel: [  16515]    26 16515   836441    42706  2310144      213             0 postgres
Jan 26 15:19:01 openqa kernel: [  16549]    26 16549   836147     3738   368640      216             0 postgres
Jan 26 15:19:01 openqa kernel: [  16574]    26 16574   856565    24086   958464      209             0 postgres
Jan 26 15:19:01 openqa kernel: [  16642]    26 16642   836430   135067  3260416      213             0 postgres
Jan 26 15:19:01 openqa kernel: [  25523]     0 25523    27439     1034   258048        0             0 sshd
Jan 26 15:19:01 openqa kernel: [  25535] 17305 25535    22226      868   204800        0             0 systemd
Jan 26 15:19:01 openqa kernel: [  25536] 17305 25536    34117     1002   290816      146             0 (sd-pam)
Jan 26 15:19:01 openqa kernel: [  25544] 17305 25544    27621      880   237568        0             0 sshd
Jan 26 15:19:01 openqa kernel: [  25546] 17305 25546    13024     2127   147456        0             0 bash
Jan 26 15:19:01 openqa kernel: [  20118]     0 20118    27439     1046   253952        0             0 sshd
Jan 26 15:19:01 openqa kernel: [  20132] 17305 20132    27439      583   245760        0             0 sshd
Jan 26 15:19:01 openqa kernel: [  20154] 17305 20154    13056     2165   151552        0             0 bash
Jan 26 15:19:01 openqa kernel: [  22426]    51 22426    21960      391   172032        0             0 pickup
Jan 26 15:19:01 openqa kernel: [  24011]  1001 24011    91550    44875   753664        0             0 openqa
Jan 26 15:19:01 openqa kernel: [  24012]    26 24012   837560   104878  4796416      213             0 postgres
Jan 26 15:19:01 openqa kernel: [  25537]  1001 25537    97313    51076   798720        0             0 openqa
Jan 26 15:19:01 openqa kernel: [  25542]    26 25542   856099   307830  6328320      207             0 postgres
Jan 26 15:19:01 openqa kernel: [  27075]    26 27075   836733     4321   425984      216             0 postgres
Jan 26 15:19:01 openqa kernel: [  27484]  1001 27484   443312   389795  3530752        0             0 openqa
Jan 26 15:19:01 openqa kernel: [  27487]    26 27487   836884     4907   421888      213             0 postgres
Jan 26 15:19:01 openqa kernel: [  27500]    26 27500   859357   285356  5971968      208             0 postgres
Jan 26 15:19:01 openqa kernel: [  28087]  1001 28087    96164    49719   790528        0             0 openqa
Jan 26 15:19:01 openqa kernel: [  28091]    26 28091   855449   292992  6307840      207             0 postgres
Jan 26 15:19:01 openqa kernel: [  31334]  1001 31334   101507    55233   831488        0             0 openqa
Jan 26 15:19:01 openqa kernel: [  31338]    26 31338   855415   251853  6246400      208             0 postgres
Jan 26 15:19:01 openqa kernel: [  31559]  1001 31559    96142    49852   790528        0             0 openqa
Jan 26 15:19:01 openqa kernel: [  31561]    26 31561   838868   246263  6082560      212             0 postgres
Jan 26 15:19:01 openqa kernel: [   1099]     0  1099    27439     1037   258048        0             0 sshd
Jan 26 15:19:01 openqa kernel: [   1116] 17285  1116    22226     1145   204800        0             0 systemd
Jan 26 15:19:01 openqa kernel: [   1117] 17285  1117    34117     1008   290816      146             0 (sd-pam)
Jan 26 15:19:01 openqa kernel: [   1133] 17285  1133    27439      453   249856        0             0 sshd
Jan 26 15:19:01 openqa kernel: [   1135] 17285  1135    12994     2070   147456        0             0 bash
Jan 26 15:19:01 openqa kernel: [   2119] 17285  2119    26150      800   249856        0             0 sudo
Jan 26 15:19:01 openqa kernel: [   2126]     0  2126    21796      668   217088        0             0 su
Jan 26 15:19:01 openqa kernel: [   2129]     0  2129    13013     2098   139264        0             0 bash
Jan 26 15:19:01 openqa kernel: [   7190]  1001  7190   101020    54582   827392        0             0 openqa
Jan 26 15:19:01 openqa kernel: [   7202]    26  7202   841415   337840  6561792      212             0 postgres
Jan 26 15:19:01 openqa kernel: [   7355]    26  7355   836916     7686   516096      213             0 postgres
Jan 26 15:19:01 openqa kernel: [   7686]  1001  7686    99144    52529   815104        0             0 openqa
Jan 26 15:19:01 openqa kernel: [   7693]    26  7693   855303   284264  6369280      207             0 postgres
Jan 26 15:19:01 openqa kernel: [   8465]    26  8465   836916     7562   503808      213             0 postgres
Jan 26 15:19:01 openqa kernel: [   8615]    26  8615   836733     4354   401408      216             0 postgres
Jan 26 15:19:01 openqa kernel: [   8845]  1001  8845    99286    52882   815104        0             0 openqa
Jan 26 15:19:01 openqa kernel: [   8847]    26  8847   838116   253521  6107136      212             0 postgres
Jan 26 15:19:01 openqa kernel: [   8951]    26  8951   836899     7501   495616      213             0 postgres
Jan 26 15:19:01 openqa kernel: [   9088]    26  9088   836881     7146   503808      213             0 postgres
Jan 26 15:19:01 openqa kernel: [   9476]  1001  9476    97574    49764   802816        0             0 openqa
Jan 26 15:19:01 openqa kernel: [   9479]    26  9479   855088   211509  6135808      208             0 postgres
Jan 26 15:19:01 openqa kernel: [   9786]  1001  9786    96388    49563   790528        0             0 openqa
Jan 26 15:19:01 openqa kernel: [   9793]    26  9793   838633   258981  6221824      212             0 postgres
Jan 26 15:19:01 openqa kernel: [  10948]  1001 10948    93831    47372   770048        0             0 openqa
Jan 26 15:19:01 openqa kernel: [  10951]    26 10951   837828   157936  5574656      213             0 postgres
Jan 26 15:19:01 openqa kernel: [  11004]    26 11004   836941     7296   516096      213             0 postgres
Jan 26 15:19:01 openqa kernel: [  12297]  1001 12297    94207    47963   774144        0             0 openqa
Jan 26 15:19:01 openqa kernel: [  12298]    26 12298   838222   229677  5836800      212             0 postgres
Jan 26 15:19:01 openqa kernel: [  12524]  1001 12524    94282    47526   774144        0             0 openqa
Jan 26 15:19:01 openqa kernel: [  12525]    26 12525   857728   178019  5820416      208             0 postgres
Jan 26 15:19:01 openqa kernel: [  13324]    26 13324   836733     4337   417792      216             0 postgres
Jan 26 15:19:01 openqa kernel: [  13798]  1001 13798    94013    47729   774144        0             0 openqa
Jan 26 15:19:01 openqa kernel: [  13802]    26 13802   837717   142999  5525504      213             0 postgres
Jan 26 15:19:01 openqa kernel: [  14203]  1001 14203    94493    47751   778240        0             0 openqa
Jan 26 15:19:01 openqa kernel: [  14206]    26 14206   857506   214835  6107136      208             0 postgres
Jan 26 15:19:01 openqa kernel: [  14771]    26 14771   836322     6739   434176      213             0 postgres
Jan 26 15:19:01 openqa kernel: [  16031]  1001 16031    93320    47126   753664        0             0 openqa
Jan 26 15:19:01 openqa kernel: [  16035]    26 16035   837742   125545  5324800      213             0 postgres
Jan 26 15:19:01 openqa kernel: [  16261]    26 16261   836733     4327   417792      216             0 postgres
Jan 26 15:19:01 openqa kernel: [  16742]     0 16742    27439     1034   249856        0             0 sshd
Jan 26 15:19:01 openqa kernel: [  16764] 17258 16764    22229     1195   208896        0             0 systemd
Jan 26 15:19:01 openqa kernel: [  16767] 17258 16767    34117     1008   290816      146             0 (sd-pam)
Jan 26 15:19:01 openqa kernel: [  16786] 17258 16786    27439      689   241664        0             0 sshd
Jan 26 15:19:01 openqa kernel: [  16791] 17258 16791    13037     2123   151552        0             0 bash
Jan 26 15:19:01 openqa kernel: [  16925] 17258 16925    26148      779   237568        0             0 sudo
Jan 26 15:19:01 openqa kernel: [  16930]     0 16930    22310      651   221184        0             0 su
Jan 26 15:19:01 openqa kernel: [  16933]  1001 16933    13072     2109   147456        0             0 bash
Jan 26 15:19:01 openqa kernel: [  17026]  1001 17026  3239487  2616717 21155840        0             0 psql
Jan 26 15:19:01 openqa kernel: [  17028]    26 17028   836286   480460  5505024      213             0 postgres
Jan 26 15:19:01 openqa kernel: [  19225]    26 19225   836733     4321   409600      216             0 postgres
Jan 26 15:19:01 openqa kernel: [  19325]  1001 19325    93766    47182   770048        0             0 openqa
Jan 26 15:19:01 openqa kernel: [  19331]    26 19331   837570   148400  5451776      213             0 postgres
Jan 26 15:19:01 openqa kernel: [  19896]   490 19896    13059      557   143360       87             0 vsftpd
Jan 26 15:19:01 openqa kernel: [  19930]    40 19930    13065      409   139264       87             0 vsftpd
Jan 26 15:19:01 openqa kernel: [  20177]   490 20177    13059      557   143360       87             0 vsftpd
Jan 26 15:19:01 openqa kernel: [  20197]    40 20197    13065      412   139264       87             0 vsftpd
Jan 26 15:19:01 openqa kernel: [  20204]    26 20204   836735     4335   401408      216             0 postgres
Jan 26 15:19:01 openqa kernel: [  20210]  1001 20210    91506    45678   741376        0             0 openqa
Jan 26 15:19:01 openqa kernel: [  20212]    26 20212   837679   159085  5488640      213             0 postgres
Jan 26 15:19:01 openqa kernel: [  23608]  1001 23608    93186    46879   765952        0             0 openqa
Jan 26 15:19:01 openqa kernel: [  23611]    26 23611   853927   118186  5074944      208             0 postgres
Jan 26 15:19:01 openqa kernel: [  25144]    26 25144   836733     4338   397312      216             0 postgres
Jan 26 15:19:01 openqa kernel: [  25479]  1001 25479    87976    43003   712704        0             0 openqa
Jan 26 15:19:01 openqa kernel: [  25482]    26 25482   836884     4956   425984      213             0 postgres
Jan 26 15:19:01 openqa kernel: [  25499]    26 25499   836367    13834  1257472      213             0 postgres
Jan 26 15:19:01 openqa kernel: [  26319]    26 26319   836733     4321   397312      216             0 postgres
Jan 26 15:19:01 openqa kernel: [  27962]    26 27962   836322     6738   434176      213             0 postgres
Jan 26 15:19:01 openqa kernel: [  28431]    51 28431    27134      744   176128        0             0 cleanup
Jan 26 15:19:01 openqa kernel: [  28432]    51 28432    27110      669   180224        0             0 trivial-rewrite
Jan 26 15:19:01 openqa kernel: [  28433]     0 28433    27119      739   176128        0             0 local
Jan 26 15:19:01 openqa kernel: [  28894] 65534 28894    27367     1653   253952       81             0 rsyncd
Jan 26 15:19:01 openqa kernel: [  28961]  1001 28961    88013    41728   708608        0             0 openqa
Jan 26 15:19:01 openqa kernel: [  28962]    26 28962   836387    14683  1212416      213             0 postgres
Jan 26 15:19:01 openqa kernel: [  28973]  1001 28973    85088    40117   692224        0             0 openqa
Jan 26 15:19:01 openqa kernel: [  28974]  1001 28974    85136    40287   692224        0             0 openqa
Jan 26 15:19:01 openqa kernel: [  28975]    26 28975   836884     4814   425984      213             0 postgres
Jan 26 15:19:01 openqa kernel: [  28976]    26 28976   836884     4891   425984      213             0 postgres
Jan 26 15:19:01 openqa kernel: [  28977]    26 28977   836327     4332   401408      213             0 postgres
Jan 26 15:19:01 openqa kernel: [  28978]    26 28978   836327     4300   438272      213             0 postgres
Jan 26 15:19:01 openqa kernel: [  28986]  1001 28986     2267      124    57344        0             0 timeout
Jan 26 15:19:01 openqa kernel: [  28989]  1001 28989     3117      500    65536        0             0 openqa-label-kn
Jan 26 15:19:01 openqa kernel: [  28997]  1001 28997     3150      490    65536        0             0 openqa-label-kn
Jan 26 15:19:01 openqa kernel: [  28998]  1001 28998     3986      137    73728        0             0 sed
Jan 26 15:19:01 openqa kernel: [  28999]  1001 28999     3150      511    65536        0             0 openqa-investig
Jan 26 15:19:01 openqa kernel: [  29015]  1001 29015     3150      279    61440        0             0 openqa-investig
Jan 26 15:19:01 openqa kernel: [  29016]  1001 29016     1233       87    57344        0             0 cat
Jan 26 15:19:01 openqa kernel: [  29017]  1001 29017     3986      118    77824        0             0 sed
Jan 26 15:19:01 openqa kernel: [  29019]  1001 29019     3150      273    61440        0             0 openqa-label-kn
Jan 26 15:19:01 openqa kernel: [  29020]  1001 29020     3150      333    61440        0             0 openqa-label-kn
Jan 26 15:19:01 openqa kernel: [  29021]  1001 29021     3150      323    61440        0             0 openqa-label-kn
Jan 26 15:19:01 openqa kernel: [  29022]  1001 29022     3150      360    61440        0             0 openqa-label-kn
Jan 26 15:19:01 openqa kernel: [  29024]  1001 29024     3150      115    61440        0             0 openqa-label-kn
Jan 26 15:19:01 openqa kernel: [  29025]  1001 29025    35713      787   180224        0             0 curl
Jan 26 15:19:01 openqa kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=psql,pid=17026,uid=1001
Jan 26 15:19:01 openqa kernel: Out of memory: Killed process 17026 (psql) total-vm:12957948kB, anon-rss:10464292kB, file-rss:2576kB, shmem-rss:0kB
Jan 26 15:19:01 openqa systemd[1]: systemd-logind.service: Watchdog timeout (limit 3min)!
Jan 26 15:19:01 openqa systemd[1]: systemd-logind.service: Killing process 720 (systemd-logind) with signal SIGABRT.
Jan 26 15:19:01 openqa systemd[1]: systemd-udevd.service: Watchdog timeout (limit 3min)!
Jan 26 15:19:01 openqa systemd[1]: systemd-udevd.service: Killing process 540 (systemd-udevd) with signal SIGABRT.
Jan 26 15:19:01 openqa kernel: oom_reaper: reaped process 17026 (psql), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Jan 26 15:19:01 openqa systemd[1]: systemd-udevd.service: Main process exited, code=dumped, status=6/ABRT
Jan 26 15:19:01 openqa systemd[1]: systemd-udevd.service: Failed with result 'watchdog'.
Jan 26 15:19:01 openqa systemd[1]: systemd-journald.service: Main process exited, code=dumped, status=6/ABRT
Jan 26 15:19:01 openqa systemd[1]: systemd-journald.service: Failed with result 'watchdog'.
Jan 26 15:19:01 openqa systemd[1]: systemd-udevd.service: Scheduled restart job, restart counter is at 1.
Jan 26 15:19:01 openqa systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1.
Jan 26 15:19:01 openqa systemd[1]: Stopping Flush Journal to Persistent Storage...
Jan 26 15:19:01 openqa systemd[1]: Stopped Rule-based Manager for Device Events and Files.
Jan 26 15:19:01 openqa systemd[1]: Starting Rule-based Manager for Device Events and Files...
Jan 26 15:19:01 openqa systemd[1]: systemd-journal-flush.service: Control process exited, code=exited, status=1/FAILURE
Jan 26 15:19:01 openqa systemd[1]: systemd-journal-flush.service: Failed with result 'exit-code'.
Jan 26 15:19:01 openqa systemd[1]: Stopped Flush Journal to Persistent Storage.
Jan 26 15:19:01 openqa systemd[1]: Stopped Journal Service.
Jan 26 15:19:01 openqa systemd[1]: Starting Journal Service...
Jan 26 15:19:01 openqa systemd[1]: systemd-logind.service: Main process exited, code=killed, status=6/ABRT
Jan 26 15:19:01 openqa systemd[1]: systemd-logind.service: Failed with result 'watchdog'.
Jan 26 15:19:01 openqa systemd[1]: systemd-logind.service: Scheduled restart job, restart counter is at 1.
Jan 26 15:19:01 openqa systemd[1]: Stopped User Login Management.
Jan 26 15:19:01 openqa systemd[1]: Starting Load Kernel Module drm...
Jan 26 15:19:01 openqa systemd[1]: modprobe@drm.service: Succeeded.
Jan 26 15:19:01 openqa systemd[1]: Finished Load Kernel Module drm.
Jan 26 15:19:01 openqa systemd[1]: Starting User Login Management...
Jan 26 15:19:01 openqa systemd[1]: Started Rule-based Manager for Device Events and Files.
Jan 26 15:19:01 openqa systemd[1]: Started User Login Management.
Jan 26 15:19:02 openqa systemd[1]: Started Journal Service.
Jan 26 15:18:53 openqa systemd[1]: systemd-journald.service: Watchdog timeout (limit 3min)!
Jan 26 15:19:02 openqa telegraf[1248]: 2022-01-26T14:18:57Z E! [outputs.influxdb] when writing to [http://openqa-monitor.qa.suse.de:8086]: Post "http://openqa-monitor.qa.suse.de:8086/write?db=telegraf": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Jan 26 15:19:02 openqa telegraf[1248]: 2022-01-26T14:18:57Z W! [agent] [inputs.http] did not complete within its interval
Jan 26 15:19:02 openqa telegraf[1248]: 2022-01-26T14:18:57Z W! [agent] ["outputs.influxdb"] did not complete within its flush interval
Jan 26 15:19:02 openqa telegraf[1248]: 2022-01-26T14:18:57Z E! [agent] Error writing to outputs.influxdb: could not write any address
Jan 26 15:19:02 openqa telegraf[1248]: 2022-01-26T14:18:57Z E! [inputs.http] Error in plugin: [url=https://openqa.suse.de/admin/influxdb/jobs]: Get "https://openqa.suse.de/admin/influxdb/jobs": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Jan 26 15:18:53 openqa chronyd[1243]: Forward time jump detected!
Jan 26 15:18:53 openqa systemd[1]: systemd-journald.service: Killing process 29178 (systemd-journal) with signal SIGABRT.
Jan 26 15:18:53 openqa chronyd[1243]: Can't synchronise: no selectable sources
Jan 26 15:18:57 openqa journalctl[29172]: Failed to connect to /run/systemd/journal/io.systemd.journal: Connection refused
Jan 26 15:18:53 openqa sshd[29059]: rexec line 119: Deprecated option UsePrivilegeSeparation
Jan 26 15:18:59 openqa systemd-udevd[29173]: Network interface NamePolicy= disabled by default.
Jan 26 15:18:53 openqa sshd[29058]: rexec line 119: Deprecated option UsePrivilegeSeparation
Jan 26 15:19:00 openqa systemd-logind[29211]: Watching system buttons on /dev/input/event3 (Power Button)
Jan 26 15:18:53 openqa sshd[29058]: Connection from 10.100.12.155 port 52736 on 10.160.0.207 port 22 rdomain ""
Jan 26 15:19:00 openqa systemd-logind[29211]: Watching system buttons on /dev/input/event0 (AT Translated Set 2 keyboard)
Jan 26 15:18:53 openqa sshd[29059]: Connection from 10.160.0.44 port 33300 on 10.160.0.207 port 22 rdomain ""
Jan 26 15:19:00 openqa systemd-logind[29211]: New seat seat0.
Jan 26 15:18:53 openqa sshd[29059]: error: kex_exchange_identification: Connection closed by remote host
Jan 26 15:19:00 openqa systemd-logind[29211]: New session 10644 of user jpupava.
Jan 26 15:18:53 openqa sshd[29059]: Connection closed by 10.160.0.44 port 33300
Jan 26 15:19:00 openqa systemd-logind[29211]: New session 10630 of user cfconrad.
Jan 26 15:18:54 openqa sshd[29058]: Accepted key RSA SHA256:RExKE8zGUlg0EaeAdtz0wy219J2IZ+xohbnF+u5Bwgc found at /home/jpupava/.ssh/authorized_keys:1
Jan 26 15:19:00 openqa systemd-logind[29211]: New session 10465 of user jlausuch.
Jan 26 15:18:54 openqa sshd[29058]: Postponed publickey for jpupava from 10.100.12.155 port 52736 ssh2 [preauth]
Jan 26 15:19:00 openqa systemd-logind[29211]: New session 10361 of user jlausuch.
Jan 26 15:18:54 openqa sshd[29058]: Accepted key RSA SHA256:RExKE8zGUlg0EaeAdtz0wy219J2IZ+xohbnF+u5Bwgc found at /home/jpupava/.ssh/authorized_keys:1
Jan 26 15:19:02 openqa systemd[1]: Starting Flush Journal to Persistent Storage...
Jan 26 15:18:54 openqa sshd[29058]: Accepted publickey for jpupava from 10.100.12.155 port 52736 ssh2: RSA SHA256:RExKE8zGUlg0EaeAdtz0wy219J2IZ+xohbnF+u5Bwgc
Jan 26 15:19:02 openqa systemd[1]: Finished Flush Journal to Persistent Storage.
Jan 26 15:18:55 openqa nrpe[29139]: Error: (!log_opts) Could not complete SSL handshake with 10.160.0.45: 5
Jan 26 15:18:56 openqa nrpe[29145]: Error: (!log_opts) Could not complete SSL handshake with 10.160.0.44: 5
Jan 26 15:18:56 openqa nrpe[29143]: Error: (!log_opts) Could not complete SSL handshake with 10.160.0.44: 5
Jan 26 15:18:58 openqa sshd[29058]: pam_systemd(sshd:session): Failed to create session: Message recipient disconnected from message bus without replying
Jan 26 15:18:58 openqa cron[29060]: pam_systemd(crond:session): Failed to create session: Message recipient disconnected from message bus without replying
Jan 26 15:18:58 openqa sshd[29058]: pam_unix(sshd:session): session opened for user jpupava by (uid=0)
Jan 26 15:18:58 openqa cron[29060]: pam_unix(crond:session): session opened for user geekotest by (uid=0)
Jan 26 15:18:58 openqa sshd[29058]: User child is on pid 29201
Jan 26 15:18:58 openqa sshd[29201]: Starting session: shell on pts/5 for jpupava from 10.100.12.155 port 52736 id 0
Jan 26 15:19:03 openqa systemd[1]: Started Session 10663 of user geekotest.
Jan 26 15:19:03 openqa cron[29441]: pam_unix(crond:session): session opened for user geekotest by (uid=0)
Jan 26 15:19:08 openqa telegraf[1248]: 2022-01-26T14:19:08Z W! [agent] [inputs.procstat] did not complete within its interval
Jan 26 15:19:09 openqa CRON[29060]: pam_unix(crond:session): session closed for user geekotest
Jan 26 15:19:09 openqa systemd[1]: Started Session 10666 of user geekotest.
Jan 26 15:19:09 openqa systemd[1]: Started Session 10664 of user geekotest.
Jan 26 15:19:09 openqa systemd[1]: Started Session 10665 of user geekotest.
Jan 26 15:19:10 openqa cron[29710]: pam_unix(crond:session): session opened for user geekotest by (uid=0)
Jan 26 15:19:10 openqa cron[29712]: pam_unix(crond:session): session opened for user geekotest by (uid=0)
Jan 26 15:19:10 openqa cron[29711]: pam_unix(crond:session): session opened for user geekotest by (uid=0)
Jan 26 15:19:10 openqa CRON[29710]: pam_unix(crond:session): session closed for user geekotest
Jan 26 15:19:10 openqa systemd[1]: session-10666.scope: Succeeded.
Jan 26 15:19:12 openqa CRON[29441]: pam_unix(crond:session): session closed for user geekotest
Jan 26 15:19:12 openqa systemd[1]: session-10663.scope: Succeeded.
Jan 26 15:19:17 openqa CRON[29712]: pam_unix(crond:session): session closed for user geekotest
Jan 26 15:19:17 openqa systemd[1]: session-10664.scope: Succeeded.
Jan 26 15:19:17 openqa postfix/pickup[29079]: 4B0ED343F2: uid=1001 from=<geekotest>
Jan 26 15:19:17 openqa postfix/cleanup[30192]: 4B0ED343F2: message-id=<20220126141917.4B0ED343F2@localhost>
Jan 26 15:19:17 openqa postfix/qmgr[1688]: 4B0ED343F2: from=<geekotest@localhost>, size=2073, nrcpt=1 (queue active)
Jan 26 15:19:17 openqa postfix/local[30203]: 4B0ED343F2: to=<geekotest@localhost>, orig_to=<geekotest>, relay=local, delay=6.5, delays=6.4/0.03/0/0.02, dsn=2.0.0, status=sent (delivered to mailbox)
Jan 26 15:19:17 openqa postfix/qmgr[1688]: 4B0ED343F2: removed
Jan 26 15:19:17 openqa CRON[29711]: pam_unix(crond:session): session closed for user geekotest
Jan 26 15:19:17 openqa systemd[1]: session-10665.scope: Succeeded.
Jan 26 15:19:17 openqa postfix/pickup[29079]: D982D343F2: uid=1001 from=<geekotest>
Jan 26 15:19:17 openqa postfix/cleanup[30192]: D982D343F2: message-id=<20220126141917.D982D343F2@localhost>
Jan 26 15:19:17 openqa postfix/qmgr[1688]: D982D343F2: from=<geekotest@localhost>, size=1297, nrcpt=1 (queue active)
Jan 26 15:19:17 openqa postfix/local[30203]: D982D343F2: to=<geekotest@localhost>, orig_to=<geekotest>, relay=local, delay=4.7, delays=4.7/0/0/0, dsn=2.0.0, status=sent (delivered to mailbox)
Jan 26 15:19:17 openqa postfix/qmgr[1688]: D982D343F2: removed
Jan 26 15:19:24 openqa sshd[29201]: Received disconnect from 10.100.12.155 port 52736:11: disconnected by user
Jan 26 15:19:24 openqa sshd[29201]: Disconnected from user jpupava 10.100.12.155 port 52736
Jan 26 15:19:24 openqa sshd[29058]: pam_unix(sshd:session): session closed for user jpupava

For simplicity I removed all output from openqa itself but I attach the full log file in case it gives us a hint why postgresql went crazy in memory usage.

Actions #5

Updated by livdywan over 2 years ago

  • Status changed from New to In Progress
  • Assignee set to livdywan
  • This could be due to postgres vaccuming
  • Maybe there was a high amount of jobs being scheduled at once
Actions #6

Updated by openqa_review over 2 years ago

  • Due date set to 2022-02-16

Setting due date based on mean cycle time of SUSE QE Tools

Actions #7

Updated by livdywan over 2 years ago

  • Due date deleted (2022-02-16)
  • Status changed from In Progress to Workable
  • Assignee deleted (livdywan)

Resetting ticket, since I wasn't actually able to just do this quickly on the side. MAye someone who's looking at related things, or just has a minute to spare can jump in.

Actions #8

Updated by livdywan over 2 years ago

  • Subject changed from [Alerting] CPU Load alert to [Alerting] CPU Load alert size:S
  • Description updated (diff)
Actions #9

Updated by jbaier_cz over 2 years ago

There is a correlation between the CPU load and "Disk I/O time for /dev/vda", so the high load could be actually caused by waiting for disk writes.

Actions #10

Updated by okurz over 2 years ago

Yes, that's very likely. A load above something like 20 I think I have only ever seen if there are processes I/O blocked.

Actions #11

Updated by livdywan over 2 years ago

It is my impression that multiple people already took a look, but we still haven't the foggiest what this issue is. And I'm not sure we even see this anymore... I suggest we take a look together in the mob session on Thursday 24 and get debug it together (or if applicable decide we can't anymore).

Actions #12

Updated by livdywan over 2 years ago

  • Status changed from Workable to Rejected
  • Assignee set to livdywan

Possible improvements to our alert handling:
- Extend log retention time
- Job-group specific alerts
- We could cap the jobs scheduled e.g. per isos post, or jobs per group
- Fail if max=1000 jobs are scheduled
- Clarify out to get from "too many jobs" alert to a meaningful conclusion
- Increase the threshold

Actions #13

Updated by okurz over 2 years ago

  • Related to action #107701: [osd] Job detail page fails to load added
Actions

Also available in: Atom PDF