|
2025-01-24T11:24:10.768506+01:00 qa-power8-3 kernel: [ C24] [c00000002114b9e0] [c00000000071770c] prune_dcache_sb+0x6c/0xa0
|
|
2025-01-24T11:24:10.768507+01:00 qa-power8-3 kernel: [ C24] [c00000002114ba30] [c0000000006ec140] super_cache_scan+0x1a0/0x240
|
|
2025-01-24T11:24:10.768509+01:00 qa-power8-3 kernel: [ C24] [c00000002114baa0] [c0000000005aa458] do_shrink_slab+0x168/0x430
|
|
2025-01-24T11:24:10.768510+01:00 qa-power8-3 kernel: [ C24] [c00000002114bb20] [c0000000005ab708] shrink_slab+0x378/0x560
|
|
2025-01-24T11:24:10.768511+01:00 qa-power8-3 kernel: [ C24] [c00000002114bc00] [c0000000005a665c] shrink_node+0x4fc/0x1070
|
|
2025-01-24T11:24:10.768512+01:00 qa-power8-3 kernel: [ C24] [c00000002114bd00] [c0000000005a79f4] balance_pgdat+0x454/0xbd0
|
|
2025-01-24T11:24:10.768514+01:00 qa-power8-3 kernel: [ C24] [c00000002114beb0] [c0000000005a83f0] kswapd+0x280/0x580
|
|
2025-01-24T11:24:10.768515+01:00 qa-power8-3 kernel: [ C24] [c00000002114bf90] [c00000000025dca4] kthread+0x134/0x140
|
|
2025-01-24T11:24:10.768516+01:00 qa-power8-3 kernel: [ C24] [c00000002114bfe0] [c00000000000de58] start_kernel_thread+0x14/0x18
|
|
2025-01-24T11:24:10.768517+01:00 qa-power8-3 kernel: [ C24] Code: 7d0a5378 f9470000 41820008 f8e80008 7c2004ac e9490000 794a07a4 f9490000 4e800020 60420000 7c210b78 7c421378 <e9490000> 714a0001 4082fff0 4bffff94
|
|
2025-01-24T11:24:30.838174+01:00 qa-power8-3 kernel: [ C0] BUG: workqueue lockup - pool cpus=24 node=0 flags=0x0 nice=0 stuck for 41562s!
|
|
2025-01-24T11:24:30.838185+01:00 qa-power8-3 kernel: [ C0] Showing busy workqueues and worker pools:
|
|
2025-01-24T11:24:30.838187+01:00 qa-power8-3 kernel: [ C0] workqueue events: flags=0x0
|
|
2025-01-24T11:24:30.838189+01:00 qa-power8-3 kernel: [ C0] pwq 98: cpus=24 node=0 flags=0x0 nice=0 active=1 refcnt=2
|
|
2025-01-24T11:24:30.838190+01:00 qa-power8-3 kernel: [ C0] pending: drain_local_stock
|
|
2025-01-24T11:24:30.838191+01:00 qa-power8-3 kernel: [ C0] workqueue events_power_efficient: flags=0x80
|
|
2025-01-24T11:24:30.838192+01:00 qa-power8-3 kernel: [ C0] pwq 98: cpus=24 node=0 flags=0x0 nice=0 active=2 refcnt=3
|
|
2025-01-24T11:24:30.838194+01:00 qa-power8-3 kernel: [ C0] pending: neigh_managed_work, neigh_periodic_work
|
|
2025-01-24T11:24:30.838199+01:00 qa-power8-3 kernel: [ C0] workqueue mm_percpu_wq: flags=0x8
|
|
2025-01-24T11:24:30.838201+01:00 qa-power8-3 kernel: [ C0] pwq 98: cpus=24 node=0 flags=0x0 nice=0 active=1 refcnt=2
|
|
2025-01-24T11:24:30.838203+01:00 qa-power8-3 kernel: [ C0] pending: vmstat_update
|
|
2025-01-24T11:24:30.838221+01:00 qa-power8-3 kernel: [ C0] workqueue iou_exit: flags=0x2
|
|
2025-01-24T11:24:30.838223+01:00 qa-power8-3 kernel: [ C0] pwq 642: cpus=0-159 node=0 flags=0x4 nice=0 active=1 refcnt=2
|
|
2025-01-24T11:24:30.838224+01:00 qa-power8-3 kernel: [ C0] in-flight: 148824:io_ring_exit_work
|
|
2025-01-24T11:24:30.838226+01:00 qa-power8-3 kernel: [ C0] pwq 648: cpus=0-159 node=1 flags=0x4 nice=0 active=3 refcnt=4
|
|
2025-01-24T11:24:30.838227+01:00 qa-power8-3 kernel: [ C0] in-flight: 118070:io_ring_exit_work ,127177:io_ring_exit_work ,129527:io_ring_exit_work
|
|
2025-01-24T11:24:30.838229+01:00 qa-power8-3 kernel: [ C0] pwq 649: cpus=0-159 node=1 flags=0x4 nice=0 active=2 refcnt=3
|
|
2025-01-24T11:24:30.838231+01:00 qa-power8-3 kernel: [ C0] in-flight: 130631:io_ring_exit_work ,129460:io_ring_exit_work
|
|
2025-01-24T11:24:30.838233+01:00 qa-power8-3 kernel: [ C0] pwq 650: cpus=0-159 node=1 flags=0x4 nice=0 active=5 refcnt=6
|
|
2025-01-24T11:24:30.838234+01:00 qa-power8-3 kernel: [ C0] in-flight: 127213:io_ring_exit_work ,127522:io_ring_exit_work ,127521:io_ring_exit_work ,147689:io_ring_exit_work ,147685:io_ring_exit_work
|
|
2025-01-24T11:24:30.838236+01:00 qa-power8-3 kernel: [ C0] pwq 651: cpus=0-159 node=16 flags=0x4 nice=0 active=7 refcnt=13
|
|
2025-01-24T11:24:30.838238+01:00 qa-power8-3 kernel: [ C0] in-flight: 153752:io_ring_exit_work ,152496:io_ring_exit_work ,152844:io_ring_exit_work ,150270:io_ring_exit_work ,153906:io_ring_exit_work ,152861:io_ring_exit_work ,152074:io_ring_exit_work
|
|
2025-01-24T11:24:30.838239+01:00 qa-power8-3 kernel: [ C0] inactive: 5*io_ring_exit_work
|
|
2025-01-24T11:24:30.838241+01:00 qa-power8-3 kernel: [ C0] pwq 652: cpus=0-159 node=16 flags=0x4 nice=0 active=1 refcnt=7
|
|
2025-01-24T11:24:30.838243+01:00 qa-power8-3 kernel: [ C0] in-flight: 151346:io_ring_exit_work
|
|
2025-01-24T11:24:30.838244+01:00 qa-power8-3 kernel: [ C0] inactive: 5*io_ring_exit_work
|
|
2025-01-24T11:24:30.838246+01:00 qa-power8-3 kernel: [ C0] pwq 653: cpus=0-159 node=16 flags=0x4 nice=0 active=1 refcnt=7
|
|
2025-01-24T11:24:30.838247+01:00 qa-power8-3 kernel: [ C0] in-flight: 147308:io_ring_exit_work
|
|
2025-01-24T11:24:30.838249+01:00 qa-power8-3 kernel: [ C0] inactive: 5*io_ring_exit_work
|
|
2025-01-24T11:24:30.838250+01:00 qa-power8-3 kernel: [ C0] pwq 654: cpus=0-159 node=16 flags=0x4 nice=0 active=5 refcnt=10
|
|
2025-01-24T11:24:30.838251+01:00 qa-power8-3 kernel: [ C0] in-flight: 148805:io_ring_exit_work ,151928:io_ring_exit_work ,152824:io_ring_exit_work ,152829:io_ring_exit_work ,152827:io_ring_exit_work
|
|
2025-01-24T11:24:30.838253+01:00 qa-power8-3 kernel: [ C0] inactive: 4*io_ring_exit_work
|
|
2025-01-24T11:24:30.838254+01:00 qa-power8-3 kernel: [ C0] pwq 655: cpus=0-159 node=16 flags=0x4 nice=0 active=2 refcnt=5
|
|
2025-01-24T11:24:30.838256+01:00 qa-power8-3 kernel: [ C0] in-flight: 152591:io_ring_exit_work ,151754:io_ring_exit_work
|
|
2025-01-24T11:24:30.838257+01:00 qa-power8-3 kernel: [ C0] inactive: 2*io_ring_exit_work
|
|
2025-01-24T11:24:30.838258+01:00 qa-power8-3 kernel: [ C0] pwq 656: cpus=0-159 node=17 flags=0x4 nice=0 active=5 refcnt=6
|
|
2025-01-24T11:24:30.838259+01:00 qa-power8-3 kernel: [ C0] in-flight: 147303:io_ring_exit_work ,130347:io_ring_exit_work ,130346:io_ring_exit_work ,129985:io_ring_exit_work ,147256:io_ring_exit_work
|
|
2025-01-24T11:24:30.838261+01:00 qa-power8-3 kernel: [ C0] pwq 658: cpus=0-159 node=17 flags=0x4 nice=0 active=1 refcnt=2
|
|
2025-01-24T11:24:30.838262+01:00 qa-power8-3 kernel: [ C0] in-flight: 147299:io_ring_exit_work
|
|
2025-01-24T11:24:30.838263+01:00 qa-power8-3 kernel: [ C0] pwq 659: cpus=0-159 node=17 flags=0x4 nice=0 active=4 refcnt=5
|
|
2025-01-24T11:24:30.838264+01:00 qa-power8-3 kernel: [ C0] in-flight: 127097:io_ring_exit_work ,130719:io_ring_exit_work ,129372:io_ring_exit_work ,147458:io_ring_exit_work
|
|
2025-01-24T11:24:30.838266+01:00 qa-power8-3 kernel: [ C0] pwq 660: cpus=0-159 node=17 flags=0x4 nice=0 active=1 refcnt=2
|
|
2025-01-24T11:24:30.838268+01:00 qa-power8-3 kernel: [ C0] in-flight: 147318:io_ring_exit_work
|
|
2025-01-24T11:24:30.838269+01:00 qa-power8-3 kernel: [ C0] pool 642: cpus=0-159 node=0 flags=0x4 nice=0 hung=0s workers=4 idle: 127904 154042 154043
|
|
2025-01-24T11:24:30.838270+01:00 qa-power8-3 kernel: [ C0] pool 648: cpus=0-159 node=1 flags=0x4 nice=0 hung=0s workers=5 idle: 147825 148763
|
|
2025-01-24T11:24:30.838271+01:00 qa-power8-3 kernel: [ C0] pool 649: cpus=0-159 node=1 flags=0x4 nice=0 hung=0s workers=4 idle: 154165 154242
|
|
2025-01-24T11:24:30.838273+01:00 qa-power8-3 kernel: [ C0] pool 650: cpus=0-159 node=1 flags=0x4 nice=0 hung=0s workers=8 idle: 147859 147693 147954
|
|
2025-01-24T11:24:30.838274+01:00 qa-power8-3 kernel: [ C0] pool 651: cpus=0-159 node=16 flags=0x4 nice=0 hung=0s workers=11 idle: 153982 154920 154915 155388
|
|
2025-01-24T11:24:30.838276+01:00 qa-power8-3 kernel: [ C0] pool 652: cpus=0-159 node=16 flags=0x4 nice=0 hung=0s workers=5 idle: 156273 155967 152855 155726
|
|
2025-01-24T11:24:30.838277+01:00 qa-power8-3 kernel: [ C0] pool 653: cpus=0-159 node=16 flags=0x4 nice=0 hung=0s workers=6 idle: 156259 153489 154759 155714 156257
|
|
2025-01-24T11:24:30.838278+01:00 qa-power8-3 kernel: [ C0] pool 654: cpus=0-159 node=16 flags=0x4 nice=0 hung=0s workers=10 idle: 156385 155190 156377 152834 152926
|
|
2025-01-24T11:24:30.838280+01:00 qa-power8-3 kernel: [ C0] pool 655: cpus=0-159 node=16 flags=0x4 nice=0 hung=0s workers=6 idle: 154694 156401 155265 155642
|
|
2025-01-24T11:24:30.838281+01:00 qa-power8-3 kernel: [ C0] pool 656: cpus=0-159 node=17 flags=0x4 nice=0 hung=0s workers=9 idle: 155382 149795 148819 155419
|
|
2025-01-24T11:24:30.838282+01:00 qa-power8-3 kernel: [ C0] pool 658: cpus=0-159 node=17 flags=0x4 nice=0 hung=0s workers=4 idle: 155872 156175 154798
|
|
2025-01-24T11:24:30.838283+01:00 qa-power8-3 kernel: [ C0] pool 659: cpus=0-159 node=17 flags=0x4 nice=0 hung=0s workers=7 idle: 155547 155771 153308
|
|
2025-01-24T11:24:30.838285+01:00 qa-power8-3 kernel: [ C0] pool 660: cpus=0-159 node=17 flags=0x4 nice=0 hung=0s workers=6 idle: 155079 156421 147520 156018 153416
|
|
2025-01-24T11:24:30.838286+01:00 qa-power8-3 kernel: [ C0] Showing backtraces of running workers in stalled CPU-bound worker pools:
|
|
2025-01-24T11:24:34.768460+01:00 qa-power8-3 kernel: [ C24] watchdog: BUG: soft lockup - CPU#24 stuck for 38712s! [kswapd0:905]
|
|
2025-01-24T11:24:34.768469+01:00 qa-power8-3 kernel: [ C24] CPU#24 Utilization every 4s during lockup:
|
|
2025-01-24T11:24:34.768470+01:00 qa-power8-3 kernel: [ C24] #1: 100% system, 0% softirq, 1% hardirq, 0% idle
|
|
2025-01-24T11:24:34.768472+01:00 qa-power8-3 kernel: [ C24] #2: 101% system, 0% softirq, 0% hardirq, 0% idle
|
|
2025-01-24T11:24:34.768473+01:00 qa-power8-3 kernel: [ C24] #3: 100% system, 0% softirq, 0% hardirq, 0% idle
|
|
2025-01-24T11:24:34.768475+01:00 qa-power8-3 kernel: [ C24] #4: 100% system, 0% softirq, 0% hardirq, 0% idle
|
|
2025-01-24T11:24:34.768476+01:00 qa-power8-3 kernel: [ C24] #5: 101% system, 0% softirq, 0% hardirq, 0% idle
|
|
2025-01-24T11:24:34.768478+01:00 qa-power8-3 kernel: [ C24] Modules linked in: nft_masq nft_chain_nat nf_tables af_packet tun bridge stp llc nfnetlink_cttimeout nfnetlink openvswitch nsh nf_conncount nf_nat rfkill nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 psample binfmt_misc aes_gcm_p10_crypto crypto_simd cryptd xts tg3 ses vmx_crypto crct10dif_vpmsum enclosure libphy scsi_transport_sas ipmi_powernv rtc_opal leds_powernv ipmi_devintf uio_pdrv_genirq ipmi_msghandler led_class ibmpowernv powernv_op_panel i2c_opal uio fuse configfs ip_tables x_tables xhci_pci ipr xhci_hcd sd_mod scsi_dh_emc scsi_dh_rdac scsi_dh_alua sg usbcore scsi_mod usb_common scsi_common btrfs blake2b_generic libcrc32c crc32c_vpmsum xor raid6_pq kvm_hv kvm
|
|
2025-01-24T11:24:34.768500+01:00 qa-power8-3 kernel: [ C24] CPU: 24 UID: 0 PID: 905 Comm: kswapd0 Tainted: G W L 6.13.0-1.g1513344-default #1 openSUSE Tumbleweed (unreleased) 067da1dff4e006e28ea31f9175684ad434e86067
|
|
2025-01-24T11:24:34.768501+01:00 qa-power8-3 kernel: [ C24] Tainted: [W]=WARN, [L]=SOFTLOCKUP
|
|
2025-01-24T11:24:34.768503+01:00 qa-power8-3 kernel: [ C24] Hardware name: 8247-22L POWER8E (raw) 0x4b0201 opal:skiboot-v5.4.12 PowerNV
|
|
2025-01-24T11:24:34.768505+01:00 qa-power8-3 kernel: [ C24] NIP: c0000000007129b8 LR: c000000000715298 CTR: c00800000f1c6240
|
|
2025-01-24T11:24:34.768507+01:00 qa-power8-3 kernel: [ C24] REGS: c00000002114b6b0 TRAP: 0900 Tainted: G W L (6.13.0-1.g1513344-default)
|
|
2025-01-24T11:24:34.768508+01:00 qa-power8-3 kernel: [ C24] MSR: 900000000280b033 <SF,HV,VEC,VSX,EE,FP,ME,IR,DR,RI,LE> CR: 48022284 XER: 00000000
|
|
2025-01-24T11:24:34.768510+01:00 qa-power8-3 kernel: [ C24] CFAR: 0000000000000000 IRQMASK: 0
|
|
2025-01-24T11:24:34.768511+01:00 qa-power8-3 kernel: [ C24] GPR00: c000000000716e94 c00000002114b950 c000000001c99200 c000000538434440
|
|
2025-01-24T11:24:34.768513+01:00 qa-power8-3 kernel: [ C24] GPR04: c00000000296d498 c000000000711c80 0000000000000031 fffffffffffe0000
|
|
2025-01-24T11:24:34.768514+01:00 qa-power8-3 kernel: [ C24] GPR08: c000ffffffffffff c0080000035a0000 0000000000000001 0000000000002200
|
|
2025-01-24T11:24:34.768515+01:00 qa-power8-3 kernel: [ C24] GPR12: 0000000000000000 c0000007fffebc80 c00000000025db78 000000000000063d
|
|
2025-01-24T11:24:34.768517+01:00 qa-power8-3 kernel: [ C24] GPR16: c00000005a8afa80 000000000000001f 0000000000000000 0000000000000000
|
|
2025-01-24T11:24:34.768518+01:00 qa-power8-3 kernel: [ C24] GPR20: c000000002996bc0 000000000001f2f0 000000000001eee4 0000000000000000
|
|
2025-01-24T11:24:34.768520+01:00 qa-power8-3 kernel: [ C24] GPR24: 000000000000f591 00000000000042b6 0000000000000047 0000000000000000
|
|
2025-01-24T11:24:34.768522+01:00 qa-power8-3 kernel: [ C24] GPR28: c00000002114ba08 c0000005384344c0 0000000000000001 c000000538434440
|
|
2025-01-24T11:24:34.768523+01:00 qa-power8-3 kernel: [ C24] NIP [c0000000007129b8] ___d_drop+0x98/0xc0
|
|
2025-01-24T11:24:34.768525+01:00 qa-power8-3 kernel: [ C24] LR [c000000000715298] __dentry_kill+0x88/0x310
|
|
2025-01-24T11:24:34.768526+01:00 qa-power8-3 kernel: [ C24] Call Trace:
|
|
2025-01-24T11:24:34.768527+01:00 qa-power8-3 kernel: [ C24] [c00000002114b950] [c00000002114b990] 0xc00000002114b990 (unreliable)
|
|
2025-01-24T11:24:34.768529+01:00 qa-power8-3 kernel: [ C24] [c00000002114b990] [c000000000716e94] shrink_dentry_list+0x84/0x150
|
|
2025-01-24T11:24:34.768530+01:00 qa-power8-3 kernel: [ C24] [c00000002114b9e0] [c00000000071770c] prune_dcache_sb+0x6c/0xa0
|
|
2025-01-24T11:24:34.768531+01:00 qa-power8-3 kernel: [ C24] [c00000002114ba30] [c0000000006ec140] super_cache_scan+0x1a0/0x240
|
|
2025-01-24T11:24:34.768532+01:00 qa-power8-3 kernel: [ C24] [c00000002114baa0] [c0000000005aa458] do_shrink_slab+0x168/0x430
|
|
2025-01-24T11:24:34.768534+01:00 qa-power8-3 kernel: [ C24] [c00000002114bb20] [c0000000005ab708] shrink_slab+0x378/0x560
|
|
2025-01-24T11:24:34.768535+01:00 qa-power8-3 kernel: [ C24] [c00000002114bc00] [c0000000005a665c] shrink_node+0x4fc/0x1070
|
|
2025-01-24T11:24:34.768536+01:00 qa-power8-3 kernel: [ C24] [c00000002114bd00] [c0000000005a79f4] balance_pgdat+0x454/0xbd0
|
|
2025-01-24T11:24:34.768538+01:00 qa-power8-3 kernel: [ C24] [c00000002114beb0] [c0000000005a83f0] kswapd+0x280/0x580
|
|
2025-01-24T11:24:34.768539+01:00 qa-power8-3 kernel: [ C24] [c00000002114bf90] [c00000000025dca4] kthread+0x134/0x140
|
|
2025-01-24T11:24:34.768540+01:00 qa-power8-3 kernel: [ C24] [c00000002114bfe0] [c00000000000de58] start_kernel_thread+0x14/0x18
|
|
2025-01-24T11:24:34.768541+01:00 qa-power8-3 kernel: [ C24] Code: 7d0a5378 f9470000 41820008 f8e80008 7c2004ac e9490000 794a07a4 f9490000 4e800020 60420000 7c210b78 7c421378 <e9490000> 714a0001 4082fff0 4bffff94
|
|
2025-01-24T11:24:58.768461+01:00 qa-power8-3 kernel: [ C24] watchdog: BUG: soft lockup - CPU#24 stuck for 38734s! [kswapd0:905]
|
|
2025-01-24T11:24:58.768472+01:00 qa-power8-3 kernel: [ C24] CPU#24 Utilization every 4s during lockup:
|
|
2025-01-24T11:24:58.768474+01:00 qa-power8-3 kernel: [ C24] #1: 100% system, 0% softirq, 0% hardirq, 0% idle
|
|
2025-01-24T11:24:58.768475+01:00 qa-power8-3 kernel: [ C24] #2: 100% system, 0% softirq, 0% hardirq, 0% idle
|
|
2025-01-24T11:24:58.768477+01:00 qa-power8-3 kernel: [ C24] #3: 101% system, 0% softirq, 0% hardirq, 0% idle
|
|
2025-01-24T11:24:58.768478+01:00 qa-power8-3 kernel: [ C24] #4: 100% system, 0% softirq, 0% hardirq, 0% idle
|
|
2025-01-24T11:24:58.768479+01:00 qa-power8-3 kernel: [ C24] #5: 100% system, 0% softirq, 0% hardirq, 0% idle
|
|
2025-01-24T11:24:58.768481+01:00 qa-power8-3 kernel: [ C24] Modules linked in: nft_masq nft_chain_nat nf_tables af_packet tun bridge stp llc nfnetlink_cttimeout nfnetlink openvswitch nsh nf_conncount nf_nat rfkill nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 psample binfmt_misc aes_gcm_p10_crypto crypto_simd cryptd xts tg3 ses vmx_crypto crct10dif_vpmsum enclosure libphy scsi_transport_sas ipmi_powernv rtc_opal leds_powernv ipmi_devintf uio_pdrv_genirq ipmi_msghandler led_class ibmpowernv powernv_op_panel i2c_opal uio fuse configfs ip_tables x_tables xhci_pci ipr xhci_hcd sd_mod scsi_dh_emc scsi_dh_rdac scsi_dh_alua sg usbcore scsi_mod usb_common scsi_common btrfs blake2b_generic libcrc32c crc32c_vpmsum xor raid6_pq kvm_hv kvm
|
|
2025-01-24T11:24:58.768486+01:00 qa-power8-3 kernel: [ C24] CPU: 24 UID: 0 PID: 905 Comm: kswapd0 Tainted: G W L 6.13.0-1.g1513344-default #1 openSUSE Tumbleweed (unreleased) 067da1dff4e006e28ea31f9175684ad434e86067
|
|
2025-01-24T11:24:58.768503+01:00 qa-power8-3 kernel: [ C24] Tainted: [W]=WARN, [L]=SOFTLOCKUP
|
|
2025-01-24T11:24:58.768505+01:00 qa-power8-3 kernel: [ C24] Hardware name: 8247-22L POWER8E (raw) 0x4b0201 opal:skiboot-v5.4.12 PowerNV
|
|
2025-01-24T11:24:58.768506+01:00 qa-power8-3 kernel: [ C24] NIP: c0000000007129b4 LR: c000000000715298 CTR: c00800000f1c6240
|
|
2025-01-24T11:24:58.768508+01:00 qa-power8-3 kernel: [ C24] REGS: c00000002114b6b0 TRAP: 0900 Tainted: G W L (6.13.0-1.g1513344-default)
|
|
2025-01-24T11:24:58.768509+01:00 qa-power8-3 kernel: [ C24] MSR: 900000000280b033 <SF,HV,VEC,VSX,EE,FP,ME,IR,DR,RI,LE> CR: 48022284 XER: 00000000
|
|
2025-01-24T11:24:58.768511+01:00 qa-power8-3 kernel: [ C24] CFAR: 0000000000000000 IRQMASK: 0
|
|
2025-01-24T11:24:58.768512+01:00 qa-power8-3 kernel: [ C24] GPR00: c000000000716e94 c00000002114b950 c000000001c99200 c000000538434440
|
|
2025-01-24T11:24:58.768514+01:00 qa-power8-3 kernel: [ C24] GPR04: c00000000296d498 c000000000711c80 0000000000000031 fffffffffffe0000
|
|
2025-01-24T11:24:58.768516+01:00 qa-power8-3 kernel: [ C24] GPR08: c000ffffffffffff c0080000035a0000 0000000000000001 0000000000002200
|
|
2025-01-24T11:24:58.768517+01:00 qa-power8-3 kernel: [ C24] GPR12: 0000000000000000 c0000007fffebc80 c00000000025db78 000000000000063d
|
|
2025-01-24T11:24:58.768518+01:00 qa-power8-3 kernel: [ C24] GPR16: c00000005a8afa80 000000000000001f 0000000000000000 0000000000000000
|
|
2025-01-24T11:24:58.768520+01:00 qa-power8-3 kernel: [ C24] GPR20: c000000002996bc0 000000000001f2f0 000000000001eee4 0000000000000000
|
|
2025-01-24T11:24:58.768521+01:00 qa-power8-3 kernel: [ C24] GPR24: 000000000000f591 00000000000042b6 0000000000000047 0000000000000000
|
|
2025-01-24T11:24:58.768523+01:00 qa-power8-3 kernel: [ C24] GPR28: c00000002114ba08 c0000005384344c0 0000000000000001 c000000538434440
|
|
2025-01-24T11:24:58.768524+01:00 qa-power8-3 kernel: [ C24] NIP [c0000000007129b4] ___d_drop+0x94/0xc0
|
|
2025-01-24T11:24:58.768526+01:00 qa-power8-3 kernel: [ C24] LR [c000000000715298] __dentry_kill+0x88/0x310
|
|
2025-01-24T11:24:58.768527+01:00 qa-power8-3 kernel: [ C24] Call Trace:
|
|
2025-01-24T11:24:58.768529+01:00 qa-power8-3 kernel: [ C24] [c00000002114b950] [c00000002114b990] 0xc00000002114b990 (unreliable)
|
|
2025-01-24T11:24:58.768530+01:00 qa-power8-3 kernel: [ C24] [c00000002114b990] [c000000000716e94] shrink_dentry_list+0x84/0x150
|
|
2025-01-24T11:24:58.768531+01:00 qa-power8-3 kernel: [ C24] [c00000002114b9e0] [c00000000071770c] prune_dcache_sb+0x6c/0xa0
|
|
2025-01-24T11:24:58.768532+01:00 qa-power8-3 kernel: [ C24] [c00000002114ba30] [c0000000006ec140] super_cache_scan+0x1a0/0x240
|
|
2025-01-24T11:24:58.768534+01:00 qa-power8-3 kernel: [ C24] [c00000002114baa0] [c0000000005aa458] do_shrink_slab+0x168/0x430
|
|
2025-01-24T11:24:58.768535+01:00 qa-power8-3 kernel: [ C24] [c00000002114bb20] [c0000000005ab708] shrink_slab+0x378/0x560
|
|
2025-01-24T11:24:58.768537+01:00 qa-power8-3 kernel: [ C24] [c00000002114bc00] [c0000000005a665c] shrink_node+0x4fc/0x1070
|
|
2025-01-24T11:24:58.768538+01:00 qa-power8-3 kernel: [ C24] [c00000002114bd00] [c0000000005a79f4] balance_pgdat+0x454/0xbd0
|
|
2025-01-24T11:24:58.768539+01:00 qa-power8-3 kernel: [ C24] [c00000002114beb0] [c0000000005a83f0] kswapd+0x280/0x580
|
|
2025-01-24T11:24:58.768540+01:00 qa-power8-3 kernel: [ C24] [c00000002114bf90] [c00000000025dca4] kthread+0x134/0x140
|
|
2025-01-24T11:24:58.768542+01:00 qa-power8-3 kernel: [ C24] [c00000002114bfe0] [c00000000000de58] start_kernel_thread+0x14/0x18
|
|
2025-01-24T11:24:58.768543+01:00 qa-power8-3 kernel: [ C24] Code: 794a07e0 7d0a5378 f9470000 41820008 f8e80008 7c2004ac e9490000 794a07a4 f9490000 4e800020 60420000 7c210b78 <7c421378> e9490000 714a0001 4082fff0
|
|
2025-01-24T11:25:00.918168+01:00 qa-power8-3 kernel: [ C0] BUG: workqueue lockup - pool cpus=24 node=0 flags=0x0 nice=0 stuck for 41592s!
|
|
2025-01-24T11:25:00.918178+01:00 qa-power8-3 kernel: [ C0] Showing busy workqueues and worker pools:
|
|
2025-01-24T11:25:00.918179+01:00 qa-power8-3 kernel: [ C0] workqueue events: flags=0x0
|
|
2025-01-24T11:25:00.918181+01:00 qa-power8-3 kernel: [ C0] pwq 98: cpus=24 node=0 flags=0x0 nice=0 active=1 refcnt=2
|
|
2025-01-24T11:25:00.918182+01:00 qa-power8-3 kernel: [ C0] pending: drain_local_stock
|
|
2025-01-24T11:25:00.918183+01:00 qa-power8-3 kernel: [ C0] workqueue events_power_efficient: flags=0x80
|
|
2025-01-24T11:25:00.918185+01:00 qa-power8-3 kernel: [ C0] pwq 98: cpus=24 node=0 flags=0x0 nice=0 active=2 refcnt=3
|
|
2025-01-24T11:25:00.918186+01:00 qa-power8-3 kernel: [ C0] pending: neigh_managed_work, neigh_periodic_work
|
|
2025-01-24T11:25:00.918190+01:00 qa-power8-3 kernel: [ C0] workqueue mm_percpu_wq: flags=0x8
|
|
2025-01-24T11:25:00.918209+01:00 qa-power8-3 kernel: [ C0] pwq 98: cpus=24 node=0 flags=0x0 nice=0 active=1 refcnt=2
|
|
2025-01-24T11:25:00.918211+01:00 qa-power8-3 kernel: [ C0] pending: vmstat_update
|
|
2025-01-24T11:25:00.918213+01:00 qa-power8-3 kernel: [ C0] workqueue iou_exit: flags=0x2
|
|
2025-01-24T11:25:00.918214+01:00 qa-power8-3 kernel: [ C0] pwq 642: cpus=0-159 node=0 flags=0x4 nice=0 active=1 refcnt=2
|
|
2025-01-24T11:25:00.918216+01:00 qa-power8-3 kernel: [ C0] in-flight: 148824:io_ring_exit_work
|
|
2025-01-24T11:25:00.918218+01:00 qa-power8-3 kernel: [ C0] pwq 648: cpus=0-159 node=1 flags=0x4 nice=0 active=3 refcnt=4
|
|
2025-01-24T11:25:00.918219+01:00 qa-power8-3 kernel: [ C0] in-flight: 118070:io_ring_exit_work ,127177:io_ring_exit_work ,129527:io_ring_exit_work
|
|
2025-01-24T11:25:00.918221+01:00 qa-power8-3 kernel: [ C0] pwq 649: cpus=0-159 node=1 flags=0x4 nice=0 active=2 refcnt=3
|
|
2025-01-24T11:25:00.918222+01:00 qa-power8-3 kernel: [ C0] in-flight: 130631:io_ring_exit_work ,129460:io_ring_exit_work
|
|
2025-01-24T11:25:00.918224+01:00 qa-power8-3 kernel: [ C0] pwq 650: cpus=0-159 node=1 flags=0x4 nice=0 active=5 refcnt=6
|
|
2025-01-24T11:25:00.918226+01:00 qa-power8-3 kernel: [ C0] in-flight: 127213:io_ring_exit_work ,127522:io_ring_exit_work ,127521:io_ring_exit_work ,147689:io_ring_exit_work ,147685:io_ring_exit_work
|
|
2025-01-24T11:25:00.918227+01:00 qa-power8-3 kernel: [ C0] pwq 651: cpus=0-159 node=16 flags=0x4 nice=0 active=7 refcnt=13
|
|
2025-01-24T11:25:00.918229+01:00 qa-power8-3 kernel: [ C0] in-flight: 153752:io_ring_exit_work ,152496:io_ring_exit_work ,152844:io_ring_exit_work ,150270:io_ring_exit_work ,153906:io_ring_exit_work ,152861:io_ring_exit_work ,152074:io_ring_exit_work
|
|
2025-01-24T11:25:00.918231+01:00 qa-power8-3 kernel: [ C0] inactive: 5*io_ring_exit_work
|
|
2025-01-24T11:25:00.918232+01:00 qa-power8-3 kernel: [ C0] pwq 652: cpus=0-159 node=16 flags=0x4 nice=0 active=1 refcnt=7
|
|
2025-01-24T11:25:00.918233+01:00 qa-power8-3 kernel: [ C0] in-flight: 151346:io_ring_exit_work
|
|
2025-01-24T11:25:00.918234+01:00 qa-power8-3 kernel: [ C0] inactive: 5*io_ring_exit_work
|
|
2025-01-24T11:25:00.918236+01:00 qa-power8-3 kernel: [ C0] pwq 653: cpus=0-159 node=16 flags=0x4 nice=0 active=1 refcnt=7
|
|
2025-01-24T11:25:00.918237+01:00 qa-power8-3 kernel: [ C0] in-flight: 147308:io_ring_exit_work
|
|
2025-01-24T11:25:00.918238+01:00 qa-power8-3 kernel: [ C0] inactive: 5*io_ring_exit_work
|
|
2025-01-24T11:25:00.918239+01:00 qa-power8-3 kernel: [ C0] pwq 654: cpus=0-159 node=16 flags=0x4 nice=0 active=5 refcnt=10
|
|
2025-01-24T11:25:00.918241+01:00 qa-power8-3 kernel: [ C0] in-flight: 148805:io_ring_exit_work ,151928:io_ring_exit_work ,152824:io_ring_exit_work ,152829:io_ring_exit_work ,152827:io_ring_exit_work
|
|
2025-01-24T11:25:00.918242+01:00 qa-power8-3 kernel: [ C0] inactive: 4*io_ring_exit_work
|
|
2025-01-24T11:25:00.918243+01:00 qa-power8-3 kernel: [ C0] pwq 655: cpus=0-159 node=16 flags=0x4 nice=0 active=2 refcnt=5
|
|
2025-01-24T11:25:00.918245+01:00 qa-power8-3 kernel: [ C0] in-flight: 152591:io_ring_exit_work ,151754:io_ring_exit_work
|
|
2025-01-24T11:25:00.918246+01:00 qa-power8-3 kernel: [ C0] inactive: 2*io_ring_exit_work
|
|
2025-01-24T11:25:00.918247+01:00 qa-power8-3 kernel: [ C0] pwq 656: cpus=0-159 node=17 flags=0x4 nice=0 active=5 refcnt=6
|
|
2025-01-24T11:25:00.918248+01:00 qa-power8-3 kernel: [ C0] in-flight: 147303:io_ring_exit_work ,130347:io_ring_exit_work ,130346:io_ring_exit_work ,129985:io_ring_exit_work ,147256:io_ring_exit_work
|
|
2025-01-24T11:25:00.918250+01:00 qa-power8-3 kernel: [ C0] pwq 658: cpus=0-159 node=17 flags=0x4 nice=0 active=1 refcnt=2
|
|
2025-01-24T11:25:00.918251+01:00 qa-power8-3 kernel: [ C0] in-flight: 147299:io_ring_exit_work
|
|
2025-01-24T11:25:00.918252+01:00 qa-power8-3 kernel: [ C0] pwq 659: cpus=0-159 node=17 flags=0x4 nice=0 active=4 refcnt=5
|
|
2025-01-24T11:25:00.918253+01:00 qa-power8-3 kernel: [ C0] in-flight: 127097:io_ring_exit_work ,130719:io_ring_exit_work ,129372:io_ring_exit_work ,147458:io_ring_exit_work
|
|
2025-01-24T11:25:00.918255+01:00 qa-power8-3 kernel: [ C0] pwq 660: cpus=0-159 node=17 flags=0x4 nice=0 active=1 refcnt=2
|
|
2025-01-24T11:25:00.918256+01:00 qa-power8-3 kernel: [ C0] in-flight: 147318:io_ring_exit_work
|
|
2025-01-24T11:25:00.918257+01:00 qa-power8-3 kernel: [ C0] pool 642: cpus=0-159 node=0 flags=0x4 nice=0 hung=0s workers=4 idle: 127904 154042 154043
|
|
2025-01-24T11:25:00.918259+01:00 qa-power8-3 kernel: [ C0] pool 648: cpus=0-159 node=1 flags=0x4 nice=0 hung=0s workers=5 idle: 147825 148763
|
|
2025-01-24T11:25:00.918260+01:00 qa-power8-3 kernel: [ C0] pool 649: cpus=0-159 node=1 flags=0x4 nice=0 hung=0s workers=4 idle: 154165 154242
|
|
2025-01-24T11:25:00.918261+01:00 qa-power8-3 kernel: [ C0] pool 650: cpus=0-159 node=1 flags=0x4 nice=0 hung=0s workers=8 idle: 147859 147693 147954
|
|
2025-01-24T11:25:00.918262+01:00 qa-power8-3 kernel: [ C0] pool 651: cpus=0-159 node=16 flags=0x4 nice=0 hung=0s workers=11 idle: 153982 154920 154915 155388
|
|
2025-01-24T11:25:00.918264+01:00 qa-power8-3 kernel: [ C0] pool 652: cpus=0-159 node=16 flags=0x4 nice=0 hung=0s workers=5 idle: 156273 155967 152855 155726
|
|
2025-01-24T11:25:00.918265+01:00 qa-power8-3 kernel: [ C0] pool 653: cpus=0-159 node=16 flags=0x4 nice=0 hung=0s workers=6 idle: 153489 156259 154759 155714 156257
|
|
2025-01-24T11:25:00.918266+01:00 qa-power8-3 kernel: [ C0] pool 654: cpus=0-159 node=16 flags=0x4 nice=0 hung=0s workers=10 idle: 155190 156385 156377 152834 152926
|
|
2025-01-24T11:25:00.918267+01:00 qa-power8-3 kernel: [ C0] pool 655: cpus=0-159 node=16 flags=0x4 nice=0 hung=0s workers=6 idle: 154694 156401 155265 155642
|
|
2025-01-24T11:25:00.918269+01:00 qa-power8-3 kernel: [ C0] pool 656: cpus=0-159 node=17 flags=0x4 nice=0 hung=0s workers=9 idle: 149795 155382 148819 155419
|
|
2025-01-24T11:25:00.918270+01:00 qa-power8-3 kernel: [ C0] pool 658: cpus=0-159 node=17 flags=0x4 nice=0 hung=0s workers=4 idle: 155872 156175 154798
|
|
2025-01-24T11:25:00.918271+01:00 qa-power8-3 kernel: [ C0] pool 659: cpus=0-159 node=17 flags=0x4 nice=0 hung=0s workers=7 idle: 155771 155547 153308
|
|
2025-01-24T11:25:00.918272+01:00 qa-power8-3 kernel: [ C0] pool 660: cpus=0-159 node=17 flags=0x4 nice=0 hung=0s workers=6 idle: 155079 156421 147520 156018 153416
|
|
2025-01-24T11:25:00.918274+01:00 qa-power8-3 kernel: [ C0] Showing backtraces of running workers in stalled CPU-bound worker pools:
|
|
2025-01-24T11:25:22.768463+01:00 qa-power8-3 kernel: [ C24] watchdog: BUG: soft lockup - CPU#24 stuck for 38757s! [kswapd0:905]
|
|
2025-01-24T11:25:22.768475+01:00 qa-power8-3 kernel: [ C24] CPU#24 Utilization every 4s during lockup:
|
|
2025-01-24T11:25:22.768477+01:00 qa-power8-3 kernel: [ C24] #1: 100% system, 0% softirq, 1% hardirq, 0% idle
|
|
2025-01-24T11:25:22.768479+01:00 qa-power8-3 kernel: [ C24] #2: 100% system, 0% softirq, 0% hardirq, 0% idle
|
|
2025-01-24T11:25:22.768480+01:00 qa-power8-3 kernel: [ C24] #3: 101% system, 0% softirq, 0% hardirq, 0% idle
|
|
2025-01-24T11:25:22.768482+01:00 qa-power8-3 kernel: [ C24] #4: 100% system, 0% softirq, 0% hardirq, 0% idle
|
|
2025-01-24T11:25:22.768483+01:00 qa-power8-3 kernel: [ C24] #5: 100% system, 0% softirq, 0% hardirq, 0% idle
|
|
2025-01-24T11:25:22.768484+01:00 qa-power8-3 kernel: [ C24] Modules linked in: nft_masq nft_chain_nat nf_tables af_packet tun bridge stp llc nfnetlink_cttimeout nfnetlink openvswitch nsh nf_conncount nf_nat rfkill nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 psample binfmt_misc aes_gcm_p10_crypto crypto_simd cryptd xts tg3 ses vmx_crypto crct10dif_vpmsum enclosure libphy scsi_transport_sas ipmi_powernv rtc_opal leds_powernv ipmi_devintf uio_pdrv_genirq ipmi_msghandler led_class ibmpowernv powernv_op_panel i2c_opal uio fuse configfs ip_tables x_tables xhci_pci ipr xhci_hcd sd_mod scsi_dh_emc scsi_dh_rdac scsi_dh_alua sg usbcore scsi_mod usb_common scsi_common btrfs blake2b_generic libcrc32c crc32c_vpmsum xor raid6_pq kvm_hv kvm
|
|
2025-01-24T11:25:22.768503+01:00 qa-power8-3 kernel: [ C24] CPU: 24 UID: 0 PID: 905 Comm: kswapd0 Tainted: G W L 6.13.0-1.g1513344-default #1 openSUSE Tumbleweed (unreleased) 067da1dff4e006e28ea31f9175684ad434e86067
|
|
2025-01-24T11:25:22.768505+01:00 qa-power8-3 kernel: [ C24] Tainted: [W]=WARN, [L]=SOFTLOCKUP
|
|
2025-01-24T11:25:22.768507+01:00 qa-power8-3 kernel: [ C24] Hardware name: 8247-22L POWER8E (raw) 0x4b0201 opal:skiboot-v5.4.12 PowerNV
|
|
2025-01-24T11:25:22.768508+01:00 qa-power8-3 kernel: [ C24] NIP: c0000000007129b0 LR: c000000000715298 CTR: c00800000f1c6240
|
|
2025-01-24T11:25:22.768510+01:00 qa-power8-3 kernel: [ C24] REGS: c00000002114b6b0 TRAP: 0900 Tainted: G W L (6.13.0-1.g1513344-default)
|
|
2025-01-24T11:25:22.768512+01:00 qa-power8-3 kernel: [ C24] MSR: 900000000280b033 <SF,HV,VEC,VSX,EE,FP,ME,IR,DR,RI,LE> CR: 48022284 XER: 00000000
|
|
2025-01-24T11:25:22.768513+01:00 qa-power8-3 kernel: [ C24] CFAR: 0000000000000000 IRQMASK: 0
|
|
2025-01-24T11:25:22.768514+01:00 qa-power8-3 kernel: [ C24] GPR00: c000000000716e94 c00000002114b950 c000000001c99200 c000000538434440
|
|
2025-01-24T11:25:22.768516+01:00 qa-power8-3 kernel: [ C24] GPR04: c00000000296d498 c000000000711c80 0000000000000031 fffffffffffe0000
|
|
2025-01-24T11:25:22.768517+01:00 qa-power8-3 kernel: [ C24] GPR08: c000ffffffffffff c0080000035a0000 0000000000000001 0000000000002200
|
|
2025-01-24T11:25:22.768518+01:00 qa-power8-3 kernel: [ C24] GPR12: 0000000000000000 c0000007fffebc80 c00000000025db78 000000000000063d
|
|
2025-01-24T11:25:22.768520+01:00 qa-power8-3 kernel: [ C24] GPR16: c00000005a8afa80 000000000000001f 0000000000000000 0000000000000000
|
|
2025-01-24T11:25:22.768521+01:00 qa-power8-3 kernel: [ C24] GPR20: c000000002996bc0 000000000001f2f0 000000000001eee4 0000000000000000
|
|
2025-01-24T11:25:22.768523+01:00 qa-power8-3 kernel: [ C24] GPR24: 000000000000f591 00000000000042b6 0000000000000047 0000000000000000
|
|
2025-01-24T11:25:22.768524+01:00 qa-power8-3 kernel: [ C24] GPR28: c00000002114ba08 c0000005384344c0 0000000000000001 c000000538434440
|
|
2025-01-24T11:25:22.768526+01:00 qa-power8-3 kernel: [ C24] NIP [c0000000007129b0] ___d_drop+0x90/0xc0
|
|
2025-01-24T11:25:22.768527+01:00 qa-power8-3 kernel: [ C24] LR [c000000000715298] __dentry_kill+0x88/0x310
|
|
2025-01-24T11:25:22.768528+01:00 qa-power8-3 kernel: [ C24] Call Trace:
|
|
2025-01-24T11:25:22.768530+01:00 qa-power8-3 kernel: [ C24] [c00000002114b950] [c00000002114b990] 0xc00000002114b990 (unreliable)
|
|
2025-01-24T11:25:22.768531+01:00 qa-power8-3 kernel: [ C24] [c00000002114b990] [c000000000716e94] shrink_dentry_list+0x84/0x150
|
|
2025-01-24T11:25:22.768532+01:00 qa-power8-3 kernel: [ C24] [c00000002114b9e0] [c00000000071770c] prune_dcache_sb+0x6c/0xa0
|
|
2025-01-24T11:25:22.768533+01:00 qa-power8-3 kernel: [ C24] [c00000002114ba30] [c0000000006ec140] super_cache_scan+0x1a0/0x240
|
|
2025-01-24T11:25:22.768534+01:00 qa-power8-3 kernel: [ C24] [c00000002114baa0] [c0000000005aa458] do_shrink_slab+0x168/0x430
|
|
2025-01-24T11:25:22.768536+01:00 qa-power8-3 kernel: [ C24] [c00000002114bb20] [c0000000005ab708] shrink_slab+0x378/0x560
|
|
2025-01-24T11:25:22.768537+01:00 qa-power8-3 kernel: [ C24] [c00000002114bc00] [c0000000005a665c] shrink_node+0x4fc/0x1070
|
|
2025-01-24T11:25:22.768539+01:00 qa-power8-3 kernel: [ C24] [c00000002114bd00] [c0000000005a79f4] balance_pgdat+0x454/0xbd0
|
|
2025-01-24T11:25:22.768540+01:00 qa-power8-3 kernel: [ C24] [c00000002114beb0] [c0000000005a83f0] kswapd+0x280/0x580
|
|
2025-01-24T11:25:22.768541+01:00 qa-power8-3 kernel: [ C24] [c00000002114bf90] [c00000000025dca4] kthread+0x134/0x140
|
|
2025-01-24T11:25:22.768542+01:00 qa-power8-3 kernel: [ C24] [c00000002114bfe0] [c00000000000de58] start_kernel_thread+0x14/0x18
|
|
2025-01-24T11:25:22.768544+01:00 qa-power8-3 kernel: [ C24] Code: 2c280000 794a07e0 7d0a5378 f9470000 41820008 f8e80008 7c2004ac e9490000 794a07a4 f9490000 4e800020 60420000 <7c210b78> 7c421378 e9490000 714a0001
|
|
2025-01-24T11:25:30.998170+01:00 qa-power8-3 kernel: [ C0] BUG: workqueue lockup - pool cpus=24 node=0 flags=0x0 nice=0 stuck for 41622s!
|
|
2025-01-24T11:25:30.998180+01:00 qa-power8-3 kernel: [ C0] Showing busy workqueues and worker pools:
|
|
2025-01-24T11:25:30.998182+01:00 qa-power8-3 kernel: [ C0] workqueue events: flags=0x0
|
|
2025-01-24T11:25:30.998184+01:00 qa-power8-3 kernel: [ C0] pwq 98: cpus=24 node=0 flags=0x0 nice=0 active=1 refcnt=2
|
|
2025-01-24T11:25:30.998185+01:00 qa-power8-3 kernel: [ C0] pending: drain_local_stock
|
|
2025-01-24T11:25:30.998186+01:00 qa-power8-3 kernel: [ C0] workqueue events_power_efficient: flags=0x80
|
|
2025-01-24T11:25:30.998188+01:00 qa-power8-3 kernel: [ C0] pwq 98: cpus=24 node=0 flags=0x0 nice=0 active=2 refcnt=3
|
|
2025-01-24T11:25:30.998189+01:00 qa-power8-3 kernel: [ C0] pending: neigh_managed_work, neigh_periodic_work
|
|
2025-01-24T11:25:30.998194+01:00 qa-power8-3 kernel: [ C0] workqueue mm_percpu_wq: flags=0x8
|
|
2025-01-24T11:25:30.998212+01:00 qa-power8-3 kernel: [ C0] pwq 98: cpus=24 node=0 flags=0x0 nice=0 active=1 refcnt=2
|
|
2025-01-24T11:25:30.998214+01:00 qa-power8-3 kernel: [ C0] pending: vmstat_update
|
|
2025-01-24T11:25:30.998216+01:00 qa-power8-3 kernel: [ C0] workqueue iou_exit: flags=0x2
|
|
2025-01-24T11:25:30.998217+01:00 qa-power8-3 kernel: [ C0] pwq 642: cpus=0-159 node=0 flags=0x4 nice=0 active=1 refcnt=2
|
|
2025-01-24T11:25:30.998219+01:00 qa-power8-3 kernel: [ C0] in-flight: 148824:io_ring_exit_work
|
|
2025-01-24T11:25:30.998221+01:00 qa-power8-3 kernel: [ C0] pwq 648: cpus=0-159 node=1 flags=0x4 nice=0 active=3 refcnt=4
|
|
2025-01-24T11:25:30.998222+01:00 qa-power8-3 kernel: [ C0] in-flight: 118070:io_ring_exit_work ,127177:io_ring_exit_work ,129527:io_ring_exit_work
|
|
2025-01-24T11:25:30.998224+01:00 qa-power8-3 kernel: [ C0] pwq 649: cpus=0-159 node=1 flags=0x4 nice=0 active=2 refcnt=3
|
|
2025-01-24T11:25:30.998226+01:00 qa-power8-3 kernel: [ C0] in-flight: 130631:io_ring_exit_work ,129460:io_ring_exit_work
|
|
2025-01-24T11:25:30.998227+01:00 qa-power8-3 kernel: [ C0] pwq 650: cpus=0-159 node=1 flags=0x4 nice=0 active=5 refcnt=6
|
|
2025-01-24T11:25:30.998229+01:00 qa-power8-3 kernel: [ C0] in-flight: 127213:io_ring_exit_work ,127522:io_ring_exit_work ,127521:io_ring_exit_work ,147689:io_ring_exit_work ,147685:io_ring_exit_work
|
|
2025-01-24T11:25:30.998230+01:00 qa-power8-3 kernel: [ C0] pwq 651: cpus=0-159 node=16 flags=0x4 nice=0 active=7 refcnt=13
|
|
2025-01-24T11:25:30.998231+01:00 qa-power8-3 kernel: [ C0] in-flight: 153752:io_ring_exit_work ,152496:io_ring_exit_work ,152844:io_ring_exit_work ,150270:io_ring_exit_work ,153906:io_ring_exit_work ,152861:io_ring_exit_work ,152074:io_ring_exit_work
|
|
2025-01-24T11:25:30.998233+01:00 qa-power8-3 kernel: [ C0] inactive: 5*io_ring_exit_work
|
|
2025-01-24T11:25:30.998235+01:00 qa-power8-3 kernel: [ C0] pwq 652: cpus=0-159 node=16 flags=0x4 nice=0 active=1 refcnt=7
|
|
2025-01-24T11:25:30.998236+01:00 qa-power8-3 kernel: [ C0] in-flight: 151346:io_ring_exit_work
|
|
2025-01-24T11:25:30.998238+01:00 qa-power8-3 kernel: [ C0] inactive: 5*io_ring_exit_work
|
|
2025-01-24T11:25:30.998239+01:00 qa-power8-3 kernel: [ C0] pwq 653: cpus=0-159 node=16 flags=0x4 nice=0 active=1 refcnt=7
|
|
2025-01-24T11:25:30.998240+01:00 qa-power8-3 kernel: [ C0] in-flight: 147308:io_ring_exit_work
|
|
2025-01-24T11:25:30.998242+01:00 qa-power8-3 kernel: [ C0] inactive: 5*io_ring_exit_work
|
|
2025-01-24T11:25:30.998243+01:00 qa-power8-3 kernel: [ C0] pwq 654: cpus=0-159 node=16 flags=0x4 nice=0 active=5 refcnt=10
|
|
2025-01-24T11:25:30.998244+01:00 qa-power8-3 kernel: [ C0] in-flight: 148805:io_ring_exit_work ,151928:io_ring_exit_work ,152824:io_ring_exit_work ,152829:io_ring_exit_work ,152827:io_ring_exit_work
|
|
2025-01-24T11:25:30.998246+01:00 qa-power8-3 kernel: [ C0] inactive: 4*io_ring_exit_work
|
|
2025-01-24T11:25:30.998248+01:00 qa-power8-3 kernel: [ C0] pwq 655: cpus=0-159 node=16 flags=0x4 nice=0 active=2 refcnt=5
|
|
2025-01-24T11:25:30.998249+01:00 qa-power8-3 kernel: [ C0] in-flight: 152591:io_ring_exit_work ,151754:io_ring_exit_work
|
|
2025-01-24T11:25:30.998250+01:00 qa-power8-3 kernel: [ C0] inactive: 2*io_ring_exit_work
|
|
2025-01-24T11:25:30.998251+01:00 qa-power8-3 kernel: [ C0] pwq 656: cpus=0-159 node=17 flags=0x4 nice=0 active=5 refcnt=6
|
|
2025-01-24T11:25:30.998253+01:00 qa-power8-3 kernel: [ C0] in-flight: 147303:io_ring_exit_work ,130347:io_ring_exit_work ,130346:io_ring_exit_work ,129985:io_ring_exit_work ,147256:io_ring_exit_work
|
|
2025-01-24T11:25:30.998254+01:00 qa-power8-3 kernel: [ C0] pwq 658: cpus=0-159 node=17 flags=0x4 nice=0 active=1 refcnt=2
|
|
2025-01-24T11:25:30.998255+01:00 qa-power8-3 kernel: [ C0] in-flight: 147299:io_ring_exit_work
|
|
2025-01-24T11:25:30.998256+01:00 qa-power8-3 kernel: [ C0] pwq 659: cpus=0-159 node=17 flags=0x4 nice=0 active=4 refcnt=5
|
|
2025-01-24T11:25:30.998257+01:00 qa-power8-3 kernel: [ C0] in-flight: 127097:io_ring_exit_work ,130719:io_ring_exit_work ,129372:io_ring_exit_work ,147458:io_ring_exit_work
|
|
2025-01-24T11:25:30.998259+01:00 qa-power8-3 kernel: [ C0] pwq 660: cpus=0-159 node=17 flags=0x4 nice=0 active=1 refcnt=2
|
|
2025-01-24T11:25:30.998260+01:00 qa-power8-3 kernel: [ C0] in-flight: 147318:io_ring_exit_work
|
|
2025-01-24T11:25:30.998262+01:00 qa-power8-3 kernel: [ C0] pool 642: cpus=0-159 node=0 flags=0x4 nice=0 hung=0s workers=4 idle: 127904 154042 154043
|
|
2025-01-24T11:25:30.998263+01:00 qa-power8-3 kernel: [ C0] pool 648: cpus=0-159 node=1 flags=0x4 nice=0 hung=0s workers=5 idle: 147825 148763
|
|
2025-01-24T11:25:30.998264+01:00 qa-power8-3 kernel: [ C0] pool 649: cpus=0-159 node=1 flags=0x4 nice=0 hung=0s workers=4 idle: 154165 154242
|
|
2025-01-24T11:25:30.998265+01:00 qa-power8-3 kernel: [ C0] pool 650: cpus=0-159 node=1 flags=0x4 nice=0 hung=0s workers=8 idle: 147859 147693 147954
|
|
2025-01-24T11:25:30.998267+01:00 qa-power8-3 kernel: [ C0] pool 651: cpus=0-159 node=16 flags=0x4 nice=0 hung=0s workers=11 idle: 153982 154920 154915 155388
|
|
2025-01-24T11:25:30.998268+01:00 qa-power8-3 kernel: [ C0] pool 652: cpus=0-159 node=16 flags=0x4 nice=0 hung=0s workers=5 idle: 156273 155967 152855 155726
|
|
2025-01-24T11:25:30.998269+01:00 qa-power8-3 kernel: [ C0] pool 653: cpus=0-159 node=16 flags=0x4 nice=0 hung=0s workers=6 idle: 153489 156259 154759 155714 156257
|
|
2025-01-24T11:25:30.998271+01:00 qa-power8-3 kernel: [ C0] pool 654: cpus=0-159 node=16 flags=0x4 nice=0 hung=0s workers=10 idle: 155190 156385 156377 152834 152926
|
|
2025-01-24T11:25:30.998272+01:00 qa-power8-3 kernel: [ C0] pool 655: cpus=0-159 node=16 flags=0x4 nice=0 hung=0s workers=6 idle: 154694 156401 155265 155642
|
|
2025-01-24T11:25:30.998273+01:00 qa-power8-3 kernel: [ C0] pool 656: cpus=0-159 node=17 flags=0x4 nice=0 hung=0s workers=9 idle: 149795 155382 148819 155419
|
|
2025-01-24T11:25:30.998274+01:00 qa-power8-3 kernel: [ C0] pool 658: cpus=0-159 node=17 flags=0x4 nice=0 hung=0s workers=5 idle: 156532 156175 154798 155872
|
|
2025-01-24T11:25:30.998276+01:00 qa-power8-3 kernel: [ C0] pool 659: cpus=0-159 node=17 flags=0x4 nice=0 hung=0s workers=7 idle: 155771 155547 153308
|
|
2025-01-24T11:25:30.998277+01:00 qa-power8-3 kernel: [ C0] pool 660: cpus=0-159 node=17 flags=0x4 nice=0 hung=0s workers=6 idle: 155079 156421 147520 156018 153416
|
|
2025-01-24T11:25:30.998278+01:00 qa-power8-3 kernel: [ C0] Showing backtraces of running workers in stalled CPU-bound worker pools:
|
|
2025-01-24T11:25:46.768461+01:00 qa-power8-3 kernel: [ C24] watchdog: BUG: soft lockup - CPU#24 stuck for 38779s! [kswapd0:905]
|
|
2025-01-24T11:25:46.768471+01:00 qa-power8-3 kernel: [ C24] CPU#24 Utilization every 4s during lockup:
|
|
2025-01-24T11:25:46.768473+01:00 qa-power8-3 kernel: [ C24] #1: 100% system, 0% softirq, 0% hardirq, 0% idle
|
|
2025-01-24T11:25:46.768475+01:00 qa-power8-3 kernel: [ C24] #2: 100% system, 1% softirq, 0% hardirq, 0% idle
|
|
2025-01-24T11:25:46.768476+01:00 qa-power8-3 kernel: [ C24] #3: 101% system, 0% softirq, 0% hardirq, 0% idle
|
|
2025-01-24T11:25:46.768478+01:00 qa-power8-3 kernel: [ C24] #4: 100% system, 0% softirq, 0% hardirq, 0% idle
|
|
2025-01-24T11:25:46.768479+01:00 qa-power8-3 kernel: [ C24] #5: 100% system, 0% softirq, 0% hardirq, 0% idle
|
|
2025-01-24T11:25:46.768480+01:00 qa-power8-3 kernel: [ C24] Modules linked in: nft_masq nft_chain_nat nf_tables af_packet tun bridge stp llc nfnetlink_cttimeout nfnetlink openvswitch nsh nf_conncount nf_nat rfkill nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 psample binfmt_misc aes_gcm_p10_crypto crypto_simd cryptd xts tg3 ses vmx_crypto crct10dif_vpmsum enclosure libphy scsi_transport_sas ipmi_powernv rtc_opal leds_powernv ipmi_devintf uio_pdrv_genirq ipmi_msghandler led_class ibmpowernv powernv_op_panel i2c_opal uio fuse configfs ip_tables x_tables xhci_pci ipr xhci_hcd sd_mod scsi_dh_emc scsi_dh_rdac scsi_dh_alua sg usbcore scsi_mod usb_common scsi_common btrfs blake2b_generic libcrc32c crc32c_vpmsum xor raid6_pq kvm_hv kvm
|
|
2025-01-24T11:25:46.768503+01:00 qa-power8-3 kernel: [ C24] CPU: 24 UID: 0 PID: 905 Comm: kswapd0 Tainted: G W L 6.13.0-1.g1513344-default #1 openSUSE Tumbleweed (unreleased) 067da1dff4e006e28ea31f9175684ad434e86067
|
|
2025-01-24T11:25:46.768505+01:00 qa-power8-3 kernel: [ C24] Tainted: [W]=WARN, [L]=SOFTLOCKUP
|
|
2025-01-24T11:25:46.768506+01:00 qa-power8-3 kernel: [ C24] Hardware name: 8247-22L POWER8E (raw) 0x4b0201 opal:skiboot-v5.4.12 PowerNV
|
|
2025-01-24T11:25:46.768508+01:00 qa-power8-3 kernel: [ C24] NIP: c0000000007129b4 LR: c000000000715298 CTR: c00800000f1c6240
|
|
2025-01-24T11:25:46.768510+01:00 qa-power8-3 kernel: [ C24] REGS: c00000002114b6b0 TRAP: 0900 Tainted: G W L (6.13.0-1.g1513344-default)
|
|
2025-01-24T11:25:46.768512+01:00 qa-power8-3 kernel: [ C24] MSR: 900000000280b033 <SF,HV,VEC,VSX,EE,FP,ME,IR,DR,RI,LE> CR: 48022284 XER: 00000000
|
|
2025-01-24T11:25:46.768513+01:00 qa-power8-3 kernel: [ C24] CFAR: 0000000000000000 IRQMASK: 0
|
|
2025-01-24T11:25:46.768515+01:00 qa-power8-3 kernel: [ C24] GPR00: c000000000716e94 c00000002114b950 c000000001c99200 c000000538434440
|
|
2025-01-24T11:25:46.768516+01:00 qa-power8-3 kernel: [ C24] GPR04: c00000000296d498 c000000000711c80 0000000000000031 fffffffffffe0000
|
|
2025-01-24T11:25:46.768517+01:00 qa-power8-3 kernel: [ C24] GPR08: c000ffffffffffff c0080000035a0000 0000000000000001 0000000000002200
|
|
2025-01-24T11:25:46.768519+01:00 qa-power8-3 kernel: [ C24] GPR12: 0000000000000000 c0000007fffebc80 c00000000025db78 000000000000063d
|
|
2025-01-24T11:25:46.768520+01:00 qa-power8-3 kernel: [ C24] GPR16: c00000005a8afa80 000000000000001f 0000000000000000 0000000000000000
|
|
2025-01-24T11:25:46.768522+01:00 qa-power8-3 kernel: [ C24] GPR20: c000000002996bc0 000000000001f2f0 000000000001eee4 0000000000000000
|
|
2025-01-24T11:25:46.768523+01:00 qa-power8-3 kernel: [ C24] GPR24: 000000000000f591 00000000000042b6 0000000000000047 0000000000000000
|
|
2025-01-24T11:25:46.768525+01:00 qa-power8-3 kernel: [ C24] GPR28: c00000002114ba08 c0000005384344c0 0000000000000001 c000000538434440
|
|
2025-01-24T11:25:46.768527+01:00 qa-power8-3 kernel: [ C24] NIP [c0000000007129b4] ___d_drop+0x94/0xc0
|
|
2025-01-24T11:25:46.768528+01:00 qa-power8-3 kernel: [ C24] LR [c000000000715298] __dentry_kill+0x88/0x310
|
|
2025-01-24T11:25:46.768529+01:00 qa-power8-3 kernel: [ C24] Call Trace:
|
|
2025-01-24T11:25:46.768531+01:00 qa-power8-3 kernel: [ C24] [c00000002114b950] [c00000002114b990] 0xc00000002114b990 (unreliable)
|
|
2025-01-24T11:25:46.768532+01:00 qa-power8-3 kernel: [ C24] [c00000002114b990] [c000000000716e94] shrink_dentry_list+0x84/0x150
|
|
2025-01-24T11:25:46.768533+01:00 qa-power8-3 kernel: [ C24] [c00000002114b9e0] [c00000000071770c] prune_dcache_sb+0x6c/0xa0
|
|
2025-01-24T11:25:46.768534+01:00 qa-power8-3 kernel: [ C24] [c00000002114ba30] [c0000000006ec140] super_cache_scan+0x1a0/0x240
|
|
2025-01-24T11:25:46.768536+01:00 qa-power8-3 kernel: [ C24] [c00000002114baa0] [c0000000005aa458] do_shrink_slab+0x168/0x430
|
|
2025-01-24T11:25:46.768537+01:00 qa-power8-3 kernel: [ C24] [c00000002114bb20] [c0000000005ab708] shrink_slab+0x378/0x560
|
|
2025-01-24T11:25:46.768539+01:00 qa-power8-3 kernel: [ C24] [c00000002114bc00] [c0000000005a665c] shrink_node+0x4fc/0x1070
|
|
2025-01-24T11:25:46.768540+01:00 qa-power8-3 kernel: [ C24] [c00000002114bd00] [c0000000005a79f4] balance_pgdat+0x454/0xbd0
|
|
2025-01-24T11:25:46.768541+01:00 qa-power8-3 kernel: [ C24] [c00000002114beb0] [c0000000005a83f0] kswapd+0x280/0x580
|
|
2025-01-24T11:25:46.768543+01:00 qa-power8-3 kernel: [ C24] [c00000002114bf90] [c00000000025dca4] kthread+0x134/0x140
|
|
2025-01-24T11:25:46.768544+01:00 qa-power8-3 kernel: [ C24] [c00000002114bfe0] [c00000000000de58] start_kernel_thread+0x14/0x18
|
|
2025-01-24T11:25:46.768545+01:00 qa-power8-3 kernel: [ C24] Code: 794a07e0 7d0a5378 f9470000 41820008 f8e80008 7c2004ac e9490000 794a07a4 f9490000 4e800020 60420000 7c210b78 <7c421378> e9490000 714a0001 4082fff0
|
|
2025-01-24T11:25:57.098384+01:00 qa-power8-3 kernel: [ C24] rcu: INFO: rcu_sched self-detected stall on CPU
|
|
2025-01-24T11:25:57.098396+01:00 qa-power8-3 kernel: [ C24] rcu: 24-....: (4164906 ticks this GP) idle=f1d4/1/0x4000000000000002 softirq=885737/885737 fqs=1768709
|
|
2025-01-24T11:25:57.098399+01:00 qa-power8-3 kernel: [ C24] rcu: hardirqs softirqs csw/system
|
|
2025-01-24T11:25:57.098401+01:00 qa-power8-3 kernel: [ C24] rcu: number: 275079 895629 0
|
|
2025-01-24T11:25:57.098402+01:00 qa-power8-3 kernel: [ C24] rcu: cputime: 14990 1288 41602771 ==> 41619050(ms)
|
|
2025-01-24T11:25:57.098404+01:00 qa-power8-3 kernel: [ C24] rcu: (t=4164907 jiffies g=1890501 q=1816167 ncpus=20)
|
|
2025-01-24T11:25:57.098406+01:00 qa-power8-3 kernel: [ C24] CPU: 24 UID: 0 PID: 905 Comm: kswapd0 Tainted: G W L 6.13.0-1.g1513344-default #1 openSUSE Tumbleweed (unreleased) 067da1dff4e006e28ea31f9175684ad434e86067
|
|
2025-01-24T11:25:57.098423+01:00 qa-power8-3 kernel: [ C24] Tainted: [W]=WARN, [L]=SOFTLOCKUP
|
|
2025-01-24T11:25:57.098425+01:00 qa-power8-3 kernel: [ C24] Hardware name: 8247-22L POWER8E (raw) 0x4b0201 opal:skiboot-v5.4.12 PowerNV
|
|
2025-01-24T11:25:57.098426+01:00 qa-power8-3 kernel: [ C24] NIP: c0000000007129b8 LR: c000000000715298 CTR: c00800000f1c6240
|
|
2025-01-24T11:25:57.098428+01:00 qa-power8-3 kernel: [ C24] REGS: c00000002114b6b0 TRAP: 0900 Tainted: G W L (6.13.0-1.g1513344-default)
|
|
2025-01-24T11:25:57.098429+01:00 qa-power8-3 kernel: [ C24] MSR: 900000000280b033 <SF,HV,VEC,VSX,EE,FP,ME,IR,DR,RI,LE> CR: 48022284 XER: 00000000
|
|
2025-01-24T11:25:57.098431+01:00 qa-power8-3 kernel: [ C24] CFAR: 0000000000000000 IRQMASK: 0
|
|
2025-01-24T11:25:57.098432+01:00 qa-power8-3 kernel: [ C24] GPR00: c000000000716e94 c00000002114b950 c000000001c99200 c000000538434440
|
|
2025-01-24T11:25:57.098434+01:00 qa-power8-3 kernel: [ C24] GPR04: c00000000296d498 c000000000711c80 0000000000000031 fffffffffffe0000
|
|
2025-01-24T11:25:57.098436+01:00 qa-power8-3 kernel: [ C24] GPR08: c000ffffffffffff c0080000035a0000 0000000000000001 0000000000002200
|
|
2025-01-24T11:25:57.098437+01:00 qa-power8-3 kernel: [ C24] GPR12: 0000000000000000 c0000007fffebc80 c00000000025db78 000000000000063d
|
|
2025-01-24T11:25:57.098439+01:00 qa-power8-3 kernel: [ C24] GPR16: c00000005a8afa80 000000000000001f 0000000000000000 0000000000000000
|
|
2025-01-24T11:25:57.098440+01:00 qa-power8-3 kernel: [ C24] GPR20: c000000002996bc0 000000000001f2f0 000000000001eee4 0000000000000000
|
|
2025-01-24T11:25:57.098442+01:00 qa-power8-3 kernel: [ C24] GPR24: 000000000000f591 00000000000042b6 0000000000000047 0000000000000000
|
|
2025-01-24T11:25:57.098443+01:00 qa-power8-3 kernel: [ C24] GPR28: c00000002114ba08 c0000005384344c0 0000000000000001 c000000538434440
|
|
2025-01-24T11:25:57.098445+01:00 qa-power8-3 kernel: [ C24] NIP [c0000000007129b8] ___d_drop+0x98/0xc0
|
|
2025-01-24T11:25:57.098446+01:00 qa-power8-3 kernel: [ C24] LR [c000000000715298] __dentry_kill+0x88/0x310
|
|
2025-01-24T11:25:57.098448+01:00 qa-power8-3 kernel: [ C24] Call Trace:
|
|
2025-01-24T11:25:57.098450+01:00 qa-power8-3 kernel: [ C24] [c00000002114b950] [c00000002114b990] 0xc00000002114b990 (unreliable)
|
|
2025-01-24T11:25:57.098451+01:00 qa-power8-3 kernel: [ C24] [c00000002114b990] [c000000000716e94] shrink_dentry_list+0x84/0x150
|
|
2025-01-24T11:25:57.098453+01:00 qa-power8-3 kernel: [ C24] [c00000002114b9e0] [c00000000071770c] prune_dcache_sb+0x6c/0xa0
|
|
2025-01-24T11:25:57.098454+01:00 qa-power8-3 kernel: [ C24] [c00000002114ba30] [c0000000006ec140] super_cache_scan+0x1a0/0x240
|
|
2025-01-24T11:25:57.098456+01:00 qa-power8-3 kernel: [ C24] [c00000002114baa0] [c0000000005aa458] do_shrink_slab+0x168/0x430
|
|
2025-01-24T11:25:57.098458+01:00 qa-power8-3 kernel: [ C24] [c00000002114bb20] [c0000000005ab708] shrink_slab+0x378/0x560
|
|
2025-01-24T11:25:57.098460+01:00 qa-power8-3 kernel: [ C24] [c00000002114bc00] [c0000000005a665c] shrink_node+0x4fc/0x1070
|
|
2025-01-24T11:25:57.098461+01:00 qa-power8-3 kernel: [ C24] [c00000002114bd00] [c0000000005a79f4] balance_pgdat+0x454/0xbd0
|
|
2025-01-24T11:25:57.098463+01:00 qa-power8-3 kernel: [ C24] [c00000002114beb0] [c0000000005a83f0] kswapd+0x280/0x580
|
|
2025-01-24T11:25:57.098465+01:00 qa-power8-3 kernel: [ C24] [c00000002114bf90] [c00000000025dca4] kthread+0x134/0x140
|
|
2025-01-24T11:25:57.098467+01:00 qa-power8-3 kernel: [ C24] [c00000002114bfe0] [c00000000000de58] start_kernel_thread+0x14/0x18
|
|
2025-01-24T11:25:57.098468+01:00 qa-power8-3 kernel: [ C24] Code: 7d0a5378 f9470000 41820008 f8e80008 7c2004ac e9490000 794a07a4 f9490000 4e800020 60420000 7c210b78 7c421378 <e9490000> 714a0001 4082fff0 4bffff94
|
|
2025-01-24T11:26:20.303092+01:00 qa-power8-3 worker[4380]: [warn] A QEMU instance using the current pool directory is still running (PID: 125502) - checking again for web UI 'https://openqa.opensuse.org' in 119.50 s
|
|
2025-01-24T11:26:22.768469+01:00 qa-power8-3 kernel: [ C24] watchdog: BUG: soft lockup - CPU#24 stuck for 38813s! [kswapd0:905]
|
|
2025-01-24T11:26:22.768481+01:00 qa-power8-3 kernel: [ C24] CPU#24 Utilization every 4s during lockup:
|
|
2025-01-24T11:26:22.768482+01:00 qa-power8-3 kernel: [ C24] #1: 100% system, 0% softirq, 0% hardirq, 0% idle
|
|
2025-01-24T11:26:22.768484+01:00 qa-power8-3 kernel: [ C24] #2: 100% system, 0% softirq, 0% hardirq, 0% idle
|
|
2025-01-24T11:26:22.768485+01:00 qa-power8-3 kernel: [ C24] #3: 101% system, 0% softirq, 0% hardirq, 0% idle
|
|
2025-01-24T11:26:22.768486+01:00 qa-power8-3 kernel: [ C24] #4: 100% system, 0% softirq, 0% hardirq, 0% idle
|
|
2025-01-24T11:26:22.768488+01:00 qa-power8-3 kernel: [ C24] #5: 100% system, 0% softirq, 0% hardirq, 0% idle
|
|
2025-01-24T11:26:22.768489+01:00 qa-power8-3 kernel: [ C24] Modules linked in: nft_masq nft_chain_nat nf_tables af_packet tun bridge stp llc nfnetlink_cttimeout nfnetlink openvswitch nsh nf_conncount nf_nat rfkill nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 psample binfmt_misc aes_gcm_p10_crypto crypto_simd cryptd xts tg3 ses vmx_crypto crct10dif_vpmsum enclosure libphy scsi_transport_sas ipmi_powernv rtc_opal leds_powernv ipmi_devintf uio_pdrv_genirq ipmi_msghandler led_class ibmpowernv powernv_op_panel i2c_opal uio fuse configfs ip_tables x_tables xhci_pci ipr xhci_hcd sd_mod scsi_dh_emc scsi_dh_rdac scsi_dh_alua sg usbcore scsi_mod usb_common scsi_common btrfs blake2b_generic libcrc32c crc32c_vpmsum xor raid6_pq kvm_hv kvm
|
|
2025-01-24T11:26:22.768509+01:00 qa-power8-3 kernel: [ C24] CPU: 24 UID: 0 PID: 905 Comm: kswapd0 Tainted: G W L 6.13.0-1.g1513344-default #1 openSUSE Tumbleweed (unreleased) 067da1dff4e006e28ea31f9175684ad434e86067
|
|
2025-01-24T11:26:22.768511+01:00 qa-power8-3 kernel: [ C24] Tainted: [W]=WARN, [L]=SOFTLOCKUP
|
|
2025-01-24T11:26:22.768512+01:00 qa-power8-3 kernel: [ C24] Hardware name: 8247-22L POWER8E (raw) 0x4b0201 opal:skiboot-v5.4.12 PowerNV
|
|
2025-01-24T11:26:22.768513+01:00 qa-power8-3 kernel: [ C24] NIP: c0000000007129b8 LR: c000000000715298 CTR: c00800000f1c6240
|
|
2025-01-24T11:26:22.768515+01:00 qa-power8-3 kernel: [ C24] REGS: c00000002114b6b0 TRAP: 0900 Tainted: G W L (6.13.0-1.g1513344-default)
|
|
2025-01-24T11:26:22.768517+01:00 qa-power8-3 kernel: [ C24] MSR: 900000000280b033 <SF,HV,VEC,VSX,EE,FP,ME,IR,DR,RI,LE> CR: 48022284 XER: 00000000
|
|
2025-01-24T11:26:22.768518+01:00 qa-power8-3 kernel: [ C24] CFAR: 0000000000000000 IRQMASK: 0
|
|
2025-01-24T11:26:22.768520+01:00 qa-power8-3 kernel: [ C24] GPR00: c000000000716e94 c00000002114b950 c000000001c99200 c000000538434440
|
|
2025-01-24T11:26:22.768522+01:00 qa-power8-3 kernel: [ C24] GPR04: c00000000296d498 c000000000711c80 0000000000000031 fffffffffffe0000
|
|
2025-01-24T11:26:22.768524+01:00 qa-power8-3 kernel: [ C24] GPR08: c000ffffffffffff c0080000035a0000 0000000000000001 0000000000002200
|
|
2025-01-24T11:26:22.768525+01:00 qa-power8-3 kernel: [ C24] GPR12: 0000000000000000 c0000007fffebc80 c00000000025db78 000000000000063d
|
|
2025-01-24T11:26:22.768527+01:00 qa-power8-3 kernel: [ C24] GPR16: c00000005a8afa80 000000000000001f 0000000000000000 0000000000000000
|
|
2025-01-24T11:26:22.768528+01:00 qa-power8-3 kernel: [ C24] GPR20: c000000002996bc0 000000000001f2f0 000000000001eee4 0000000000000000
|
|
2025-01-24T11:26:22.768530+01:00 qa-power8-3 kernel: [ C24] GPR24: 000000000000f591 00000000000042b6 0000000000000047 0000000000000000
|
|
2025-01-24T11:26:22.768532+01:00 qa-power8-3 kernel: [ C24] GPR28: c00000002114ba08 c0000005384344c0 0000000000000001 c000000538434440
|
|
2025-01-24T11:26:22.768533+01:00 qa-power8-3 kernel: [ C24] NIP [c0000000007129b8] ___d_drop+0x98/0xc0
|
|
2025-01-24T11:26:22.768535+01:00 qa-power8-3 kernel: [ C24] LR [c000000000715298] __dentry_kill+0x88/0x310
|
|
2025-01-24T11:26:22.768536+01:00 qa-power8-3 kernel: [ C24] Call Trace:
|
|
2025-01-24T11:26:22.768537+01:00 qa-power8-3 kernel: [ C24] [c00000002114b950] [c00000002114b990] 0xc00000002114b990 (unreliable)
|
|
2025-01-24T11:26:22.768538+01:00 qa-power8-3 kernel: [ C24] [c00000002114b990] [c000000000716e94] shrink_dentry_list+0x84/0x150
|
|
2025-01-24T11:26:22.768540+01:00 qa-power8-3 kernel: [ C24] [c00000002114b9e0] [c00000000071770c] prune_dcache_sb+0x6c/0xa0
|
|
2025-01-24T11:26:22.768541+01:00 qa-power8-3 kernel: [ C24] [c00000002114ba30] [c0000000006ec140] super_cache_scan+0x1a0/0x240
|
|
2025-01-24T11:26:22.768542+01:00 qa-power8-3 kernel: [ C24] [c00000002114baa0] [c0000000005aa458] do_shrink_slab+0x168/0x430
|
|
2025-01-24T11:26:22.768543+01:00 qa-power8-3 kernel: [ C24] [c00000002114bb20] [c0000000005ab708] shrink_slab+0x378/0x560
|
|
2025-01-24T11:26:22.768545+01:00 qa-power8-3 kernel: [ C24] [c00000002114bc00] [c0000000005a665c] shrink_node+0x4fc/0x1070
|
|
2025-01-24T11:26:22.768546+01:00 qa-power8-3 kernel: [ C24] [c00000002114bd00] [c0000000005a79f4] balance_pgdat+0x454/0xbd0
|
|
2025-01-24T11:26:22.768547+01:00 qa-power8-3 kernel: [ C24] [c00000002114beb0] [c0000000005a83f0] kswapd+0x280/0x580
|
|
2025-01-24T11:26:22.768548+01:00 qa-power8-3 kernel: [ C24] [c00000002114bf90] [c00000000025dca4] kthread+0x134/0x140
|
|
2025-01-24T11:26:22.768550+01:00 qa-power8-3 kernel: [ C24] [c00000002114bfe0] [c00000000000de58] start_kernel_thread+0x14/0x18
|
|
2025-01-24T11:26:22.768551+01:00 qa-power8-3 kernel: [ C24] Code: 7d0a5378 f9470000 41820008 f8e80008 7c2004ac e9490000 794a07a4 f9490000 4e800020 60420000 7c210b78 7c421378 <e9490000> 714a0001 4082fff0 4bffff94
|
|
2025-01-24T11:26:25.202498+01:00 qa-power8-3 worker[4378]: [warn] A QEMU instance using the current pool directory is still running (PID: 124692) - checking again for web UI 'https://openqa.opensuse.org' in 234.66 s
|
|
2025-01-24T11:26:31.148164+01:00 qa-power8-3 kernel: [ C0] BUG: workqueue lockup - pool cpus=24 node=0 flags=0x0 nice=0 stuck for 41682s!
|
|
2025-01-24T11:26:31.148170+01:00 qa-power8-3 kernel: [ C0] Showing busy workqueues and worker pools:
|
|
2025-01-24T11:26:31.148172+01:00 qa-power8-3 kernel: [ C0] workqueue events: flags=0x0
|
|
2025-01-24T11:26:31.148173+01:00 qa-power8-3 kernel: [ C0] pwq 98: cpus=24 node=0 flags=0x0 nice=0 active=1 refcnt=2
|
|
2025-01-24T11:26:31.148174+01:00 qa-power8-3 kernel: [ C0] pending: drain_local_stock
|
|
2025-01-24T11:26:31.148185+01:00 qa-power8-3 kernel: [ C0] workqueue events_power_efficient: flags=0x80
|
|
2025-01-24T11:26:31.148187+01:00 qa-power8-3 kernel: [ C0] pwq 98: cpus=24 node=0 flags=0x0 nice=0 active=2 refcnt=3
|
|
2025-01-24T11:26:31.148188+01:00 qa-power8-3 kernel: [ C0] pending: neigh_managed_work, neigh_periodic_work
|
|
2025-01-24T11:26:31.148190+01:00 qa-power8-3 kernel: [ C0] workqueue mm_percpu_wq: flags=0x8
|
|
2025-01-24T11:26:31.148191+01:00 qa-power8-3 kernel: [ C0] pwq 98: cpus=24 node=0 flags=0x0 nice=0 active=1 refcnt=2
|
|
2025-01-24T11:26:31.148192+01:00 qa-power8-3 kernel: [ C0] pending: vmstat_update
|
|
2025-01-24T11:26:31.148194+01:00 qa-power8-3 kernel: [ C0] workqueue iou_exit: flags=0x2
|
|
2025-01-24T11:26:31.148195+01:00 qa-power8-3 kernel: [ C0] pwq 642: cpus=0-159 node=0 flags=0x4 nice=0 active=1 refcnt=2
|
|
2025-01-24T11:26:31.148196+01:00 qa-power8-3 kernel: [ C0] in-flight: 148824:io_ring_exit_work
|
|
2025-01-24T11:26:31.148198+01:00 qa-power8-3 kernel: [ C0] pwq 648: cpus=0-159 node=1 flags=0x4 nice=0 active=3 refcnt=4
|
|
2025-01-24T11:26:31.148199+01:00 qa-power8-3 kernel: [ C0] in-flight: 118070:io_ring_exit_work ,127177:io_ring_exit_work ,129527:io_ring_exit_work
|
|
2025-01-24T11:26:31.148200+01:00 qa-power8-3 kernel: [ C0] pwq 649: cpus=0-159 node=1 flags=0x4 nice=0 active=2 refcnt=3
|
|
2025-01-24T11:26:31.148202+01:00 qa-power8-3 kernel: [ C0] in-flight: 130631:io_ring_exit_work ,129460:io_ring_exit_work
|
|
2025-01-24T11:26:31.148203+01:00 qa-power8-3 kernel: [ C0] pwq 650: cpus=0-159 node=1 flags=0x4 nice=0 active=5 refcnt=6
|
|
2025-01-24T11:26:31.148204+01:00 qa-power8-3 kernel: [ C0] in-flight: 127213:io_ring_exit_work ,127522:io_ring_exit_work ,127521:io_ring_exit_work ,147689:io_ring_exit_work ,147685:io_ring_exit_work
|
|
2025-01-24T11:26:31.148206+01:00 qa-power8-3 kernel: [ C0] pwq 651: cpus=0-159 node=16 flags=0x4 nice=0 active=7 refcnt=13
|
|
2025-01-24T11:26:31.148207+01:00 qa-power8-3 kernel: [ C0] in-flight: 153752:io_ring_exit_work ,152496:io_ring_exit_work ,152844:io_ring_exit_work ,150270:io_ring_exit_work ,153906:io_ring_exit_work ,152861:io_ring_exit_work ,152074:io_ring_exit_work
|
|
2025-01-24T11:26:31.148208+01:00 qa-power8-3 kernel: [ C0] inactive: 5*io_ring_exit_work
|
|
2025-01-24T11:26:31.148209+01:00 qa-power8-3 kernel: [ C0] pwq 652: cpus=0-159 node=16 flags=0x4 nice=0 active=1 refcnt=7
|
|
2025-01-24T11:26:31.148211+01:00 qa-power8-3 kernel: [ C0] in-flight: 151346:io_ring_exit_work
|
|
2025-01-24T11:26:31.148212+01:00 qa-power8-3 kernel: [ C0] inactive: 5*io_ring_exit_work
|
|
2025-01-24T11:26:31.148213+01:00 qa-power8-3 kernel: [ C0] pwq 653: cpus=0-159 node=16 flags=0x4 nice=0 active=1 refcnt=7
|
|
2025-01-24T11:26:31.148214+01:00 qa-power8-3 kernel: [ C0] in-flight: 147308:io_ring_exit_work
|
|
2025-01-24T11:26:31.148216+01:00 qa-power8-3 kernel: [ C0] inactive: 5*io_ring_exit_work
|
|
2025-01-24T11:26:31.148217+01:00 qa-power8-3 kernel: [ C0] pwq 654: cpus=0-159 node=16 flags=0x4 nice=0 active=5 refcnt=10
|
|
2025-01-24T11:26:31.148218+01:00 qa-power8-3 kernel: [ C0] in-flight: 148805:io_ring_exit_work ,151928:io_ring_exit_work ,152824:io_ring_exit_work ,152829:io_ring_exit_work ,152827:io_ring_exit_work
|
|
2025-01-24T11:26:31.148220+01:00 qa-power8-3 kernel: [ C0] inactive: 4*io_ring_exit_work
|
|
2025-01-24T11:26:31.148221+01:00 qa-power8-3 kernel: [ C0] pwq 655: cpus=0-159 node=16 flags=0x4 nice=0 active=2 refcnt=5
|
|
2025-01-24T11:26:31.148222+01:00 qa-power8-3 kernel: [ C0] in-flight: 152591:io_ring_exit_work ,151754:io_ring_exit_work
|
|
2025-01-24T11:26:31.148224+01:00 qa-power8-3 kernel: [ C0] inactive: 2*io_ring_exit_work
|
|
2025-01-24T11:26:31.148225+01:00 qa-power8-3 kernel: [ C0] pwq 656: cpus=0-159 node=17 flags=0x4 nice=0 active=5 refcnt=6
|
|
2025-01-24T11:26:31.148226+01:00 qa-power8-3 kernel: [ C0] in-flight: 147303:io_ring_exit_work ,130347:io_ring_exit_work ,130346:io_ring_exit_work ,129985:io_ring_exit_work ,147256:io_ring_exit_work
|
|
2025-01-24T11:26:31.148227+01:00 qa-power8-3 kernel: [ C0] pwq 658: cpus=0-159 node=17 flags=0x4 nice=0 active=1 refcnt=2
|
|
2025-01-24T11:26:31.148229+01:00 qa-power8-3 kernel: [ C0] in-flight: 147299:io_ring_exit_work
|
|
2025-01-24T11:26:31.148230+01:00 qa-power8-3 kernel: [ C0] pwq 659: cpus=0-159 node=17 flags=0x4 nice=0 active=4 refcnt=5
|
|
2025-01-24T11:26:31.148231+01:00 qa-power8-3 kernel: [ C0] in-flight: 127097:io_ring_exit_work ,130719:io_ring_exit_work ,129372:io_ring_exit_work ,147458:io_ring_exit_work
|
|
2025-01-24T11:26:31.148232+01:00 qa-power8-3 kernel: [ C0] pwq 660: cpus=0-159 node=17 flags=0x4 nice=0 active=1 refcnt=2
|
|
2025-01-24T11:26:31.148234+01:00 qa-power8-3 kernel: [ C0] in-flight: 147318:io_ring_exit_work
|
|
2025-01-24T11:26:31.148236+01:00 qa-power8-3 kernel: [ C0] pool 642: cpus=0-159 node=0 flags=0x4 nice=0 hung=0s workers=4 idle: 127904 154042 154043
|
|
2025-01-24T11:26:31.148237+01:00 qa-power8-3 kernel: [ C0] pool 648: cpus=0-159 node=1 flags=0x4 nice=0 hung=0s workers=5 idle: 147825 148763
|
|
2025-01-24T11:26:31.148239+01:00 qa-power8-3 kernel: [ C0] pool 649: cpus=0-159 node=1 flags=0x4 nice=0 hung=0s workers=4 idle: 154165 154242
|
|
2025-01-24T11:26:31.148240+01:00 qa-power8-3 kernel: [ C0] pool 650: cpus=0-159 node=1 flags=0x4 nice=0 hung=0s workers=8 idle: 147859 147693 147954
|
|
2025-01-24T11:26:31.148241+01:00 qa-power8-3 kernel: [ C0] pool 651: cpus=0-159 node=16 flags=0x4 nice=0 hung=0s workers=11 idle: 153982 154920 154915 155388
|
|
2025-01-24T11:26:31.148243+01:00 qa-power8-3 kernel: [ C0] pool 652: cpus=0-159 node=16 flags=0x4 nice=0 hung=0s workers=4 idle: 156273 155967 152855
|
|
2025-01-24T11:26:31.148244+01:00 qa-power8-3 kernel: [ C0] pool 653: cpus=0-159 node=16 flags=0x4 nice=0 hung=0s workers=6 idle: 153489 156259 154759 155714 156257
|
|
2025-01-24T11:26:31.148245+01:00 qa-power8-3 kernel: [ C0] pool 654: cpus=0-159 node=16 flags=0x4 nice=0 hung=0s workers=9 idle: 155190 156385 156377 152834
|
|
2025-01-24T11:26:31.148247+01:00 qa-power8-3 kernel: [ C0] pool 655: cpus=0-159 node=16 flags=0x4 nice=0 hung=0s workers=4 idle: 154694 156401
|
|
2025-01-24T11:26:31.148248+01:00 qa-power8-3 kernel: [ C0] pool 656: cpus=0-159 node=17 flags=0x4 nice=0 hung=0s workers=9 idle: 149795 155382 148819 155419
|
|
2025-01-24T11:26:31.148250+01:00 qa-power8-3 kernel: [ C0] pool 658: cpus=0-159 node=17 flags=0x4 nice=0 hung=0s workers=5 idle: 156532 156175 154798 155872
|
|
2025-01-24T11:26:31.148252+01:00 qa-power8-3 kernel: [ C0] pool 659: cpus=0-159 node=17 flags=0x4 nice=0 hung=0s workers=7 idle: 155547 155771 153308
|
|
2025-01-24T11:26:31.148253+01:00 qa-power8-3 kernel: [ C0] pool 660: cpus=0-159 node=17 flags=0x4 nice=0 hung=0s workers=6 idle: 155079 156421 147520 156018 153416
|
|
2025-01-24T11:26:31.148255+01:00 qa-power8-3 kernel: [ C0] Showing backtraces of running workers in stalled CPU-bound worker pools:
|
|
2025-01-24T11:26:46.768464+01:00 qa-power8-3 kernel: [ C24] watchdog: BUG: soft lockup - CPU#24 stuck for 38835s! [kswapd0:905]
|
|
2025-01-24T11:26:46.768475+01:00 qa-power8-3 kernel: [ C24] CPU#24 Utilization every 4s during lockup:
|
|
2025-01-24T11:26:46.768476+01:00 qa-power8-3 kernel: [ C24] #1: 100% system, 0% softirq, 0% hardirq, 0% idle
|
|
2025-01-24T11:26:46.768478+01:00 qa-power8-3 kernel: [ C24] #2: 100% system, 0% softirq, 0% hardirq, 0% idle
|
|
2025-01-24T11:26:46.768479+01:00 qa-power8-3 kernel: [ C24] #3: 101% system, 0% softirq, 0% hardirq, 0% idle
|
|
2025-01-24T11:26:46.768480+01:00 qa-power8-3 kernel: [ C24] #4: 100% system, 0% softirq, 0% hardirq, 0% idle
|
|
2025-01-24T11:26:46.768482+01:00 qa-power8-3 kernel: [ C24] #5: 100% system, 0% softirq, 1% hardirq, 0% idle
|
|
2025-01-24T11:26:46.768483+01:00 qa-power8-3 kernel: [ C24] Modules linked in: nft_masq nft_chain_nat nf_tables af_packet tun bridge stp llc nfnetlink_cttimeout nfnetlink openvswitch nsh nf_conncount nf_nat rfkill nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 psample binfmt_misc aes_gcm_p10_crypto crypto_simd cryptd xts tg3 ses vmx_crypto crct10dif_vpmsum enclosure libphy scsi_transport_sas ipmi_powernv rtc_opal leds_powernv ipmi_devintf uio_pdrv_genirq ipmi_msghandler led_class ibmpowernv powernv_op_panel i2c_opal uio fuse configfs ip_tables x_tables xhci_pci ipr xhci_hcd sd_mod scsi_dh_emc scsi_dh_rdac scsi_dh_alua sg usbcore scsi_mod usb_common scsi_common btrfs blake2b_generic libcrc32c crc32c_vpmsum xor raid6_pq kvm_hv kvm
|
|
2025-01-24T11:26:46.768506+01:00 qa-power8-3 kernel: [ C24] CPU: 24 UID: 0 PID: 905 Comm: kswapd0 Tainted: G W L 6.13.0-1.g1513344-default #1 openSUSE Tumbleweed (unreleased) 067da1dff4e006e28ea31f9175684ad434e86067
|
|
2025-01-24T11:26:46.768508+01:00 qa-power8-3 kernel: [ C24] Tainted: [W]=WARN, [L]=SOFTLOCKUP
|
|
2025-01-24T11:26:46.768510+01:00 qa-power8-3 kernel: [ C24] Hardware name: 8247-22L POWER8E (raw) 0x4b0201 opal:skiboot-v5.4.12 PowerNV
|
|
2025-01-24T11:26:46.768511+01:00 qa-power8-3 kernel: [ C24] NIP: c0000000007129b8 LR: c000000000715298 CTR: c00800000f1c6240
|
|
2025-01-24T11:26:46.768513+01:00 qa-power8-3 kernel: [ C24] REGS: c00000002114b6b0 TRAP: 0900 Tainted: G W L (6.13.0-1.g1513344-default)
|
|
2025-01-24T11:26:46.768515+01:00 qa-power8-3 kernel: [ C24] MSR: 900000000280b033 <SF,HV,VEC,VSX,EE,FP,ME,IR,DR,RI,LE> CR: 48022284 XER: 00000000
|
|
2025-01-24T11:26:46.768516+01:00 qa-power8-3 kernel: [ C24] CFAR: 0000000000000000 IRQMASK: 0
|
|
2025-01-24T11:26:46.768518+01:00 qa-power8-3 kernel: [ C24] GPR00: c000000000716e94 c00000002114b950 c000000001c99200 c000000538434440
|
|
2025-01-24T11:26:46.768519+01:00 qa-power8-3 kernel: [ C24] GPR04: c00000000296d498 c000000000711c80 0000000000000031 fffffffffffe0000
|
|
2025-01-24T11:26:46.768520+01:00 qa-power8-3 kernel: [ C24] GPR08: c000ffffffffffff c0080000035a0000 0000000000000001 0000000000002200
|
|
2025-01-24T11:26:46.768522+01:00 qa-power8-3 kernel: [ C24] GPR12: 0000000000000000 c0000007fffebc80 c00000000025db78 000000000000063d
|
|
2025-01-24T11:26:46.768523+01:00 qa-power8-3 kernel: [ C24] GPR16: c00000005a8afa80 000000000000001f 0000000000000000 0000000000000000
|
|
2025-01-24T11:26:46.768524+01:00 qa-power8-3 kernel: [ C24] GPR20: c000000002996bc0 000000000001f2f0 000000000001eee4 0000000000000000
|
|
2025-01-24T11:26:46.768526+01:00 qa-power8-3 kernel: [ C24] GPR24: 000000000000f591 00000000000042b6 0000000000000047 0000000000000000
|
|
2025-01-24T11:26:46.768528+01:00 qa-power8-3 kernel: [ C24] GPR28: c00000002114ba08 c0000005384344c0 0000000000000001 c000000538434440
|
|
2025-01-24T11:26:46.768529+01:00 qa-power8-3 kernel: [ C24] NIP [c0000000007129b8] ___d_drop+0x98/0xc0
|
|
2025-01-24T11:26:46.768530+01:00 qa-power8-3 kernel: [ C24] LR [c000000000715298] __dentry_kill+0x88/0x310
|
|
2025-01-24T11:26:46.768532+01:00 qa-power8-3 kernel: [ C24] Call Trace:
|
|
2025-01-24T11:26:46.768533+01:00 qa-power8-3 kernel: [ C24] [c00000002114b950] [c00000002114b990] 0xc00000002114b990 (unreliable)
|
|
2025-01-24T11:26:46.768534+01:00 qa-power8-3 kernel: [ C24] [c00000002114b990] [c000000000716e94] shrink_dentry_list+0x84/0x150
|
|
2025-01-24T11:26:46.768535+01:00 qa-power8-3 kernel: [ C24] [c00000002114b9e0] [c00000000071770c] prune_dcache_sb+0x6c/0xa0
|
|
2025-01-24T11:26:46.768537+01:00 qa-power8-3 kernel: [ C24] [c00000002114ba30] [c0000000006ec140] super_cache_scan+0x1a0/0x240
|
|
2025-01-24T11:26:46.768538+01:00 qa-power8-3 kernel: [ C24] [c00000002114baa0] [c0000000005aa458] do_shrink_slab+0x168/0x430
|
|
2025-01-24T11:26:46.768540+01:00 qa-power8-3 kernel: [ C24] [c00000002114bb20] [c0000000005ab708] shrink_slab+0x378/0x560
|
|
2025-01-24T11:26:46.768541+01:00 qa-power8-3 kernel: [ C24] [c00000002114bc00] [c0000000005a665c] shrink_node+0x4fc/0x1070
|
|
2025-01-24T11:26:46.768542+01:00 qa-power8-3 kernel: [ C24] [c00000002114bd00] [c0000000005a79f4] balance_pgdat+0x454/0xbd0
|
|
2025-01-24T11:26:46.768543+01:00 qa-power8-3 kernel: [ C24] [c00000002114beb0] [c0000000005a83f0] kswapd+0x280/0x580
|
|
2025-01-24T11:26:46.768545+01:00 qa-power8-3 kernel: [ C24] [c00000002114bf90] [c00000000025dca4] kthread+0x134/0x140
|
|
2025-01-24T11:26:46.768546+01:00 qa-power8-3 kernel: [ C24] [c00000002114bfe0] [c00000000000de58] start_kernel_thread+0x14/0x18
|
|
2025-01-24T11:26:46.768547+01:00 qa-power8-3 kernel: [ C24] Code: 7d0a5378 f9470000 41820008 f8e80008 7c2004ac e9490000 794a07a4 f9490000 4e800020 60420000 7c210b78 7c421378 <e9490000> 714a0001 4082fff0 4bffff94
|
|
2025-01-24T11:26:57.398136+01:00 qa-power8-3 kernel: [ T18] rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 24-.... } 2052344 jiffies s: 2813 root: 0x2/.
|
|
2025-01-24T11:26:57.398147+01:00 qa-power8-3 kernel: [ T18] rcu: blocking rcu_node structures (internal RCU debug): l=1:16-31:0x100/.
|
|
2025-01-24T11:26:57.398150+01:00 qa-power8-3 kernel: [ T18] Sending NMI from CPU 56 to CPUs 24:
|
|
2025-01-24T11:26:57.398151+01:00 qa-power8-3 kernel: [ C24] NMI backtrace for cpu 24
|
|
2025-01-24T11:26:57.398152+01:00 qa-power8-3 kernel: [ C24] CPU: 24 UID: 0 PID: 905 Comm: kswapd0 Tainted: G W L 6.13.0-1.g1513344-default #1 openSUSE Tumbleweed (unreleased) 067da1dff4e006e28ea31f9175684ad434e86067
|
|
2025-01-24T11:26:57.398158+01:00 qa-power8-3 kernel: [ C24] Tainted: [W]=WARN, [L]=SOFTLOCKUP
|
|
2025-01-24T11:26:57.398174+01:00 qa-power8-3 kernel: [ C24] Hardware name: 8247-22L POWER8E (raw) 0x4b0201 opal:skiboot-v5.4.12 PowerNV
|
|
2025-01-24T11:26:57.398176+01:00 qa-power8-3 kernel: [ C24] NIP: c0000000007129b8 LR: c000000000715298 CTR: c00800000f1c6240
|
|
2025-01-24T11:26:57.398177+01:00 qa-power8-3 kernel: [ C24] REGS: c00000002114b6b0 TRAP: 0500 Tainted: G W L (6.13.0-1.g1513344-default)
|
|
2025-01-24T11:26:57.398179+01:00 qa-power8-3 kernel: [ C24] MSR: 900000000280b033 <SF,HV,VEC,VSX,EE,FP,ME,IR,DR,RI,LE> CR: 48022284 XER: 00000000
|
|
2025-01-24T11:26:57.398180+01:00 qa-power8-3 kernel: [ C24] CFAR: c0000000007129c0 IRQMASK: 0
|
|
2025-01-24T11:26:57.398181+01:00 qa-power8-3 kernel: [ C24] GPR00: c000000000716e94 c00000002114b950 c000000001c99200 c000000538434440
|
|
2025-01-24T11:26:57.398183+01:00 qa-power8-3 kernel: [ C24] GPR04: c00000000296d498 c000000000711c80 0000000000000031 fffffffffffe0000
|
|
2025-01-24T11:26:57.398184+01:00 qa-power8-3 kernel: [ C24] GPR08: c000ffffffffffff c0080000035a0000 0000000000000001 0000000000002200
|
|
2025-01-24T11:26:57.398186+01:00 qa-power8-3 kernel: [ C24] GPR12: 0000000000000000 c0000007fffebc80 c00000000025db78 000000000000063d
|
|
2025-01-24T11:26:57.398187+01:00 qa-power8-3 kernel: [ C24] GPR16: c00000005a8afa80 000000000000001f 0000000000000000 0000000000000000
|
|
2025-01-24T11:26:57.398188+01:00 qa-power8-3 kernel: [ C24] GPR20: c000000002996bc0 000000000001f2f0 000000000001eee4 0000000000000000
|
|
2025-01-24T11:26:57.398190+01:00 qa-power8-3 kernel: [ C24] GPR24: 000000000000f591 00000000000042b6 0000000000000047 0000000000000000
|
|
2025-01-24T11:26:57.398191+01:00 qa-power8-3 kernel: [ C24] GPR28: c00000002114ba08 c0000005384344c0 0000000000000001 c000000538434440
|
|
2025-01-24T11:26:57.398193+01:00 qa-power8-3 kernel: [ C24] NIP [c0000000007129b8] ___d_drop+0x98/0xc0
|
|
2025-01-24T11:26:57.398194+01:00 qa-power8-3 kernel: [ C24] LR [c000000000715298] __dentry_kill+0x88/0x310
|
|
2025-01-24T11:26:57.398196+01:00 qa-power8-3 kernel: [ C24] Call Trace:
|
|
2025-01-24T11:26:57.398197+01:00 qa-power8-3 kernel: [ C24] [c00000002114b950] [c00000002114b990] 0xc00000002114b990 (unreliable)
|
|
2025-01-24T11:26:57.398199+01:00 qa-power8-3 kernel: [ C24] [c00000002114b990] [c000000000716e94] shrink_dentry_list+0x84/0x150
|
|
2025-01-24T11:26:57.398200+01:00 qa-power8-3 kernel: [ C24] [c00000002114b9e0] [c00000000071770c] prune_dcache_sb+0x6c/0xa0
|
|
2025-01-24T11:26:57.398202+01:00 qa-power8-3 kernel: [ C24] [c00000002114ba30] [c0000000006ec140] super_cache_scan+0x1a0/0x240
|
|
2025-01-24T11:26:57.398203+01:00 qa-power8-3 kernel: [ C24] [c00000002114baa0] [c0000000005aa458] do_shrink_slab+0x168/0x430
|
|
2025-01-24T11:26:57.398205+01:00 qa-power8-3 kernel: [ C24] [c00000002114bb20] [c0000000005ab708] shrink_slab+0x378/0x560
|
|
2025-01-24T11:26:57.398206+01:00 qa-power8-3 kernel: [ C24] [c00000002114bc00] [c0000000005a665c] shrink_node+0x4fc/0x1070
|
|
2025-01-24T11:26:57.398208+01:00 qa-power8-3 kernel: [ C24] [c00000002114bd00] [c0000000005a79f4] balance_pgdat+0x454/0xbd0
|
|
2025-01-24T11:26:57.398209+01:00 qa-power8-3 kernel: [ C24] [c00000002114beb0] [c0000000005a83f0] kswapd+0x280/0x580
|
|
2025-01-24T11:26:57.398211+01:00 qa-power8-3 kernel: [ C24] [c00000002114bf90] [c00000000025dca4] kthread+0x134/0x140
|
|
2025-01-24T11:26:57.398213+01:00 qa-power8-3 kernel: [ C24] [c00000002114bfe0] [c00000000000de58] start_kernel_thread+0x14/0x18
|
|
2025-01-24T11:26:57.398215+01:00 qa-power8-3 kernel: [ C24] Code: 7d0a5378 f9470000 41820008 f8e80008 7c2004ac e9490000 794a07a4 f9490000 4e800020 60420000 7c210b78 7c421378 <e9490000> 714a0001 4082fff0 4bffff94
|
|
2025-01-24T11:27:01.228164+01:00 qa-power8-3 kernel: [ C0] BUG: workqueue lockup - pool cpus=24 node=0 flags=0x0 nice=0 stuck for 41712s!
|
|
2025-01-24T11:27:01.228170+01:00 qa-power8-3 kernel: [ C0] Showing busy workqueues and worker pools:
|
|
2025-01-24T11:27:01.228172+01:00 qa-power8-3 kernel: [ C0] workqueue events: flags=0x0
|
|
2025-01-24T11:27:01.228173+01:00 qa-power8-3 kernel: [ C0] pwq 98: cpus=24 node=0 flags=0x0 nice=0 active=1 refcnt=2
|
|
2025-01-24T11:27:01.228174+01:00 qa-power8-3 kernel: [ C0] pending: drain_local_stock
|
|
2025-01-24T11:27:01.228186+01:00 qa-power8-3 kernel: [ C0] workqueue events_power_efficient: flags=0x80
|
|
2025-01-24T11:27:01.228187+01:00 qa-power8-3 kernel: [ C0] pwq 98: cpus=24 node=0 flags=0x0 nice=0 active=2 refcnt=3
|
|
2025-01-24T11:27:01.228188+01:00 qa-power8-3 kernel: [ C0] pending: neigh_managed_work, neigh_periodic_work
|
|
2025-01-24T11:27:01.228190+01:00 qa-power8-3 kernel: [ C0] workqueue mm_percpu_wq: flags=0x8
|
|
2025-01-24T11:27:01.228191+01:00 qa-power8-3 kernel: [ C0] pwq 98: cpus=24 node=0 flags=0x0 nice=0 active=1 refcnt=2
|
|
2025-01-24T11:27:01.228193+01:00 qa-power8-3 kernel: [ C0] pending: vmstat_update
|
|
2025-01-24T11:27:01.228194+01:00 qa-power8-3 kernel: [ C0] workqueue iou_exit: flags=0x2
|
|
2025-01-24T11:27:01.228195+01:00 qa-power8-3 kernel: [ C0] pwq 642: cpus=0-159 node=0 flags=0x4 nice=0 active=1 refcnt=2
|
|
2025-01-24T11:27:01.228197+01:00 qa-power8-3 kernel: [ C0] in-flight: 148824:io_ring_exit_work
|
|
2025-01-24T11:27:01.228198+01:00 qa-power8-3 kernel: [ C0] pwq 648: cpus=0-159 node=1 flags=0x4 nice=0 active=3 refcnt=4
|
|
2025-01-24T11:27:01.228199+01:00 qa-power8-3 kernel: [ C0] in-flight: 118070:io_ring_exit_work ,127177:io_ring_exit_work ,129527:io_ring_exit_work
|
|
2025-01-24T11:27:01.228201+01:00 qa-power8-3 kernel: [ C0] pwq 649: cpus=0-159 node=1 flags=0x4 nice=0 active=2 refcnt=3
|
|
2025-01-24T11:27:01.228202+01:00 qa-power8-3 kernel: [ C0] in-flight: 130631:io_ring_exit_work ,129460:io_ring_exit_work
|
|
2025-01-24T11:27:01.228203+01:00 qa-power8-3 kernel: [ C0] pwq 650: cpus=0-159 node=1 flags=0x4 nice=0 active=5 refcnt=6
|
|
2025-01-24T11:27:01.228204+01:00 qa-power8-3 kernel: [ C0] in-flight: 127213:io_ring_exit_work ,127522:io_ring_exit_work ,127521:io_ring_exit_work ,147689:io_ring_exit_work ,147685:io_ring_exit_work
|
|
2025-01-24T11:27:01.228206+01:00 qa-power8-3 kernel: [ C0] pwq 651: cpus=0-159 node=16 flags=0x4 nice=0 active=7 refcnt=13
|
|
2025-01-24T11:27:01.228207+01:00 qa-power8-3 kernel: [ C0] in-flight: 153752:io_ring_exit_work ,152496:io_ring_exit_work ,152844:io_ring_exit_work ,150270:io_ring_exit_work ,153906:io_ring_exit_work ,152861:io_ring_exit_work ,152074:io_ring_exit_work
|
|
2025-01-24T11:27:01.228208+01:00 qa-power8-3 kernel: [ C0] inactive: 5*io_ring_exit_work
|
|
2025-01-24T11:27:01.228210+01:00 qa-power8-3 kernel: [ C0] pwq 652: cpus=0-159 node=16 flags=0x4 nice=0 active=1 refcnt=7
|
|
2025-01-24T11:27:01.228211+01:00 qa-power8-3 kernel: [ C0] in-flight: 151346:io_ring_exit_work
|
|
2025-01-24T11:27:01.228212+01:00 qa-power8-3 kernel: [ C0] inactive: 5*io_ring_exit_work
|
|
2025-01-24T11:27:01.228214+01:00 qa-power8-3 kernel: [ C0] pwq 653: cpus=0-159 node=16 flags=0x4 nice=0 active=1 refcnt=7
|
|
2025-01-24T11:27:01.228215+01:00 qa-power8-3 kernel: [ C0] in-flight: 147308:io_ring_exit_work
|
|
2025-01-24T11:27:01.228216+01:00 qa-power8-3 kernel: [ C0] inactive: 5*io_ring_exit_work
|
|
2025-01-24T11:27:01.228217+01:00 qa-power8-3 kernel: [ C0] pwq 654: cpus=0-159 node=16 flags=0x4 nice=0 active=5 refcnt=10
|
|
2025-01-24T11:27:01.228218+01:00 qa-power8-3 kernel: [ C0] in-flight: 148805:io_ring_exit_work ,151928:io_ring_exit_work ,152824:io_ring_exit_work ,152829:io_ring_exit_work ,152827:io_ring_exit_work
|
|
2025-01-24T11:27:01.228220+01:00 qa-power8-3 kernel: [ C0] inactive: 4*io_ring_exit_work
|
|
2025-01-24T11:27:01.228221+01:00 qa-power8-3 kernel: [ C0] pwq 655: cpus=0-159 node=16 flags=0x4 nice=0 active=2 refcnt=5
|
|
2025-01-24T11:27:01.228222+01:00 qa-power8-3 kernel: [ C0] in-flight: 152591:io_ring_exit_work ,151754:io_ring_exit_work
|
|
2025-01-24T11:27:01.228224+01:00 qa-power8-3 kernel: [ C0] inactive: 2*io_ring_exit_work
|
|
2025-01-24T11:27:01.228225+01:00 qa-power8-3 kernel: [ C0] pwq 656: cpus=0-159 node=17 flags=0x4 nice=0 active=5 refcnt=6
|
|
2025-01-24T11:27:01.228226+01:00 qa-power8-3 kernel: [ C0] in-flight: 147303:io_ring_exit_work ,130347:io_ring_exit_work ,130346:io_ring_exit_work ,129985:io_ring_exit_work ,147256:io_ring_exit_work
|
|
2025-01-24T11:27:01.228228+01:00 qa-power8-3 kernel: [ C0] pwq 658: cpus=0-159 node=17 flags=0x4 nice=0 active=1 refcnt=2
|
|
2025-01-24T11:27:01.228229+01:00 qa-power8-3 kernel: [ C0] in-flight: 147299:io_ring_exit_work
|
|
2025-01-24T11:27:01.228230+01:00 qa-power8-3 kernel: [ C0] pwq 659: cpus=0-159 node=17 flags=0x4 nice=0 active=4 refcnt=5
|
|
2025-01-24T11:27:01.228231+01:00 qa-power8-3 kernel: [ C0] in-flight: 127097:io_ring_exit_work ,130719:io_ring_exit_work ,129372:io_ring_exit_work ,147458:io_ring_exit_work
|
|
2025-01-24T11:27:01.228232+01:00 qa-power8-3 kernel: [ C0] pwq 660: cpus=0-159 node=17 flags=0x4 nice=0 active=1 refcnt=2
|
|
2025-01-24T11:27:01.228234+01:00 qa-power8-3 kernel: [ C0] in-flight: 147318:io_ring_exit_work
|
|
2025-01-24T11:27:01.228235+01:00 qa-power8-3 kernel: [ C0] pool 642: cpus=0-159 node=0 flags=0x4 nice=0 hung=0s workers=4 idle: 127904 154042 154043
|
|
2025-01-24T11:27:01.228237+01:00 qa-power8-3 kernel: [ C0] pool 648: cpus=0-159 node=1 flags=0x4 nice=0 hung=0s workers=5 idle: 147825 148763
|
|
2025-01-24T11:27:01.228238+01:00 qa-power8-3 kernel: [ C0] pool 649: cpus=0-159 node=1 flags=0x4 nice=0 hung=0s workers=4 idle: 154165 154242
|
|
2025-01-24T11:27:01.228239+01:00 qa-power8-3 kernel: [ C0] pool 650: cpus=0-159 node=1 flags=0x4 nice=0 hung=0s workers=8 idle: 147859 147693 147954
|
|
2025-01-24T11:27:01.228240+01:00 qa-power8-3 kernel: [ C0] pool 651: cpus=0-159 node=16 flags=0x4 nice=0 hung=0s workers=11 idle: 154920 153982 154915 155388
|
|
2025-01-24T11:27:01.228242+01:00 qa-power8-3 kernel: [ C0] pool 652: cpus=0-159 node=16 flags=0x4 nice=0 hung=0s workers=4 idle: 155967 156273 152855
|
|
2025-01-24T11:27:01.228243+01:00 qa-power8-3 kernel: [ C0] pool 653: cpus=0-159 node=16 flags=0x4 nice=0 hung=0s workers=7 idle: 156257 153489 156591 156259 154759 155714
|
|
2025-01-24T11:27:01.228244+01:00 qa-power8-3 kernel: [ C0] pool 654: cpus=0-159 node=16 flags=0x4 nice=0 hung=0s workers=9 idle: 156385 155190 156377 152834
|
|
2025-01-24T11:27:01.228246+01:00 qa-power8-3 kernel: [ C0] pool 655: cpus=0-159 node=16 flags=0x4 nice=0 hung=0s workers=4 idle: 154694 156401
|
|
2025-01-24T11:27:01.228247+01:00 qa-power8-3 kernel: [ C0] pool 656: cpus=0-159 node=17 flags=0x4 nice=0 hung=0s workers=9 idle: 155382 149795 148819 155419
|
|
2025-01-24T11:27:01.228248+01:00 qa-power8-3 kernel: [ C0] pool 658: cpus=0-159 node=17 flags=0x4 nice=0 hung=0s workers=5 idle: 156532 156175 154798 155872
|
|
2025-01-24T11:27:01.228249+01:00 qa-power8-3 kernel: [ C0] pool 659: cpus=0-159 node=17 flags=0x4 nice=0 hung=0s workers=7 idle: 155771 155547 153308
|
|
2025-01-24T11:27:01.228250+01:00 qa-power8-3 kernel: [ C0] pool 660: cpus=0-159 node=17 flags=0x4 nice=0 hung=0s workers=6 idle: 155079 156421 147520 156018 153416
|
|
2025-01-24T11:27:01.228252+01:00 qa-power8-3 kernel: [ C0] Showing backtraces of running workers in stalled CPU-bound worker pools:
|
|
2025-01-24T11:27:10.768456+01:00 qa-power8-3 kernel: [ C24] watchdog: BUG: soft lockup - CPU#24 stuck for 38857s! [kswapd0:905]
|
|
2025-01-24T11:27:10.768465+01:00 qa-power8-3 kernel: [ C24] CPU#24 Utilization every 4s during lockup:
|
|
2025-01-24T11:27:10.768467+01:00 qa-power8-3 kernel: [ C24] #1: 100% system, 0% softirq, 0% hardirq, 0% idle
|
|
2025-01-24T11:27:10.768469+01:00 qa-power8-3 kernel: [ C24] #2: 100% system, 0% softirq, 0% hardirq, 0% idle
|
|
2025-01-24T11:27:10.768470+01:00 qa-power8-3 kernel: [ C24] #3: 101% system, 0% softirq, 0% hardirq, 0% idle
|
|
2025-01-24T11:27:10.768471+01:00 qa-power8-3 kernel: [ C24] #4: 100% system, 0% softirq, 0% hardirq, 0% idle
|
|
2025-01-24T11:27:10.768473+01:00 qa-power8-3 kernel: [ C24] #5: 100% system, 0% softirq, 0% hardirq, 0% idle
|
|
2025-01-24T11:27:10.768475+01:00 qa-power8-3 kernel: [ C24] Modules linked in: nft_masq nft_chain_nat nf_tables af_packet tun bridge stp llc nfnetlink_cttimeout nfnetlink openvswitch nsh nf_conncount nf_nat rfkill nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 psample binfmt_misc aes_gcm_p10_crypto crypto_simd cryptd xts tg3 ses vmx_crypto crct10dif_vpmsum enclosure libphy scsi_transport_sas ipmi_powernv rtc_opal leds_powernv ipmi_devintf uio_pdrv_genirq ipmi_msghandler led_class ibmpowernv powernv_op_panel i2c_opal uio fuse configfs ip_tables x_tables xhci_pci ipr xhci_hcd sd_mod scsi_dh_emc scsi_dh_rdac scsi_dh_alua sg usbcore scsi_mod usb_common scsi_common btrfs blake2b_generic libcrc32c crc32c_vpmsum xor raid6_pq kvm_hv kvm
|
|
2025-01-24T11:27:10.768493+01:00 qa-power8-3 kernel: [ C24] CPU: 24 UID: 0 PID: 905 Comm: kswapd0 Tainted: G W L 6.13.0-1.g1513344-default #1 openSUSE Tumbleweed (unreleased) 067da1dff4e006e28ea31f9175684ad434e86067
|
|
2025-01-24T11:27:10.768494+01:00 qa-power8-3 kernel: [ C24] Tainted: [W]=WARN, [L]=SOFTLOCKUP
|
|
2025-01-24T11:27:10.768496+01:00 qa-power8-3 kernel: [ C24] Hardware name: 8247-22L POWER8E (raw) 0x4b0201 opal:skiboot-v5.4.12 PowerNV
|
|
2025-01-24T11:27:10.768497+01:00 qa-power8-3 kernel: [ C24] NIP: c0000000007129b8 LR: c000000000715298 CTR: c00800000f1c6240
|
|
2025-01-24T11:27:10.768499+01:00 qa-power8-3 kernel: [ C24] REGS: c00000002114b6b0 TRAP: 0900 Tainted: G W L (6.13.0-1.g1513344-default)
|
|
2025-01-24T11:27:10.768501+01:00 qa-power8-3 kernel: [ C24] MSR: 900000000280b033 <SF,HV,VEC,VSX,EE,FP,ME,IR,DR,RI,LE> CR: 48022284 XER: 00000000
|
|
2025-01-24T11:27:10.768502+01:00 qa-power8-3 kernel: [ C24] CFAR: 0000000000000000 IRQMASK: 0
|
|
2025-01-24T11:27:10.768504+01:00 qa-power8-3 kernel: [ C24] GPR00: c000000000716e94 c00000002114b950 c000000001c99200 c000000538434440
|
|
2025-01-24T11:27:10.768505+01:00 qa-power8-3 kernel: [ C24] GPR04: c00000000296d498 c000000000711c80 0000000000000031 fffffffffffe0000
|
|
2025-01-24T11:27:10.768506+01:00 qa-power8-3 kernel: [ C24] GPR08: c000ffffffffffff c0080000035a0000 0000000000000001 0000000000002200
|
|
2025-01-24T11:27:10.768507+01:00 qa-power8-3 kernel: [ C24] GPR12: 0000000000000000 c0000007fffebc80 c00000000025db78 000000000000063d
|
|
2025-01-24T11:27:10.768509+01:00 qa-power8-3 kernel: [ C24] GPR16: c00000005a8afa80 000000000000001f 0000000000000000 0000000000000000
|
|
2025-01-24T11:27:10.768510+01:00 qa-power8-3 kernel: [ C24] GPR20: c000000002996bc0 000000000001f2f0 000000000001eee4 0000000000000000
|
|
2025-01-24T11:27:10.768512+01:00 qa-power8-3 kernel: [ C24] GPR24: 000000000000f591 00000000000042b6 0000000000000047 0000000000000000
|
|
2025-01-24T11:27:10.768514+01:00 qa-power8-3 kernel: [ C24] GPR28: c00000002114ba08 c0000005384344c0 0000000000000001 c000000538434440
|
|
2025-01-24T11:27:10.768515+01:00 qa-power8-3 kernel: [ C24] NIP [c0000000007129b8] ___d_drop+0x98/0xc0
|
|
2025-01-24T11:27:10.768516+01:00 qa-power8-3 kernel: [ C24] LR [c000000000715298] __dentry_kill+0x88/0x310
|
|
2025-01-24T11:27:10.768517+01:00 qa-power8-3 kernel: [ C24] Call Trace:
|
|
2025-01-24T11:27:10.768519+01:00 qa-power8-3 kernel: [ C24] [c00000002114b950] [c00000002114b990] 0xc00000002114b990 (unreliable)
|
|
2025-01-24T11:27:10.768520+01:00 qa-power8-3 kernel: [ C24] [c00000002114b990] [c000000000716e94] shrink_dentry_list+0x84/0x150
|
|
2025-01-24T11:27:10.768522+01:00 qa-power8-3 kernel: [ C24] [c00000002114b9e0] [c00000000071770c] prune_dcache_sb+0x6c/0xa0
|
|
2025-01-24T11:27:10.768523+01:00 qa-power8-3 kernel: [ C24] [c00000002114ba30] [c0000000006ec140] super_cache_scan+0x1a0/0x240
|
|
2025-01-24T11:27:10.768524+01:00 qa-power8-3 kernel: [ C24] [c00000002114baa0] [c0000000005aa458] do_shrink_slab+0x168/0x430
|
|
2025-01-24T11:27:10.768526+01:00 qa-power8-3 kernel: [ C24] [c00000002114bb20] [c0000000005ab708] shrink_slab+0x378/0x560
|
|
2025-01-24T11:27:10.768527+01:00 qa-power8-3 kernel: [ C24] [c00000002114bc00] [c0000000005a665c] shrink_node+0x4fc/0x1070
|
|
2025-01-24T11:27:10.768528+01:00 qa-power8-3 kernel: [ C24] [c00000002114bd00] [c0000000005a79f4] balance_pgdat+0x454/0xbd0
|
|
2025-01-24T11:27:10.768530+01:00 qa-power8-3 kernel: [ C24] [c00000002114beb0] [c0000000005a83f0] kswapd+0x280/0x580
|
|
2025-01-24T11:27:10.768531+01:00 qa-power8-3 kernel: [ C24] [c00000002114bf90] [c00000000025dca4] kthread+0x134/0x140
|
|
2025-01-24T11:27:10.768532+01:00 qa-power8-3 kernel: [ C24] [c00000002114bfe0] [c00000000000de58] start_kernel_thread+0x14/0x18
|
|
2025-01-24T11:27:10.768534+01:00 qa-power8-3 kernel: [ C24] Code: 7d0a5378 f9470000 41820008 f8e80008 7c2004ac e9490000 794a07a4 f9490000 4e800020 60420000 7c210b78 7c421378 <e9490000> 714a0001 4082fff0 4bffff94
|
|
2025-01-24T11:27:16.652429+01:00 qa-power8-3 worker[4382]: [warn] A QEMU instance using the current pool directory is still running (PID: 125500) - checking again for web UI 'https://openqa.opensuse.org' in 227.59 s
|
|
2025-01-24T11:27:31.318171+01:00 qa-power8-3 kernel: [ C0] BUG: workqueue lockup - pool cpus=24 node=0 flags=0x0 nice=0 stuck for 41743s!
|
|
2025-01-24T11:27:31.318184+01:00 qa-power8-3 kernel: [ C0] Showing busy workqueues and worker pools:
|
|
2025-01-24T11:27:31.318185+01:00 qa-power8-3 kernel: [ C0] workqueue events: flags=0x0
|
|
2025-01-24T11:27:31.318187+01:00 qa-power8-3 kernel: [ C0] pwq 98: cpus=24 node=0 flags=0x0 nice=0 active=1 refcnt=2
|
|
2025-01-24T11:27:31.318188+01:00 qa-power8-3 kernel: [ C0] pending: drain_local_stock
|
|
2025-01-24T11:27:31.318189+01:00 qa-power8-3 kernel: [ C0] workqueue events_power_efficient: flags=0x80
|
|
2025-01-24T11:27:31.318191+01:00 qa-power8-3 kernel: [ C0] pwq 98: cpus=24 node=0 flags=0x0 nice=0 active=2 refcnt=3
|
|
2025-01-24T11:27:31.318192+01:00 qa-power8-3 kernel: [ C0] pending: neigh_managed_work, neigh_periodic_work
|
|
2025-01-24T11:27:31.318197+01:00 qa-power8-3 kernel: [ C0] workqueue mm_percpu_wq: flags=0x8
|
|
2025-01-24T11:27:31.318211+01:00 qa-power8-3 kernel: [ C0] pwq 98: cpus=24 node=0 flags=0x0 nice=0 active=1 refcnt=2
|
|
2025-01-24T11:27:31.318213+01:00 qa-power8-3 kernel: [ C0] pending: vmstat_update
|
|
2025-01-24T11:27:31.318215+01:00 qa-power8-3 kernel: [ C0] workqueue iou_exit: flags=0x2
|
|
2025-01-24T11:27:31.318216+01:00 qa-power8-3 kernel: [ C0] pwq 642: cpus=0-159 node=0 flags=0x4 nice=0 active=1 refcnt=2
|
|
2025-01-24T11:27:31.318218+01:00 qa-power8-3 kernel: [ C0] in-flight: 148824:io_ring_exit_work
|
|
2025-01-24T11:27:31.318219+01:00 qa-power8-3 kernel: [ C0] pwq 648: cpus=0-159 node=1 flags=0x4 nice=0 active=3 refcnt=4
|
|
2025-01-24T11:27:31.318221+01:00 qa-power8-3 kernel: [ C0] in-flight: 118070:io_ring_exit_work ,127177:io_ring_exit_work ,129527:io_ring_exit_work
|
|
2025-01-24T11:27:31.318223+01:00 qa-power8-3 kernel: [ C0] pwq 649: cpus=0-159 node=1 flags=0x4 nice=0 active=2 refcnt=3
|
|
2025-01-24T11:27:31.318224+01:00 qa-power8-3 kernel: [ C0] in-flight: 130631:io_ring_exit_work ,129460:io_ring_exit_work
|
|
2025-01-24T11:27:31.318226+01:00 qa-power8-3 kernel: [ C0] pwq 650: cpus=0-159 node=1 flags=0x4 nice=0 active=5 refcnt=6
|
|
2025-01-24T11:27:31.318228+01:00 qa-power8-3 kernel: [ C0] in-flight: 127213:io_ring_exit_work ,127522:io_ring_exit_work ,127521:io_ring_exit_work ,147689:io_ring_exit_work ,147685:io_ring_exit_work
|
|
2025-01-24T11:27:31.318229+01:00 qa-power8-3 kernel: [ C0] pwq 651: cpus=0-159 node=16 flags=0x4 nice=0 active=7 refcnt=13
|
|
2025-01-24T11:27:31.318231+01:00 qa-power8-3 kernel: [ C0] in-flight: 153752:io_ring_exit_work ,152496:io_ring_exit_work ,152844:io_ring_exit_work ,150270:io_ring_exit_work ,153906:io_ring_exit_work ,152861:io_ring_exit_work ,152074:io_ring_exit_work
|
|
2025-01-24T11:27:31.318233+01:00 qa-power8-3 kernel: [ C0] inactive: 5*io_ring_exit_work
|
|
2025-01-24T11:27:31.318234+01:00 qa-power8-3 kernel: [ C0] pwq 652: cpus=0-159 node=16 flags=0x4 nice=0 active=1 refcnt=7
|
|
2025-01-24T11:27:31.318236+01:00 qa-power8-3 kernel: [ C0] in-flight: 151346:io_ring_exit_work
|
|
2025-01-24T11:27:31.318237+01:00 qa-power8-3 kernel: [ C0] inactive: 5*io_ring_exit_work
|
|
2025-01-24T11:27:31.318238+01:00 qa-power8-3 kernel: [ C0] pwq 653: cpus=0-159 node=16 flags=0x4 nice=0 active=1 refcnt=7
|
|
2025-01-24T11:27:31.318240+01:00 qa-power8-3 kernel: [ C0] in-flight: 147308:io_ring_exit_work
|
|
2025-01-24T11:27:31.318241+01:00 qa-power8-3 kernel: [ C0] inactive: 5*io_ring_exit_work
|
|
2025-01-24T11:27:31.318242+01:00 qa-power8-3 kernel: [ C0] pwq 654: cpus=0-159 node=16 flags=0x4 nice=0 active=5 refcnt=10
|
|
2025-01-24T11:27:31.318243+01:00 qa-power8-3 kernel: [ C0] in-flight: 148805:io_ring_exit_work ,151928:io_ring_exit_work ,152824:io_ring_exit_work ,152829:io_ring_exit_work ,152827:io_ring_exit_work
|
|
2025-01-24T11:27:31.318245+01:00 qa-power8-3 kernel: [ C0] inactive: 4*io_ring_exit_work
|
|
2025-01-24T11:27:31.318246+01:00 qa-power8-3 kernel: [ C0] pwq 655: cpus=0-159 node=16 flags=0x4 nice=0 active=2 refcnt=5
|
|
2025-01-24T11:27:31.318248+01:00 qa-power8-3 kernel: [ C0] in-flight: 152591:io_ring_exit_work ,151754:io_ring_exit_work
|
|
2025-01-24T11:27:31.318249+01:00 qa-power8-3 kernel: [ C0] inactive: 2*io_ring_exit_work
|
|
2025-01-24T11:27:31.318250+01:00 qa-power8-3 kernel: [ C0] pwq 656: cpus=0-159 node=17 flags=0x4 nice=0 active=5 refcnt=6
|
|
2025-01-24T11:27:31.318251+01:00 qa-power8-3 kernel: [ C0] in-flight: 147303:io_ring_exit_work ,130347:io_ring_exit_work ,130346:io_ring_exit_work ,129985:io_ring_exit_work ,147256:io_ring_exit_work
|
|
2025-01-24T11:27:31.318253+01:00 qa-power8-3 kernel: [ C0] pwq 658: cpus=0-159 node=17 flags=0x4 nice=0 active=1 refcnt=2
|
|
2025-01-24T11:27:31.318254+01:00 qa-power8-3 kernel: [ C0] in-flight: 147299:io_ring_exit_work
|
|
2025-01-24T11:27:31.318255+01:00 qa-power8-3 kernel: [ C0] pwq 659: cpus=0-159 node=17 flags=0x4 nice=0 active=4 refcnt=5
|
|
2025-01-24T11:27:31.318256+01:00 qa-power8-3 kernel: [ C0] in-flight: 127097:io_ring_exit_work ,130719:io_ring_exit_work ,129372:io_ring_exit_work ,147458:io_ring_exit_work
|
|
2025-01-24T11:27:31.318258+01:00 qa-power8-3 kernel: [ C0] pwq 660: cpus=0-159 node=17 flags=0x4 nice=0 active=1 refcnt=2
|
|
2025-01-24T11:27:31.318259+01:00 qa-power8-3 kernel: [ C0] in-flight: 147318:io_ring_exit_work
|
|
2025-01-24T11:27:31.318261+01:00 qa-power8-3 kernel: [ C0] pool 642: cpus=0-159 node=0 flags=0x4 nice=0 hung=0s workers=4 idle: 127904 154042 154043
|
|
2025-01-24T11:27:31.318262+01:00 qa-power8-3 kernel: [ C0] pool 648: cpus=0-159 node=1 flags=0x4 nice=0 hung=0s workers=5 idle: 147825 148763
|
|
2025-01-24T11:27:31.318263+01:00 qa-power8-3 kernel: [ C0] pool 649: cpus=0-159 node=1 flags=0x4 nice=0 hung=0s workers=4 idle: 154165 154242
|
|
2025-01-24T11:27:31.318265+01:00 qa-power8-3 kernel: [ C0] pool 650: cpus=0-159 node=1 flags=0x4 nice=0 hung=0s workers=8 idle: 147859 147693 147954
|
|
2025-01-24T11:27:31.318266+01:00 qa-power8-3 kernel: [ C0] pool 651: cpus=0-159 node=16 flags=0x4 nice=0 hung=0s workers=11 idle: 154920 153982 154915 155388
|
|
2025-01-24T11:27:31.318267+01:00 qa-power8-3 kernel: [ C0] pool 652: cpus=0-159 node=16 flags=0x4 nice=0 hung=0s workers=4 idle: 155967 156273 152855
|
|
2025-01-24T11:27:31.318268+01:00 qa-power8-3 kernel: [ C0] pool 653: cpus=0-159 node=16 flags=0x4 nice=0 hung=0s workers=7 idle: 156591 156259 153489 156257 154759 155714
|
|
2025-01-24T11:27:31.318270+01:00 qa-power8-3 kernel: [ C0] pool 654: cpus=0-159 node=16 flags=0x4 nice=0 hung=0s workers=9 idle: 156385 155190 156377 152834
|
|
2025-01-24T11:27:31.318271+01:00 qa-power8-3 kernel: [ C0] pool 655: cpus=0-159 node=16 flags=0x4 nice=0 hung=0s workers=4 idle: 154694 156401
|
|
2025-01-24T11:27:31.318272+01:00 qa-power8-3 kernel: [ C0] pool 656: cpus=0-159 node=17 flags=0x4 nice=0 hung=0s workers=9 idle: 149795 155382 148819 155419
|
|
2025-01-24T11:27:31.318273+01:00 qa-power8-3 kernel: [ C0] pool 658: cpus=0-159 node=17 flags=0x4 nice=0 hung=0s workers=5 idle: 156532 156175 154798 155872
|
|
2025-01-24T11:27:31.318275+01:00 qa-power8-3 kernel: [ C0] pool 659: cpus=0-159 node=17 flags=0x4 nice=0 hung=0s workers=7 idle: 155771 155547 153308
|
|
2025-01-24T11:27:31.318276+01:00 qa-power8-3 kernel: [ C0] pool 660: cpus=0-159 node=17 flags=0x4 nice=0 hung=0s workers=6 idle: 156421 155079 147520 156018 153416
|
|
2025-01-24T11:27:31.318277+01:00 qa-power8-3 kernel: [ C0] Showing backtraces of running workers in stalled CPU-bound worker pools:
|
|
2025-01-24T11:27:34.768458+01:00 qa-power8-3 kernel: [ C24] watchdog: BUG: soft lockup - CPU#24 stuck for 38880s! [kswapd0:905]
|
|
2025-01-24T11:27:34.768468+01:00 qa-power8-3 kernel: [ C24] CPU#24 Utilization every 4s during lockup:
|
|
2025-01-24T11:27:34.768470+01:00 qa-power8-3 kernel: [ C24] #1: 100% system, 0% softirq, 0% hardirq, 0% idle
|
|
2025-01-24T11:27:34.768471+01:00 qa-power8-3 kernel: [ C24] #2: 100% system, 0% softirq, 0% hardirq, 0% idle
|
|
2025-01-24T11:27:34.768472+01:00 qa-power8-3 kernel: [ C24] #3: 101% system, 0% softirq, 0% hardirq, 0% idle
|
|
2025-01-24T11:27:34.768474+01:00 qa-power8-3 kernel: [ C24] #4: 100% system, 0% softirq, 0% hardirq, 0% idle
|
|
2025-01-24T11:27:34.768475+01:00 qa-power8-3 kernel: [ C24] #5: 100% system, 0% softirq, 0% hardirq, 0% idle
|
|
2025-01-24T11:27:34.768477+01:00 qa-power8-3 kernel: [ C24] Modules linked in: nft_masq nft_chain_nat nf_tables af_packet tun bridge stp llc nfnetlink_cttimeout nfnetlink openvswitch nsh nf_conncount nf_nat rfkill nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 psample binfmt_misc aes_gcm_p10_crypto crypto_simd cryptd xts tg3 ses vmx_crypto crct10dif_vpmsum enclosure libphy scsi_transport_sas ipmi_powernv rtc_opal leds_powernv ipmi_devintf uio_pdrv_genirq ipmi_msghandler led_class ibmpowernv powernv_op_panel i2c_opal uio fuse configfs ip_tables x_tables xhci_pci ipr xhci_hcd sd_mod scsi_dh_emc scsi_dh_rdac scsi_dh_alua sg usbcore scsi_mod usb_common scsi_common btrfs blake2b_generic libcrc32c crc32c_vpmsum xor raid6_pq kvm_hv kvm
|
|
2025-01-24T11:27:34.768496+01:00 qa-power8-3 kernel: [ C24] CPU: 24 UID: 0 PID: 905 Comm: kswapd0 Tainted: G W L 6.13.0-1.g1513344-default #1 openSUSE Tumbleweed (unreleased) 067da1dff4e006e28ea31f9175684ad434e86067
|
|
2025-01-24T11:27:34.768498+01:00 qa-power8-3 kernel: [ C24] Tainted: [W]=WARN, [L]=SOFTLOCKUP
|
|
2025-01-24T11:27:34.768499+01:00 qa-power8-3 kernel: [ C24] Hardware name: 8247-22L POWER8E (raw) 0x4b0201 opal:skiboot-v5.4.12 PowerNV
|
|
2025-01-24T11:27:34.768501+01:00 qa-power8-3 kernel: [ C24] NIP: c0000000007129b8 LR: c000000000715298 CTR: c00800000f1c6240
|
|
2025-01-24T11:27:34.768503+01:00 qa-power8-3 kernel: [ C24] REGS: c00000002114b6b0 TRAP: 0900 Tainted: G W L (6.13.0-1.g1513344-default)
|
|
2025-01-24T11:27:34.768504+01:00 qa-power8-3 kernel: [ C24] MSR: 900000000280b033 <SF,HV,VEC,VSX,EE,FP,ME,IR,DR,RI,LE> CR: 48022284 XER: 00000000
|
|
2025-01-24T11:27:34.768506+01:00 qa-power8-3 kernel: [ C24] CFAR: 0000000000000000 IRQMASK: 0
|
|
2025-01-24T11:27:34.768508+01:00 qa-power8-3 kernel: [ C24] GPR00: c000000000716e94 c00000002114b950 c000000001c99200 c000000538434440
|
|
2025-01-24T11:27:34.768509+01:00 qa-power8-3 kernel: [ C24] GPR04: c00000000296d498 c000000000711c80 0000000000000031 fffffffffffe0000
|
|
2025-01-24T11:27:34.768510+01:00 qa-power8-3 kernel: [ C24] GPR08: c000ffffffffffff c0080000035a0000 0000000000000001 0000000000002200
|
|
2025-01-24T11:27:34.768512+01:00 qa-power8-3 kernel: [ C24] GPR12: 0000000000000000 c0000007fffebc80 c00000000025db78 000000000000063d
|
|
2025-01-24T11:27:34.768513+01:00 qa-power8-3 kernel: [ C24] GPR16: c00000005a8afa80 000000000000001f 0000000000000000 0000000000000000
|
|
2025-01-24T11:27:34.768514+01:00 qa-power8-3 kernel: [ C24] GPR20: c000000002996bc0 000000000001f2f0 000000000001eee4 0000000000000000
|
|
2025-01-24T11:27:34.768516+01:00 qa-power8-3 kernel: [ C24] GPR24: 000000000000f591 00000000000042b6 0000000000000047 0000000000000000
|
|
2025-01-24T11:27:34.768518+01:00 qa-power8-3 kernel: [ C24] GPR28: c00000002114ba08 c0000005384344c0 0000000000000001 c000000538434440
|
|
2025-01-24T11:27:34.768520+01:00 qa-power8-3 kernel: [ C24] NIP [c0000000007129b8] ___d_drop+0x98/0xc0
|
|
2025-01-24T11:27:34.768521+01:00 qa-power8-3 kernel: [ C24] LR [c000000000715298] __dentry_kill+0x88/0x310
|
|
2025-01-24T11:27:34.768522+01:00 qa-power8-3 kernel: [ C24] Call Trace:
|
|
2025-01-24T11:27:34.768523+01:00 qa-power8-3 kernel: [ C24] [c00000002114b950] [c00000002114b990] 0xc00000002114b990 (unreliable)
|
|
2025-01-24T11:27:34.768524+01:00 qa-power8-3 kernel: [ C24] [c00000002114b990] [c000000000716e94] shrink_dentry_list+0x84/0x150
|
|
2025-01-24T11:27:34.768526+01:00 qa-power8-3 kernel: [ C24] [c00000002114b9e0] [c00000000071770c] prune_dcache_sb+0x6c/0xa0
|
|
2025-01-24T11:27:34.768527+01:00 qa-power8-3 kernel: [ C24] [c00000002114ba30] [c0000000006ec140] super_cache_scan+0x1a0/0x240
|
|
2025-01-24T11:27:34.768528+01:00 qa-power8-3 kernel: [ C24] [c00000002114baa0] [c0000000005aa458] do_shrink_slab+0x168/0x430
|
|
2025-01-24T11:27:34.768529+01:00 qa-power8-3 kernel: [ C24] [c00000002114bb20] [c0000000005ab708] shrink_slab+0x378/0x560
|
|
2025-01-24T11:27:34.768531+01:00 qa-power8-3 kernel: [ C24] [c00000002114bc00] [c0000000005a665c] shrink_node+0x4fc/0x1070
|
|
2025-01-24T11:27:34.768532+01:00 qa-power8-3 kernel: [ C24] [c00000002114bd00] [c0000000005a79f4] balance_pgdat+0x454/0xbd0
|
|
2025-01-24T11:27:34.768533+01:00 qa-power8-3 kernel: [ C24] [c00000002114beb0] [c0000000005a83f0] kswapd+0x280/0x580
|
|
2025-01-24T11:27:34.768535+01:00 qa-power8-3 kernel: [ C24] [c00000002114bf90] [c00000000025dca4] kthread+0x134/0x140
|
|
2025-01-24T11:27:34.768536+01:00 qa-power8-3 kernel: [ C24] [c00000002114bfe0] [c00000000000de58] start_kernel_thread+0x14/0x18
|
|
2025-01-24T11:27:34.768537+01:00 qa-power8-3 kernel: [ C24] Code: 7d0a5378 f9470000 41820008 f8e80008 7c2004ac e9490000 794a07a4 f9490000 4e800020 60420000 7c210b78 7c421378 <e9490000> 714a0001 4082fff0 4bffff94
|
|
2025-01-24T11:27:58.768462+01:00 qa-power8-3 kernel: [ C24] watchdog: BUG: soft lockup - CPU#24 stuck for 38902s! [kswapd0:905]
|
|
2025-01-24T11:27:58.768474+01:00 qa-power8-3 kernel: [ C24] CPU#24 Utilization every 4s during lockup:
|
|
2025-01-24T11:27:58.768476+01:00 qa-power8-3 kernel: [ C24] #1: 100% system, 0% softirq, 0% hardirq, 0% idle
|
|
2025-01-24T11:27:58.768477+01:00 qa-power8-3 kernel: [ C24] #2: 100% system, 0% softirq, 0% hardirq, 0% idle
|
|
2025-01-24T11:27:58.768478+01:00 qa-power8-3 kernel: [ C24] #3: 101% system, 0% softirq, 0% hardirq, 0% idle
|
|
2025-01-24T11:27:58.768480+01:00 qa-power8-3 kernel: [ C24] #4: 100% system, 0% softirq, 0% hardirq, 0% idle
|
|
2025-01-24T11:27:58.768481+01:00 qa-power8-3 kernel: [ C24] #5: 100% system, 0% softirq, 0% hardirq, 0% idle
|
|
2025-01-24T11:27:58.768482+01:00 qa-power8-3 kernel: [ C24] Modules linked in: nft_masq nft_chain_nat nf_tables af_packet tun bridge stp llc nfnetlink_cttimeout nfnetlink openvswitch nsh nf_conncount nf_nat rfkill nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 psample binfmt_misc aes_gcm_p10_crypto crypto_simd cryptd xts tg3 ses vmx_crypto crct10dif_vpmsum enclosure libphy scsi_transport_sas ipmi_powernv rtc_opal leds_powernv ipmi_devintf uio_pdrv_genirq ipmi_msghandler led_class ibmpowernv powernv_op_panel i2c_opal uio fuse configfs ip_tables x_tables xhci_pci ipr xhci_hcd sd_mod scsi_dh_emc scsi_dh_rdac scsi_dh_alua sg usbcore scsi_mod usb_common scsi_common btrfs blake2b_generic libcrc32c crc32c_vpmsum xor raid6_pq kvm_hv kvm
|
|
2025-01-24T11:27:58.768488+01:00 qa-power8-3 kernel: [ C24] CPU: 24 UID: 0 PID: 905 Comm: kswapd0 Tainted: G W L 6.13.0-1.g1513344-default #1 openSUSE Tumbleweed (unreleased) 067da1dff4e006e28ea31f9175684ad434e86067
|
|
2025-01-24T11:27:58.768490+01:00 qa-power8-3 kernel: [ C24] Tainted: [W]=WARN, [L]=SOFTLOCKUP
|
|
2025-01-24T11:27:58.768491+01:00 qa-power8-3 kernel: [ C24] Hardware name: 8247-22L POWER8E (raw) 0x4b0201 opal:skiboot-v5.4.12 PowerNV
|
|
2025-01-24T11:27:58.768493+01:00 qa-power8-3 kernel: [ C24] NIP: c0000000007129b4 LR: c000000000715298 CTR: c00800000f1c6240
|
|
2025-01-24T11:27:58.768512+01:00 qa-power8-3 kernel: [ C24] REGS: c00000002114b6b0 TRAP: 0900 Tainted: G W L (6.13.0-1.g1513344-default)
|
|
2025-01-24T11:27:58.768515+01:00 qa-power8-3 kernel: [ C24] MSR: 900000000280b033 <SF,HV,VEC,VSX,EE,FP,ME,IR,DR,RI,LE> CR: 48022284 XER: 00000000
|
|
2025-01-24T11:27:58.768516+01:00 qa-power8-3 kernel: [ C24] CFAR: 0000000000000000 IRQMASK: 0
|
|
2025-01-24T11:27:58.768518+01:00 qa-power8-3 kernel: [ C24] GPR00: c000000000716e94 c00000002114b950 c000000001c99200 c000000538434440
|
|
2025-01-24T11:27:58.768519+01:00 qa-power8-3 kernel: [ C24] GPR04: c00000000296d498 c000000000711c80 0000000000000031 fffffffffffe0000
|
|
2025-01-24T11:27:58.768521+01:00 qa-power8-3 kernel: [ C24] GPR08: c000ffffffffffff c0080000035a0000 0000000000000001 0000000000002200
|
|
2025-01-24T11:27:58.768522+01:00 qa-power8-3 kernel: [ C24] GPR12: 0000000000000000 c0000007fffebc80 c00000000025db78 000000000000063d
|
|
2025-01-24T11:27:58.768524+01:00 qa-power8-3 kernel: [ C24] GPR16: c00000005a8afa80 000000000000001f 0000000000000000 0000000000000000
|
|
2025-01-24T11:27:58.768525+01:00 qa-power8-3 kernel: [ C24] GPR20: c000000002996bc0 000000000001f2f0 000000000001eee4 0000000000000000
|
|
2025-01-24T11:27:58.768526+01:00 qa-power8-3 kernel: [ C24] GPR24: 000000000000f591 00000000000042b6 0000000000000047 0000000000000000
|
|
2025-01-24T11:27:58.768528+01:00 qa-power8-3 kernel: [ C24] GPR28: c00000002114ba08 c0000005384344c0 0000000000000001 c000000538434440
|
|
2025-01-24T11:27:58.768529+01:00 qa-power8-3 kernel: [ C24] NIP [c0000000007129b4] ___d_drop+0x94/0xc0
|
|
2025-01-24T11:27:58.768531+01:00 qa-power8-3 kernel: [ C24] LR [c000000000715298] __dentry_kill+0x88/0x310
|
|
2025-01-24T11:27:58.768532+01:00 qa-power8-3 kernel: [ C24] Call Trace:
|
|
2025-01-24T11:27:58.768534+01:00 qa-power8-3 kernel: [ C24] [c00000002114b950] [c00000002114b990] 0xc00000002114b990 (unreliable)
|
|
2025-01-24T11:27:58.768535+01:00 qa-power8-3 kernel: [ C24] [c00000002114b990] [c000000000716e94] shrink_dentry_list+0x84/0x150
|
|
2025-01-24T11:27:58.768537+01:00 qa-power8-3 kernel: [ C24] [c00000002114b9e0] [c00000000071770c] prune_dcache_sb+0x6c/0xa0
|
|
2025-01-24T11:27:58.768538+01:00 qa-power8-3 kernel: [ C24] [c00000002114ba30] [c0000000006ec140] super_cache_scan+0x1a0/0x240
|
|
2025-01-24T11:27:58.768539+01:00 qa-power8-3 kernel: [ C24] [c00000002114baa0] [c0000000005aa458] do_shrink_slab+0x168/0x430
|
|
2025-01-24T11:27:58.768541+01:00 qa-power8-3 kernel: [ C24] [c00000002114bb20] [c0000000005ab708] shrink_slab+0x378/0x560
|
|
2025-01-24T11:27:58.768542+01:00 qa-power8-3 kernel: [ C24] [c00000002114bc00] [c0000000005a665c] shrink_node+0x4fc/0x1070
|
|
2025-01-24T11:27:58.768543+01:00 qa-power8-3 kernel: [ C24] [c00000002114bd00] [c0000000005a79f4] balance_pgdat+0x454/0xbd0
|
|
2025-01-24T11:27:58.768545+01:00 qa-power8-3 kernel: [ C24] [c00000002114beb0] [c0000000005a83f0] kswapd+0x280/0x580
|
|
2025-01-24T11:27:58.768546+01:00 qa-power8-3 kernel: [ C24] [c00000002114bf90] [c00000000025dca4] kthread+0x134/0x140
|
|
2025-01-24T11:27:58.768547+01:00 qa-power8-3 kernel: [ C24] [c00000002114bfe0] [c00000000000de58] start_kernel_thread+0x14/0x18
|
|
2025-01-24T11:27:58.768548+01:00 qa-power8-3 kernel: [ C24] Code: 794a07e0 7d0a5378 f9470000 41820008 f8e80008 7c2004ac e9490000 794a07a4 f9490000 4e800020 60420000 7c210b78 <7c421378> e9490000 714a0001 4082fff0
|
|
2025-01-24T11:28:01.398166+01:00 qa-power8-3 kernel: [ C0] BUG: workqueue lockup - pool cpus=24 node=0 flags=0x0 nice=0 stuck for 41773s!
|
|
2025-01-24T11:28:01.398172+01:00 qa-power8-3 kernel: [ C0] Showing busy workqueues and worker pools:
|
|
2025-01-24T11:28:01.398173+01:00 qa-power8-3 kernel: [ C0] workqueue events: flags=0x0
|
|
2025-01-24T11:28:01.398175+01:00 qa-power8-3 kernel: [ C0] pwq 98: cpus=24 node=0 flags=0x0 nice=0 active=1 refcnt=2
|
|
2025-01-24T11:28:01.398176+01:00 qa-power8-3 kernel: [ C0] pending: drain_local_stock
|
|
2025-01-24T11:28:01.398187+01:00 qa-power8-3 kernel: [ C0] workqueue events_power_efficient: flags=0x80
|
|
2025-01-24T11:28:01.398188+01:00 qa-power8-3 kernel: [ C0] pwq 98: cpus=24 node=0 flags=0x0 nice=0 active=2 refcnt=3
|
|
2025-01-24T11:28:01.398189+01:00 qa-power8-3 kernel: [ C0] pending: neigh_managed_work, neigh_periodic_work
|
|
2025-01-24T11:28:01.398191+01:00 qa-power8-3 kernel: [ C0] workqueue mm_percpu_wq: flags=0x8
|
|
2025-01-24T11:28:01.398192+01:00 qa-power8-3 kernel: [ C0] pwq 98: cpus=24 node=0 flags=0x0 nice=0 active=1 refcnt=2
|
|
2025-01-24T11:28:01.398194+01:00 qa-power8-3 kernel: [ C0] pending: vmstat_update
|
|
2025-01-24T11:28:01.398195+01:00 qa-power8-3 kernel: [ C0] workqueue iou_exit: flags=0x2
|
|
2025-01-24T11:28:01.398196+01:00 qa-power8-3 kernel: [ C0] pwq 642: cpus=0-159 node=0 flags=0x4 nice=0 active=1 refcnt=2
|
|
2025-01-24T11:28:01.398198+01:00 qa-power8-3 kernel: [ C0] in-flight: 148824:io_ring_exit_work
|
|
2025-01-24T11:28:01.398199+01:00 qa-power8-3 kernel: [ C0] pwq 648: cpus=0-159 node=1 flags=0x4 nice=0 active=3 refcnt=4
|
|
2025-01-24T11:28:01.398200+01:00 qa-power8-3 kernel: [ C0] in-flight: 118070:io_ring_exit_work ,127177:io_ring_exit_work ,129527:io_ring_exit_work
|
|
2025-01-24T11:28:01.398202+01:00 qa-power8-3 kernel: [ C0] pwq 649: cpus=0-159 node=1 flags=0x4 nice=0 active=2 refcnt=3
|
|
2025-01-24T11:28:01.398203+01:00 qa-power8-3 kernel: [ C0] in-flight: 130631:io_ring_exit_work ,129460:io_ring_exit_work
|
|
2025-01-24T11:28:01.398204+01:00 qa-power8-3 kernel: [ C0] pwq 650: cpus=0-159 node=1 flags=0x4 nice=0 active=5 refcnt=6
|
|
2025-01-24T11:28:01.398205+01:00 qa-power8-3 kernel: [ C0] in-flight: 127213:io_ring_exit_work ,127522:io_ring_exit_work ,127521:io_ring_exit_work ,147689:io_ring_exit_work ,147685:io_ring_exit_work
|
|
2025-01-24T11:28:01.398207+01:00 qa-power8-3 kernel: [ C0] pwq 651: cpus=0-159 node=16 flags=0x4 nice=0 active=7 refcnt=13
|
|
2025-01-24T11:28:01.398208+01:00 qa-power8-3 kernel: [ C0] in-flight: 153752:io_ring_exit_work ,152496:io_ring_exit_work ,152844:io_ring_exit_work ,150270:io_ring_exit_work ,153906:io_ring_exit_work ,152861:io_ring_exit_work ,152074:io_ring_exit_work
|
|
2025-01-24T11:28:01.398209+01:00 qa-power8-3 kernel: [ C0] inactive: 5*io_ring_exit_work
|
|
2025-01-24T11:28:01.398211+01:00 qa-power8-3 kernel: [ C0] pwq 652: cpus=0-159 node=16 flags=0x4 nice=0 active=1 refcnt=7
|
|
2025-01-24T11:28:01.398212+01:00 qa-power8-3 kernel: [ C0] in-flight: 151346:io_ring_exit_work
|
|
2025-01-24T11:28:01.398213+01:00 qa-power8-3 kernel: [ C0] inactive: 5*io_ring_exit_work
|
|
2025-01-24T11:28:01.398214+01:00 qa-power8-3 kernel: [ C0] pwq 653: cpus=0-159 node=16 flags=0x4 nice=0 active=1 refcnt=7
|
|
2025-01-24T11:28:01.398216+01:00 qa-power8-3 kernel: [ C0] in-flight: 147308:io_ring_exit_work
|
|
2025-01-24T11:28:01.398217+01:00 qa-power8-3 kernel: [ C0] inactive: 5*io_ring_exit_work
|
|
2025-01-24T11:28:01.398218+01:00 qa-power8-3 kernel: [ C0] pwq 654: cpus=0-159 node=16 flags=0x4 nice=0 active=5 refcnt=10
|
|
2025-01-24T11:28:01.398219+01:00 qa-power8-3 kernel: [ C0] in-flight: 148805:io_ring_exit_work ,151928:io_ring_exit_work ,152824:io_ring_exit_work ,152829:io_ring_exit_work ,152827:io_ring_exit_work
|
|
2025-01-24T11:28:01.398221+01:00 qa-power8-3 kernel: [ C0] inactive: 4*io_ring_exit_work
|
|
2025-01-24T11:28:01.398222+01:00 qa-power8-3 kernel: [ C0] pwq 655: cpus=0-159 node=16 flags=0x4 nice=0 active=2 refcnt=5
|
|
2025-01-24T11:28:01.398223+01:00 qa-power8-3 kernel: [ C0] in-flight: 152591:io_ring_exit_work ,151754:io_ring_exit_work
|
|
2025-01-24T11:28:01.398225+01:00 qa-power8-3 kernel: [ C0] inactive: 2*io_ring_exit_work
|
|
2025-01-24T11:28:01.398226+01:00 qa-power8-3 kernel: [ C0] pwq 656: cpus=0-159 node=17 flags=0x4 nice=0 active=5 refcnt=6
|
|
2025-01-24T11:28:01.398227+01:00 qa-power8-3 kernel: [ C0] in-flight: 147303:io_ring_exit_work ,130347:io_ring_exit_work ,130346:io_ring_exit_work ,129985:io_ring_exit_work ,147256:io_ring_exit_work
|
|
2025-01-24T11:28:01.398228+01:00 qa-power8-3 kernel: [ C0] pwq 658: cpus=0-159 node=17 flags=0x4 nice=0 active=1 refcnt=2
|
|
2025-01-24T11:28:01.398230+01:00 qa-power8-3 kernel: [ C0] in-flight: 147299:io_ring_exit_work
|
|
2025-01-24T11:28:01.398231+01:00 qa-power8-3 kernel: [ C0] pwq 659: cpus=0-159 node=17 flags=0x4 nice=0 active=4 refcnt=5
|
|
2025-01-24T11:28:01.398232+01:00 qa-power8-3 kernel: [ C0] in-flight: 127097:io_ring_exit_work ,130719:io_ring_exit_work ,129372:io_ring_exit_work ,147458:io_ring_exit_work
|
|
2025-01-24T11:28:01.398233+01:00 qa-power8-3 kernel: [ C0] pwq 660: cpus=0-159 node=17 flags=0x4 nice=0 active=1 refcnt=2
|
|
2025-01-24T11:28:01.398235+01:00 qa-power8-3 kernel: [ C0] in-flight: 147318:io_ring_exit_work
|
|
2025-01-24T11:28:01.398237+01:00 qa-power8-3 kernel: [ C0] pool 642: cpus=0-159 node=0 flags=0x4 nice=0 hung=0s workers=4 idle: 127904 154042 154043
|
|
2025-01-24T11:28:01.398238+01:00 qa-power8-3 kernel: [ C0] pool 648: cpus=0-159 node=1 flags=0x4 nice=0 hung=0s workers=5 idle: 147825 148763
|
|
2025-01-24T11:28:01.398239+01:00 qa-power8-3 kernel: [ C0] pool 649: cpus=0-159 node=1 flags=0x4 nice=0 hung=0s workers=4 idle: 154165 154242
|
|
2025-01-24T11:28:01.398240+01:00 qa-power8-3 kernel: [ C0] pool 650: cpus=0-159 node=1 flags=0x4 nice=0 hung=0s workers=8 idle: 147859 147693 147954
|
|
2025-01-24T11:28:01.398242+01:00 qa-power8-3 kernel: [ C0] pool 651: cpus=0-159 node=16 flags=0x4 nice=0 hung=0s workers=11 idle: 154920 153982 154915 155388
|
|
2025-01-24T11:28:01.398243+01:00 qa-power8-3 kernel: [ C0] pool 652: cpus=0-159 node=16 flags=0x4 nice=0 hung=0s workers=4 idle: 155967 156273 152855
|
|
2025-01-24T11:28:01.398244+01:00 qa-power8-3 kernel: [ C0] pool 653: cpus=0-159 node=16 flags=0x4 nice=0 hung=0s workers=7 idle: 156591 153489 156259 154759 156257 155714
|
|
2025-01-24T11:28:01.398246+01:00 qa-power8-3 kernel: [ C0] pool 654: cpus=0-159 node=16 flags=0x4 nice=0 hung=0s workers=9 idle: 156385 155190 156377 152834
|
|
2025-01-24T11:28:01.398247+01:00 qa-power8-3 kernel: [ C0] pool 655: cpus=0-159 node=16 flags=0x4 nice=0 hung=0s workers=4 idle: 154694 156401
|
|
2025-01-24T11:28:01.398249+01:00 qa-power8-3 kernel: [ C0] pool 656: cpus=0-159 node=17 flags=0x4 nice=0 hung=0s workers=9 idle: 155382 149795 148819 155419
|
|
2025-01-24T11:28:01.398250+01:00 qa-power8-3 kernel: [ C0] pool 658: cpus=0-159 node=17 flags=0x4 nice=0 hung=0s workers=5 idle: 156532 156175 154798 155872
|
|
2025-01-24T11:28:01.398252+01:00 qa-power8-3 kernel: [ C0] pool 659: cpus=0-159 node=17 flags=0x4 nice=0 hung=0s workers=7 idle: 155771 155547 153308
|
|
2025-01-24T11:28:01.398253+01:00 qa-power8-3 kernel: [ C0] pool 660: cpus=0-159 node=17 flags=0x4 nice=0 hung=0s workers=6 idle: 156421 155079 147520 156018 153416
|
|
2025-01-24T11:28:01.398255+01:00 qa-power8-3 kernel: [ C0] Showing backtraces of running workers in stalled CPU-bound worker pools:
|
|
|
