Project

General

Profile

Actions
Copy link

action #160487

closed

Failed systemd services alert (unreal6 kdump) size:S

Added by tinita 7 months ago. Updated 7 months ago.

Status:
Resolved
Priority:
Urgent
Assignee:
dheidler
Category:
Regressions/Crashes
Target version:
openQA Project (public) - Ready
Start date:
2024-05-17
Due date:
% Done:

0%

Estimated time:
Tags:
alert, infra, reactive work

Description

Observation

2024-05-17 10:59:50 unreal6 kdump
https://stats.openqa-monitor.qa.suse.de/d/KToPYLEWz/failed-systemd-services?orgId=1

Suggestions

  • DONE Check on unreal6 journalctl -u kdump (it's empty, due to #160655)
  • Lookup older related ticket or email response, possibly "backup-qam"
  • Try it out by starting the service multiple times and multiple reboots
  • Apply mitigations, retries, monitor, report upstream, etc.

Related issues 1 (1 open — 0 closed)

Copied to openQA Infrastructure (public) - action #160655: No persistent journal on unreal6New2024-05-17

Actions
Actions
Copy link
 #1

Updated by okurz 7 months ago

  • Priority changed from High to Urgent
Actions
Copy link
 #2

Updated by okurz 7 months ago

  • Tags set to infra, alert, reactive work
Actions
Copy link
 #3

Updated by okurz 7 months ago

Actions
Copy link
 #4

Updated by livdywan 7 months ago

  • Subject changed from Failed systemd services alert (unreal6 kdump) to Failed systemd services alert (unreal6 kdump) size:S
  • Description updated (diff)
  • Status changed from New to Workable
Actions
Copy link
 #5

Updated by dheidler 7 months ago

  • Status changed from Workable to In Progress
  • Assignee set to dheidler
Actions
Copy link
 #6

Updated by dheidler 7 months ago

It seems that around 2024-05-09 unreal6 was upgraded to Leap 15.6.

Actions
Copy link
 #7

Updated by dheidler 7 months ago 

2024-05-17T03:32:24.281360+02:00 unreal6 sh[18901]: (537/566) Installing: kdump-2.0.3+git10.gfdb71b2-150600.1.2.x86_64 [...
2024-05-17T03:32:24.281604+02:00 unreal6 sh[18901]: Updating /etc/sysconfig/kdump ...
2024-05-17T03:32:24.334296+02:00 unreal6 systemd[1]: Reloading requested from client PID 18611 ('systemctl') (unit auto-upgrade.service)...
2024-05-17T03:32:24.334360+02:00 unreal6 systemd[1]: Reloading...
2024-05-17T03:32:24.924343+02:00 unreal6 systemd[1]: Reloading finished in 589 ms.
2024-05-17T03:32:24.961835+02:00 unreal6 systemd[1]: Reloading requested from client PID 18656 ('systemctl') (unit auto-upgrade.service)...
2024-05-17T03:32:24.961918+02:00 unreal6 systemd[1]: Reloading...
2024-05-17T03:32:25.630446+02:00 unreal6 systemd[1]: Reloading finished in 667 ms.
2024-05-17T03:32:25.666952+02:00 unreal6 systemd[1]: Reloading requested from client PID 18704 ('systemctl') (unit auto-upgrade.service)...
2024-05-17T03:32:25.667019+02:00 unreal6 systemd[1]: Reloading...
2024-05-17T03:32:26.369634+02:00 unreal6 systemd[1]: Reloading finished in 702 ms.
2024-05-17T03:32:26.439702+02:00 unreal6 sh[18901]: Removed "/etc/systemd/system/multi-user.target.wants/kdump.service".
2024-05-17T03:32:26.439788+02:00 unreal6 systemd[1]: Reloading requested from client PID 18754 ('systemctl') (unit auto-upgrade.service)...
2024-05-17T03:32:26.439877+02:00 unreal6 systemd[1]: Reloading...
2024-05-17T03:32:27.110180+02:00 unreal6 systemd[1]: Reloading finished in 669 ms.
2024-05-17T03:32:27.144729+02:00 unreal6 [RPM][18604]: install kdump-2.0.3+git10.gfdb71b2-150600.1.2.x86_64: success
2024-05-17T03:32:27.149403+02:00 unreal6 sh[18901]: Removed "/etc/systemd/system/multi-user.target.wants/kdump-early.service".
2024-05-17T03:32:27.149452+02:00 unreal6 sh[18901]: Removed "/etc/systemd/system/multi-user.target.wants/kdump-notify.service".
2024-05-17T03:32:27.149507+02:00 unreal6 sh[18901]: Created symlink /etc/systemd/system/multi-user.target.wants/kdump.service -> /usr/lib/systemd/system/kdump.service.
2024-05-17T03:32:27.149542+02:00 unreal6 sh[18901]: Created symlink /etc/systemd/system/multi-user.target.wants/kdump-early.service -> /usr/lib/systemd/system/kdump-early.service.
2024-05-17T03:32:27.149574+02:00 unreal6 sh[18901]: Created symlink /etc/systemd/system/multi-user.target.wants/kdump-notify.service -> /usr/lib/systemd/system/kdump-notify.service.
2024-05-17T03:32:27.149596+02:00 unreal6 sh[18901]: Stopping kdump ...
2024-05-17T03:32:27.169840+02:00 unreal6 systemd[1]: Reloading requested from client PID 18804 ('systemctl') (unit auto-upgrade.service)...
2024-05-17T03:32:27.169906+02:00 unreal6 systemd[1]: Reloading...
2024-05-17T03:32:27.792978+02:00 unreal6 systemd[1]: Reloading finished in 622 ms.
2024-05-17T03:32:27.854093+02:00 unreal6 systemd[1]: Stopping Load kdump kernel and initrd...
2024-05-17T03:32:27.872108+02:00 unreal6 unload.sh[18863]: config option KDUMP_COPY_KERNEL is deprecated, ignoring
2024-05-17T03:32:27.896361+02:00 unreal6 systemd[1]: kdump.service: Deactivated successfully.
2024-05-17T03:32:27.930224+02:00 unreal6 systemd[1]: Stopped Load kdump kernel and initrd.
2024-05-17T03:32:27.978698+02:00 unreal6 systemd[1]: Starting Load kdump kernel and initrd...
2024-05-17T03:32:27.994187+02:00 unreal6 load.sh[18867]: config option KDUMP_COPY_KERNEL is deprecated, ignoring
2024-05-17T03:32:30.191833+02:00 unreal6 auditd[826]: Audit daemon rotating log files
2024-05-17T03:32:49.966979+02:00 unreal6 load.sh[22533]: console=com3 in Xen commandline not handled
2024-05-17T03:32:49.980421+02:00 unreal6 load.sh[18867]: Starting kdump kernel load; kexec cmdline: /sbin/kexec -p /var/lib/kdump/kernel --append=" console=ttyS2,115200n8 sysrq=yes reset_devices acpi_no_memhotplug cgroup_disable=memory nokaslr numa=off irqpoll nr_cpus=1 root=kdump rootflags=bind rd.udev.children-max=8 disable_cpu_apicid=0   panic=1" --initrd=/var/lib/kdump/initrd  -a
2024-05-17T03:32:50.552928+02:00 unreal6 kernel: [300981.080213][T22543] kexec: page allocation failure: order:4, mode:0x40dc0(GFP_KERNEL|__GFP_COMP|__GFP_ZERO), nodemask=(null),cpuset=/,mems_allowed=0
2024-05-17T03:32:50.552957+02:00 unreal6 kernel: [300981.093661][T22543] CPU: 5 PID: 22543 Comm: kexec Tainted: G                   n 6.4.0-150600.17-default #1 SLE15-SP6 8d7122a8d4d10b24a1d15f496e70a80423123bfb
2024-05-17T03:32:50.686496+02:00 unreal6 kernel: [300981.107928][T22543] Hardware name: Supermicro X10SLD-F/HF/X10SLD, BIOS 3.2 05/10/2018
2024-05-17T03:32:50.686507+02:00 unreal6 kernel: [300981.107930][T22543] Call Trace:
2024-05-17T03:32:50.686508+02:00 unreal6 kernel: [300981.107932][T22543]  <TASK>
2024-05-17T03:32:50.686509+02:00 unreal6 kernel: [300981.107936][T22543]  dump_stack_lvl+0x44/0x60
2024-05-17T03:32:50.686509+02:00 unreal6 kernel: [300981.107942][T22543]  warn_alloc+0x116/0x190
2024-05-17T03:32:50.686521+02:00 unreal6 kernel: [300981.107946][T22543]  __alloc_pages_slowpath.constprop.76+0xd21/0xda0
2024-05-17T03:32:50.686522+02:00 unreal6 kernel: [300981.107949][T22543]  ? mas_alloc_nodes+0x58/0x200
2024-05-17T03:32:50.686523+02:00 unreal6 kernel: [300981.142000][T22543]  __alloc_pages+0x306/0x350
2024-05-17T03:32:50.686523+02:00 unreal6 kernel: [300981.142018][T22543]  ? privcmd_buf_mmap+0x40/0x140 [xen_privcmd 934db45836cbae60a50a9c21606f8d0a484d9c16]
2024-05-17T03:32:50.686524+02:00 unreal6 kernel: [300981.142021][T22543]  __kmalloc_large_node+0x7a/0x140
2024-05-17T03:32:50.686525+02:00 unreal6 kernel: [300981.142023][T22543]  __kmalloc+0xbe/0x130
2024-05-17T03:32:50.686532+02:00 unreal6 kernel: [300981.142025][T22543]  privcmd_buf_mmap+0x40/0x140 [xen_privcmd 934db45836cbae60a50a9c21606f8d0a484d9c16]
2024-05-17T03:32:50.686535+02:00 unreal6 kernel: [300981.142027][T22543]  mmap_region+0x26c/0xa70
2024-05-17T03:32:50.686535+02:00 unreal6 kernel: [300981.179256][T22543]  do_mmap+0x3c4/0x550
2024-05-17T03:32:50.686536+02:00 unreal6 kernel: [300981.179262][T22543]  vm_mmap_pgoff+0xe1/0x1a0
2024-05-17T03:32:50.686537+02:00 unreal6 kernel: [300981.179266][T22543]  ksys_mmap_pgoff+0x1a1/0x1e0
2024-05-17T03:32:50.686537+02:00 unreal6 kernel: [300981.179268][T22543]  do_syscall_64+0x5b/0x80
2024-05-17T03:32:50.686538+02:00 unreal6 kernel: [300981.179271][T22543]  ? syscall_exit_to_user_mode+0x1e/0x40
2024-05-17T03:32:50.686547+02:00 unreal6 kernel: [300981.179273][T22543]  ? do_syscall_64+0x67/0x80
2024-05-17T03:32:50.686549+02:00 unreal6 kernel: [300981.179274][T22543]  ? irq_exit_rcu+0x40/0xc0
2024-05-17T03:32:50.686550+02:00 unreal6 kernel: [300981.211457][T22543]  ? xen_pv_evtchn_do_upcall+0x8f/0xa0
2024-05-17T03:32:50.686550+02:00 unreal6 kernel: [300981.211471][T22543]  entry_SYSCALL_64_after_hwframe+0x77/0xe1
2024-05-17T03:32:50.686551+02:00 unreal6 kernel: [300981.211474][T22543] RIP: 0033:0x7ff96012a5f2
2024-05-17T03:32:50.686552+02:00 unreal6 kernel: [300981.227143][T22543] Code: 90 90 90 90 90 90 90 90 41 f7 c1 ff 0f 00 00 75 27 55 89 cd 53 48 89 fb 48 85 ff 74 3b 41 89 ea 48 89 df b8 09 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 76 5b 5d c3 0f 1f 00 48 8b 05 f9 47 0d 00 64
2024-05-17T03:32:50.686558+02:00 unreal6 kernel: [300981.227145][T22543] RSP: 002b:00007fffe14c8538 EFLAGS: 00000246 ORIG_RAX: 0000000000000009
2024-05-17T03:32:50.686559+02:00 unreal6 kernel: [300981.227157][T22543] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff96012a5f2
2024-05-17T03:32:50.686559+02:00 unreal6 kernel: [300981.227158][T22543] RDX: 0000000000000003 RSI: 00000000012ea000 RDI: 0000000000000000
2024-05-17T03:32:50.686560+02:00 unreal6 kernel: [300981.227159][T22543] RBP: 0000000000000001 R08: 0000000000000004 R09: 0000000000000000
2024-05-17T03:32:50.686561+02:00 unreal6 kernel: [300981.227159][T22543] R10: 0000000000000001 R11: 0000000000000246 R12: 00000000000012ea
2024-05-17T03:32:50.686561+02:00 unreal6 kernel: [300981.227160][T22543] R13: 000055d5af63eb20 R14: 000055d5af63f480 R15: 000055d5af63fa80
2024-05-17T03:32:50.686566+02:00 unreal6 kernel: [300981.227163][T22543]  </TASK>
2024-05-17T03:32:50.686576+02:00 unreal6 kernel: [300981.227246][T22543] Mem-Info:
2024-05-17T03:32:50.806334+02:00 unreal6 kernel: [300981.301023][T22543] active_anon:27322 inactive_anon:41870 isolated_anon:0
2024-05-17T03:32:50.806343+02:00 unreal6 kernel: [300981.301023][T22543]  active_file:26410 inactive_file:35814 isolated_file:0
2024-05-17T03:32:50.806344+02:00 unreal6 kernel: [300981.301023][T22543]  unevictable:16761 dirty:196 writeback:0
2024-05-17T03:32:50.806345+02:00 unreal6 kernel: [300981.301023][T22543]  slab_reclaimable:22564 slab_unreclaimable:25083
2024-05-17T03:32:50.806365+02:00 unreal6 kernel: [300981.301023][T22543]  mapped:15869 shmem:1831 pagetables:5681
2024-05-17T03:32:50.806368+02:00 unreal6 kernel: [300981.301023][T22543]  sec_pagetables:0 bounce:0
2024-05-17T03:32:50.806368+02:00 unreal6 kernel: [300981.301023][T22543]  kernel_misc_reclaimable:0
2024-05-17T03:32:50.806369+02:00 unreal6 kernel: [300981.301023][T22543]  free:11073 free_pcp:149 free_cma:0
2024-05-17T03:32:50.806370+02:00 unreal6 kernel: [300981.301027][T22543] Node 0 active_anon:109288kB inactive_anon:167480kB active_file:105640kB inactive_file:143256kB unevictable:67044kB isolated(anon):0kB isolated(file):0kB mapped:63476kB dirty:784kB writeback:0kB shmem:7324kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB kernel_stack:10260kB pagetables:22724kB sec_pagetables:0kB all_unreclaimable? no
2024-05-17T03:32:50.806370+02:00 unreal6 kernel: [300981.301030][T22543] Node 0 DMA free:3464kB boost:0kB min:64kB low:80kB high:96kB reserved_highatomic:0KB active_anon:1056kB inactive_anon:1960kB active_file:1272kB inactive_file:1300kB unevictable:12kB writepending:0kB present:15968kB managed:15908kB mlocked:12kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
2024-05-17T03:32:50.806377+02:00 unreal6 kernel: [300981.301034][T22543] lowmem_reserve[]: 0 849 849 849 849
2024-05-17T03:32:50.806378+02:00 unreal6 kernel: [300981.301036][T22543] Node 0 DMA32 free:40828kB boost:0kB min:3696kB low:4620kB high:5544kB reserved_highatomic:2048KB active_anon:108348kB inactive_anon:165472kB active_file:104448kB inactive_file:141932kB unevictable:67032kB writepending:720kB present:1032604kB managed:898080kB mlocked:63960kB bounce:0kB free_pcp:600kB local_pcp:0kB free_cma:0kB
2024-05-17T03:32:50.806379+02:00 unreal6 kernel: [300981.301040][T22543] lowmem_reserve[]: 0 0 0 0 0
2024-05-17T03:32:50.908415+02:00 unreal6 load.sh[22543]: xencall: error: alloc_pages: mmap (,4842*4096,...) [bufdev] failed: Cannot allocate memory
2024-05-17T03:32:50.908545+02:00 unreal6 load.sh[22543]: kexec_load failed: Cannot allocate memory
2024-05-17T03:32:50.908589+02:00 unreal6 load.sh[22543]: entry       = 0x41a49f740 flags = 0x3e0001
2024-05-17T03:32:50.908627+02:00 unreal6 load.sh[22543]: nr_segments = 7
2024-05-17T03:32:50.908663+02:00 unreal6 load.sh[22543]: segment[0].buf   = 0x55d5af5902a0
2024-05-17T03:32:50.908695+02:00 unreal6 load.sh[22543]: segment[0].bufsz = 0x30
2024-05-17T03:32:50.908729+02:00 unreal6 load.sh[22543]: segment[0].mem   = 0x40d346000
2024-05-17T03:32:50.908762+02:00 unreal6 load.sh[22543]: segment[0].memsz = 0x1000
2024-05-17T03:32:50.908796+02:00 unreal6 load.sh[22543]: segment[1].buf   = 0x7ff95df8d010
2024-05-17T03:32:50.908831+02:00 unreal6 load.sh[22543]: segment[1].bufsz = 0x12e94dc
2024-05-17T03:32:50.908863+02:00 unreal6 load.sh[22543]: segment[1].mem   = 0x415716000
2024-05-17T03:32:50.908908+02:00 unreal6 load.sh[22543]: segment[1].memsz = 0x12ea000
2024-05-17T03:32:50.908944+02:00 unreal6 load.sh[22543]: segment[2].buf   = 0x7ff95f27b010
2024-05-17T03:32:50.908978+02:00 unreal6 load.sh[22543]: segment[2].bufsz = 0xd841e0
2024-05-17T03:32:50.909013+02:00 unreal6 load.sh[22543]: segment[2].mem   = 0x416a00000
2024-05-17T03:32:50.909048+02:00 unreal6 load.sh[22543]: segment[2].memsz = 0x39d8000
2024-05-17T03:32:50.909086+02:00 unreal6 load.sh[22543]: segment[3].buf   = 0x55d5af63a7c0
2024-05-17T03:32:50.909119+02:00 unreal6 load.sh[22543]: segment[3].bufsz = 0x4121
2024-05-17T03:32:50.909148+02:00 unreal6 load.sh[22543]: segment[3].mem   = 0x41a49a000
2024-05-17T03:32:50.909179+02:00 unreal6 load.sh[22543]: segment[3].memsz = 0x5000
2024-05-17T03:32:50.909210+02:00 unreal6 load.sh[22543]: segment[4].buf   = 0x55d5af6334b0
2024-05-17T03:32:50.909241+02:00 unreal6 load.sh[22543]: segment[4].bufsz = 0x70e0
2024-05-17T03:32:50.909272+02:00 unreal6 load.sh[22543]: segment[4].mem   = 0x41a49f000
2024-05-17T03:32:50.909346+02:00 unreal6 kernel: [300981.301042][T22543] Node 0 DMA: 40*4kB (UME) 29*8kB (UME) 36*16kB (UE) 14*32kB (UME) 4*64kB (UME) 4*128kB (UE) 3*256kB (U) 1*512kB (U) 0*1024kB 0*2048kB 0*4096kB = 3464kB
2024-05-17T03:32:50.909351+02:00 unreal6 kernel: [300981.448605][T22543] Node 0 DMA32: 7108*4kB (UMEH) 1421*8kB (UMEH) 50*16kB (UMH) 3*32kB (H) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 40696kB
2024-05-17T03:32:50.909364+02:00 unreal6 kernel: [300981.448612][T22543] 72778 total pagecache pages
2024-05-17T03:32:50.909367+02:00 unreal6 kernel: [300981.448613][T22543] 4164 pages in swap cache
2024-05-17T03:32:50.909368+02:00 unreal6 kernel: [300981.448614][T22543] Free swap  = 1067936kB
2024-05-17T03:32:50.909369+02:00 unreal6 kernel: [300981.448614][T22543] Total swap = 2099648kB
2024-05-17T03:32:50.909375+02:00 unreal6 kernel: [300981.448615][T22543] 262143 pages RAM
2024-05-17T03:32:50.909376+02:00 unreal6 kernel: [300981.448616][T22543] 0 pages HighMem/MovableOnly
2024-05-17T03:32:50.909382+02:00 unreal6 kernel: [300981.448616][T22543] 33646 pages reserved
2024-05-17T03:32:50.909383+02:00 unreal6 kernel: [300981.448617][T22543] 0 pages cma reserved
2024-05-17T03:32:50.909384+02:00 unreal6 kernel: [300981.448617][T22543] 0 pages hwpoisoned
2024-05-17T03:32:50.909328+02:00 unreal6 load.sh[22543]: segment[4].memsz = 0x9000
2024-05-17T03:32:50.909407+02:00 unreal6 load.sh[22543]: segment[5].buf   = 0x55d5af6315e0
2024-05-17T03:32:50.909444+02:00 unreal6 load.sh[22543]: segment[5].bufsz = 0x800
2024-05-17T03:32:50.909475+02:00 unreal6 load.sh[22543]: segment[5].mem   = 0x41a4a8000
2024-05-17T03:32:50.909505+02:00 unreal6 load.sh[22543]: segment[5].memsz = 0x4000
2024-05-17T03:32:50.909537+02:00 unreal6 load.sh[22543]: segment[6].buf   = 0x55d5af5979d0
2024-05-17T03:32:50.909568+02:00 unreal6 load.sh[22543]: segment[6].bufsz = 0x99c00
2024-05-17T03:32:50.909596+02:00 unreal6 load.sh[22543]: segment[6].mem   = 0x41a4ac000
2024-05-17T03:32:50.909625+02:00 unreal6 load.sh[22543]: segment[6].memsz = 0x9a000
2024-05-17T03:32:50.912398+02:00 unreal6 load.sh[18867]: kexec failed.
2024-05-17T03:32:50.916526+02:00 unreal6 systemd[1]: kdump.service: Main process exited, code=exited, status=255/EXCEPTION
2024-05-17T03:32:50.916594+02:00 unreal6 systemd[1]: kdump.service: Failed with result 'exit-code'.
2024-05-17T03:32:50.941787+02:00 unreal6 systemd[1]: Failed to start Load kdump kernel and initrd.
2024-05-17T03:32:50.942145+02:00 unreal6 systemd[1]: kdump.service: Consumed 20.770s CPU time.
2024-05-17T03:32:50.978629+02:00 unreal6 sh[18901]: ..
2024-05-17T03:32:50.978699+02:00 unreal6 sh[18901]: Job for kdump.service failed because the control process exited with error code.
Actions
Copy link
 #8

Updated by dheidler 7 months ago

Looks like as if the memory was full. The DOM-0 has only 892MB.

2024-05-17T03:32:50.908415+02:00 unreal6 load.sh[22543]: xencall: error: alloc_pages: mmap (,4842*4096,...) [bufdev] failed: Cannot allocate memory
2024-05-17T03:32:50.908545+02:00 unreal6 load.sh[22543]: kexec_load failed: Cannot allocate memory
Actions
Copy link
 #9

Updated by dheidler 7 months ago

I opened https://bugzilla.suse.com/show_bug.cgi?id=1224805 just to be sure that this is not a product bug.

Actions
Copy link
 #10

Updated by dheidler 7 months ago ยท Edited

/etc/default/grub:

-GRUB_CMDLINE_XEN_DEFAULT="console=com3 com3=115200 dom0_mem=1024M,max:1024M loglvl=all guest_loglvl=all loglvl=all guest_loglvl=all"
+GRUB_CMDLINE_XEN_DEFAULT="console=com3 com3=115200 dom0_mem=3072M,max:3072M loglvl=all guest_loglvl=all loglvl=all guest_loglvl=all"

update-bootloader

Actions
Copy link
 #11

Updated by dheidler 7 months ago 

dheidler@unreal6:~> free -m
               total        used        free      shared  buff/cache   available
Mem:            2895         520        1714          11         733        2375
Swap:           2050           0        2050

# xl list
Name                                        ID   Mem VCPUs  State   Time(s)
Domain-0                                     0  3072     8     r-----     151.1
Xenstore                                     1    32     1     -b----       0.0

The DOM-0 should have sufficient memory now.

Actions
Copy link
 #12

Updated by dheidler 7 months ago

  • Status changed from In Progress to Resolved
Actions
Copy link

Also available in: Atom PDF