QA » openQA Project » openQA Tests

The machine is already configured for 4GB of RAM.

I think there are already product bugs: http://fastzilla.suse.de/?q=kdump+OOM+openQA returns some. I thought there would be more specific ones. Maybe http://fastzilla.suse.de/?q=kdump+openQA is a better search.

bsc#1039527 "[Build 0387] kdump endup with "Out of memory" error" looks like the best candidate. At least related looks https://bugzilla.suse.com/show_bug.cgi?id=1075937 . https://bugzilla.suse.com/show_bug.cgi?id=998544 is an older one about general instability of the test/component. But now I found https://bugzilla.suse.com/show_bug.cgi?id=1075945 which is describing what looks like the same problem. The bug is marked as RESOLVED FIXED so not verified. Maybe the submission does not fix it or never made it to SLE15.

https://bugzilla.suse.com/show_bug.cgi?id=1070397 is also about OOM in kdump. https://bugzilla.suse.com/show_bug.cgi?id=986196 is also about memory and kdump, getting old but still open.

Btw, I am not agreeing with the suggestion in general. It might be necessary here but we should be careful to not hide product regressions.

According with Peter, he has been able to save dumps on VMs with only 2G of RAM, approx. 130M reserved for kdump. 3x refers to refers to user-space runtime requirements, not to the total RAM requirements. I crosschecked other scenarios, and even with 4 modules installed like in this case, is not necessary to have more than 2GB or 4GB. Increase the memory here seems to be to hide some error here for ppc.

Very good evaluation. Thank you.

It could be that there actually are too many unexpected applications running in the background in the test, depending on what the previous test modules did within that openQA jobs that failed. Our post_fail_hook in many cases include a process table being collected. https://openqa.suse.de/tests/1549130#step/kdump_and_crash/43 shows another problem on top which is that the post_fail_hook does not execute successfully because the log console tty, tty5, is not found. If you have another example of a job failing maybe you can gather the data from there.

This ticket should be related to other kdump tests and mloviska is also working on one. Of course it might be different issues but why not work together on one first and then pick up the next one?

We are working together in both issues. We think that the post_fail_hook does not work due to the moment when it happens, during the creation of the dump, so makes sense not to be able have logs as in this moment we are creating the dump which is critical for the system. No more jobs with info was found, but what is clear is that even when is checking the status of the service, the memory issue can be read there, before the dump. Added a couple of checking of the memory with this pull request:
https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/4709 so we can know where could be a potential leak.

In fact I've just realized that available memory is displayed in the test in YaST: https://openqa.suse.de/tests/1574541#step/kdump_and_crash/24 but that is the system memory, my best guess is that the memory we need to increment is the one for kdump on previous screen as a workaround, but only for ppc and probably open a bug.

[WIP] PR: https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/4758
256MB didn't work, trying with 512MB.

It didn't work and I can still reproduce the memory issue. I found in https://www.suse.com/support/kb/doc/?id=3374462 that "For the PPC64 architecture: crashkernel=128M@32M" I could try to add to boot options.

Boot params are volatile after reboot so it didn't help.
After trying several approachs, starting with a fresh testing by reproducing it with the latest build. Latest shows that the dumps are created, the memory issue is present when checking the status of the service, but the tool that is failing is the one for reading the dump (not the one to created): crash utility. Trying to reproduce it with my remote worker, but it is failing in the creation of the dump. Possible explanation could be based on free -m output that the machine does not have enough physical RAM (is almost fully consuming its 5GB), could it be the same cause on the osd worker?

Not sure if it's in feedback, or how we should proceed here. Let's discuss.

PR: https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/4984

PRs to fix typo 1 and type 2.

Running VR: https://openqa.suse.de/tests/overview?distri=sle&version=15&build=611.1%40tests_toolchain_ppc_jeriveramoya&groupid=96

Same error is displayed in https://openqa.suse.de/tests/1684402#step/kdump_and_crash/56 from the previous round of statistics and now finally is provided to the developer.

Provided more stats to the bug, it seems that there are two issues (1) crash utility failing (1-2 out of 5 runs) because kernel file is corrupted and (2) OOM much easier to reproduce (5-4 out of 5 runs).

http://e13.suse.de/tests/10952#step/kdump_and_crash/33 shows problem with string "Enterprise" which is broken... quite strange

http://openqa.suse.de/tests/2295805/file/serial0.txt shows:

sysrq: SysRq : Trigger a crash
Oops: Kernel access of bad area, sig: 11 [#1]
SMP NR_CPUS=2048

Later I see some call trace. Need to understand these relationship.

http://e13.suse.de/tests/10979#step/kdump_and_crash/29 shows problem with fluting output for core dump on console.

The last one is about yast coredumping... which is a different issue... in previous screens nokogiri failed to install... just not sure if its blocked

https://openqa.suse.de/tests/2295805#step/kdump_and_crash
worker_log:

Can't download sle-15-SP1-ppc64le-107.5-textmode@ppc64le-uefi-vars.qcow2

It seems to be an issue on worker side.

one issue more on my remote worker:

ps64vt1069:~ #
Message from syslogd@ps64vt1069 at Dec 3 09:12:16 ...
kernel:NMI watchdog: BUG: soft lockup - CPU#6 stuck for 65s! [systemd-coredum:6597]

Message from syslogd@ps64vt1069 at Dec 3 09:13:40 ...
kernel:NMI watchdog: BUG: soft lockup - CPU#4 stuck for 42s! [swapper/4:0]

this is a known issue on sles 12 sp3.

atm the failed because of send_key issue or yast2_console is not finished: http://f40.suse.de/tests/112#step/kdump_and_crash/27

This happens now on osd. We need to fix this problem at first.

sub activate_kdump {
# activate kdump
type_string "echo \"remove potential harmful nokogiri package boo#1047449\"\n";
zypper_call('rm -y ruby2.1-rubygem-nokogiri', exitcode => [0, 104]);
script_run 'yast2 kdump', 0;
my @tags = qw(yast2-kdump-disabled yast2-kdump-enabled yast2-kdump-restart-info yast2-missing_package yast2_console-finished);
do {
assert_screen \@tags, 300;
# enable kdump if it is not already
wait_screen_change { send_key 'alt-u' } if match_has_tag('yast2-kdump-disabled');
wait_screen_change { send_key 'alt-o' } if match_has_tag('yast2-kdump-enabled');
wait_screen_change { send_key 'alt-o' } if match_has_tag('yast2-kdump-restart-info');
wait_screen_change { send_key 'alt-i' } if match_has_tag('yast2-missing_package');
} until (match_has_tag('yast2_console-finished'));

needle issue fixed but test got incomplete, no screenshots available: http://f40.suse.de/tests/123

increase QEMURAM to 6GB and try...

I don't think you can solve "[2019-01-09T07:34:09.420 EST] [debug] QEMU: qemu-system-ppc64: cannot set up guest memory 'ppc_spapr.ram': Cannot allocate memory" by increasing the memory on the VM side. I guess the hypervisor can not provide the memory. Btw, please see #33376#note-48 as it could be that we need to adjust the kernel command line parameters for the right memory that is available to the crash handling kernel to prevent an OOM within the kdump process.

@okurz will talk to kgw for this

fix at first needle match issue, needle PR:
https://gitlab.suse.de/openqa/os-autoinst-needles-sles/merge_requests/1041

http://f40.suse.de/tests/145

increased kdump memory to 320 MB, but it got stuck later at:
echo c > /proc/sysrq-trigger

--
[2019-01-10T07:43:23.962 EST] [debug] /var/lib/openqa/cache/f40.suse.de/tests/sle/tests/console/kdump_and_crash.pm:44 called opensusebasetest::wait_boot
[2019-01-10T07:43:23.962 EST] [debug] <<< testapi::check_screen(mustmatch=[
'grub2',
'bootloader',
'inst-bootmenu'
], timeout=100)
[2019-01-10T07:43:24.666 EST] [debug] load of /var/lib/openqa/cache/f40.suse.de/tests/sle/products/sle/needles/inst-bootmenu-20160309.png took 0.13 seconds
[2019-01-10T07:43:24.683 EST] [debug] WARNING: check_asserted_screen took 0.71 seconds for 30 candidate needles - make your needles more specific
[2019-01-10T07:43:24.683 EST] [debug] no match: 99.9s, best candidate: bootloader-ofw-grub2-leanos-bsc1055166-20170823 (0.00)
[2019-01-10T07:43:24.971 EST] [debug] no change: 98.9s
[2019-01-10T07:43:25.971 EST] [debug] no change: 97.9s
[2019-01-10T07:43:26.972 EST] [debug] no change: 96.9s
[2019-01-10T07:43:29.942 EST] [debug] WARNING: check_asserted_screen took 1.97 seconds for 30 candidate needles - make your needles more specific
[2019-01-10T07:43:30.557 EST] [debug] no match: 95.9s, best candidate: bootloader-ofw-grub2-leanos-bsc1055166-20170823 (0.00)
[2019-01-10T07:43:30.558 EST] [debug] considering VNC stalled, no update for 6.60 seconds

--

the problem is also that there is no logs available because of (qemu crash?)...

increased to 640 MB
but http://f40.suse.de/tests/148#step/kdump_and_crash/38 shows known issue with BUG: soft lockup

atm I face the problem with qemu crash. And it is not possible atm to test on osd with new ASSET with my code changes:

https://openqa.suse.de/tests/2367754#step/kdump_and_crash/48

changes kdump memory itself works well on my pkvm remote worker.
but sysrq-trigger crashes every time qemu instance.
We cannot test directly on osd via new ASSERT-URL because of shared worker
configuration on osd.

So we need to merge this change now and see if this works or not.

PR:
https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/6531

found bsc#957053 and bsc#1120566 which are related to this kdump issue. This happened also on aarch64 as well, so add this change for aarch64 too.
update PR now.

need to check https://openqa.suse.de/tests/2439423 and see whether PR works or not.

we experience at moment issue with SCC ...

https://openqa.suse.de/tests/2439423#step/install/8

https://openqa.suse.de/tests/2440868/file/autoinst-log.txt shows syntax error

create https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/6725 and remove unnecessary bugref.

https://openqa.suse.de/tests/2441710 shows expected test results and PR works.