QA » openQA Project » openQA Tests

There is another typo error ( missing space between Y2DEBUG and linux location ) on ppc64le https://openqa.suse.de/tests/2052952#step/bootloader/12.

ppc typos:

• missing space
• missing space between linux location and textmode=1
• incomplete y2debug
• missing space
• incomplete y2debug
• incomplete y2debug
• incomplete y2debug
• incomplete y2debug

All of those failed in typing the regurl correctly (confirmed with logs + video):

https://openqa.suse.de/tests/2060935#step/bootloader/8
https://openqa.suse.de/tests/2060924#step/reboot_and_install/19
https://openqa.suse.de/tests/2060892#step/bootloader/8
https://openqa.suse.de/tests/2060963#step/reboot_and_install/19

As thehejik pointed out: also with passwords - https://openqa.suse.de/tests/2061324#step/bootloader/17

It also destroys each and every ppc64 test on 15SP1 because it fails to append parameters after the kernel. Example: https://openqa.suse.de/tests/2058405#step/welcome/1
The code itself shouldn't cause this problem (note the leading space in front of Y2DEBUG=1):

[2018-09-16T16:17:54.0144 CEST] [debug] <<< testapi::send_key(key='end', do_wait=0)
[2018-09-16T16:17:54.0351 CEST] [debug] waiting for screen change: 0 1000000
[2018-09-16T16:17:54.0862 CEST] [debug] waiting for screen change: 0 25.2867820069347
[2018-09-16T16:17:54.0862 CEST] [debug] >>> testapi::wait_screen_change: screen change seen at 0
[2018-09-16T16:17:54.0862 CEST] [debug] /var/lib/openqa/cache/tests/sle/tests/installation/bootloader.pm:27 called bootloader_setup::bootmenu_default_params
[2018-09-16T16:17:54.0862 CEST] [debug] <<< testapi::type_string(string=' Y2DEBUG=1 ', max_interval=4, wait_screen_changes=0, wait_still_screen=0)

• test changes: No relevant changes, checking logs we type the correct string with still the correct speed
• product changes: We see the same also on 15SP1 so I doubt it is related to product changes
• We find passing tests on all malbec, qa-power8-5, qa-power8-4, powerqaworker-qam-1

https://openqa.suse.de/tests/2058508#step/bootloader/12 is an example where not the space before Y2DEBUG is missing but the starting character is lowercase. We assume it is in the end the same issue because AFAIK for typing it presses the shift-key and then the next characters.

As I have found no failure on a machine different than qa-worker8-4 I assume that machine is at fault.

https://gitlab.suse.de/openqa/salt-pillars-openqa/merge_requests/118 prepared to exclude the machine until the investigation is done.

Tested on osd with sudo salt 'QA-Power8-4-kvm*' state.apply openqa.worker test=True

     Changes:   
              ----------
              diff:
                  --- 
                  +++ 
                  @@ -6,7 +6,7 @@
                   HOST=openqa.suse.de 
                   CACHEDIRECTORY=/var/lib/openqa/cache
                   QEMUTHREADS=1
                  -WORKER_CLASS=qemu_ppc64le_disabled_investigate_poo40670,qemu_ppc64le_no_tmpfs
                  +WORKER_CLASS=qemu_ppc64le,qemu_ppc64le_no_tmpfs
                   WORKER_HOSTNAME=10.162.6.201

                   [openqa.suse.de]

And executed again in hot.

side-task: zluo created "investigation" jobs but left them in the production job group. To move them to the development job group I used the following commands

for i in $(openqa_client_osd --json-output jobs build=0393 arch=ppc64le groupid=139 test=check_bootloader | jq '.jobs | .[] | .id') ;do openqa_client_osd jobs/$i put --json-data '{"group_id": 132}' ; done
for i in $(openqa_client_osd --json-output jobs build=0393 arch=ppc64le groupid=139 test="check_bootloader@ppc64le" | jq '.jobs | .[] | .id') ;do openqa_client_osd jobs/$i put --json-data '{"group_id": 132}' ; done

Retriggered all tests that failed on SLE12SP4 in the corresponding same step. With this I feel we have reduced the urgency give back the task to the backlog and involve the tools team. I doubt zluo can easily drive the work here. I suggest to investigate first by gathering better data, e.g. trigger a corresponding test scenario confined to that machine with WORKER_CLASS=qemu_ppc64le_disabled_investigate_poo40670

I did not find anything related on the workers that differs, at least not in the parameters of the kernel modules "kvm" or "kvm_hv"

okurz@openqa:~> sudo salt 'QA-Power8*' cmd.run 'for i in kvm kvm_hv ; do modinfo $i ; done'
QA-Power8-4-kvm.qa.suse.de:
    filename:       /lib/modules/4.4.140-94.42-default/kernel/arch/powerpc/kvm/kvm.ko
    license:        GPL
    author:         Qumranet
    srcversion:     AB9D9B9EA7191D23D5C11C5
    depends:        
    supported:      yes
    intree:         Y
    vermagic:       4.4.140-94.42-default SMP mod_unload modversions 
    signer:         SUSE Linux Enterprise Secure Boot Signkey
    sig_key:        3F:B0:77:B6:CE:BC:6F:F2:52:2E:1C:14:8C:57:C7:77:C7:88:E3:E7
    sig_hashalgo:   sha256
    parm:           halt_poll_ns:uint
    parm:           halt_poll_ns_grow:int
    parm:           halt_poll_ns_shrink:int
    filename:       /lib/modules/4.4.140-94.42-default/kernel/arch/powerpc/kvm/kvm-hv.ko
    alias:          devname:kvm
    alias:          char-major-10-232
    license:        GPL
    srcversion:     C42262D681C35A7BF4CEDCE
    depends:        kvm
    supported:      yes
    intree:         Y
    vermagic:       4.4.140-94.42-default SMP mod_unload modversions 
    signer:         SUSE Linux Enterprise Secure Boot Signkey
    sig_key:        3F:B0:77:B6:CE:BC:6F:F2:52:2E:1C:14:8C:57:C7:77:C7:88:E3:E7
    sig_hashalgo:   sha256
    parm:           dynamic_mt_modes:Set of allowed dynamic micro-threading modes: 0 (= none), 2, 4, or 6 (= 2 or 4) (int)
    parm:           target_smt_mode:Target threads per core (0 = max) (int)
QA-Power8-5-kvm.qa.suse.de:
    filename:       /lib/modules/4.4.140-94.42-default/kernel/arch/powerpc/kvm/kvm.ko
    license:        GPL
    author:         Qumranet
    srcversion:     AB9D9B9EA7191D23D5C11C5
    depends:        
    supported:      yes
    intree:         Y
    vermagic:       4.4.140-94.42-default SMP mod_unload modversions 
    signer:         SUSE Linux Enterprise Secure Boot Signkey
    sig_key:        3F:B0:77:B6:CE:BC:6F:F2:52:2E:1C:14:8C:57:C7:77:C7:88:E3:E7
    sig_hashalgo:   sha256
    parm:           halt_poll_ns:uint
    parm:           halt_poll_ns_grow:int
    parm:           halt_poll_ns_shrink:int
    filename:       /lib/modules/4.4.140-94.42-default/kernel/arch/powerpc/kvm/kvm-hv.ko
    alias:          devname:kvm
    alias:          char-major-10-232
    license:        GPL
    srcversion:     C42262D681C35A7BF4CEDCE
    depends:        kvm
    supported:      yes
    intree:         Y
    vermagic:       4.4.140-94.42-default SMP mod_unload modversions 
    signer:         SUSE Linux Enterprise Secure Boot Signkey
    sig_key:        3F:B0:77:B6:CE:BC:6F:F2:52:2E:1C:14:8C:57:C7:77:C7:88:E3:E7
    sig_hashalgo:   sha256
    parm:           dynamic_mt_modes:Set of allowed dynamic micro-threading modes: 0 (= none), 2, 4, or 6 (= 2 or 4) (int)
    parm:           target_smt_mode:Target threads per core (0 = max) (int)
okurz@openqa:~> sudo salt 'QA-Power8*' cmd.run 'cat /sys/module/kvm*/parameters/*'
QA-Power8-4-kvm.qa.suse.de:
    500000
    2
    0
    6
    0
QA-Power8-5-kvm.qa.suse.de:
    500000
    2
    0
    6
    0
okurz@openqa:~> sudo salt 'malbec*' cmd.run 'cat /sys/module/kvm*/parameters/*'
malbec.arch.suse.de:
    500000
    2
    0
    6
    0

Cross checking with other worker needed for final proof that this is a worker issue, with a rate of 85% 10 runs should be enough, still have to figure out which worker class to use for this
'QA-Power8-4-kvm*' is the questioned worker
"qemu_ppc64le" is the class to use here since the affected 8-4-kvm worker is still changed to the other

'GROUP="SLE15 Development" TEST=poo40670_crosscheck WORKER_CLASS=qemu_ppc64le'

And all of them passed the bootloader finely, additionally they ran on more than one worker (excluding 8-4-kvm).
with an initial error quote of 85% 10 tests should be proofing that this is a worker specific issue.

jorauch wrote:

with an initial error quote of 85% 10 tests should be proofing that this is a worker specific issue.

I created another ticket for a similar issue on x86_64 which is #42275 so even if we find out that the ppc64le worker is special here we still have a problem on x86_64.

I suggest to still try with even slower typing in the bootloader screen, not the test variable but even slower typing explicitly in the bootloader screen. I can see in https://openqa.suse.de/tests/2164495/modules/bootloader/steps/1/src that we already give an explicit parameter to the line registration_bootloader_params(utils::VERY_SLOW_TYPING_SPEED); so we can schedule tests even on production but override that individual test module with one that has even slower typing, e.g. registration_bootloader_params(2); . This can be scheduled using the parameter "SCHEDULE" of os-autoinst and supplying the patched test module as a downloadable asset. You can put the patched module on a webserver, e.g. your export folder and then download it into the job and schedule, e.g. openqa-clone-job … ASSET_1_URL=https://w3.nue.suse.com/~slindomansilla/bootloader_poo40670.pm SCHEDULE=bootloader_poo40670,tests/installation/welcome

okurz suggested to create a constant with a lower value.

PR here: https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/5998

Ok, so even slower typing does not help. You can revert your PR then I guess.

Could you please crosscheck with an older released product, e.g. SLE15-GM-668.1, to rule out that product changes caused this?

PR to revert changes: https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/6004

Info: http://christian.weblog.heimdaheim.de/wp-content/uploads/sites/2/2009/01/linuxrc.html

Sample file created: https://w3.nue.suse.com/~slindomansilla/tmp/linuxrc_parameters.info

Same issue here: https://openqa.suse.de/tests/2199334# , wrong url to the profile is typed.

The file format for the info file is very tricky, and I haven't got it working through https (but probably we don't need https for that).

A working sample file can be seen here: https://w3.nue.suse.com/~slindomansilla/tmp/linuxrc_parameters.info

@SLindoMansilla In #40670#note-7 we mentioned that basically ppc64le is unusable due to this. This does not seem to be the case anymore. I have the feeling that we see now quite seldomly the originally observed problem but I do not know the reason. https://w3.nue.suse.com/~okurz/openqa_suse_de_status.html mentions poo#40670 a single time. Don't know what changed but I guess it is not urgent anymore, we should reduce to high and re-discuss what approach to follow. How would you assess the current situation?

And in https://openqa.suse.de/tests/2213493/file/autoinst-log.txt I see one problem. The regurl again is typed with not "very slow" but only "slow" which can explain the problems to me.

I will handle that special situation -> https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/6094

The info-approach should improve the situation significantly (since it would avoid typing long URLs and such) but doesn't avoid typing inside the bootloader completely.
Definitely worth a try. If it doesn't help, here is another idea:

We could have a specific bootloader per worker (e.g. ipxe) which always chainloads its configuration. This configuration could be written by openqa beforehand. This way we could eliminate all typing in the bootloader completely. However, it would involve creating the script for the bootloader and teaching openQA to write these configurations. So it would be a bigger rewrite of the backend there.

again we seem to be back to a single specific problem: https://w3.nue.suse.com/~okurz/openqa_suse_de_status.html mentions this ticket in only two jobs, https://openqa.suse.de/tests/2228503 and https://openqa.suse.de/tests/2228489 both in SLE15SP1 and both showing the same problem, " Y2DEBUG=1" is typed without the leading space. This is not even the graphical grub screen. I assume what is happening here is that the previous key sequence with the key "end" is sent and then we immediately type the next character without waiting enough to actually reach the end of the line. Can we handle just that special case for now please?

Yes, fixing this specific issue.

PR: https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/6129

PR to have more shared workers (PPC): https://gitlab.suse.de/qa-sle/shared-development-worker/merge_requests/4

The investigation doesn't show the issue of missing white space before Y2DEBUG.

PR merged.

Preparing statistical investigation on OSD.

Waiting for verification run: https://openqa.suse.de/tests/overview?build=poo40670_mistyped_y2debug

0/101 passed, I think we have this particular issue covered. I created #43655 to cover the info-file approach. I guess we can close this ticket here as we do not seem to see any current, reproducible bootloader typing parameter issues anymore.

yes, this is what I meant, this is what I thought. What I wrote is stupid

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: RAID5
https://openqa.suse.de/tests/2244625