QA » openQA Project » openQA Tests

We from the product side also looked further into this issue: interesting points are that neither Leap (same openqa instane) nor SLE (different instance) seem to show this kind of issue - which in turn seems to point towards a product issue.

Lately, the issue shows up much more often, as I removed ungeneric needles (matching all of the user_settings screen_ against one matching also the name explicitly. By digging through history, this issue seems to have started around snapshot 1214 - which co-incides with the checkin date of Qt 5.12

Qt 5.12 is thus a suspect for this failure, BUT this theory somewhat is disproven by the fact that Krypton does not show the issue

Turns out Krypton is not free from this error: https://openqa.opensuse.org/tests/822087#step/user_settings/2 (one suspicion was that Krypton would not see it due to the higher memory availability)

okurz wrote:

but no urgent issue anymore.

And why did you come to this conclusion?
I also think it is still an urgent issue until we have statistics about this

need to investigate this issue with needle match.

@zluo please do not make this ticket something completely different. If currently "inst-userpasswdtoosimple" does not match then it's a different issue. Btw, where did you see this?
it is a different issue, please check the labeled bugref... https://bugzilla.opensuse.org/show_bug.cgi?id=1122181

hmm there is already a softfail needle for this: https://openqa.opensuse.org/tests/831713#step/user_settings/2
question is if this is acceptable as urgency removal or not - the two builds afterwards looked fine though
@okurz your take on this?

if low characters don't appear now, it mean we have a different issue, right. so this is not urgent anyway.

zluo wrote:

if low characters don't appear now, it mean we have a different issue, right. so this is not urgent anyway.

anyway, following #note-4 I would try to check statistics on Builds before Qt5 submission and compare it to the ones after the Qt5 submission

I've triggered 50 jobs overnight to get some statistics...

also trying with type_string_slow

For further reference because I know that all of us will have forgotten in some months what you or we did here: Please also reference the commands you used to trigger tests and provide links to the openQA test overviews from which you gathered your observations.

This has nothing to do with the "tools"-team and you know that they would not be able to look into this anytime soon given that the team consists of just one guy that actually has openQA experience and he works on the webUI mainly.

#46190#note-4 already points in the direction of Qt submission within openSUSE Factory/Tumbleweed as we did not see this problem yet in Leap nor SLE AFAIK. There should be a corresponding product bug but most likely this will not be fixed without our help in regards of having openQA showing the error and only then providing a workaround.

Created bug report https://bugzilla.opensuse.org/show_bug.cgi?id=1122804 which you can use now in all needle references or the record_soft_failure.

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: minimal+role_minimal
https://openqa.suse.de/tests/3462442

To prevent further reminder comments one of the following options should be followed:

1. The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
2. The openQA job group is moved to "Released"
3. The label in the openQA scenario is removed

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: minimal+role_minimal
https://openqa.suse.de/tests/3462442

To prevent further reminder comments one of the following options should be followed:

1. The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
2. The openQA job group is moved to "Released"
3. The label in the openQA scenario is removed

It seems to happen at every run currently on sle 15sp2/aarch64, even in cases where it seems to have worked, like here or here

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: yast_no_self_update
https://openqa.suse.de/tests/3526342

To prevent further reminder comments one of the following options should be followed:

1. The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
2. The openQA job group is moved to "Released"
3. The label in the openQA scenario is removed

Other mistyping issues in user_settings. Race condition of events happening before previous characters are typed:

• https://openqa.suse.de/tests/3549343#step/user_settings/2
• https://openqa.suse.de/tests/3549170#step/user_settings/3
• https://openqa.suse.de/tests/3546669#step/user_settings/3
• https://openqa.suse.de/tests/3549551#step/user_settings/3
• https://openqa.suse.de/tests/3549094#step/user_settings/3

With https://github.com/os-autoinst/os-autoinst/pull/1260 typing in general is configured by default to be slower and also especially the modifier keys are handled now more human-like. This might fix it already. Please check test results for the status accordingly.

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: RAID1
https://openqa.suse.de/tests/3624277

To prevent further reminder comments one of the following options should be followed:

1. The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
2. The openQA job group is moved to "Released"
3. The label in the openQA scenario is removed

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: cryptlvm+activate_existing
https://openqa.suse.de/tests/3686994

To prevent further reminder comments one of the following options should be followed:

1. The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
2. The openQA job group is moved to "Released"
3. The label in the openQA scenario is removed

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: skip_registration+workaround_modules
https://openqa.suse.de/tests/3731135

To prevent further reminder comments one of the following options should be followed:

1. The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
2. The openQA job group is moved to "Released"
3. The label in the openQA scenario is removed

the issue mentioned in #54 is performance related: see Stall detection text in first_boot.

https://openqa.suse.de/tests/3731135#step/user_settings/3 shows softfail and uppercase 'Bernhard M Wiedemann' is used. and the workaround needle is not correct here.

I cannot see any issue here.

I believe the real issue with tying issue is performance on aarch64 worker. We have Stall detection sometimes. for example:

https://openqa.suse.de/tests/3766064#step/accept_license/3

for TW we see clearly that we don't have this issue anymore, however the needle which used is not accurate:

https://openqa.opensuse.org/tests/1138820#step/user_settings/2

zluo wrote:

I believe the real issue with tying issue is performance on aarch64 worker

If it all this is a different issue than the originally described on which happened on i586, at least a sub-issue, which can not explain all failures. If you want to follow on with this hypothesis I suggest to do it in a different (sub-)ticket.

http://f40.suse.de/tests/5995#step/user_settings/2 looks much better with VNC_TYPING_LIMIT=18 in this case, but I found match level of the needle is not set, so this needs to be changed anyway.

http://f40.suse.de/tests/5998#step/user_settings/2 shows now 100% match and with VNC_TYPING_LIMIT=18

https://progress.opensuse.org/issues/60833 needs to be handled as well to get for a good performance.

Together with VNC_TYPING_LIMIT and a new needle (with proper match level), I think it can prevent and resolve this issue.

zluo wrote:

Together with VNC_TYPING_LIMIT […] it can prevent and resolve this issue.

VNC_TYPING_LIMIT should not be considered a fix unless it is used for specific workers or machines. It can be used for investigation purposes but slowing down the complete test flow to fix a single test module is not a good idea. Where necessary slower typing can be instructed with the according test API calls or the helper functions we have in os-autoinst-distri-opensuse.

I think we live without VNC_TYPING_LIMIT if we can make sure that poor performance will not cause typing issue.

https://gitlab.suse.de/openqa/os-autoinst-needles-sles/merge_requests/1301

zluo wrote:

I think we live without VNC_TYPING_LIMIT if we can make sure that poor performance will not cause typing issue.

Unless you can define "poor performance" better I doubt we can "fix it". Tests and os-autoinst should be designed to work regardless of a slow worker host though.

https://gitlab.suse.de/openqa/os-autoinst-needles-sles/merge_requests/1302 re-worked.

We have all the time poor performance for aarch64 on osd:

https://openqa.suse.de/tests/3776484#next_previous
https://openqa.suse.de/tests/3776484#next_previous

Just look at first_boot and see a lot of Stall detection.

So poor performance of aarch64 workers on osd is the root cause.
If we don't want to use VNC_TYPING_LIMIT, then we need to solve the issue at first or we have to leave with this kind of sporadic issue.

https://openqa.suse.de/tests/3803671#step/first_boot/7 shows still issue with stall detection:

related ticket: #25864

to check later:

https://openqa.suse.de/tests/3817019#next_previous

it looks really bad.

https://openqa.suse.de/tests/3817022#step/accept_license/4 show issue with stall detection.

I guess when we can actually detect "stalls" then we should regard it as a different issue. We have #25864 for OSD worker specific issues. I suggest to keep this ticket specific when we see the symptoms of wrong username typing in the user_settings test module without seeing other symptoms, e.g. no stall detection message.

zluo wrote:

1. delete the workaround needle which matches even uppercase

if you found false-matches, yes.

1. create new needle tag for a softfail for misstyping and we need then information for the product issue.

I am not sure what you see this as necessary for. Keep in mind that for openSUSE we already track this issue since about 1 year and there are workaround needles in place which correctly detect the error condition and accept the mistyped username as acceptable. What IIRC the test code does not do is to try to fix the mistyping, e.g. detecting mistyping, deleting wrong characters and retype until typing is correct before continuing. This is already implemented but depends on the needle tag 'boo#1122804' which maybe you still need to add for the corresponding SLE needles.

https://openqa.suse.de/tests/3694753#step/user_settings/2 shows that workaround doesn't come up for matching. inst-userinfostyped-sle15-2018022 matches however it has typing issue with lowercase. We should delete this needle anyway.

For TW we have also issue with workaround needle:
https://openqa.opensuse.org/tests/1150786#step/user_settings/3 typing correct with uppercase, but screen before this it shows lowercase for m. and it got matched as by workaround needle already, really weird. So what is the typing issue here? We have actually then uppercase for M and this got also matched correctly, but little bit later.

So in this case the workaround needle is not needed.

zluo wrote:

https://openqa.suse.de/tests/3694753#step/user_settings/2 shows that workaround doesn't come up for matching. inst-userinfostyped-sle15-2018022 matches however it has typing issue with lowercase. We should delete this needle anyway.

agreed. Replace it by a more strict version.

For TW we have also issue with workaround needle:
https://openqa.opensuse.org/tests/1150786#step/user_settings/3 typing correct with uppercase, but screen before this it shows lowercase for m. and it got matched as by workaround needle already, really weird. So what is the typing issue here? We have actually then uppercase for M and this got also matched correctly, but little bit later.

So in this case the workaround needle is not needed.

No, the workaround needle is needed but the test code correctly detects this and corrects the mistyping. We already have the correct code since af67fc8e7 to retry on typing mistakes. see #46190#note-24

PR to improve workaround: https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/9469 (merged)

Wouldn't be the fastest way to simplify the username?
This "mistyping" looks a lot like a tool issue, not like a product issue.
And if it's "just" the same issue again and a gain, using less letters is the easiest way to reduce the failures.
(Yes, this requires adaptions on some testcases/images, but this might be faster than everything else)

sbehlert wrote:

Wouldn't be the fastest way to simplify the username?
This "mistyping" looks a lot like a tool issue, not like a product issue.

I am still convinced this is a product issue: https://bugzilla.opensuse.org/show_bug.cgi?id=1122804

This ticket is only about a workaround for the real issue. Simplifying the username would definitely be a possible workaround as well but with no possibility to detect the original issue. Seeing how much effort has gone into this ticket and apparently none into a bug fix it might have been a better choice from the beginning. Would be cool if someone would have made the call in the bug to state: "Yes, this is clearly a product regression but we will not fix it due to low customer impact and too high implementation effort". Then probably someone would have arrived to the decision sooner to accept any "workaround" as the new "happy path".

sbehlert wrote:

Wouldn't be the fastest way to simplify the username?
This "mistyping" looks a lot like a tool issue, not like a product issue.
And if it's "just" the same issue again and a gain, using less letters is the easiest way to reduce the failures.
(Yes, this requires adaptions on some testcases/images, but this might be faster than everything else)

If the bug ticket gets "resolved: won't fix", we can remove the soft-fail. Then our support team can tell customers that when they use a VM with 1GB RAM, they should pick short full names and don't press the shift key.

Hello Sergio, thanks for your comment. I have raised your question to Stefan Behlert directly (again). If he confirms we can remove a lot of soft failures but I also agree with Oli Kurz, we could have saved a lot of time....

maritawerner wrote:

Hello Sergio, thanks for your comment. I have raised your question to Stefan Behlert directly (again). If he confirms we can remove a lot of soft failures but I also agree with Oli Kurz, we could have saved a lot of time....

Hm, I fail to see what the question is? As mentioned, I don't think this is a product bug, and currently I see or hear no evidence to the contrary.

Reducing the namelength might reduce the times the issue happen, but of course if its an openQA issue it still should be fixed. (For a product issue, I see no real life occurance yet, which also points to the "no product issue" part...)

sbehlert wrote:

[…]
Hm, I fail to see what the question is? As mentioned, I don't think this is a product bug.

It is a product bug. https://bugzilla.opensuse.org/show_bug.cgi?id=1122804 is the report about it. The bug report is currently in status "CONFIRMED". As long as that is the case we should consider it as a product bug.

[…] and currently I see or hear no evidence to the contrary.

slindomansilla already manually reproduced the problem which appears when a VM is configured with just 1 GB. My suggestion:

• Adjust product requirements and recommend >= 2 GB for GUI interactive installer and keep existing requirements for textmode or non-interactive installer
• Adjust test plan accordingly, e.g. run openQA tests using interactive installer against 2GB variants, 1GB for others

side-note: https://gitlab.suse.de/openqa/os-autoinst-needles-sles/merge_requests/1316 merged to prevent some false-positives and links to the progress ticket when instead we should directly link to the bug report.

Also for x86_64: https://openqa.suse.de/tests/3929374#step/user_settings/6
But, aarch64 is the most affected.

I have talked to Alex Herzig and he has talked to Stefan Behlert -> they do not agree on OLi's suggestion and now they try to get a QT expert who will fix the bug.

maritawerner wrote:

I have talked to Alex Herzig and he has talked to Stefan Behlert -> they do not agree on OLi's suggestion and now they try to get a QT expert who will fix the bug.

Even better then :D I never ruled out the option to have the real bug fixed instead ;)

Waiting for verification on next build

Isolate product bug to specific scenarios: https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/9808 (merged)

Setting ASSERT_BSC1122804=1 added to default test suite.

Reduce max_interval to minimum: https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/9804