Project

General

Profile

action #45650

[functional][u][aarch64] test fails in first_boot because of username letters not written in capital case during installation user_settings

Added by ggardet_arm over 2 years ago. Updated over 1 year ago.

Status:
Resolved
Priority:
High
Assignee:
Category:
Bugs in existing tests
Target version:
SUSE QA - Milestone 23
Start date:
2019-01-02
Due date:
% Done:

0%

Estimated time:
Difficulty:

Description

Observation

openQA test in scenario opensuse-Tumbleweed-DVD-aarch64-create_hdd_gnome@aarch64 fails in
first_boot

Reproducible

Fails since (at least) Build 20181231

Expected result

Last good: 20181224 (or more recent)

Further details

Always latest result in this scenario: latest

The problem is earlier in installer where the M and W are not typed as upper cases in username: https://openqa.opensuse.org/tests/822544#step/user_settings/2

This is probably related to the worker (which is running Leap 15.0) since a local test in os-autoinst with an up-to-date Tumbleweed aarch64 host works fine.

snapper_diff_109..115 (11.8 KB) snapper_diff_109..115 okurz, 2019-01-02 16:04

Related issues

Related to openQA Tests - action #46190: [functional][u] test fails in user_settings - mistyping in Username (lowercase instead of uppercase) Resolved2019-01-15

Copied to openQA Tests - action #45713: [functional][u] test for nested virtualization / qemu / typing over VNCResolved2019-01-02

History

#1 Updated by okurz over 2 years ago

See attached the difference of the two root filesystem snapshots "109" from 2018-12-23 and the current one. The complete change of packages between the oldest still available snapshot 105 from 2018-12-19 and the current one is:

# diff -Naur <(rpm -r /.snapshots/105/snapshot/ -qa) <(rpm -qa) | grep '^[+-]' | sort
+++ /dev/fd/62  2019-01-02 17:07:26.626769544 +0100
--- /dev/fd/63  2019-01-02 17:07:26.626769544 +0100
+dracut-044.1-lp150.14.12.1.aarch64
-dracut-044.1-lp150.14.9.1.aarch64
-git-core-2.16.4-lp150.2.6.1.aarch64
+git-core-2.16.4-lp150.2.9.1.aarch64
-git-gui-2.16.4-lp150.2.6.1.aarch64
+git-gui-2.16.4-lp150.2.9.1.aarch64
-gitk-2.16.4-lp150.2.6.1.aarch64
+gitk-2.16.4-lp150.2.9.1.aarch64
+grub2-2.02-lp150.13.10.1.aarch64
-grub2-2.02-lp150.13.7.1.aarch64
+grub2-arm64-efi-2.02-lp150.13.10.1.aarch64
-grub2-arm64-efi-2.02-lp150.13.7.1.aarch64
+grub2-snapper-plugin-2.02-lp150.13.10.1.noarch
-grub2-snapper-plugin-2.02-lp150.13.7.1.noarch
+grub2-systemd-sleep-plugin-2.02-lp150.13.10.1.noarch
-grub2-systemd-sleep-plugin-2.02-lp150.13.7.1.noarch
-libbluetooth3-5.48-lp150.4.3.1.aarch64
+libbluetooth3-5.48-lp150.4.6.1.aarch64
-libfreebl3-3.36.6-lp150.2.6.1.aarch64
+libfreebl3-3.40.1-lp150.2.10.2.aarch64
-libgcrypt20-1.8.2-lp150.5.3.1.aarch64
+libgcrypt20-1.8.2-lp150.5.6.1.aarch64
-libhogweed4-3.4-lp150.2.3.aarch64
+libhogweed4-3.4-lp150.3.3.1.aarch64
-libmtp9-1.1.15-lp150.1.2.aarch64
+libmtp9-1.1.16-lp150.2.3.1.aarch64
-libmtp-udev-1.1.15-lp150.1.2.aarch64
+libmtp-udev-1.1.16-lp150.2.3.1.aarch64
-libnettle6-3.4-lp150.2.3.aarch64
+libnettle6-3.4-lp150.3.3.1.aarch64
-libsoftokn3-3.36.6-lp150.2.6.1.aarch64
+libsoftokn3-3.40.1-lp150.2.10.2.aarch64
-mozilla-nspr-4.19-lp150.1.2.aarch64
+mozilla-nspr-4.20-lp150.2.3.1.aarch64
-mozilla-nss-3.36.6-lp150.2.6.1.aarch64
+mozilla-nss-3.40.1-lp150.2.10.2.aarch64
-mozilla-nss-certs-3.36.6-lp150.2.6.1.aarch64
+mozilla-nss-certs-3.40.1-lp150.2.10.2.aarch64
-nfs-client-2.1.1-lp150.4.3.1.aarch64
+nfs-client-2.1.1-lp150.4.6.1.aarch64
-openQA-client-4.6.1545139030.66c0b50f-lp150.1023.1.noarch
+openQA-client-4.6.1545406149.53968c1e-lp150.1054.1.noarch
-openQA-common-4.6.1545139030.66c0b50f-lp150.1023.1.noarch
+openQA-common-4.6.1545406149.53968c1e-lp150.1054.1.noarch
-openQA-worker-4.6.1545139030.66c0b50f-lp150.1023.1.noarch
+openQA-worker-4.6.1545406149.53968c1e-lp150.1054.1.noarch
-os-autoinst-4.5.1544691921.44e93d8d-lp150.8.1.aarch64
+os-autoinst-4.5.1545369866.fc084a6a-lp150.16.1.aarch64
-ovmf-2017+git1510945757.b2662641d5-lp150.4.6.1.aarch64
+ovmf-2017+git1510945757.b2662641d5-lp150.4.9.1.aarch64
-perl-5.26.1-lp150.6.3.1.aarch64
+perl-5.26.1-lp150.6.6.1.aarch64
-perl-base-5.26.1-lp150.6.3.1.aarch64
+perl-base-5.26.1-lp150.6.6.1.aarch64
-perl-Bootloader-0.921-lp150.3.3.1.aarch64
+perl-Bootloader-0.923-lp150.3.6.1.aarch64
-polkit-default-privs-13.2-lp150.8.6.1.noarch
+polkit-default-privs-13.2-lp150.8.9.1.noarch
-psmisc-23.0-lp150.4.3.1.aarch64
+psmisc-23.0-lp150.4.6.1.aarch64
-psmisc-lang-23.0-lp150.4.3.1.noarch
+psmisc-lang-23.0-lp150.4.6.1.noarch
-qemu-ovmf-x86_64-2017+git1510945757.b2662641d5-lp150.4.6.1.noarch
+qemu-ovmf-x86_64-2017+git1510945757.b2662641d5-lp150.4.9.1.noarch
-qemu-uefi-aarch64-2017+git1510945757.b2662641d5-lp150.4.6.1.noarch
+qemu-uefi-aarch64-2017+git1510945757.b2662641d5-lp150.4.9.1.noarch
-suse-module-tools-15.0.1-lp150.2.3.1.aarch64
+suse-module-tools-15.0.2-lp150.2.6.1.aarch64

From this list I suspect one of

os-autoinst-4.5.1545369866.fc084a6a-lp150.16.1.aarch64
ovmf-2017+git1510945757.b2662641d5-lp150.4.9.1.aarch64
qemu-ovmf-x86_64-2017+git1510945757.b2662641d5-lp150.4.9.1.noarch
qemu-uefi-aarch64-2017+git1510945757.b2662641d5-lp150.4.9.1.noarch

to be the culprit.

Maybe you can use os-autoinst in a local container of openSUSE Leap 15.0 with the above package versions to see if the original problem can be reproduced?

#2 Updated by ggardet_arm over 2 years ago

Test run on 2018-12-26 was fine: https://openqa.opensuse.org/tests/821310#step/user_settings/2
So, the problem is probably in the very last update. What is the timestamp of the last snapshot (115)?

I will try to reproduce locally with a 15.0 system.

#3 Updated by ggardet_arm over 2 years ago

I restarted gnome test from snapshot 20181224 and it passes just fine: https://openqa.opensuse.org/tests/822688#step/user_settings/2

#4 Updated by ggardet_arm over 2 years ago

And now, it passes again: https://openqa.opensuse.org/tests/822692#step/user_settings/2
The machine has been updated or something since the last failing?

#5 Updated by okurz over 2 years ago

  • Subject changed from [aarch64] test fails in first_boot to [aarch64] test fails in first_boot because of username letters not written in capital case during installation user_settings

Yes, the worker is auto-updated every night depending on update availability.

# diff -Naur <(rpm -r /.snapshots/115/snapshot/ -qa) <(rpm -qa) | grep '^[+-]' | sort
-openQA-client-4.6.1545406149.53968c1e-lp150.1054.1.noarch
+openQA-client-4.6.1545406149.53968c1e-lp150.1055.1.noarch
-openQA-common-4.6.1545406149.53968c1e-lp150.1054.1.noarch
+openQA-common-4.6.1545406149.53968c1e-lp150.1055.1.noarch
-openQA-worker-4.6.1545406149.53968c1e-lp150.1054.1.noarch
+openQA-worker-4.6.1545406149.53968c1e-lp150.1055.1.noarch

That is just a rebuild of the openQA packages without any software changes in openQA itself so I doubt the package changes really make a difference. If you like I recommend you conduct a https://progress.opensuse.org/projects/openqatests/wiki/Wiki#Statistical-investigation by triggering a bigger set of tests to crosscheck exactly the originally reported symptoms. This week might be a good time anyway as the ressources on o3 are mainly underused ;)

Regarding testing the "typing stability" in general I discussed this topic today with lnussel and dimstar and we mentioned the idea to include better tests for the VNC typing emulation in OBS packages, could be the os-autoinst unit tests or also in e.g. qemu.

#6 Updated by ggardet_arm over 2 years ago

I know it is updated each night (if updates are available), but it started to work yesterday afternoon, so before the auto-update...

#7 Updated by ggardet_arm over 2 years ago

20 builds: https://openqa.opensuse.org/tests/overview?version=Tumbleweed&distri=opensuse&build=poo45650_investigation

Results (with QEMU_COMPRESS_LEVEL=6 QEMU_COMPRESS_THREADS=6 and VNC_TYPING_LIMIT=10):

  • succeeded: only 7 times (out of 20)
  • first_boot error (due to this current problem in user settings in installation): 7 times (out of 20)
  • problems due to missed key(s): 6 times (out of 20)

#8 Updated by okurz over 2 years ago

  • Subject changed from [aarch64] test fails in first_boot because of username letters not written in capital case during installation user_settings to [functional][u][aarch64] test fails in first_boot because of username letters not written in capital case during installation user_settings
  • Status changed from New to In Progress
  • Assignee set to okurz
  • Priority changed from Normal to High
  • Target version set to Milestone 22
[03/01/2019 15:16:28] <guillaume_g> okurz: how are handled upper cases in qemu/vnc for openQA? Is it a shift+key? If so, it could be just the same 'missed keys' problem. [03/01/2019 15:18:03] <okurz> guillaume_g: yes, it's a combination of keys and it could be that we just "miss" the shift. if you take a look in progress tickets you should be able to find others. Most severe the issue seems to be on virtio+wayland.
[03/01/2019 15:18:48] <okurz> guillaume_g: did you crosscheck to run about 20 jobs of the previous snapshot to find out if it's worker or product changes?
[03/01/2019 15:20:00] <guillaume_g> okurz: no, currently I scheduled 20 jobs for latest snapshot only. I will do it for previous snapshot right now.
[03/01/2019 15:22:43] <guillaume_g> okurz: gnome tests encouter more errors than other test. Not sure where wayland is used or not.
[03/01/2019 15:24:36] <okurz> guillaume_g: I think it's not using wayland here, in https://openqa.opensuse.org/tests/822800#step/xorg_vt/3 it says "X" and not "Xwayland"
[03/01/2019 15:26:57] <guillaume_g> okurz: Indeed
[03/01/2019 15:27:49] <guillaume_g> okurz: kde uses X too: https://openqa.opensuse.org/tests/822330#step/xorg_vt/3
[03/01/2019 15:28:01] <okurz> yes it does
[03/01/2019 15:32:38] <guillaume_g> okurz: previous snapshot (1224) testing: https://openqa.opensuse.org/tests/overview?build=poo45650_investigation_snap1224&distri=opensuse&version=Tumbleweed
[03/01/2019 15:37:23] <guillaume_g> okurz: do we have the same 'missed keys' errors on x86 or ppc?
[03/01/2019 15:44:06] <okurz> in principle yes but I do not know how the probability of appearance compares
[03/01/2019 15:45:32] <guillaume_g> okurz: it seems that latest x86 snapshot has also lots of errors ;)
[03/01/2019 15:45:56] <okurz> same symptoms? I'm not sure
[03/01/2019 15:48:49] <guillaume_g> okurz: At least here: https://openqa.opensuse.org/tests/822879#step/user_settings/2  but as auto-login works on x86, it can be hidden
[03/01/2019 15:51:40] <guillaume_g> okurz: qemu update: https://build.opensuse.org/package/rdiff/openSUSE:Leap:15.0:Update/qemu?linkrev=base&rev=5 seems to be a good candidate though
[03/01/2019 15:56:39] <guillaume_g> or not. the char retry patches are for serial, which is not used here.
[03/01/2019 16:18:47] <okurz> guillaume_g: https://openqa.opensuse.org/tests/823125#step/zypper_ref/8 fails because the repo is most likely already pruned. You may want to skip these test modules with "SKIP_MODULES=…" or we just regard the jobs failing here as "passed" because the installation at least passed as well as first_boot
[03/01/2019 16:31:15] <guillaume_g> okurz: yes, I will just ignore this failure
[03/01/2019 16:32:07] <guillaume_g> but it fails later with keys missed: https://openqa.opensuse.org/tests/823123#step/prepare_system_for_update_tests/2
[03/01/2019 17:02:13] <okurz> guillaume_g: so this is supporting the hypothesis that the updates on the worker caused this problem. I can try to revert to an older root filesystem snapshot on the worker host, Should I try that and maybe go back to the oldest snapshot 2018-12-19T00:14:59 ?
[03/01/2019 17:09:55] <guillaume_g> ok,yeah, they are all failing so far! So, please try to revert to oldest snapshot.

On the aarch64 machine:

# snapper list -a
Type   | #   | Pre # | Date                    | User | Cleanup | Description             | Userdata     
-------+-----+-------+-------------------------+------+---------+-------------------------+--------------
single | 0   |       |                         | root |         | current                 |              
single | 105 |       | 2018-12-19T00:14:59 CET | root | number  | Snapshot Update of #104 | important=yes
single | 106 |       | 2018-12-20T00:40:39 CET | root | number  | Snapshot Update of #105 | important=yes
single | 107 |       | 2018-12-21T00:17:16 CET | root | number  | Snapshot Update of #106 | important=yes
single | 108 |       | 2018-12-22T00:52:40 CET | root | number  | Snapshot Update of #107 | important=yes
single | 109 |       | 2018-12-23T00:53:52 CET | root | number  | Snapshot Update of #108 | important=yes
single | 110 |       | 2018-12-25T01:18:09 CET | root | number  | Snapshot Update of #109 | important=yes
single | 111 |       | 2018-12-26T01:57:43 CET | root | number  | Snapshot Update of #110 | important=yes
single | 112 |       | 2018-12-28T00:34:33 CET | root | number  | Snapshot Update of #111 | important=yes
single | 113 |       | 2018-12-29T00:34:04 CET | root | number  | Snapshot Update of #112 | important=yes
single | 114 |       | 2018-12-30T01:51:22 CET | root | number  | Snapshot Update of #113 | important=yes
single | 115 |       | 2019-01-01T01:45:06 CET | root |         | Snapshot Update of #114 |              
single | 116 |       | 2019-01-03T01:14:56 CET | root |         | Snapshot Update of #115 |              
# transactional-update rollback 105
transactional-update 2.11 started
Options: rollback 105
Separate /var detected.
Separate /etc detected.
Rollback to snapshot 105...
Please reboot to finish rollback!
# reboot

After reboot:

# zypper info openQA-worker
Loading repository data...
Reading installed packages...


Information for package openQA-worker:
--------------------------------------
Repository     : Providing openQA dependencies (openSUSE_Leap_15.0)                  
Name           : openQA-worker                                                       
Version        : 4.6.1545406149.53968c1e-lp150.1055.1                                
Arch           : noarch                                                              
Vendor         : obs://build.opensuse.org/devel:openQA                               
Installed Size : 13.1 KiB                                                            
Installed      : Yes                                                                 
Status         : out-of-date (version 4.6.1545139030.66c0b50f-lp150.1023.1 installed)
Source package : openQA-4.6.1545406149.53968c1e-lp150.1055.1.src                     
Summary        : The openQA worker                                                   
Description    :                                                                     
    The openQA worker manages test engine (provided by os-autoinst package).

so I assume we are on the old version again. I am not sure what will happen during the next update window if that snapshot will be preserved or if the system still auto-updates to the most recent state but anyway we should have enough time to run tests and gather statistics. Some jobs already started again.

Triggered tests again with:

for i in {001..100}; do openqa-clone-job --from https://openqa.opensuse.org --host https://openqa.opensuse.org --skip-download --skip-chained-deps 822796 TEST=ggardet_poo45650_$i _GROUP="Development Tumbleweed" BUILD="poo45650_investigation_aarch64_worker_snapshot_105_2018-12-19T00:14:59"; done

-> https://openqa.opensuse.org/tests/overview?build=poo45650_investigation_aarch64_worker_snapshot_105_2018-12-19T00:14:59&distri=opensuse&version=Tumbleweed&groupid=38

#9 Updated by okurz over 2 years ago

  • Status changed from In Progress to Feedback

With the tests running on the old worker packages we found already two cases of mistyping: https://openqa.opensuse.org/tests/823316#step/user_settings/2 and https://openqa.opensuse.org/tests/823318#step/user_settings/2 , see the "m." and "w".

Created new needle "inst-userinfostyped-99p_match-20190103" with a little lower match area and 99% match level which should prevent the false-matches like https://openqa.opensuse.org/tests/823316#step/user_settings/2 and https://openqa.opensuse.org/tests/823318#step/user_settings/2
Proposed deletion of the old one in https://github.com/os-autoinst/os-autoinst-needles-opensuse/pull/492

@waitfor results of https://openqa.opensuse.org/tests/overview?build=poo45650_investigation_aarch64_worker_snapshot_105_2018-12-19T00:14:59&distri=opensuse&version=Tumbleweed&groupid=38 and the next update window of the worker host if it has any effect.

#10 Updated by okurz over 2 years ago

  • Assignee changed from okurz to ggardet_arm

Over night the worker was updated automatically again so the rollback was temporary. 53/100 jobs have been obsoleted, probably due to a more recent openSUSE Tumbleweed snapshot. Many jobs failed. The most recent job was https://openqa.opensuse.org/tests/823366 that failed at 00:18 UTC so way before the automatic worker update meaning that all jobs have been executed with the old state of worker packages. 26/39 failed so 67% so therefore I reject the hypothesis that any package updates on the worker installed since 2018-12-19 cause any regression.

back to ggardet_arm for the next ideas ;)

#11 Updated by okurz over 2 years ago

  • Copied to action #45713: [functional][u] test for nested virtualization / qemu / typing over VNC added

#12 Updated by ggardet_arm over 2 years ago

Setting 'QEMUMACHINE=virt,gic-version=host' to use host GIC version seems to improve the current problem in 'user_settings', but still fails later, where 'echo ' become 'ech o' in prepare_system_for_update_tests. See: https://openqa.opensuse.org/tests/overview?build=poo45650_investigation_gic_version&distri=opensuse&version=Tumbleweed

Lower VNC_TYPING_LIMIT improve things too, but I must find the right value to improve things, without becoming too slow. 10 triggers the problem, 1 seems to be fine, but very slow.

#13 Updated by okurz over 2 years ago

The idea was raised to compare against the state on the openqa.suse.de aarch64 workers as the package updates should be equivalent. However we have to keep in mind that the SLE workers are based on SLE12SP3. The latest deployment with package updates was on 2018-12-10 (from /var/log/zypp/history) and latest SLE15SP1 tests were conducted 17 days ago. I suggest to wait for the next deployment with package updates and see how the systems compare.

ggardet_arm wrote:

I will try to reproduce locally with a 15.0 system.

Any results?

#14 Updated by ggardet_arm over 2 years ago

okurz wrote:

ggardet_arm wrote:

I will try to reproduce locally with a 15.0 system.

Any results?

No, I had problems to set-up a working 15.0 chroot with qemu (because of missing noarch qemu-ipxe package in 15.0 in aarch64 repo). Now, it is fixed, tests are running. :)

#15 Updated by ggardet_arm over 2 years ago

I managed to reproduce this kind of failure locally (missing key), but not this specific failure.
BTW, setting 'QEMUMACHINE=virt,gic-version=host' to use host GIC version instead of the default GICv2, improves things a lot. (aarch64 machine updated accordingly)

If I copy a huge file over network to the worker while worker is testing, then I got a 'cat /etc/reeeeeeeeeeeeesolv.conf', with lots of 'e'.

#16 Updated by ggardet_arm over 2 years ago

Something to note is aarch64 kernel has CONFIG_PREEMPT_NONE=y instead of CONFIG_PREEMPT=y (contrary to most archs) so it is less responsive and could cause this kind of failure.

#17 Updated by mgriessmeier over 2 years ago

  • Related to action #46190: [functional][u] test fails in user_settings - mistyping in Username (lowercase instead of uppercase) added

#18 Updated by mgriessmeier over 2 years ago

also happens on TW on 32bit, see https://progress.opensuse.org/issues/46190 for details

#19 Updated by okurz over 2 years ago

  • Target version changed from Milestone 22 to Milestone 23

ggardet_arm after a month, how would you describe the current state?

#20 Updated by ggardet_arm over 2 years ago

okurz wrote:

ggardet_arm after a month, how would you describe the current state?

This problem does not occur very often (at all?) those days on o3. And the problem is now caught earlier in user_settings, which avoid a useless installation if it occurs.

#21 Updated by okurz over 2 years ago

alright. So what do you suggest to do to continue with this ticket?

#22 Updated by ggardet_arm over 2 years ago

  • Status changed from Feedback to Resolved

okurz wrote:

alright. So what do you suggest to do to continue with this ticket?

Set to resolved.

#23 Updated by okurz over 1 year ago

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: btrfs_libstorage-ng
https://openqa.suse.de/tests/3953797

To prevent further reminder comments one of the following options should be followed:

  1. The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
  2. The openQA job group is moved to "Released"
  3. The label in the openQA scenario is removed

Also available in: Atom PDF