openQA Project (public)

It seems to be ThunderX1 and ThunderX2 machines. From my experience, they are not good as openQA workers and better to be used to build packages.

You can check/try:

• to disable hardware threading in firmware (it will lower the number of CPU seen by the kernel)
• check actual CPU frequency
• check temperature (cpu throttling could slow down cpu freq and you get lower perfs)
• Use single socket instead of dual sockets (may be configurable in the firmware)
• Use a distribution without LSE-atomics (known to be slow on TX2)
• Disable mitigation (KPTI, etc.)
• enable/disable huge pages
• You can also run sudo perf stat while the system is busy with openQA tests

Thanks. I'll look into these points.

Because it came up as well: The failing tests seem to be a mix of typing issues and connection issues. Here are some examples:

• https://openqa.suse.de/tests/7631673
• https://openqa.suse.de/tests/7631458
• https://openqa.suse.de/tests/7632312
• https://openqa.suse.de/tests/7632463
• https://openqa.suse.de/tests/7632073
• https://openqa.suse.de/tests/7632006
• https://openqa.suse.de/tests/7631965

(via select id, t_finished, result, reason from jobs where (select host from workers where id = assigned_worker_id) = 'openqaworker-arm-4' and (result = 'failed') and t_finished >= '2021-08-24T00:00:00')

You can also play with openQA settings:

• VNC_TYPING_LIMIT
• TIMEOUT_SCALE
• QEMU_COMPRESS_LEVEL
• QEMU_COMPRESS_THREADS
• QEMUCPU (I recommend host)
• QEMUMACHINE (I recommend virt,gic-version=host)

and of courses, the number of parallel openQA jobs.

journalctl | grep stuck prints nothing on arm-4/5 (unlike to aarch64) and the logs reach back to November. So we definitely see a different problem than on aarch64 (where we frequently see message like kernel:[480826.136444] watchdog: BUG: soft lockup - CPU#49 stuck for 26s! [qemu-system-aar:13803]).

Instead we get lots of Nov 13 22:55:19 openqaworker-arm-5 kernel: ACPI CPPC: PCC check channel failed for ss: 0. ret=-110 and Nov 13 22:55:20 openqaworker-arm-5 kernel: CPPC Cpufreq:cppc_scale_freq_workfn: failed to read perf counters messages (on arm-4 and arm-5).

Apparently we're not the only ones seeing those messages with a ThunderX2 CPU: https://bugzilla.kernel.org/show_bug.cgi?id=208785

@ggardet_arm suggested to disable frequency scaling when I brought up these error messages (also see https://wiki.archlinux.org/title/CPU_frequency_scaling).

Considering https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1763817 the firmware setting cppccontrol=1 is relevant so we might try disabling this setting.

Independent of fixing the actual performance problemsI think it can help in general for openQA tests if we configure the systems we test in with a disabled key repeat to mitigate the issues, i.e. "xset r off" within X11 sessions or whatever can be seen as equivalent in non-X11 getty terminals. According to fvogt "There's also an ioctl for the vtcon". The command "kbdrate" allows to configure something but says only for "Intel" and does not allow to disable key repeat completely, only increase times, which could already help enough. By default keyboard delay is "250 ms". Any different setting likely needs to be applied per VT. So in a true text tty one could use kbdrate -d 1000 and in X xset r off whenever we start a new terminal.

• Looks like there are nevertheless still problems with the network connection: https://openqa.suse.de/tests/8639166#step/setup/29
• I've also already spotted typing issues: https://openqa.suse.de/tests/8639164#step/setup/27 (character typed too often), https://openqa.suse.de/tests/8639174 (character " missing)

With cppccontrol=0 the ACPI error mentioned in #109232#note-10 isn't occurring again. Considering the job results I've seen so far it likely still doesn't help with the typing issues, though. However, I'll wait for all jobs to finish for a conclusion (as also the other arm workers have some typing issues).

Further typing issues:

• https://openqa.suse.de/tests/8639168#step/system_prepare#1/11 (not 100% sure but likely some typing issue)
• https://openqa.suse.de/tests/8639169#step/patch_sle/128 (missing characters)
• https://openqa.suse.de/tests/8639188#step/conman/1 (missing ()
• https://openqa.suse.de/tests/8639283#step/installation_overview/3 (missing o)
• https://openqa.suse.de/tests/8639302#step/hostname_inst/6 (missing u)
• https://openqa.suse.de/tests/8639271#step/setup/41 (missing d)

33 % failures is not good (within a set of jobs that passed on other arm workers before):

openqa=> with test_jobs as (select distinct id, result from jobs where build = 'test-arm4-2') select result, count(id) * 100. / (select count(id) from test_jobs) as ratio from test_jobs group by test_jobs.result order by ratio desc;
      result      |         ratio          
------------------+------------------------
 passed           |    40.8510638297872340
 failed           |    33.6170212765957447
 parallel_failed  |    14.8936170212765957
 incomplete       |     8.5106382978723404
 timeout_exceeded | 0.85106382978723404255
 user_cancelled   | 0.85106382978723404255
 softfailed       | 0.42553191489361702128
(7 Zeilen)

I now disabled cpu frequency scaling via the firmware menu:

CAVM_CN99xx# env set progcpufreq 0
CPU Freq set by SYS will be used 
Env Var progcpufreq set with Value 0 
Execute 'env save' Command to make the changes persistent 
CAVM_CN99xx# env save
drivername snor
snor_erase: off=0x3ff0000, len=0x10000

-----------------------------------
       ENV Variable Settings 
-----------------------------------
Name                  : Value 
-----------------------------------
turbo                 : 2 
smt                   : 4 
corefreq              : 2199 
numcores              : 32 
icispeed              : 1 
socnclk               : 666 
socsclk               : 1199 
memclk                : 2199 
ddrspeed_auto         : 1 
ddrspeed              : 2400 
progcpufreq           : 0 
progdevfreq           : 1 
dmc_node_channel_mask : 0000ffff 
thermcontrol          : 1 
thermlimit            : 110 
enter_debug_shell     : 0 
dbg_speed_up_ddr_lvl  : 0 
enable_dram_scrub     : 0 
ipmbcontrol           : 1
ddr_dmt_advanced      : 0 
cppccontrol           : 0
loglevel              : 0
uart_params           : 115200/8-N-1 none
core_feature_mask     : 0
sys_feature_mask      : 0x00000000
ddr_refresh_rate      : 1
fw_feature_mask       : 0x00000000
dram_ce_threshold     : 1
dram_ce_step_threshold: 0
dram_ce_record_max    : 10
dram_ce_window        : 60 sec
dram_ce_leak_rate     : 2000 msec/error
pcie_ce_threshold     : 1
pcie_ce_window        : 30 sec
pcie_ce_leak_rate     : 15000 msec/error
-----------------------------------

Another round of test jobs: https://openqa.suse.de/tests/overview?build=test-arm4-4

And it already looks bad - so here some more typing issues:

• https://openqa.suse.de/tests/8646939#step/setup/3 (casing)
• https://openqa.suse.de/tests/8646941#step/patch_sle/14 (casing)
• https://openqa.suse.de/tests/8646960#step/before_test/14 (missing A)
• https://openqa.suse.de/tests/8646948#step/wait_children/3 (missing O)

Disabling cpu frequency scaling didn't help much, also considering all the test jobs:

openqa=> with test_jobs as (select distinct id, result from jobs where build = 'test-arm4-4') select result, count(id) * 100. / (select count(id) from test_jobs) as ratio from test_jobs group by test_jobs.result order by ratio desc;
     result      |         ratio          
-----------------+------------------------
 failed          |    45.2054794520547945
 passed          |    37.8995433789954338
 parallel_failed |    11.8721461187214612
 incomplete      |     4.1095890410958904
 softfailed      | 0.91324200913242009132
(5 Zeilen)