QA » openQA Project » openQA Infrastructure

I've now added kernel parameters that we also have on the o3 worker aarch64:

martchus@openqaworker-arm-4:~> cat /proc/cmdline
BOOT_IMAGE=/boot/Image-5.3.18-150300.59.63-default root=UUID=be776b2a-53e6-458c-9ab6-c35b63e4a834 console=tty0 console=ttyAMA0,115200 nospec kvm.nested=1 kvm_intel.nested=1 kvm_amd.nested=1 kvm-arm.nested=1 crashkernel=210M mitigations=off default_hugepagesz=1G hugepagesz=1G hugepages=64 enforcing=0

So now mitigations are disabled and huge pages are enabled similarly to aarch64.

I've been cloning the last 100 passing jobs from OSD to see whether it makes a difference: https://openqa.suse.de/tests/overview?build=test-arm4-3

A short review of the test results we've go so far already reveals quite a lot of typing issues.

Typing issues:

• https://openqa.suse.de/tests/8798813#step/setup/82
• https://openqa.suse.de/tests/8798628#step/before_test/18
• https://openqa.suse.de/tests/8799018#step/before_test/10
• https://openqa.suse.de/tests/8799075#step/installation_overview/3
• https://openqa.suse.de/tests/8798877#step/force_scheduled_tasks/8
• https://openqa.suse.de/tests/8798697#step/barrier_init/15
• https://openqa.suse.de/tests/8798644#step/setup/73
• https://openqa.suse.de/tests/8800122#step/snapper_thin_lvm/12
• https://openqa.suse.de/tests/8800121#step/consoletest_setup/71
• https://openqa.suse.de/tests/8799373#step/pam_su/6

Possibly typing issue:

• https://openqa.suse.de/tests/8800124#step/pam_faillock/28

Other failures / not sure about the cause:

• https://openqa.suse.de/tests/8798760#step/consoletest_setup/8
• https://openqa.suse.de/tests/8798752#step/iscsi_client/22
• https://openqa.suse.de/tests/8799090#step/hostname_inst/6
• https://openqa.suse.de/tests/8798979#step/scc_registration/6
• https://openqa.suse.de/tests/8799369#step/command_not_found/18
• https://openqa.suse.de/tests/8799133#step/update_kernel/87
• https://openqa.suse.de/tests/8799127#step/madvise06/7

The overall fail rate is still quite high:

openqa=> with test_jobs as (select distinct id, state, result from jobs where build = 'test-arm4-3') select state, result, count(id) * 100. / (select count(id) from test_jobs) as ratio from test_jobs group by test_jobs.state, test_jobs.result order by ratio desc;
 state |     result      |         ratio          
-------+-----------------+------------------------
 done  | passed          |    59.1836734693877551
 done  | failed          |    24.4897959183673469
 done  | parallel_failed |    15.6462585034013605
 done  | softfailed      | 0.68027210884353741497
(4 Zeilen)

I've updated the previous comment. I guess the number of typing issues is still too high to consider using mitigations=off default_hugepagesz=1G hugepagesz=1G hugepages=64 kernel parameters an improvement.

I now disabled progdevfreq in the firmware (after previously only disabling progcpufreq). Not sure how I'd disable hardware threading in firmware (as mentioned in suggestions). The same counts for useing a single socket instead of dual sockets.

So that's what the current firmware settings are:

CAVM_CN99xx# env save
drivername snor
snor_erase: off=0x3ff0000, len=0x10000

-----------------------------------
       ENV Variable Settings 
-----------------------------------
Name                  : Value 
-----------------------------------
turbo                 : 2 
smt                   : 4 
corefreq              : 2199 
numcores              : 32 
icispeed              : 1 
socnclk               : 666 
socsclk               : 1199 
memclk                : 2199 
ddrspeed_auto         : 1 
ddrspeed              : 2400 
progcpufreq           : 0 
progdevfreq           : 0 
dmc_node_channel_mask : 0000ffff 
thermcontrol          : 1 
thermlimit            : 110 
enter_debug_shell     : 0 
dbg_speed_up_ddr_lvl  : 0 
enable_dram_scrub     : 0 
ipmbcontrol           : 1
ddr_dmt_advanced      : 0 
cppccontrol           : 0
loglevel              : 0
uart_params           : 115200/8-N-1 none
core_feature_mask     : 0
sys_feature_mask      : 0x00000000
ddr_refresh_rate      : 1
fw_feature_mask       : 0x00000000
dram_ce_threshold     : 1
dram_ce_step_threshold: 0
dram_ce_record_max    : 10
dram_ce_window        : 60 sec
dram_ce_leak_rate     : 2000 msec/error
pcie_ce_threshold     : 1
pcie_ce_window        : 30 sec
pcie_ce_leak_rate     : 15000 msec/error
-----------------------------------

Btw, I've just found: https://en.opensuse.org/HCL:ThunderX2 - Somehow I doubt these are "the best processors".

I invoked cvmrundiag in the hope it would maybe print something useful. However, it left the system in a broken state where neither power cycle nor power reset help. I'm currently trying a factory reset (hopefully preserving most of the settings for authentication/IPMI).

found https://www.gigabyte.com/Enterprise/ARM-Server/R181-T92-rev-100#Support-Bios . Downloaded server_system_boot_mt91-fsx_f34.zip, extracted from that "image.RBU" and flashed that over https://ipmi.openqaworker-arm-4.qa.suse.de/#maintenance/firmware_update_wizard selecting "Update Type: BIOS". There could be options for CPDLD and BMC itself it seems.

Updated, system behaves the same as reported in https://progress.opensuse.org/issues/110545#note-10 . I wonder how we can configure boot devices.

I saved all configuration from BMC to a local ZIP file. Now restorting factory defaults. This saves all entries listed in a checklist so if this has no effect we likely need to configure the system to not preserve that much, then try again.

No effect, same behaviour. Configured on https://ipmi.openqaworker-arm-4.qa.suse.de/#maintenance/preserve_configuration to not preserve anything except IPMI(+network),Authentication so that our remote access password should be preserved.

Conducted a "factory reset", no difference. Then with nsinger being my witness I connected to https://ipmi.openqaworker-arm-5.qa.suse.de/ and in the remote control "h5viewer", likely HTML5 viewer which actually looks quite nice and usable that showed the picture of a getty session with some linux messages on the screen and the serial console showed a getty as well, so working nicely. Then I triggered a "power reset" and the machine started a reboot and it looked like there would be similar output as #110545#note-10 but with some "ff ff" included or some non-zero output. Then I triggered a "power cycle" (while the system was still booting) to resemble what mkittler reported he did on arm-4 and we actually ended up with the same symptoms, system does not boot anymore and "Header" shows only zeroes

okurz wrote:

Asked BuildOPS if they would be interested for a trade, see #110539

The answer for #110539 is "No". awaiting response from szarate in https://suse.slack.com/archives/C02CANHLANP/p1664966773773939

(Oliver Kurz) @Santiago Zarate for https://progress.opensuse.org/issues/110545, what's the result regarding ARM workers discussion with afaerber?

There was no update by szarate yet so I reminded them in
https://suse.slack.com/archives/C02CANHLANP/p1665563709438689?thread_ts=1664966773.773939&cid=C02CANHLANP

@Santiago Zarate still missing the update from above, plz