Project

General

Profile

Actions

action #162272

closed

[qe-tools][RPi3] test fails in prepare_firstboot - use retries with exponential back-off in os-autoinst size:S

Added by zluo 5 months ago. Updated 5 months ago.

Status:
Resolved
Priority:
High
Assignee:
Category:
Feature requests
Target version:
Start date:
2024-06-14
Due date:
% Done:

0%

Estimated time:

Description

Observation

I did a manual installation on my RPi 3 without any issue. It looks like a setup issue on openQA.

openQA test in scenario sle-15-SP6-JeOS-for-RaspberryPi-aarch64-jeos-realhw-RPi@RPi3B fails in
prepare_firstboot

Test suite description

Run a test on real RPi hardware. To be used eg. with RPi4 machine.
Setup: https://confluence.suse.com/pages/viewpage.action?spaceKey=~dheidler&title=Hardware+Automation

Reproducible

Fails since (at least) Build 6.75 (current job)

Expected result

Last good: 6.74 (or more recent)

Suggestions

  • Look into what was found in #158404 which was about the last time this test failed ~1 month ago
  • This looks to be failing sporadically only? Try and reproduce it - take care to only run first_reboot to avoid reflashing the disk
  • Meanwhile the scenario is ok again but we want to improve. The call select_console('root-ssh') establishes an ssh connection within the os-autoinst backend and as visible in https://openqa.suse.de/tests/14266260/logfile?filename=autoinst-log.txt there is already retrying but seemingly only 5 times up to 1m, not longer. We should ensure that there are longer retries with exponential backoff up to let's say 30m
  • Add exponential retry in https://github.com/os-autoinst/os-autoinst/blob/master/backend/baseclass.pm#L1211
  • Consider using a higher value than 5 for the specific test scenario or machine related to RPi

Further details

Always latest result in this scenario: latest

Actions #1

Updated by okurz 5 months ago

  • Tags changed from bugbusters to bugbusters, reactive work, infra, rpi
  • Category set to Regressions/Crashes
  • Assignee deleted (dheidler@suse.de)
  • Target version set to Ready
Actions #2

Updated by livdywan 5 months ago

  • Subject changed from [qe-tools][RPi3] test fails in prepare_firstboot to [qe-tools][RPi3] test fails in prepare_firstboot size:S
  • Description updated (diff)
  • Status changed from New to Workable
Actions #3

Updated by okurz 5 months ago

  • Tags changed from bugbusters, reactive work, infra, rpi to bugbusters, reactive work, rpi
  • Subject changed from [qe-tools][RPi3] test fails in prepare_firstboot size:S to [qe-tools][RPi3] test fails in prepare_firstboot - use retries with exponential back-off in os-autoinst size:S
  • Category changed from Regressions/Crashes to Feature requests
Actions #4

Updated by livdywan 5 months ago

  • Status changed from Workable to In Progress
  • Assignee set to livdywan

I'm looking into the exponential retry on the backend side.

Actions #5

Updated by livdywan 5 months ago

  • Status changed from In Progress to Feedback
Actions #6

Updated by livdywan 5 months ago

  • Consider using a higher value than 5 for the specific test scenario or machine related to RPi

https://gitlab.suse.de/qac/qac-openqa-yaml/-/merge_requests/1703

Actions #7

Updated by livdywan 5 months ago

  • Status changed from Feedback to Resolved

All merged. Please re-open the ticket should this not work, but I assume we're good here.

Actions

Also available in: Atom PDF