Project

General

Profile

coordination #60518

[functional][u][rpi][epic] Add support for real systems without IPMI or KVM-IP in openQA/os-autoinst

Added by ggardet_arm over 3 years ago. Updated over 2 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
ggardet_arm
Category:
Spike/Research
Target version:
-
Start date:
2019-12-04
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Difficulty:

Description

The goal of this ticket is to track progress of the support of real systems without IPMI or KVM-IP, in openQA/os-autoinst, such as Raspberry Pi boards and other ARM boards.
We need to:

  1. Power ON/OFF the board (already supported by the generalhw backend, we just need hardware and scripts) Done
  2. Flash the board (use another board with USB mass storage gadget to emulate a USB disk or use a USB switch to share a USB stick across hosts) Done
  3. Use SSH consoles instead of VNC Done
  4. Use real serial connection (optional, but useful before ssh is available) Done

The generalhw backend is designed for real systems connected to kvm-ip (keyboard, video, mouse over IP) systems with VNC support.
So, we could reuse this backend and make the VNC part optional and switch to SSH and serial consoles.

Subtasks

action #60650: [functional][u][rpi] Add a system to flash a remote USB disk (and or µSD card) for generalhw backendResolvedggardet_arm

action #60707: [functional][rpi] Add a way to remotely POWERON/POWEROFF RPi boardResolvedggardet_arm

Related issues

Blocks openQA Tests - action #61801: [opensuse][aarch64] Tests to run on aarch64 real hardware (RPi3/4)In Progress2020-01-06

History

#1 Updated by ggardet_arm over 3 years ago

  • Subject changed from Add support for real systems without IPMI in openQA/os-autoinst to Add support for real systems without IPMI or KVM-IP in openQA/os-autoinst
  • Description updated (diff)

#2 Updated by ggardet_arm over 3 years ago

1st patches (with problems to be fixed):
https://github.com/os-autoinst/os-autoinst/pull/1304
https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/9061

  • Changing console is broken: it tries to open a new ssh connection whereas it should re-use root-ssh, and it does not handle ssh Password prompt
  • If isotovideo waits for ssh connection to be available (for serial?) as the SUT is still booting, you cannot get a working ssh connection Fixed.

#3 Updated by ggardet_arm over 3 years ago

Ok, above problems are now fixed.
Time for deeper tests.

Sometimes tests are failing with:

[2019-12-04T16:50:42.852 CET] [debug] /var/lib/openqa/share/tests/opensuse/tests/console/force_scheduled_tasks.pm:31 called testapi::wait_serial
[2019-12-04T16:50:42.853 CET] [debug] <<< testapi::wait_serial(buffer_size=undef, no_regex=0, timeout=1005, expect_not_found=0, quiet=undef, record_output=undef, regexp='TOP-DONE')
[2019-12-04T16:50:42.920 CET] [debug] Driver backend collected unknown process with pid 27334 and exit status: 1
[2019-12-04T16:50:42.920 CET] [debug] Driver backend collected unknown process with pid 27336 and exit status: 84
[2019-12-04T16:50:42.920 CET] [debug] Driver backend collected unknown process with pid 27337 and exit status: 0
[2019-12-04T16:50:42.921 CET] [debug] Driver backend collected unknown process with pid 27338 and exit status: 0
[2019-12-04T16:50:43.163 CET] [debug] Backend process died, backend errors are reported below in the following lines:
unexpected end of data at /usr/lib/os-autoinst/consoles/VNC.pm line 934.

It seems that some processes (ssh connection? local Xvnc? ...?) are exiting unexpectedly.

#4 Updated by ggardet_arm over 3 years ago

  • Status changed from New to In Progress
  • Assignee set to ggardet_arm

#5 Updated by ggardet_arm over 3 years ago

ggardet_arm wrote:

Sometimes tests are failing with:

[2019-12-04T16:50:42.852 CET] [debug] /var/lib/openqa/share/tests/opensuse/tests/console/force_scheduled_tasks.pm:31 called testapi::wait_serial
[2019-12-04T16:50:42.853 CET] [debug] <<< testapi::wait_serial(buffer_size=undef, no_regex=0, timeout=1005, expect_not_found=0, quiet=undef, record_output=undef, regexp='TOP-DONE')
[2019-12-04T16:50:42.920 CET] [debug] Driver backend collected unknown process with pid 27334 and exit status: 1
[2019-12-04T16:50:42.920 CET] [debug] Driver backend collected unknown process with pid 27336 and exit status: 84
[2019-12-04T16:50:42.920 CET] [debug] Driver backend collected unknown process with pid 27337 and exit status: 0
[2019-12-04T16:50:42.921 CET] [debug] Driver backend collected unknown process with pid 27338 and exit status: 0
[2019-12-04T16:50:43.163 CET] [debug] Backend process died, backend errors are reported below in the following lines:
unexpected end of data at /usr/lib/os-autoinst/consoles/VNC.pm line 934.

It seems to be fixed by: https://github.com/os-autoinst/os-autoinst/pull/1310

#6 Updated by pvorel over 3 years ago

Great plan, thanks for working on it :).

#8 Updated by ggardet_arm over 3 years ago

Here is my current setup working properly so far (this is not from a full openQA setup, but I run manually isotovideo from worker machine):
We could merge worker and flasher if we find an aarch64 machine with a working OTG port.

#9 Updated by ggardet_arm over 3 years ago

I added the scripts I used to grab serial, flash image and power on/off the SUT to https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/9061

#10 Updated by mgriessmeier over 3 years ago

  • Subject changed from Add support for real systems without IPMI or KVM-IP in openQA/os-autoinst to [functional][u][rpi][epic] Add support for real systems without IPMI or KVM-IP in openQA/os-autoinst

#11 Updated by ggardet_arm over 3 years ago

  • Blocks action #61801: [opensuse][aarch64] Tests to run on aarch64 real hardware (RPi3/4) added

#12 Updated by ggardet_arm over 3 years ago

It should be ok now.
Hardware need to be added to o3.

#13 Updated by ggardet_arm over 3 years ago

  • Description updated (diff)

#14 Updated by ggardet_arm over 3 years ago

Here is an updated version where SUT serial is connected to flasher instead of openQA worker.

It requires:

#15 Updated by ggardet_arm about 3 years ago

I run tests on RPi3 (B and Bplus) on my local openQA setup since months now, so we can close this ticket and work on adding missing hardware to o3 with https://progress.opensuse.org/issues/61801

#16 Updated by ggardet_arm about 3 years ago

  • Status changed from In Progress to Resolved

#17 Updated by szarate over 2 years ago

  • Tracker changed from action to coordination

Also available in: Atom PDF