Project

General

Profile

action #23554

[sle][functional][tools][yast][y][hard] test fails in iscsi_configuration because of missing updated needle iscsi-ibft / missing iscsi server on osd?

Added by Anonymous almost 3 years ago. Updated about 2 years ago.

Status:
Resolved
Priority:
High
Assignee:
Category:
Bugs in existing tests
Start date:
2017-08-23
Due date:
2018-05-22
% Done:

0%

Estimated time:
Difficulty:
hard
Duration: 195

Description

Observation

Needles should be updated to make sure test runs.

openQA test in scenario sle-15-Leanos-DVD-x86_64-iscsi_ibft@64bit fails in
iscsi_configuration

and now the tab 'iBFT' does not show up anymore

Reproducible

Fails since (at least) Build 93.17

Expected result

Last good: SLE 12 SP3 GM

Suggestions

  • Read bsc#1077629
  • Ensure the iSCSI testing infrastructure (server service on osd?) is working properly. As we had a problem recently that the samba server "vanished" some time between 2017-07 and 2017-12 and I had to reinstall it I assume the same could have happened for other ressources. At least we can see it works sometimes or at least 17 days ago: #23554#note-2
  • Confirm it manually/locally
  • Create updated needles, adapt to recent SLE15 behaviour if any
  • Crosscheck SLE12SP4 and SLE15 test

Further details

Always latest result in this scenario: latest, formerly latest Leanos-DVD


Related issues

Related to openQA Tests - action #28276: [sle][functional][medium][yast][y] test fails in iscsi_server - needs needle update and further investigationResolved2017-11-232018-05-22

Related to openQA Tests - action #32452: [sle][funtional][y][easy] test fails in iscsi_client need adaptionResolved2018-02-28

Related to openQA Tests - action #36060: [functional][y][tools][fast] Investigate iscsi_ibft test scenario failureResolved2018-05-102018-05-22

Has duplicate openQA Tests - action #30823: [sle][functional][tools][iscsi][hard] test is stuck in linuxrc asking if dhcp should be used - since worker upgrade/qemu/os-autoinst?Resolved2018-01-252018-03-27

Blocks openQA Tests - action #23558: [sle][functional][sle15][easy][y] test fails in module partitioning_iscsi - probably needs adaption to testcodeResolved2017-08-232018-05-22

Blocks openQA Tests - action #30685: [sle][functional][sle12 sp4][u] test fails in firefox - firefox_readerview_window doesn't matchResolved2018-01-232018-06-19

History

#1 Updated by okurz over 2 years ago

  • Subject changed from test fails in iscsi_configuration because of missing updated needle iscsi-ibft to [sle][functional][medium][tools]test fails in iscsi_configuration because of missing updated needle iscsi-ibft / missing iscsi server on osd?
  • Description updated (diff)
  • Due date set to 2018-03-13
  • Category changed from Enhancement to existing tests to Bugs in existing tests
  • Status changed from New to Workable
  • Target version set to Milestone 15

The latest job in the same scenario for "Leanos" still showed the tab https://openqa.suse.de/tests/1192623#step/iscsi_configuration/5 with missing needle but now we have a different problem also described in bsc#1077629 which seems to be related to our infrastructure.

Link to latest updated to "Installer-DVD"

#2 Updated by okurz over 2 years ago

  • Blocks action #23558: [sle][functional][sle15][easy][y] test fails in module partitioning_iscsi - probably needs adaption to testcode added

#3 Updated by okurz over 2 years ago

  • Related to action #28276: [sle][functional][medium][yast][y] test fails in iscsi_server - needs needle update and further investigation added

#4 Updated by okurz over 2 years ago

  • Related to action #30823: [sle][functional][tools][iscsi][hard] test is stuck in linuxrc asking if dhcp should be used - since worker upgrade/qemu/os-autoinst? added

#5 Updated by okurz over 2 years ago

  • Description updated (diff)

… But actually it works sometimes so it's not that the server is really missing or if at all, it's only missing since 17 days, see https://openqa.suse.de/tests/1404134 progressing further to partitioning_iscsi

#6 Updated by okurz over 2 years ago

  • Blocks action #30685: [sle][functional][sle12 sp4][u] test fails in firefox - firefox_readerview_window doesn't match added

#7 Updated by okurz over 2 years ago

  • Subject changed from [sle][functional][medium][tools]test fails in iscsi_configuration because of missing updated needle iscsi-ibft / missing iscsi server on osd? to [sle][functional][medium][tools][yast]test fails in iscsi_configuration because of missing updated needle iscsi-ibft / missing iscsi server on osd?
  • Target version changed from Milestone 15 to Milestone 14

#8 Updated by okurz about 2 years ago

  • Related to action #32452: [sle][funtional][y][easy] test fails in iscsi_client need adaption added

#9 Updated by riafarov about 2 years ago

  • Assignee set to riafarov

#10 Updated by riafarov about 2 years ago

  • Status changed from Workable to Blocked

So, as result of investigation I've assigned original bug to qemu dev to take a look as it's our best guess. I was able to reproduce the issue. We have workers in place with iscsi configured and that's not the problem. I left my orthos instance running, so iscsi can be accessed there (iscsi:10.161.8.141::3260:0:iqn.2016-02.openqa.de:for.openqa). See no way to proceed here, but I was able to make this functionality work with tap setup, which we may apply on OSD, but that's not cheap solution. Let's track updates on https://bugzilla.suse.com/show_bug.cgi?id=1077629

#11 Updated by riafarov about 2 years ago

  • Related to deleted (action #30823: [sle][functional][tools][iscsi][hard] test is stuck in linuxrc asking if dhcp should be used - since worker upgrade/qemu/os-autoinst?)

#12 Updated by riafarov about 2 years ago

  • Has duplicate action #30823: [sle][functional][tools][iscsi][hard] test is stuck in linuxrc asking if dhcp should be used - since worker upgrade/qemu/os-autoinst? added

#13 Updated by riafarov about 2 years ago

  • Status changed from Blocked to In Progress

#14 Updated by riafarov about 2 years ago

ibft setup is correct, I also was able to make it work in my local setup with iscsi: http://gershwin.arch.suse.de/tests/133#step/iscsi_configuration/3
Seems, something wrong in production, maybe FW rules, which are different. Same ipxe kernel file works fine in local setup.

#15 Updated by riafarov about 2 years ago

  • Subject changed from [sle][functional][medium][tools][yast]test fails in iscsi_configuration because of missing updated needle iscsi-ibft / missing iscsi server on osd? to [sle][functional][hard][tools][yast] test fails in iscsi_configuration because of missing updated needle iscsi-ibft / missing iscsi server on osd?

#16 Updated by riafarov about 2 years ago

  • Due date changed from 2018-03-13 to 2018-03-27

#17 Updated by riafarov about 2 years ago

Reproduced manually onopenqaworker6, firewall is not affecting results, works fine if using 10.0.2.2 as iscsi ip address. We will need to change openQA to use that value, as cannot overwrite WORKER_HOSTNAME in the test suite.

#18 Updated by okurz about 2 years ago

  • Target version changed from Milestone 14 to Milestone 15

#19 Updated by riafarov about 2 years ago

Investigated further with szarate. I've spawn the VM and it worked when using 10.0.2.2 ip address for iscsi as well as the one set in WORKER_HOSTNAME. With local hot patch we've performed test run https://openqa.suse.de/tests/1538954 but it failed even with 10.0.2.2. Seems to be related to GRE tunnels.

#20 Updated by riafarov about 2 years ago

  • Status changed from In Progress to Workable

Will continue on this one on Monday with @szarate.

#21 Updated by riafarov about 2 years ago

  • Status changed from Workable to Blocked

Santiago is on sick leave, so not possible to progress on this ticket as of now.

#22 Updated by riafarov about 2 years ago

  • Due date changed from 2018-03-27 to 2018-04-10
  • Status changed from Blocked to Workable
  • Priority changed from Normal to High

#23 Updated by okurz about 2 years ago

  • Due date changed from 2018-04-10 to 2018-04-24
  • Assignee deleted (riafarov)

We will not be able to do this one efficiently, szarate will not be available. It's ok to do it later which will of course delay our chain so I will update the blocked ticket accordingly as well.

Next step: Wait for szarate and fix server on osd infrastructure with him.

#24 Updated by okurz about 2 years ago

  • Subject changed from [sle][functional][hard][tools][yast] test fails in iscsi_configuration because of missing updated needle iscsi-ibft / missing iscsi server on osd? to [sle][functional][hard][tools][yast][y] test fails in iscsi_configuration because of missing updated needle iscsi-ibft / missing iscsi server on osd?

#25 Updated by okurz about 2 years ago

  • Subject changed from [sle][functional][hard][tools][yast][y] test fails in iscsi_configuration because of missing updated needle iscsi-ibft / missing iscsi server on osd? to [sle][functional][tools][yast][y][hard] test fails in iscsi_configuration because of missing updated needle iscsi-ibft / missing iscsi server on osd?
  • Difficulty set to hard

#26 Updated by riafarov about 2 years ago

  • Assignee set to riafarov

We've tried test on openqaworker2, where we don't have GRE tunnels, still didn't work. Also launched on arm: https://openqa.suse.de/tests/1631090#settings
As a next step we'll try to replicate the setup and check closely, as we don't want to break production env.

#27 Updated by riafarov about 2 years ago

So, what we have, is that when we try to configure iBFT we have eth0 not configured:
09:08:24 : exec: wicked ifup all = 0
09:08:24 : stdout + stderr:
lo up
09:08:25 : net_update_state:
09:08:25 : lo: up,
09:08:25 : eth0: device-unconfigured
09:08:25 :
Which is not the case in successful runs. Meaning that iPXE cannot successfully configure network. Looking for the root cause of this behavior.

#28 Updated by riafarov about 2 years ago

  • Status changed from Workable to In Progress

#29 Updated by riafarov about 2 years ago

  • Status changed from In Progress to Feedback

#30 Updated by mgriessmeier about 2 years ago

  • Due date changed from 2018-04-24 to 2018-05-08
  • Target version changed from Milestone 15 to Milestone 16

as agreed with riafarov, moving to Sprint 16

#31 Updated by riafarov about 2 years ago

Changes will be deployed tomorrow.

#32 Updated by okurz about 2 years ago

needles merged

#33 Updated by okurz about 2 years ago

  • Status changed from Feedback to Workable
  • Assignee deleted (riafarov)

someone else please pick up, optionally discuss with szarate what's the current state and next step, and continue

#34 Updated by riafarov about 2 years ago

  • Assignee set to riafarov

Changes not yet deployed to osd.

#35 Updated by riafarov about 2 years ago

  • Due date changed from 2018-05-08 to 2018-05-22

Deployment is planned for Wednesday (09/05/2018). After that can be resolved (of course if works).

#36 Updated by riafarov about 2 years ago

  • Status changed from Workable to In Progress

#37 Updated by riafarov about 2 years ago

  • Status changed from In Progress to Feedback

Test fails sporadically, as can have existing installations on iscsi disk.
https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/4999

#38 Updated by riafarov about 2 years ago

  • Status changed from Feedback to Resolved

#39 Updated by okurz about 2 years ago

  • Related to action #36060: [functional][y][tools][fast] Investigate iscsi_ibft test scenario failure added

Also available in: Atom PDF