Project

General

Profile

Actions

action #23554

closed

[sle][functional][tools][yast][y][hard] test fails in iscsi_configuration because of missing updated needle iscsi-ibft / missing iscsi server on osd?

Added by Anonymous over 7 years ago. Updated over 6 years ago.

Status:
Resolved
Priority:
High
Assignee:
Category:
Bugs in existing tests
Start date:
2017-08-23
Due date:
2018-05-22
% Done:

0%

Estimated time:
Difficulty:
hard

Description

Observation

Needles should be updated to make sure test runs.

openQA test in scenario sle-15-Leanos-DVD-x86_64-iscsi_ibft@64bit fails in
iscsi_configuration

and now the tab 'iBFT' does not show up anymore

Reproducible

Fails since (at least) Build 93.17

Expected result

Last good: SLE 12 SP3 GM

Suggestions

  • Read bsc#1077629
  • Ensure the iSCSI testing infrastructure (server service on osd?) is working properly. As we had a problem recently that the samba server "vanished" some time between 2017-07 and 2017-12 and I had to reinstall it I assume the same could have happened for other ressources. At least we can see it works sometimes or at least 17 days ago: #23554#note-2
  • Confirm it manually/locally
  • Create updated needles, adapt to recent SLE15 behaviour if any
  • Crosscheck SLE12SP4 and SLE15 test

Further details

Always latest result in this scenario: latest, formerly latest Leanos-DVD


Related issues 6 (0 open6 closed)

Related to openQA Tests (public) - action #28276: [sle][functional][medium][yast][y] test fails in iscsi_server - needs needle update and further investigationResolveddheidler2017-11-232018-05-22

Actions
Related to openQA Tests (public) - action #32452: [sle][funtional][y][easy] test fails in iscsi_client need adaptionResolvedokurz2018-02-28

Actions
Related to openQA Tests (public) - action #36060: [functional][y][tools][fast] Investigate iscsi_ibft test scenario failureResolvedriafarov2018-05-102018-05-22

Actions
Has duplicate openQA Tests (public) - action #30823: [sle][functional][tools][iscsi][hard] test is stuck in linuxrc asking if dhcp should be used - since worker upgrade/qemu/os-autoinst?Resolvedokurz2018-01-252018-03-27

Actions
Blocks openQA Tests (public) - action #23558: [sle][functional][sle15][easy][y] test fails in module partitioning_iscsi - probably needs adaption to testcodeResolvedriafarov2017-08-232018-05-22

Actions
Blocks openQA Tests (public) - action #30685: [sle][functional][sle12 sp4][u] test fails in firefox - firefox_readerview_window doesn't matchResolved2018-01-232018-06-19

Actions
Actions #1

Updated by okurz almost 7 years ago

  • Subject changed from test fails in iscsi_configuration because of missing updated needle iscsi-ibft to [sle][functional][medium][tools]test fails in iscsi_configuration because of missing updated needle iscsi-ibft / missing iscsi server on osd?
  • Description updated (diff)
  • Due date set to 2018-03-13
  • Category changed from Enhancement to existing tests to Bugs in existing tests
  • Status changed from New to Workable
  • Target version set to Milestone 15

The latest job in the same scenario for "Leanos" still showed the tab https://openqa.suse.de/tests/1192623#step/iscsi_configuration/5 with missing needle but now we have a different problem also described in bsc#1077629 which seems to be related to our infrastructure.

Link to latest updated to "Installer-DVD"

Actions #2

Updated by okurz almost 7 years ago

  • Blocks action #23558: [sle][functional][sle15][easy][y] test fails in module partitioning_iscsi - probably needs adaption to testcode added
Actions #3

Updated by okurz almost 7 years ago

  • Related to action #28276: [sle][functional][medium][yast][y] test fails in iscsi_server - needs needle update and further investigation added
Actions #4

Updated by okurz almost 7 years ago

  • Related to action #30823: [sle][functional][tools][iscsi][hard] test is stuck in linuxrc asking if dhcp should be used - since worker upgrade/qemu/os-autoinst? added
Actions #5

Updated by okurz almost 7 years ago

  • Description updated (diff)

… But actually it works sometimes so it's not that the server is really missing or if at all, it's only missing since 17 days, see https://openqa.suse.de/tests/1404134 progressing further to partitioning_iscsi

Actions #6

Updated by okurz almost 7 years ago

  • Blocks action #30685: [sle][functional][sle12 sp4][u] test fails in firefox - firefox_readerview_window doesn't match added
Actions #7

Updated by okurz almost 7 years ago

  • Subject changed from [sle][functional][medium][tools]test fails in iscsi_configuration because of missing updated needle iscsi-ibft / missing iscsi server on osd? to [sle][functional][medium][tools][yast]test fails in iscsi_configuration because of missing updated needle iscsi-ibft / missing iscsi server on osd?
  • Target version changed from Milestone 15 to Milestone 14
Actions #8

Updated by okurz almost 7 years ago

  • Related to action #32452: [sle][funtional][y][easy] test fails in iscsi_client need adaption added
Actions #9

Updated by riafarov almost 7 years ago

  • Assignee set to riafarov
Actions #10

Updated by riafarov almost 7 years ago

  • Status changed from Workable to Blocked

So, as result of investigation I've assigned original bug to qemu dev to take a look as it's our best guess. I was able to reproduce the issue. We have workers in place with iscsi configured and that's not the problem. I left my orthos instance running, so iscsi can be accessed there (iscsi:10.161.8.141::3260:0:iqn.2016-02.openqa.de:for.openqa). See no way to proceed here, but I was able to make this functionality work with tap setup, which we may apply on OSD, but that's not cheap solution. Let's track updates on https://bugzilla.suse.com/show_bug.cgi?id=1077629

Actions #11

Updated by riafarov almost 7 years ago

  • Related to deleted (action #30823: [sle][functional][tools][iscsi][hard] test is stuck in linuxrc asking if dhcp should be used - since worker upgrade/qemu/os-autoinst?)
Actions #12

Updated by riafarov almost 7 years ago

  • Has duplicate action #30823: [sle][functional][tools][iscsi][hard] test is stuck in linuxrc asking if dhcp should be used - since worker upgrade/qemu/os-autoinst? added
Actions #13

Updated by riafarov almost 7 years ago

  • Status changed from Blocked to In Progress
Actions #14

Updated by riafarov almost 7 years ago

ibft setup is correct, I also was able to make it work in my local setup with iscsi: http://gershwin.arch.suse.de/tests/133#step/iscsi_configuration/3
Seems, something wrong in production, maybe FW rules, which are different. Same ipxe kernel file works fine in local setup.

Actions #15

Updated by riafarov almost 7 years ago

  • Subject changed from [sle][functional][medium][tools][yast]test fails in iscsi_configuration because of missing updated needle iscsi-ibft / missing iscsi server on osd? to [sle][functional][hard][tools][yast] test fails in iscsi_configuration because of missing updated needle iscsi-ibft / missing iscsi server on osd?
Actions #16

Updated by riafarov almost 7 years ago

  • Due date changed from 2018-03-13 to 2018-03-27
Actions #17

Updated by riafarov almost 7 years ago

Reproduced manually onopenqaworker6, firewall is not affecting results, works fine if using 10.0.2.2 as iscsi ip address. We will need to change openQA to use that value, as cannot overwrite WORKER_HOSTNAME in the test suite.

Actions #18

Updated by okurz almost 7 years ago

  • Target version changed from Milestone 14 to Milestone 15
Actions #19

Updated by riafarov almost 7 years ago

Investigated further with szarate. I've spawn the VM and it worked when using 10.0.2.2 ip address for iscsi as well as the one set in WORKER_HOSTNAME. With local hot patch we've performed test run https://openqa.suse.de/tests/1538954 but it failed even with 10.0.2.2. Seems to be related to GRE tunnels.

Actions #20

Updated by riafarov almost 7 years ago

  • Status changed from In Progress to Workable

Will continue on this one on Monday with @szarate.

Actions #21

Updated by riafarov almost 7 years ago

  • Status changed from Workable to Blocked

Santiago is on sick leave, so not possible to progress on this ticket as of now.

Actions #22

Updated by riafarov over 6 years ago

  • Due date changed from 2018-03-27 to 2018-04-10
  • Status changed from Blocked to Workable
  • Priority changed from Normal to High
Actions #23

Updated by okurz over 6 years ago

  • Due date changed from 2018-04-10 to 2018-04-24
  • Assignee deleted (riafarov)

We will not be able to do this one efficiently, szarate will not be available. It's ok to do it later which will of course delay our chain so I will update the blocked ticket accordingly as well.

Next step: Wait for @szarate and fix server on osd infrastructure with him.

Actions #24

Updated by okurz over 6 years ago

  • Subject changed from [sle][functional][hard][tools][yast] test fails in iscsi_configuration because of missing updated needle iscsi-ibft / missing iscsi server on osd? to [sle][functional][hard][tools][yast][y] test fails in iscsi_configuration because of missing updated needle iscsi-ibft / missing iscsi server on osd?
Actions #25

Updated by okurz over 6 years ago

  • Subject changed from [sle][functional][hard][tools][yast][y] test fails in iscsi_configuration because of missing updated needle iscsi-ibft / missing iscsi server on osd? to [sle][functional][tools][yast][y][hard] test fails in iscsi_configuration because of missing updated needle iscsi-ibft / missing iscsi server on osd?
  • Difficulty set to hard
Actions #26

Updated by riafarov over 6 years ago

  • Assignee set to riafarov

We've tried test on openqaworker2, where we don't have GRE tunnels, still didn't work. Also launched on arm: https://openqa.suse.de/tests/1631090#settings
As a next step we'll try to replicate the setup and check closely, as we don't want to break production env.

Actions #27

Updated by riafarov over 6 years ago

So, what we have, is that when we try to configure iBFT we have eth0 not configured:
09:08:24 : exec: wicked ifup all = 0
09:08:24 : stdout + stderr:
lo up
09:08:25 : net_update_state:
09:08:25 : lo: up,
09:08:25 : eth0: device-unconfigured
09:08:25 :
Which is not the case in successful runs. Meaning that iPXE cannot successfully configure network. Looking for the root cause of this behavior.

Actions #28

Updated by riafarov over 6 years ago

  • Status changed from Workable to In Progress
Actions #29

Updated by riafarov over 6 years ago

  • Status changed from In Progress to Feedback
Actions #30

Updated by mgriessmeier over 6 years ago

  • Due date changed from 2018-04-24 to 2018-05-08
  • Target version changed from Milestone 15 to Milestone 16

as agreed with riafarov, moving to Sprint 16

Actions #31

Updated by riafarov over 6 years ago

Changes will be deployed tomorrow.

Actions #32

Updated by okurz over 6 years ago

needles merged

Actions #33

Updated by okurz over 6 years ago

  • Status changed from Feedback to Workable
  • Assignee deleted (riafarov)

someone else please pick up, optionally discuss with @szarate what's the current state and next step, and continue

Actions #34

Updated by riafarov over 6 years ago

  • Assignee set to riafarov

Changes not yet deployed to osd.

Actions #35

Updated by riafarov over 6 years ago

  • Due date changed from 2018-05-08 to 2018-05-22

Deployment is planned for Wednesday (09/05/2018). After that can be resolved (of course if works).

Actions #36

Updated by riafarov over 6 years ago

  • Status changed from Workable to In Progress
Actions #37

Updated by riafarov over 6 years ago

  • Status changed from In Progress to Feedback

Test fails sporadically, as can have existing installations on iscsi disk.
https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/4999

Actions #38

Updated by riafarov over 6 years ago

  • Status changed from Feedback to Resolved
Actions #39

Updated by okurz over 6 years ago

  • Related to action #36060: [functional][y][tools][fast] Investigate iscsi_ibft test scenario failure added
Actions

Also available in: Atom PDF