Project

General

Profile

action #21038

[qam] test fails in bootloader_s390 - missing install image on ftp server

Added by pcervinka about 4 years ago. Updated 7 months ago.

Status:
Workable
Priority:
Normal
Assignee:
-
Category:
Infrastructure
Target version:
-
Start date:
2017-08-03
Due date:
% Done:

0%

Estimated time:
Difficulty:

Description

Observation

openQA test in scenario sle-12-Server-DVD-Incidents-Minimal-s390x-qam-minimal@s390x-zVM-vswitch-l2 fails in
bootloader_s390

Reproducible

Fails since (at least) Build :5299:kernel-source.1501485659 (current job)

Expected result

Last good: :5299:kernel-source.1501437271 (or more recent)

Further details

Always latest result in this scenario: latest

There is missing install media on ftp ftp 10.160.0.207 SLE-12-Server-DVD-s390x-GM-DVD1.

'227 Entering Passive Mode (10,160,0,207,117,87).                                ',
'>>>RETR SLE-12-Server-DVD-s390x-GM-DVD1/suse.ins                                ',
'550 Failed to open file.   

Related issues

Related to openQA Tests - action #20904: [qam] - test fails in bootloader_s390 - ftp is missing install mediumResolved2017-07-28

Blocks openQA Tests - action #25286: [sle][functional][u][qam][tools] Wrong URL for repositories -> repo symlinks deleted on osd prematurely?Resolved2017-09-14

History

#1 Updated by okurz about 4 years ago

  • Related to action #20904: [qam] - test fails in bootloader_s390 - ftp is missing install medium added

#2 Updated by pcervinka about 4 years ago

  • Status changed from New to Resolved

Fixed by Oliver.

#3 Updated by okurz about 4 years ago

  • Status changed from Resolved to Feedback
  • Assignee set to okurz

So it worked 4 days ago meaning the repo SLE-12-Server-DVD-s390x-GM-DVD1 was still there. I assume it was cleaned up by the gru cleanup task. Anyway for now I "fixed" that - probably only temporarily - by linking the repo from the fixed path with

/var/lib/openqa/share/factory/repo/
sudo -u geekotest ln -s fixed/SLE-12-Server-DVD-s390x-GM-DVD1/

but I want to monitor this and try to understand better what is happening here. I don't really see the issue as fixed for now.

#4 Updated by pcervinka about 4 years ago

There was missing SLE-12-SP3-Server-DVD-s390x-GM-DVD1 few days ago.
It was SLE-12-Server-DVD-s390x-GM-DVD1 this morning.
And now is missing SLE-12-SP1-Server-DVD-s390x-GM-DVD1, recent failure https://openqa.suse.de/tests/1090205#step/bootloader_s390/45 .

#5 Updated by okurz about 4 years ago

ok, I did now

ln -s fixed/SLE-12-SP2-Server-DVD-s390x-GM-DVD1/
ln -s fixed/SLE-12-SP2-Server-DVD-s390x-GM-DVD1/
ln -s fixed/SLE-12-SP3-Server-DVD-s390x-GM-DVD1/

#6 Updated by mgriessmeier about 4 years ago

PR provided as suggested fix to always take the boot-repository from fixed/ subfolder
https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/3382

#7 Updated by okurz about 4 years ago

symlinks for SLE-12-SP2 and SLE-12-SP3 disappeared again. In the logs I could find

/var/log/openqa-20170808.xz:[Tue Aug  8 10:02:38 2017] [22817:info] GRU: removing /var/lib/openqa/share/factory/repo/SLE-12-SP2-Server-DVD-s390x-GM-DVD1

so I created the symlinks again with

geekotest@openqa:~/share/factory/repo> ln -s fixed/SLE-12-SP2-Server-DVD-s390x-GM-DVD1/
geekotest@openqa:~/share/factory/repo> ln -s fixed/SLE-12-SP3-Server-DVD-s390x-GM-DVD1/

so that is no long-term solution.

#8 Updated by riafarov about 4 years ago

okurz as I've did the same for SDK builds for sp2 and sp3, luckily there are still not removed yet. Should I change permissions to root, so gru cold not remove those?

#9 Updated by okurz about 4 years ago

I did that right now for the Server symlinks (not for SDK, let's see). If this is really the way to go I would like to have it documented somewhere else, not only ticket comments.

#10 Updated by okurz about 4 years ago

symlinks were pruned again, creating them as root won't help, still cleaned. Put in a workaround in
osd:/etc/cron.d/sles

# okurz: 2017-08-17: https://progress.opensuse.org/issues/21038 [qam] test fails in bootloader_s390 - missing install image on ftp server
*/5 * * * * geekotest (cd /var/lib/openqa/share/factory/repo/ && for i in fixed/SLE-12-{,SP1-,SP2-,SP3-}Server-DVD-s390x-GM-DVD1/ ; do ln -sf $i ; done)

#11 Updated by okurz about 4 years ago

  • Priority changed from Urgent to Normal

ok, with the workaround applied we can live with it for the time being but still not perfect. Let's give coolo some grace time to evaluate what he thinks about this ticket.

#12 Updated by okurz about 4 years ago

  • Related to action #25286: [sle][functional][u][qam][tools] Wrong URL for repositories -> repo symlinks deleted on osd prematurely? added

#13 Updated by okurz almost 4 years ago

  • Related to deleted (action #25286: [sle][functional][u][qam][tools] Wrong URL for repositories -> repo symlinks deleted on osd prematurely?)

#14 Updated by okurz almost 4 years ago

  • Blocks action #25286: [sle][functional][u][qam][tools] Wrong URL for repositories -> repo symlinks deleted on osd prematurely? added

#15 Updated by okurz over 3 years ago

  • Subject changed from [qam] test fails in bootloader_s390 - missing install image on ftp server to [qam][tools] test fails in bootloader_s390 - missing install image on ftp server
  • Category changed from Bugs in existing tests to Infrastructure
  • Assignee deleted (okurz)

The workaround is still in place and ok but I won't find the time to look into how openQA behaves and why the links are pruned.

#16 Updated by pcervinka almost 3 years ago

  • Status changed from Feedback to Resolved

Resolved for few months.

#17 Updated by okurz 7 months ago

  • Subject changed from [qam][tools] test fails in bootloader_s390 - missing install image on ftp server to [qam] test fails in bootloader_s390 - missing install image on ftp server
  • Status changed from Resolved to Workable

How do you think it was "Resolved"? According to my comments above this was never really solved but just a workaround put in place which seems to have failed again today, see https://chat.suse.de/direct/2FFxjQXvCCPbj5kbFyGwmADGBuMWnkHedE?msg=ZiasaYDwDpYr2g7LB

#18 Updated by pcervinka 7 months ago

I don't think anything, as it os more that two years ago. And proper way... if you think the issue still persist open new issue and relate this one.

#19 Updated by okurz 7 months ago

Sorry, I don't understand. What should the new issue be for? For me it's still the very same issue: s390x references a path that is not provided by assets synced from OBS and the symlinks need to be maintained by a hacky workaround of a regurlarly running cron job. Either the tests need to reference an asset that is provided by https://github.com/os-autoinst/openqa-trigger-from-obs or https://github.com/os-autoinst/openqa-trigger-from-obs needs to be adapted to provide what the s390x tests need. As has been shown the periodic symlink recreation is not stable and not reliable.

#20 Updated by pcervinka 7 months ago

Original problem was for specific QAM s390 failing job, which was solved by that workaround. If you think, that it needs better solution, it should have poo with better generic name and overall description.

#21 Updated by okurz 7 months ago

you are welcome to do so. I myself feel too lazy to extract the relevant information from this ticket into another one though. Sorry about that :)

#22 Updated by pcervinka 7 months ago

I see, that you are open about your work attitude, so I will be as well. My statement is that this issue is out of the scope of my work, so I don't care about it anymore.

Also available in: Atom PDF