action #21038
open[qam] test fails in bootloader_s390 - missing install image on ftp server
Added by pcervinka almost 7 years ago. Updated about 2 months ago.
0%
Description
Observation¶
openQA test in scenario sle-12-Server-DVD-Incidents-Minimal-s390x-qam-minimal@s390x-zVM-vswitch-l2 fails in
bootloader_s390
Reproducible¶
Fails since (at least) Build :5299:kernel-source.1501485659 (current job)
Expected result¶
Last good: :5299:kernel-source.1501437271 (or more recent)
Further details¶
Always latest result in this scenario: latest
There is missing install media on ftp ftp 10.160.0.207 SLE-12-Server-DVD-s390x-GM-DVD1.
'227 Entering Passive Mode (10,160,0,207,117,87). ',
'>>>RETR SLE-12-Server-DVD-s390x-GM-DVD1/suse.ins ',
'550 Failed to open file.
Updated by okurz almost 7 years ago
- Related to action #20904: [qam] - test fails in bootloader_s390 - ftp is missing install medium added
Updated by okurz almost 7 years ago
- Status changed from Resolved to Feedback
- Assignee set to okurz
So it worked 4 days ago meaning the repo SLE-12-Server-DVD-s390x-GM-DVD1 was still there. I assume it was cleaned up by the gru cleanup task. Anyway for now I "fixed" that - probably only temporarily - by linking the repo from the fixed path with
/var/lib/openqa/share/factory/repo/
sudo -u geekotest ln -s fixed/SLE-12-Server-DVD-s390x-GM-DVD1/
but I want to monitor this and try to understand better what is happening here. I don't really see the issue as fixed for now.
Updated by pcervinka almost 7 years ago
There was missing SLE-12-SP3-Server-DVD-s390x-GM-DVD1 few days ago.
It was SLE-12-Server-DVD-s390x-GM-DVD1 this morning.
And now is missing SLE-12-SP1-Server-DVD-s390x-GM-DVD1, recent failure https://openqa.suse.de/tests/1090205#step/bootloader_s390/45 .
Updated by okurz almost 7 years ago
ok, I did now
ln -s fixed/SLE-12-SP2-Server-DVD-s390x-GM-DVD1/
ln -s fixed/SLE-12-SP2-Server-DVD-s390x-GM-DVD1/
ln -s fixed/SLE-12-SP3-Server-DVD-s390x-GM-DVD1/
Updated by mgriessmeier almost 7 years ago
PR provided as suggested fix to always take the boot-repository from fixed/ subfolder
https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/3382
Updated by okurz over 6 years ago
symlinks for SLE-12-SP2 and SLE-12-SP3 disappeared again. In the logs I could find
/var/log/openqa-20170808.xz:[Tue Aug 8 10:02:38 2017] [22817:info] GRU: removing /var/lib/openqa/share/factory/repo/SLE-12-SP2-Server-DVD-s390x-GM-DVD1
so I created the symlinks again with
geekotest@openqa:~/share/factory/repo> ln -s fixed/SLE-12-SP2-Server-DVD-s390x-GM-DVD1/
geekotest@openqa:~/share/factory/repo> ln -s fixed/SLE-12-SP3-Server-DVD-s390x-GM-DVD1/
so that is no long-term solution.
Updated by riafarov over 6 years ago
@okurz as I've did the same for SDK builds for sp2 and sp3, luckily there are still not removed yet. Should I change permissions to root, so gru cold not remove those?
Updated by okurz over 6 years ago
I did that right now for the Server symlinks (not for SDK, let's see). If this is really the way to go I would like to have it documented somewhere else, not only ticket comments.
Updated by okurz over 6 years ago
symlinks were pruned again, creating them as root won't help, still cleaned. Put in a workaround in
osd:/etc/cron.d/sles
# okurz: 2017-08-17: https://progress.opensuse.org/issues/21038 [qam] test fails in bootloader_s390 - missing install image on ftp server
*/5 * * * * geekotest (cd /var/lib/openqa/share/factory/repo/ && for i in fixed/SLE-12-{,SP1-,SP2-,SP3-}Server-DVD-s390x-GM-DVD1/ ; do ln -sf $i ; done)
Updated by okurz over 6 years ago
- Priority changed from Urgent to Normal
ok, with the workaround applied we can live with it for the time being but still not perfect. Let's give coolo some grace time to evaluate what he thinks about this ticket.
Updated by okurz over 6 years ago
- Related to action #25286: [sle][functional][u][qam][tools] Wrong URL for repositories -> repo symlinks deleted on osd prematurely? added
Updated by okurz over 6 years ago
- Related to deleted (action #25286: [sle][functional][u][qam][tools] Wrong URL for repositories -> repo symlinks deleted on osd prematurely?)
Updated by okurz over 6 years ago
- Blocks action #25286: [sle][functional][u][qam][tools] Wrong URL for repositories -> repo symlinks deleted on osd prematurely? added
Updated by okurz over 6 years ago
- Subject changed from [qam] test fails in bootloader_s390 - missing install image on ftp server to [qam][tools] test fails in bootloader_s390 - missing install image on ftp server
- Category changed from Bugs in existing tests to Infrastructure
- Assignee deleted (
okurz)
The workaround is still in place and ok but I won't find the time to look into how openQA behaves and why the links are pruned.
Updated by pcervinka over 5 years ago
- Status changed from Feedback to Resolved
Resolved for few months.
Updated by okurz about 3 years ago
- Subject changed from [qam][tools] test fails in bootloader_s390 - missing install image on ftp server to [qam] test fails in bootloader_s390 - missing install image on ftp server
- Status changed from Resolved to Workable
How do you think it was "Resolved"? According to my comments above this was never really solved but just a workaround put in place which seems to have failed again today, see https://chat.suse.de/direct/2FFxjQXvCCPbj5kbFyGwmADGBuMWnkHedE?msg=ZiasaYDwDpYr2g7LB
Updated by pcervinka about 3 years ago
I don't think anything, as it os more that two years ago. And proper way... if you think the issue still persist open new issue and relate this one.
Updated by okurz about 3 years ago
Sorry, I don't understand. What should the new issue be for? For me it's still the very same issue: s390x references a path that is not provided by assets synced from OBS and the symlinks need to be maintained by a hacky workaround of a regurlarly running cron job. Either the tests need to reference an asset that is provided by https://github.com/os-autoinst/openqa-trigger-from-obs or https://github.com/os-autoinst/openqa-trigger-from-obs needs to be adapted to provide what the s390x tests need. As has been shown the periodic symlink recreation is not stable and not reliable.
Updated by pcervinka about 3 years ago
Original problem was for specific QAM s390 failing job, which was solved by that workaround. If you think, that it needs better solution, it should have poo with better generic name and overall description.
Updated by okurz about 3 years ago
you are welcome to do so. I myself feel too lazy to extract the relevant information from this ticket into another one though. Sorry about that :)
Updated by pcervinka about 3 years ago
I see, that you are open about your work attitude, so I will be as well. My statement is that this issue is out of the scope of my work, so I don't care about it anymore.
Updated by slo-gin over 1 year ago
This ticket was set to Normal priority but was not updated within the SLO period. Please consider picking up this ticket or just set the ticket to the next lower priority.
Updated by slo-gin about 2 months ago
This ticket was set to Normal priority but was not updated within the SLO period. Please consider picking up this ticket or just set the ticket to the next lower priority.