action #44690
closed[tools] Repos in http://openqa.suse.de/assets/repo/fixed/ got deleted from time to time after new openqa deployment
Added by xlai over 5 years ago. Updated over 5 years ago.
0%
Description
Hi folks,
There were repos for old products sle11sp4/sle12sp3/sle15 GM image under http://openqa.suse.de/assets/repo/fixed/ last week. They are still needed for virtualization sle15sp1 test.
Anyone know why are they cleaned up? Would you please help to recover? And make possible marks if any to avoid future such cleanup? Thank you!
Updated by xlai over 5 years ago
- Subject changed from [infra] Wrong repo cleanup in http://openqa.suse.de/assets/repo/fixed/. to [infra] Recover repos in http://openqa.suse.de/assets/repo/fixed/.
Updated by xlai over 5 years ago
Please let the repo's name follow up rules like :
SLE-12-SP3-Server-DVD-x86_64-GM-DVD1
SLE-11-SP4-Server-DVD-x86_64-GM-DVD1
SLE-15-Server-DVD-x86_64-GM-DVD1
Thank you!
Updated by mitiao over 5 years ago
- Status changed from New to Feedback
ftp://openqa.suse.de/fixed/SLE-12-SP3-Server-DVD-x86_64-GM-DVD1/
ftp://openqa.suse.de/fixed/SLE-15-Installer-DVD-x86_64-GM-DVD1/
and
ftp://openqa.suse.de/fixed/SLES-11-SP4-DVD-x86_64-GM-DVD1/
I guess sle11sp4 should follow REPO_0_TO_INSTALL in https://openqa.suse.de/tests/2296332#settings
Updated by okurz over 5 years ago
@mitiao did you find out who/what deleted the assets? I think the subject tag "[infra]" does not help as this repo is the "infrastructure" ticket tracker, maybe "[tools]"?
Updated by mitiao over 5 years ago
- Subject changed from [infra] Recover repos in http://openqa.suse.de/assets/repo/fixed/. to [tools] Recover repos in http://openqa.suse.de/assets/repo/fixed/
@okurz no idea who/what deleted them
Updated by mitiao over 5 years ago
Almost repos in ftp://openqa.suse.de/fixed/ were deleted again.
I was looking into /var/log/openqa to find out something interesting:
not registering asset fixed/SLE-12-SP3-Server-DVD-x86_64-GM-DVD1 containing /
not registering asset fixed/SLE-15-Installer-DVD-x86_64-GM-Media1 containing /
...
Seems openqa recognized the assets with none registration and deleted them, need some more investigation...
Updated by xlai over 5 years ago
- Subject changed from [tools] Recover repos in http://openqa.suse.de/assets/repo/fixed/ to [tools] Repos in http://openqa.suse.de/assets/repo/fixed/ got deleted from time to time after new openqa deployment
Change the title since it is not manual error, but tools issue.
Need to find root cause for it and fix.
Updated by xlai over 5 years ago
- Status changed from Feedback to Workable
@mitiao, This cleanup repo issue keeps happening which blocks openqa testing. SO I change the issue subject. If you think you are not the proper person to fix it , please help to reassign. Thank you.
Updated by riafarov over 5 years ago
For builds 107.5 108.1 we got repos removed, gru doesn't have it in the logs, so it was something else causing it. For fixed it's even worse, as gru doesn't touch those.
Updated by szarate over 5 years ago
- Assignee changed from mitiao to coolo
- Priority changed from High to Urgent
Coolo seems to be looking at this problem atm...
osukup | sh.. why is /var/lib/openqa/factory/repo/fixed on osd empty ?
coolo | something is fishy - as the free disk space didn't increase
Updated by szarate over 5 years ago
- Status changed from Workable to In Progress
Updated by coolo over 5 years ago
- Project changed from openQA Infrastructure to openQA Project
- Category set to Regressions/Crashes
- Assignee deleted (
coolo) - Priority changed from Urgent to Immediate
- Target version set to Current Sprint
[2018-12-06T07:11:24.0264 CET] [info] Removing asset repo/
Just as https://openqa.suse.de/tests/2307982#settings created an 'iso/' (which would have removed all isos tomorrow), https://openqa.suse.de/tests/2300713#settings created a 'repo/' asset, that removed all repos today.
Can we please ban all experiments from production environment?
Updated by coolo over 5 years ago
xlai, I guess leli is in your office? Could you please hit him hard? ;-)
Updated by xlai over 5 years ago
coolo wrote:
xlai, I guess leli is in your office? Could you please hit him hard? ;-)
coolo, I guess @mitiao is a more suitable person ;-)
Updated by xlai over 5 years ago
Why does the auto recovery only recovers sle11sp4 x86 repo, and others sle12sp3/sle12sp4/sle15 x86 not? If it is recorded somewhere, can anyone help to add these 3 to the list of fixed repos?
@mitiao, would you please help to recover them manually one more time as talked before?
Updated by coolo over 5 years ago
there is no auto-recovery. sle11 was left behind because it's owned by root - and the s390x repos were copied back by qam/ondrej.
Updated by mitiao over 5 years ago
xlai wrote:
Why does the auto recovery only recovers sle11sp4 x86 repo, and others sle12sp3/sle12sp4/sle15 x86 not? If it is recorded somewhere, can anyone help to add these 3 to the list of fixed repos?
@mitiao, would you please help to recover them manually one more time as talked before?
please check ftp://openqa.suse.de/fixed/
Updated by riafarov over 5 years ago
coolo wrote:
[2018-12-06T07:11:24.0264 CET] [info] Removing asset repo/
Just as https://openqa.suse.de/tests/2307982#settings created an 'iso/' (which would have removed all isos tomorrow), https://openqa.suse.de/tests/2300713#settings created a 'repo/' asset, that removed all repos today.
Can we please ban all experiments from production environment?
Even though I totally agree with the statement above, but I would not call setting ISO to empty value an experiment and I doubt that anyone would recognize how dangerous it is. So if already know about similar issues, we need to warn people. Would be interesting to try something like '../' =)
Updated by SLindoMansilla over 5 years ago
For test suite suse_patches-systemd_testsuite (https://openqa.suse.de/admin/test_suites), the settings ISO is set to empty since at least 24 days (seen in audit log https://openqa.suse.de/admin/auditlog?eventid=1703340, that is the first time I cloned the s390x job into OSD.
And, I would say that I set that 4 months ago because of: https://progress.opensuse.org/issues/38807#note-7
If it is true that the empty value deletes all ISOs, this is happening since long time ago.
Updated by SLindoMansilla over 5 years ago
Found, I set ISO to empty 8 months ago: https://openqa.suse.de/admin/auditlog?eventid=1110195
FYI: I removed the setting from the test suite, to avoid possible future problems.
Updated by mkittler over 5 years ago
Updated by SLindoMansilla over 5 years ago
Setting ISO= restored in test suite suse_patches-systemd_testsuite on https://openqa.suse.de/admin/test_suites