Project

General

Profile

Actions

action #44690

closed

[tools] Repos in http://openqa.suse.de/assets/repo/fixed/ got deleted from time to time after new openqa deployment

Added by xlai over 5 years ago. Updated over 5 years ago.

Status:
Resolved
Priority:
Immediate
Assignee:
Category:
Regressions/Crashes
Target version:
Start date:
2018-12-04
Due date:
% Done:

0%

Estimated time:

Description

Hi folks,
There were repos for old products sle11sp4/sle12sp3/sle15 GM image under http://openqa.suse.de/assets/repo/fixed/ last week. They are still needed for virtualization sle15sp1 test.

Anyone know why are they cleaned up? Would you please help to recover? And make possible marks if any to avoid future such cleanup? Thank you!

Actions #1

Updated by xlai over 5 years ago

  • Subject changed from [infra] Wrong repo cleanup in http://openqa.suse.de/assets/repo/fixed/. to [infra] Recover repos in http://openqa.suse.de/assets/repo/fixed/.
Actions #2

Updated by xlai over 5 years ago

Please let the repo's name follow up rules like :
SLE-12-SP3-Server-DVD-x86_64-GM-DVD1
SLE-11-SP4-Server-DVD-x86_64-GM-DVD1
SLE-15-Server-DVD-x86_64-GM-DVD1

Thank you!

Actions #3

Updated by xlai over 5 years ago

  • Assignee set to mitiao
Actions #5

Updated by okurz over 5 years ago

@mitiao did you find out who/what deleted the assets? I think the subject tag "[infra]" does not help as this repo is the "infrastructure" ticket tracker, maybe "[tools]"?

Actions #6

Updated by mitiao over 5 years ago

  • Subject changed from [infra] Recover repos in http://openqa.suse.de/assets/repo/fixed/. to [tools] Recover repos in http://openqa.suse.de/assets/repo/fixed/

@okurz no idea who/what deleted them

Actions #7

Updated by mitiao over 5 years ago

Almost repos in ftp://openqa.suse.de/fixed/ were deleted again.
I was looking into /var/log/openqa to find out something interesting:

not registering asset fixed/SLE-12-SP3-Server-DVD-x86_64-GM-DVD1 containing /
not registering asset fixed/SLE-15-Installer-DVD-x86_64-GM-Media1 containing /
...

Seems openqa recognized the assets with none registration and deleted them, need some more investigation...

Actions #8

Updated by xlai over 5 years ago

  • Subject changed from [tools] Recover repos in http://openqa.suse.de/assets/repo/fixed/ to [tools] Repos in http://openqa.suse.de/assets/repo/fixed/ got deleted from time to time after new openqa deployment

Change the title since it is not manual error, but tools issue.

Need to find root cause for it and fix.

Actions #9

Updated by xlai over 5 years ago

  • Status changed from Feedback to Workable

@mitiao, This cleanup repo issue keeps happening which blocks openqa testing. SO I change the issue subject. If you think you are not the proper person to fix it , please help to reassign. Thank you.

Actions #10

Updated by riafarov over 5 years ago

For builds 107.5 108.1 we got repos removed, gru doesn't have it in the logs, so it was something else causing it. For fixed it's even worse, as gru doesn't touch those.

Actions #11

Updated by szarate over 5 years ago

  • Assignee changed from mitiao to coolo
  • Priority changed from High to Urgent

Coolo seems to be looking at this problem atm...

 osukup | sh.. why is /var/lib/openqa/factory/repo/fixed on osd empty ?
 coolo | something is fishy - as the free disk space didn't increase
Actions #12

Updated by szarate over 5 years ago

  • Status changed from Workable to In Progress
Actions #13

Updated by coolo over 5 years ago

  • Project changed from openQA Infrastructure to openQA Project
  • Category set to Regressions/Crashes
  • Assignee deleted (coolo)
  • Priority changed from Urgent to Immediate
  • Target version set to Current Sprint

[2018-12-06T07:11:24.0264 CET] [info] Removing asset repo/

Just as https://openqa.suse.de/tests/2307982#settings created an 'iso/' (which would have removed all isos tomorrow), https://openqa.suse.de/tests/2300713#settings created a 'repo/' asset, that removed all repos today.

Can we please ban all experiments from production environment?

Actions #14

Updated by coolo over 5 years ago

xlai, I guess leli is in your office? Could you please hit him hard? ;-)

Actions #15

Updated by xlai over 5 years ago

coolo wrote:

xlai, I guess leli is in your office? Could you please hit him hard? ;-)

coolo, I guess @mitiao is a more suitable person ;-)

Actions #16

Updated by xlai over 5 years ago

Why does the auto recovery only recovers sle11sp4 x86 repo, and others sle12sp3/sle12sp4/sle15 x86 not? If it is recorded somewhere, can anyone help to add these 3 to the list of fixed repos?

@mitiao, would you please help to recover them manually one more time as talked before?

Actions #17

Updated by coolo over 5 years ago

there is no auto-recovery. sle11 was left behind because it's owned by root - and the s390x repos were copied back by qam/ondrej.

Actions #18

Updated by mitiao over 5 years ago

xlai wrote:

Why does the auto recovery only recovers sle11sp4 x86 repo, and others sle12sp3/sle12sp4/sle15 x86 not? If it is recorded somewhere, can anyone help to add these 3 to the list of fixed repos?

@mitiao, would you please help to recover them manually one more time as talked before?

please check ftp://openqa.suse.de/fixed/

Actions #19

Updated by riafarov over 5 years ago

coolo wrote:

[2018-12-06T07:11:24.0264 CET] [info] Removing asset repo/

Just as https://openqa.suse.de/tests/2307982#settings created an 'iso/' (which would have removed all isos tomorrow), https://openqa.suse.de/tests/2300713#settings created a 'repo/' asset, that removed all repos today.

Can we please ban all experiments from production environment?

Even though I totally agree with the statement above, but I would not call setting ISO to empty value an experiment and I doubt that anyone would recognize how dangerous it is. So if already know about similar issues, we need to warn people. Would be interesting to try something like '../' =)

Actions #20

Updated by SLindoMansilla over 5 years ago

For test suite suse_patches-systemd_testsuite (https://openqa.suse.de/admin/test_suites), the settings ISO is set to empty since at least 24 days (seen in audit log https://openqa.suse.de/admin/auditlog?eventid=1703340, that is the first time I cloned the s390x job into OSD.

And, I would say that I set that 4 months ago because of: https://progress.opensuse.org/issues/38807#note-7

If it is true that the empty value deletes all ISOs, this is happening since long time ago.

Actions #21

Updated by SLindoMansilla over 5 years ago

Found, I set ISO to empty 8 months ago: https://openqa.suse.de/admin/auditlog?eventid=1110195

FYI: I removed the setting from the test suite, to avoid possible future problems.

Actions #22

Updated by coolo over 5 years ago

possibly we're smarter with ISO type.

Actions #23

Updated by mkittler over 5 years ago

  • Assignee set to mkittler
Actions #25

Updated by mkittler over 5 years ago

  • Status changed from In Progress to Resolved

PR has been merged

Actions #26

Updated by SLindoMansilla over 5 years ago

Setting ISO= restored in test suite suse_patches-systemd_testsuite on https://openqa.suse.de/admin/test_suites

Actions

Also available in: Atom PDF