action #44690

[tools] Repos in http://openqa.suse.de/assets/repo/fixed/ got deleted from time to time after new openqa deployment

Added by xlai over 1 year ago. Updated over 1 year ago.

Status:ResolvedStart date:04/12/2018
Priority:ImmediateDue date:
Assignee:mkittler% Done:

0%

Category:Concrete Bugs
Target version:Current Sprint
Difficulty:
Duration:

Description

Hi folks,
There were repos for old products sle11sp4/sle12sp3/sle15 GM image under http://openqa.suse.de/assets/repo/fixed/ last week. They are still needed for virtualization sle15sp1 test.

Anyone know why are they cleaned up? Would you please help to recover? And make possible marks if any to avoid future such cleanup? Thank you!

History

#1 Updated by xlai over 1 year ago

  • Subject changed from [infra] Wrong repo cleanup in http://openqa.suse.de/assets/repo/fixed/. to [infra] Recover repos in http://openqa.suse.de/assets/repo/fixed/.

#2 Updated by xlai over 1 year ago

Please let the repo's name follow up rules like :
SLE-12-SP3-Server-DVD-x86_64-GM-DVD1
SLE-11-SP4-Server-DVD-x86_64-GM-DVD1
SLE-15-Server-DVD-x86_64-GM-DVD1

Thank you!

#3 Updated by xlai over 1 year ago

  • Assignee set to mitiao

#5 Updated by okurz over 1 year ago

@mitiao did you find out who/what deleted the assets? I think the subject tag "[infra]" does not help as this repo is the "infrastructure" ticket tracker, maybe "[tools]"?

#6 Updated by mitiao over 1 year ago

  • Subject changed from [infra] Recover repos in http://openqa.suse.de/assets/repo/fixed/. to [tools] Recover repos in http://openqa.suse.de/assets/repo/fixed/

@okurz no idea who/what deleted them

#7 Updated by mitiao over 1 year ago

Almost repos in ftp://openqa.suse.de/fixed/ were deleted again.
I was looking into /var/log/openqa to find out something interesting:

not registering asset fixed/SLE-12-SP3-Server-DVD-x86_64-GM-DVD1 containing /
not registering asset fixed/SLE-15-Installer-DVD-x86_64-GM-Media1 containing /
...

Seems openqa recognized the assets with none registration and deleted them, need some more investigation...

#8 Updated by xlai over 1 year ago

  • Subject changed from [tools] Recover repos in http://openqa.suse.de/assets/repo/fixed/ to [tools] Repos in http://openqa.suse.de/assets/repo/fixed/ got deleted from time to time after new openqa deployment

Change the title since it is not manual error, but tools issue.

Need to find root cause for it and fix.

#9 Updated by xlai over 1 year ago

  • Status changed from Feedback to Workable

@mitiao, This cleanup repo issue keeps happening which blocks openqa testing. SO I change the issue subject. If you think you are not the proper person to fix it , please help to reassign. Thank you.

#10 Updated by riafarov over 1 year ago

For builds 107.5 108.1 we got repos removed, gru doesn't have it in the logs, so it was something else causing it. For fixed it's even worse, as gru doesn't touch those.

#11 Updated by szarate over 1 year ago

  • Assignee changed from mitiao to coolo
  • Priority changed from High to Urgent

Coolo seems to be looking at this problem atm...

 osukup | sh.. why is /var/lib/openqa/factory/repo/fixed on osd empty ?
 coolo | something is fishy - as the free disk space didn't increase

#12 Updated by szarate over 1 year ago

  • Status changed from Workable to In Progress

#13 Updated by coolo over 1 year ago

  • Project changed from openQA Infrastructure to openQA Project
  • Category set to Concrete Bugs
  • Assignee deleted (coolo)
  • Priority changed from Urgent to Immediate
  • Target version set to Current Sprint

[2018-12-06T07:11:24.0264 CET] [info] Removing asset repo/

Just as https://openqa.suse.de/tests/2307982#settings created an 'iso/' (which would have removed all isos tomorrow), https://openqa.suse.de/tests/2300713#settings created a 'repo/' asset, that removed all repos today.

Can we please ban all experiments from production environment?

#14 Updated by coolo over 1 year ago

xlai, I guess leli is in your office? Could you please hit him hard? ;-)

#15 Updated by xlai over 1 year ago

coolo wrote:

xlai, I guess leli is in your office? Could you please hit him hard? ;-)

coolo, I guess @mitiao is a more suitable person ;-)

#16 Updated by xlai over 1 year ago

Why does the auto recovery only recovers sle11sp4 x86 repo, and others sle12sp3/sle12sp4/sle15 x86 not? If it is recorded somewhere, can anyone help to add these 3 to the list of fixed repos?

@mitiao, would you please help to recover them manually one more time as talked before?

#17 Updated by coolo over 1 year ago

there is no auto-recovery. sle11 was left behind because it's owned by root - and the s390x repos were copied back by qam/ondrej.

#18 Updated by mitiao over 1 year ago

xlai wrote:

Why does the auto recovery only recovers sle11sp4 x86 repo, and others sle12sp3/sle12sp4/sle15 x86 not? If it is recorded somewhere, can anyone help to add these 3 to the list of fixed repos?


@mitiao, would you please help to recover them manually one more time as talked before?

please check ftp://openqa.suse.de/fixed/

#19 Updated by riafarov over 1 year ago

coolo wrote:

[2018-12-06T07:11:24.0264 CET] [info] Removing asset repo/


Just as https://openqa.suse.de/tests/2307982#settings created an 'iso/' (which would have removed all isos tomorrow), https://openqa.suse.de/tests/2300713#settings created a 'repo/' asset, that removed all repos today.


Can we please ban all experiments from production environment?

Even though I totally agree with the statement above, but I would not call setting ISO to empty value an experiment and I doubt that anyone would recognize how dangerous it is. So if already know about similar issues, we need to warn people. Would be interesting to try something like '../' =)

#20 Updated by SLindoMansilla over 1 year ago

For test suite suse_patches-systemd_testsuite (https://openqa.suse.de/admin/test_suites), the settings ISO is set to empty since at least 24 days (seen in audit log https://openqa.suse.de/admin/auditlog?eventid=1703340, that is the first time I cloned the s390x job into OSD.

And, I would say that I set that 4 months ago because of: https://progress.opensuse.org/issues/38807#note-7

If it is true that the empty value deletes all ISOs, this is happening since long time ago.

#21 Updated by SLindoMansilla over 1 year ago

Found, I set ISO to empty 8 months ago: https://openqa.suse.de/admin/auditlog?eventid=1110195

FYI: I removed the setting from the test suite, to avoid possible future problems.

#22 Updated by coolo over 1 year ago

possibly we're smarter with ISO type.

#23 Updated by mkittler over 1 year ago

  • Assignee set to mkittler

#25 Updated by mkittler over 1 year ago

  • Status changed from In Progress to Resolved

PR has been merged

#26 Updated by SLindoMansilla over 1 year ago

Setting ISO= restored in test suite suse_patches-systemd_testsuite on https://openqa.suse.de/admin/test_suites

Also available in: Atom PDF