tickets #56690

tumbleweed-snapshot package nuked from download.o.o machine

Added by jberry 6 months ago. Updated 6 months ago.

Status:ResolvedStart date:
Priority:NormalDue date:
Assignee:-% Done:

30%

Category:-
Target version:-
Duration:

Description

As noted here https://lists.opensuse.org/opensuse-factory/2019-09/
msg00105.html and my subsequent response it appears someone nuked the package
and config.

Jimmy

History

#1 Updated by jberry 6 months ago

  • Private changed from Yes to No

#2 Updated by jberry 6 months ago

73509:2019-09-05 07:52:18 <1> pontifex2(20265) [zypper:req++] SolverRequester.cc(remove):358 Marking for deletion: I__s_o(832)tumbleweed-snapshot-0.4.3-3.1.noarch(@System)
73521:2019-09-05 07:52:18 <1> pontifex2(20265) [zypp] SATResolver.cc(resolvePool):719 Delete I_Tu_o(832)tumbleweed-snapshot-0.4.3-3.1.noarch(@System)
74178:2019-09-05 07:52:18 <1> pontifex2(20265) [libsolv] PoolImpl.cc(logSat):121 job: user installed tumbleweed-snapshot
74243:2019-09-05 07:52:18 <1> pontifex2(20265) [libsolv] PoolImpl.cc(logSat):121 job: erase tumbleweed-snapshot-0.4.3-3.1.noarch
74245:2019-09-05 07:52:18 <1> pontifex2(20265) [libsolv] PoolImpl.cc(logSat):121     !tumbleweed-snapshot-0.4.3-3.1.noarch [832]I (w1)
74263:2019-09-05 07:52:18 <1> pontifex2(20265) [zypper++] Summary.cc(readPool):127 <uninstall> I_Tu_ou(832)tumbleweed-snapshot-0.4.3-3.1.noarch(@System)
74303:2019-09-05 07:52:20 <1> pontifex2(20265) [zypp] RpmDb.cc(doRemovePackage):2139 RpmDb::doRemovePackage(tumbleweed-snapshot-0.4.3-3.1.noarch,0x00000008)
74304:2019-09-05 07:52:20 <1> pontifex2(20265) [zypp++] ExternalProgram.cc(start_program):249 Executing 'rpm' '--root' '/' '--dbpath' '/var/lib/rpm' '-e' '--allmatches' '--nodeps' '--' 'tumbleweed-snapshot-0.4.3-3.1.noarch'
74307:2019-09-05 07:52:20 <1> pontifex2(20265) [Progress++] ProgressData.cc(report):88 {#7|Removing tumbleweed-snapshot-0.4.3-3.1.noarch}END

The openSUSE:infrastructure no longer able to download metadata. Heading out so as far as I investigated.

#3 Updated by jberry 6 months ago

  • Subject changed from tumbleweed-snapshot package nukes from download.o.o machine to tumbleweed-snapshot package nuked from download.o.o machine

#4 Updated by dimstar 6 months ago

So, it seems it was 'explicitly' removed, not accidentally:

2019-09-05 07:52:20|command|root@pontifex2|'zypper' 'rm' 'tumbleweed-snapshot'|
# 2019-09-05 07:52:20 tumbleweed-snapshot-0.4.3-3.1.noarch removed ok
# Warning: Stopping tumbleweed-snapshot.service, but it can still be activated by:
#   tumbleweed-snapshot.timer
# warning: /etc/tumbleweed-snapshot.conf saved as /etc/tumbleweed-snapshot.conf.rpmsave
2019-09-05 07:52:20|remove |tumbleweed-snapshot|0.4.3-3.1|noarch|root@pontifex2|

and yeah - fun stuff: the journal starts pretty much right after that only:

# journalctl 
-- Logs begin at Thu 2019-09-05 08:19:26 UTC, end at Tue 2019-09-10 12:15:01 UTC. --

making it hard to see WHO performed the action

#5 Updated by dimstar 6 months ago

right after the rm, a zypper dup was called - and the first user to login after the system was back up (very likely post update verification!?!) came from

Sep 05 08:33:34 pontifex2 sshd[17781]: Accepted publickey for root from 192.168.253.203 port 48722 ssh2: RSA SHA256:nYkj0QLB0lNAkMEfYXFsDQRdB6P2HVitZef6OZRHyv4

#6 Updated by jberry 6 months ago

Reinstalled

zypper in tumbleweed-snapshot

and reinstated config

SOURCE_DIR="/srv/ftp/pub/opensuse"
SNAPSHOT_DIR="/srv/ftp/pub/opensuse/history"
SNAPSHOT_COUNT_MAX=20
RSYNC_INCLUDE="/usr/share/tumbleweed-snapshot/rsync/include-tumbleweed-only-repo.txt"
SOURCE_LOCK="/home/mirror/var/run/withlock/LOCK-publish"

enabled timer

#7 Updated by cboltz 6 months ago

and yeah - fun stuff: the journal starts pretty much right after that only: [...]

making it hard to see WHO performed the action

/var/log/messages-*.xz exists ;-)

Reinstalled [...]

Now that you have written down how to setup tumbleweed-snapshot, what about moving that note to salt? ;-) (should be easy, but if needed, I can help - just ping me on IRC)

#8 Updated by jberry 6 months ago

It was written in the original p.o.o issue as well. And short of custom config rather self documented by providing a proper package unlike 99% of the rest of the contents of the box.

I tracked down the SSH key used just before to mcaj@nibbler from authorized_keys file on root

#9 Updated by jberry 6 months ago

Who knows who that is and so we can have a discussion to understand what happened and avoid this from happening again?

#10 Updated by duncreg 6 months ago

jberry wrote:

Who knows who that is and so we can have a discussion to understand what happened and avoid this from happening again?

Martin Caj.

https://cz.linkedin.com/in/martin-caj-36a03399

#11 Updated by mcaj 6 months ago

  • Assignee set to mcaj

#12 Updated by mcaj 6 months ago

  • Status changed from New to In Progress
  • % Done changed from 0 to 30

Hi Jimmy,

Yes I worked on the machine recently. The machine has old
unsupported OS. I did the updat, but there ware some problematic dependencies..
During the work on it I probably removed the tumbleweed-snapshot package.
For That I'm sorry.
I checked the machine one more time today and it looks good.

Can you please check the server functionality and let me know if there is any problem with
it ?

Thank you.
Martin

#13 Updated by jberry 6 months ago

  • Status changed from In Progress to Resolved
  • Assignee deleted (mcaj)

Everything appears to be functioning at the moment. If there are dependency issues with the package file an issue and I'll take care of them, but tumbleweed-snapshot has very minimal dependencies. Removing packages, especially ones with the name of an openSUSE product in them on a openSUSE download server seems a bit haphazard to be done without investigating to see systemd services of the same name as well.

#14 Updated by jberry 6 months ago

To be clear the dependencies of tumbleweed-snapshot:

Requires: coreutils
Requires: findutils
Requires: grep
Requires: procmail
Requires: rsync

That's absolutely nothing. Given an explicit zypper rm tumbleweed-snapshot was run I find this whole thing fishy.

Also available in: Atom PDF