openSUSE admin

73509:2019-09-05 07:52:18 <1> pontifex2(20265) [zypper:req++] SolverRequester.cc(remove):358 Marking for deletion: I__s_o(832)tumbleweed-snapshot-0.4.3-3.1.noarch(@System)
73521:2019-09-05 07:52:18 <1> pontifex2(20265) [zypp] SATResolver.cc(resolvePool):719 Delete I_Tu_o(832)tumbleweed-snapshot-0.4.3-3.1.noarch(@System)
74178:2019-09-05 07:52:18 <1> pontifex2(20265) [libsolv] PoolImpl.cc(logSat):121 job: user installed tumbleweed-snapshot
74243:2019-09-05 07:52:18 <1> pontifex2(20265) [libsolv] PoolImpl.cc(logSat):121 job: erase tumbleweed-snapshot-0.4.3-3.1.noarch
74245:2019-09-05 07:52:18 <1> pontifex2(20265) [libsolv] PoolImpl.cc(logSat):121     !tumbleweed-snapshot-0.4.3-3.1.noarch [832]I (w1)
74263:2019-09-05 07:52:18 <1> pontifex2(20265) [zypper++] Summary.cc(readPool):127 <uninstall> I_Tu_ou(832)tumbleweed-snapshot-0.4.3-3.1.noarch(@System)
74303:2019-09-05 07:52:20 <1> pontifex2(20265) [zypp] RpmDb.cc(doRemovePackage):2139 RpmDb::doRemovePackage(tumbleweed-snapshot-0.4.3-3.1.noarch,0x00000008)
74304:2019-09-05 07:52:20 <1> pontifex2(20265) [zypp++] ExternalProgram.cc(start_program):249 Executing 'rpm' '--root' '/' '--dbpath' '/var/lib/rpm' '-e' '--allmatches' '--nodeps' '--' 'tumbleweed-snapshot-0.4.3-3.1.noarch'
74307:2019-09-05 07:52:20 <1> pontifex2(20265) [Progress++] ProgressData.cc(report):88 {#7|Removing tumbleweed-snapshot-0.4.3-3.1.noarch}END

The openSUSE:infrastructure no longer able to download metadata. Heading out so as far as I investigated.

So, it seems it was 'explicitly' removed, not accidentally:

2019-09-05 07:52:20|command|root@pontifex2|'zypper' 'rm' 'tumbleweed-snapshot'|
# 2019-09-05 07:52:20 tumbleweed-snapshot-0.4.3-3.1.noarch removed ok
# Warning: Stopping tumbleweed-snapshot.service, but it can still be activated by:
#   tumbleweed-snapshot.timer
# warning: /etc/tumbleweed-snapshot.conf saved as /etc/tumbleweed-snapshot.conf.rpmsave
2019-09-05 07:52:20|remove |tumbleweed-snapshot|0.4.3-3.1|noarch|root@pontifex2|

and yeah - fun stuff: the journal starts pretty much right after that only:

# journalctl 
-- Logs begin at Thu 2019-09-05 08:19:26 UTC, end at Tue 2019-09-10 12:15:01 UTC. --

making it hard to see WHO performed the action

right after the rm, a zypper dup was called - and the first user to login after the system was back up (very likely post update verification!?!) came from

Sep 05 08:33:34 pontifex2 sshd[17781]: Accepted publickey for root from 192.168.253.203 port 48722 ssh2: RSA SHA256:nYkj0QLB0lNAkMEfYXFsDQRdB6P2HVitZef6OZRHyv4

Reinstalled

zypper in tumbleweed-snapshot

and reinstated config

SOURCE_DIR="/srv/ftp/pub/opensuse"
SNAPSHOT_DIR="/srv/ftp/pub/opensuse/history"
SNAPSHOT_COUNT_MAX=20
RSYNC_INCLUDE="/usr/share/tumbleweed-snapshot/rsync/include-tumbleweed-only-repo.txt"
SOURCE_LOCK="/home/mirror/var/run/withlock/LOCK-publish"

enabled timer

and yeah - fun stuff: the journal starts pretty much right after that only: [...]
making it hard to see WHO performed the action

/var/log/messages-*.xz exists ;-)

Reinstalled [...]

Now that you have written down how to setup tumbleweed-snapshot, what about moving that note to salt? ;-) (should be easy, but if needed, I can help - just ping me on IRC)

It was written in the original p.o.o issue as well. And short of custom config rather self documented by providing a proper package unlike 99% of the rest of the contents of the box.

I tracked down the SSH key used just before to mcaj@nibbler from authorized_keys file on root

Who knows who that is and so we can have a discussion to understand what happened and avoid this from happening again?

jberry wrote:

Who knows who that is and so we can have a discussion to understand what happened and avoid this from happening again?

Martin Caj.

https://cz.linkedin.com/in/martin-caj-36a03399

To be clear the dependencies of tumbleweed-snapshot:

Requires: coreutils
Requires: findutils
Requires: grep
Requires: procmail
Requires: rsync

That's absolutely nothing. Given an explicit zypper rm tumbleweed-snapshot was run I find this whole thing fishy.