openSUSE admin

The main part (mirrorcache) is fixed, and I reverted the workarounds in the haproxy config yesterday.

However, the things listed under "bonus points" are still not fixed - which means nobody will be able to login (and be able to fix things) if the server fails next time.

andriinikitin wrote:

cboltz wrote:

At around 15:00 CEST today, I noticed that mirrorcache.o.o is down.

The problem was that WebUI couldn't connect to database at startup after reboot. This should be fixed by https://github.com/openSUSE/MirrorCache/commit/278719a6753

I guess that means you have mysql/mariadb running locally?

I am not sure if I should put retry there: I have impression that it may cause some damage to DB in extreme cases, but maybe will add it at some point.

I don't know the details, but in general, I can't imagine that retrying to connect to the database would cause damage.

Bonus points if you

• make sure the server connects to the saltmaster

It should be fixed now. (I was testing salt on mirrorcache some time ago, so minion was configured to local only).

Confirmed, test.ping works now :-)

• apply at least the basic salt highstate so that ssh logins etc. work

I am not sure which exact state should be applied here, but at least cmd.run does work, so I guess it can be marked as complete.

cmd.run (aka "the salt backdoor") is a workaround, and I added my ssh key to more than one machine this way ;-) - but it's far from perfect.

role.base makes sure that for example ssh logins as user (via ldap/freeipa) work, and that sudo works for users in the wheel group. It also does some other useful config we want everywhere, see the salt repo for details.

So - please run a highstate (maybe first with test=True) to apply these things. (Note: This needs the pillar/id/ files, so you'll have to wait until your MR gets merged.)

• add the root password of all mirrorcache* machines to the infra pass (gitlab.infra.opensuse.org/infra/pass.git)

I don't know current root password. But since it is now possible to add ssh key with salt - this can be skipped? (Or I will appreciate if somebody does it or instruct me how to do it).

Since you can login as root via ssh, you could simply set a new password ;-)

cboltz wrote:

andriinikitin wrote:

I am not sure if I should put retry there: I have impression that it may cause some damage to DB in extreme cases, but maybe will add it at some point.

I don't know the details, but in general, I can't imagine that retrying to connect to the database would cause damage.

Me neither, surely every application will always have some reconnect code.