Project

General

Profile

Actions

tickets #139139

closed

State of nuka (affects Weblate / i18n.o.o / l10n.o.o)

Added by crameleon 6 months ago. Updated 5 months ago.

Status:
Resolved
Priority:
High
Assignee:
Category:
Weblate
Target version:
-
Start date:
2023-11-06
Due date:
% Done:

0%

Estimated time:

Description

Hi,

upon migrating nuka.infra.opensuse.org, it was discovered that this machine seems to still not be maintained.

It runs openSUSE Leap 15.3, an operating system which is two versions behind and considered End Of Life. Said operating seems to not be receiving patches either, as the machine showed an uptime of almost 300 days. This is not only a security risk for the rest of the infrastructure, but also a maintenance burden as it causes issues with integration into our Salt infrastructure.
After the migration it was not possible to re-enroll the machine to Salt, and the Weblate app on https://l10n.opensuse.org seems to show errors upon trying to log in. The machine also tries to talk to various services on the internet, which would need to be allowed on a case by case basis, but seems unreasonable given the lack of security updates.

Please upgrade this machine to a supported version of openSUSE Leap including the latest patches and let me know which, if any, outside communication is required afterwards.

Thanks,
Georg

Actions #1

Updated by crameleon 6 months ago

  • Priority changed from Normal to High
  • Private changed from Yes to No
Actions #2

Updated by crameleon 6 months ago

  • Subject changed from State of nuka to State of nuka (affects Weblate / i18n.o.o / l10n.o.o)
Actions #3

Updated by javierllorente 6 months ago

Logging in via the Web interface is still not possible. However, logging in via the REST API works (ie: Weblate FX).

Actions #4

Updated by crameleon 6 months ago

Hi @sbrabec @ateixeira,

two weeks have passed with no response. Please bring the software to an up to date state this month.
Otherwise the machine will be shut down.

Thanks,
Georg

Actions #5

Updated by ateixeira 6 months ago

  • Assignee changed from sbrabec to ateixeira

Hello @crameleon

First of all, I'm very sorry for the delay in responding, I must have missed your previous e-mail somehow.

I agree the state of this machine is not ideal. We have plans to move it to weblate cloud, and while that is confirmed it looks like it's taking a bit longer than I expected it would.
Unfortunately, upgrading this to a supported version of Leap would be a big effort due to python dependencies for the weblate service.

Would it be acceptable if I applied all available security updates for 15.3 but still kept the machine on this Leap version? At least to keep it running until we can finalize the migration to weblate cloud?

Other than that, do you have any more info on the migration issues that caused the currently present login failures? Any idea what we could do to get this working again?

As for external services, I don't really have an exhaustive list of everything it requires access to, I'll have to check and get back to you.

Actions #6

Updated by crameleon 6 months ago

Hi @ateixeira,

thanks for getting back!

Installing updates would be good, but it's not sufficient, as we still have issues applying our Salt states on this machine due to mismatching packages or incompatible Salt master<->minion versions.

When exactly will you decommission the machine? The migration plans already existed back in July.

Would openSUSE Tumbleweed be easier for you to maintain? It requires slightly more maintenance but would offer you a more modern Python stack.
What is important to us, is that the OS is maintained and supported by our surrounding automation infrastructure. Currently our surrounding infrastructure supports openSUSE Leap 15.5 and openSUSE Tumbleweed.

Other than that, do you have any more info on the migration issues that caused the currently present login failures? Any idea what we could do to get this working again?

If I understand correctly it tries to talk to id.opensuse.org (not sure why, as the user should be redirected there client-sided, maybe it tries to query metadata?). As announced (https://lists.opensuse.org/archives/list/heroes@lists.opensuse.org/thread/4YV256BAZPZ4ILA3MI76MVO2BSKBB265/), all traffic needs to be explicitly whitelisted, but given the state of the machine we did not make any efforts in identifying and satisfying its needs.

Please understand that in its current state, the machine is operating in violation with our policy (https://en.opensuse.org/openSUSE:Infrastructure_policy).

Best,
Georg

Actions #7

Updated by ateixeira 6 months ago

Hello.

I'm currently unable to get root access in this machine, as it won't accept my user password. I'm assuming this is related to the Salt integration issues.
Could you help me with that so that I can look into upgrading it?

Actions #8

Updated by crameleon 6 months ago

Hi,

you should be able to log in with your openSUSE Heroes/IPA account. From the Heroes VPN, it should be:

ssh ateixeira@nuka.infra.opensuse.org
Actions #9

Updated by crameleon 6 months ago

Passphrase authentication should take the same passphrase you use for authenticating to the VPN, alternatively the SSH key configured via https://freeipa.infra.opensuse.org.

Actions #10

Updated by ateixeira 6 months ago

crameleon wrote in #note-9:

Passphrase authentication should take the same passphrase you use for authenticating to the VPN, alternatively the SSH key configured via https://freeipa.infra.opensuse.org.

I'm able to login with my SSH key, the problem is that while logged in, if I try to get root access via sudo, password authentication is not working. I'm using the same password as when authenticating to the VPN.

Actions #11

Updated by crameleon 6 months ago ยท Edited

Ah, I understand:

pam_sss(sudo:auth): received for user ateixeira: 9 (Authentication service cannot retrieve authentication info)

It's missing some Salt deployed configuration for the new environment indeed.

For the time being, I added NOPASSWD, so it shouldn't prompt you.

Actions #12

Updated by ateixeira 5 months ago

I've updated the machine to Leap 15.5, and redeployed the weblate service. It looks to be working mostly fine, including login.
Here is a list of URLs we need access to:
github.com (port 22)
www.icewm.org (port 80)
uyuni-project.org (port 443)
rofi.roger-ferrer.org (port 80)
www.opensuse.org (port 80)
documentation.suse.com (port 443)
weblate.org (port 443)
build.opensuse.org (port 443)

Actions #13

Updated by crameleon 5 months ago

Thank you for the intervention.
I now allowed the requested traffic.

Actions #14

Updated by crameleon 5 months ago

I was now able to add the machine to Salt again. The highstate applied with no errors. :-)
Note that this also reverted the temporary sudo patch from earlier, you should be able to sudo with your passphrase now.

Actions #15

Updated by crameleon 5 months ago

I noticed that https://l10n.opensuse.org works again (at least it let me log in), hence I set the component on https://status.opensuse.org to "Operational" again.

Actions #16

Updated by ateixeira 5 months ago

crameleon wrote in #note-13:

Thank you for the intervention.
I now allowed the requested traffic.

Can you double check if github.com access is allowed on port 22?

I'm still getting errors like this:
Nov 28 05:47:36 nuka weblate[25456]: WARNING Could not update the repository: RepositoryException: ssh: connect to host github.com port 22: Network is unreachable

Actions #17

Updated by ateixeira 5 months ago

ateixeira wrote in #note-16:

crameleon wrote in #note-13:

Thank you for the intervention.
I now allowed the requested traffic.

Can you double check if github.com access is allowed on port 22?

I'm still getting errors like this:
Nov 28 05:47:36 nuka weblate[25456]: WARNING Could not update the repository: RepositoryException: ssh: connect to host github.com port 22: Network is unreachable

Hello.

In addition to the mentioned issues with accessing github.com on port 22, Weblate now seems to also not be able to access https://id.opensuse.org and https://www.opensuse.org/openid/user/ with error "Network is unreachable".

This means it's again impossible to login to Weblate. Plus most projects there depend on github access anyway so we really need both problems fixed.

Actions #18

Updated by crameleon 5 months ago

Hi,

I misread github.com as port 443. I now changed it to 22 as desired and also added HTTPS to id.opensuse.org.
Note that www.o.o/openid is a legacy endpoint, let me know if you need it, but ideally you shouldn't if you use id.o.o.

Cheers,
Georg

Actions #19

Updated by crameleon 5 months ago

  • Status changed from New to Feedback

Hi,

what's the status here?

Actions #20

Updated by ateixeira 5 months ago

  • Status changed from Feedback to Resolved

I got feedback from the translation team that everything seems to be working. I'm changing status to Resolved.

Actions #21

Updated by crameleon 5 months ago

Great news, thank you!

Actions

Also available in: Atom PDF