action #159060
closedcoordination #155182: [epic] Participate in alpha-testing of new version of velociraptor-client
Rollback/switch to officially installed velociraptor-client repo and server size:S
Added by okurz about 1 year ago. Updated 3 months ago.
0%
Description
Motivation¶
#155179 is done. Crosscheck with the velociraptor-client development team and rollback our salt managed machines to use official packages and the non-development server again
Acceptance Criteria¶
- AC1: salt-controlled machines use the stable velociraptor repo
Suggestions¶
- Look at https://gitlab.suse.de/openqa/salt-pillars-openqa/-/merge_requests/736 for previous changes
- Clarify which repo to use, see https://suse.slack.com/archives/C02NJAA1PEC/p1713696437484159
https://download.suse.de/ibs/SUSE:/Velociraptor/15.5/ only offers x86_64. I thought we should use that repo unconditionally but what about aarch64, s390x, ppc64le? Until we found a better place to install aarch64/ppc64le/s390x Leap 15.5 packages from I will revert to the official Leap repository content. Soon we will upgrade to Leap 15.6 anyway- Check that we use the correct endpoint
Updated by okurz about 1 year ago
- Copied from action #155179: Participate in alpha-testing of new version of velociraptor-client added
Updated by okurz about 1 year ago
- Subject changed from Participate in alpha-testing of new version of velociraptor-client to Rollback to officially installed velociraptor-client repo and server
- Status changed from New to Feedback
- Target version changed from Tools - Next to future
https://suse.slack.com/archives/C02NJAA1PEC/p1713272166072159
(Oliver Kurz) Hi, can we switch the LSG QE machines back from https://download.opensuse.org/repositories/security:/sensor/15.5/ to official Leap packages and the server target URL accordingly?
(Marcela Mašláňová) All machines will need to install from IBS. We are just waiting for the official announcement
(Oliver Kurz) that sounds like a different problem but I will wait for such announcement then
Updated by okurz about 1 year ago
- Subject changed from Rollback to officially installed velociraptor-client repo and server to Rollback/switch to officially installed velociraptor-client repo and server
- Target version changed from future to Ready
announcement was sent by Jeff Mahoney in https://suse.slack.com/archives/C02NJAA1PEC/p1713535789090039 pointing to https://confluence.suse.com/display/CS/Sensor+-+Linux+Endpoint+Protection+Agent with deployment instructions on https://gitlab.suse.de/linux-security-sensor/suse-client-deployment and https://confluence.suse.com/display/CS/Deploying+the+Sensor+Client
https://suse.slack.com/archives/C02NJAA1PEC/p1713537499171589
(Oliver Kurz) @Jeff Mahoney 1. "It includes virtual machines and systems that are subject to frequent redeployment, like those in […] test systems." So do the 1..10k automated openQA tests need to install velociraptor-client and report to the server during openQA test runs? […] 3. "any Linux servers maintained by BCL" and "must switch to the IBS release" is not possible for the openqa.opensuse.org infrastructure. I don't know how to fulfill such requirements
Updated by okurz about 1 year ago
I checked the config mentioned on the documentation pages and could confirm that we are just expected to go back to https://sec-velociraptor.prg.suse.com:8000 so created https://gitlab.suse.de/openqa/salt-pillars-openqa/-/merge_requests/783 (merged)
Unfortunately we need to switch to an internal repo but ok:
https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/1160
Updated by livdywan about 1 year ago
https://gitlab.suse.de/openqa/salt-states-openqa/-/jobs/2513164
ID: security-sensor.repo
Function: pkgrepo.managed
Result: False
Comment: Failed to configure repo 'security-sensor.repo': refresh_db() got multiple values for keyword argument 'root'
Started: 18:31:20.487570
Duration: 1268.534 ms
Changes:
Updated by okurz about 1 year ago
https://suse.slack.com/archives/C02NJAA1PEC/p1713696437484159
(Oliver Kurz) https://download.suse.de/ibs/SUSE:/Velociraptor/15.5/ only offers x86_64. I thought we should use that repo unconditionally but what about aarch64, s390x, ppc64le? Until we found a better place to install aarch64/ppc64le/s390x Leap 15.5 packages from I will revert to the official Leap repository content. Soon we will upgrade to Leap 15.6 anyway
Updated by livdywan about 1 year ago
- Subject changed from Rollback/switch to officially installed velociraptor-client repo and server to Rollback/switch to officially installed velociraptor-client repo and server size:S
- Description updated (diff)
Updated by okurz about 1 year ago
- Due date changed from 2024-05-05 to 2024-05-12
Asked in https://suse.slack.com/archives/C02NJAA1PEC/p1714397276453239 now again
Updated by okurz about 1 year ago
- Due date deleted (
2024-05-12) - Status changed from Feedback to Blocked
- Target version changed from Ready to future
Updated by okurz 6 months ago
This is related to #169546 now.
In
https://suse.slack.com/archives/C02NJAA1PEC/p1731488334523429
I asked
Hi, https://jira.suse.com/browse/SENS-111 is unresolved since about 7 months and now we come to more problems due to the common criteria related network changes. With that machines which are not in a CC-compliant location can not access repositories on download.suse.de anymore. Can we switch to an official repository like Leap update channel again? Or second best an OBS repository
Updated by okurz 6 months ago · Edited
- Status changed from Blocked to In Progress
No news yet but since #170368 machines like tumblesle are again connecting to the salt master openqa.suse.de and in consequence show up more problematic regarding not being able to reach repositories like download.suse.de as we do not have and should not have wireguard tunnels there. With that I will disable IBS repositories and effectively downgrade the velociraptor-client to a version possibly incompatible with the current data collecting server. Let's see if someone brings this up as a problem and expedites the work to upstream the new version of the security sensor.
Based on sudo salt '*.qe.nue2.suse.org' cmd.run 'zypper ref security-sensor.repo' this affects the following machines: tumblesle, baremetal-support, jenkins, backup-vm, osiris, schort-server, qamaster, unreal6, backup-qam
Updated by okurz 6 months ago
- Status changed from In Progress to Blocked
I called
sudo salt -L 'backup-qam.qe.nue2.suse.org,osiris-1.qe.nue2.suse.org,unreal6.qe.nue2.suse.org,qamaster.qe.nue2.suse.org,schort-server.qe.nue2.suse.org,jenkins.qe.nue2.suse.org,tumblesle.qe.nue2.suse.org' cmd.run 'echo "needs_wireguard: False" >> /etc/salt/grains'
as we shouldn't need wireguard on those, maybe on unreal6, maybe backup-qam.
Then I disabled the security sensor repo for all NUE2 machines not currently having wireguard
sudo salt '*.qe.nue2.suse.org' cmd.run 'zypper mr -d --name "Server Monitoring Software - disabled due to https://progress.opensuse.org/issues/159060" security-sensor.repo'
and then
sudo salt '*.qe.nue2.suse.org' cmd.run 'zypper --gpg-auto-import-keys ref'
completed fine again.
Back to blocking on https://jira.suse.com/browse/SENS-111
Updated by okurz 5 months ago
- Status changed from Blocked to Workable
- Assignee deleted (
okurz) - Priority changed from Low to High
Apparently what I did was not enough or seems to be at least partially reverted. From tumblesle I found:
Repository 'Server Monitoring Software' is invalid.
[security-sensor.repo|http://download.suse.de/ibs/SUSE:/Velociraptor/15.6] Failed to retrieve new repository metadata.
History:
- [|] Error trying to read from 'http://download.suse.de/ibs/SUSE:/Velociraptor/15.6'
- Timeout exceeded when accessing 'http://download.suse.de/ibs/SUSE:/Velociraptor/15.6/content'.
Warning: Skipping repository 'Server Monitoring Software' because of the above error.
we should ensure that http://download.suse.de/ibs/SUSE:/Velociraptor/15.6 is at best removed or at least disabled and not re-enabled by salt or something.
Updated by mkittler 5 months ago · Edited
Judging by https://suse.slack.com/archives/C02NJAA1PEC/p1713696437484159 and https://jira.suse.com/browse/SENS-111 IBS is still the way to go.
It looks like we also use that consistently. Considering we don't have Wireguard on all NUE2 hosts (e.g. we don't have it on tumblesle) we might indeed want to exclude them using a command like sudo salt '*.qe.nue2.suse.org' cmd.run 'zypper mr -d --name "Server Monitoring Software - disabled due to https://progress.opensuse.org/issues/159060" security-sensor.repo' but we also need to ensure Salt knows about it.
Updated by openqa_review 5 months ago
- Due date set to 2024-12-28
Setting due date based on mean cycle time of SUSE QE Tools
Updated by jbaier_cz 5 months ago
From https://gitlab.suse.de/openqa/salt-states-openqa/-/jobs/3541473 it also seems, that the new package has different name and conflicts with the current one (which needs to be uninstalled).
backup-vm.qe.nue2.suse.org:
----------
ID: security-sensor.repo
Function: pkg.latest
Name: velociraptor-client
Result: False
Comment: An error was encountered while installing package(s): Zypper command failure: Running scope as unit: run-rf7ce53766060495a91eb212eb63b2e5a.scopeLoading repository data...
Reading installed packages...
Resolving package dependencies...
Problem: 1: the installed velociraptor-0.6.7.4~git63.4a1ed09d-bp156.2.21.x86_64 conflicts with 'velociraptor-client' provided by the to be installed velociraptor-client-0.6.7.4~git63.4a1ed09d-bp156.2.21.x86_64
Solution 1: deinstallation of velociraptor-0.6.7.4~git63.4a1ed09d-bp156.2.21.x86_64
Solution 2: do not install velociraptor-client-0.6.7.4~git63.4a1ed09d-bp156.2.21.x86_64
Choose from above solutions by number or cancel [1/2/c/d/?] (c): c
Started: 10:54:59.136414
Duration: 7939.167 ms
Changes:
Summary for backup-vm.qe.nue2.suse.org
Updated by mkittler 5 months ago
- Status changed from In Progress to Feedback
The package conflict is because the hosts without the IBS repo will now automatically vendor-change to the version of velociraptor-client. I think this is actually what we want and the only problem is that on the backup VM velociraptor (not -client) is also installed. It looks like we don't need it so I uninstalled it resolving the conflict. I applied the salt states again on NBG2 hosts and it seems that the backup VM was the only affected host.
This was the vendor switch:
martchus@backup-vm:~> sudo zypper in velociraptor-client
Loading repository data...
Reading installed packages...
Resolving package dependencies...
Problem: 1: the installed velociraptor-0.6.7.4~git63.4a1ed09d-bp156.2.21.x86_64 conflicts with 'velociraptor-client' provided by the to be installed velociraptor-client-0.6.7.4~git63.4a1ed09d-bp156.2.21.x86_64
Solution 1: deinstallation of velociraptor-0.6.7.4~git63.4a1ed09d-bp156.2.21.x86_64
Solution 2: do not install velociraptor-client-0.6.7.4~git63.4a1ed09d-bp156.2.21.x86_64
Choose from above solutions by number or cancel [1/2/c/d/?] (c): ^C
Trying to exit gracefully...
martchus@backup-vm:~> sudo zypper in velociraptor-client
Loading repository data...
Reading installed packages...
Resolving package dependencies...
Problem: 1: the installed velociraptor-0.6.7.4~git63.4a1ed09d-bp156.2.21.x86_64 conflicts with 'velociraptor-client' provided by the to be installed velociraptor-client-0.6.7.4~git63.4a1ed09d-bp156.2.21.x86_64
Solution 1: deinstallation of velociraptor-0.6.7.4~git63.4a1ed09d-bp156.2.21.x86_64
Solution 2: do not install velociraptor-client-0.6.7.4~git63.4a1ed09d-bp156.2.21.x86_64
Choose from above solutions by number or cancel [1/2/c/d/?] (c): 1
Resolving dependencies...
Resolving package dependencies...
The following NEW package is going to be installed:
velociraptor-client
The following package is going to be REMOVED:
velociraptor
1 new package to install, 1 to remove.
Package download size: 15.0 MiB
Package install size change:
| 55.9 MiB required by packages that will be installed
-1.5 MiB | - 57.4 MiB released by packages that will be removed
Backend: classic_rpmtrans
Continue? [y/n/v/...? shows all options] (y):
Retrieving: velociraptor-client-0.6.7.4~git63.4a1ed09d-bp156.2.21.x86_64 (Main Repository (OSS)) (1/1), 15.0 MiB
Retrieving: velociraptor-client-0.6.7.4~git63.4a1ed09d-bp156.2.21.x86_64.rpm .................................................................................[done (41.4 MiB/s)]
Checking for file conflicts: ..............................................................................................................................................[done]
Removed "/etc/systemd/system/multi-user.target.wants/velociraptor-client.service".
warning: /etc/velociraptor/client.config saved as /etc/velociraptor/client.config.rpmsave
(1/2) Removing: velociraptor-0.6.7.4~git63.4a1ed09d-bp156.2.21.x86_64 .....................................................................................................[done]
Updating /etc/sysconfig/velociraptor-client ...
(2/2) Installing: velociraptor-client-0.6.7.4~git63.4a1ed09d-bp156.2.21.x86_64 ............................................................................................[done]
I'm mentioning this because it means we actually didn't lose much by that. The upstream version is still 0.6.7.4~git63.4a1ed09d - only the downstream version changed.
Does this mean we can actually just use the Leap provided version at this point everywhere? (The installed version is still from 2 years ago btw, see https://build.opensuse.org/package/show/openSUSE:Leap:15.6/velociraptor. Just for comparison: The current version in Factory would only be 6 month old, see https://build.opensuse.org/package/show/openSUSE:Factory/velociraptor.)
I'm now also wondering why the title of this ticket says "… client repo and server". We don't install velociraptor (the server) via salt (where we only install velociraptor-client) and I'd expect the client is also sufficient for us.
Updated by jbaier_cz 5 months ago
I spotted new related failure in the salt deployment, not sure if that's from zypper or salt-zypper module or what:
worker40.oqa.prg2.suse.org:
----------
ID: security-sensor.repo
Function: pkg.latest
Name: velociraptor-client
Result: False
Comment: An exception occurred in this state: Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/salt/state.py", line 2402, in call
*cdata["args"], **cdata["kwargs"]
File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 149, in __call__
return self.loader.run(run_func, *args, **kwargs)
File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1234, in run
return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
File "/usr/lib/python3.6/site-packages/contextvars/__init__.py", line 38, in run
return callable(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1249, in _run_as
ret = _func_or_method(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1285, in wrapper
return f(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/salt/states/pkg.py", line 2659, in latest
*desired_pkgs, fromrepo=fromrepo, refresh=refresh, **kwargs
File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 149, in __call__
return self.loader.run(run_func, *args, **kwargs)
File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1234, in run
return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
File "/usr/lib/python3.6/site-packages/contextvars/__init__.py", line 38, in run
return callable(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1249, in _run_as
ret = _func_or_method(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/salt/modules/zypperpkg.py", line 828, in latest_version
package_info = info_available(*names, **kwargs)
File "/usr/lib/python3.6/site-packages/salt/modules/zypperpkg.py", line 752, in info_available
"info", "-t", "package", *batch[:batch_size]
File "/usr/lib/python3.6/site-packages/salt/modules/zypperpkg.py", line 439, in __call
salt.utils.stringutils.to_str(self.__call_result["stdout"])
File "/usr/lib64/python3.6/xml/dom/minidom.py", line 1968, in parseString
return expatbuilder.parseString(string)
File "/usr/lib64/python3.6/xml/dom/expatbuilder.py", line 925, in parseString
return builder.parseString(string)
File "/usr/lib64/python3.6/xml/dom/expatbuilder.py", line 223, in parseString
parser.Parse(string, True)
xml.parsers.expat.ExpatError: syntax error: line 1, column 0
Started: 13:14:41.455083
Duration: 2986.675 ms
Changes:
----------
Updated by mkittler 5 months ago
- Status changed from Feedback to Blocked
Back to blocking on https://jira.suse.com/browse/SENS-111
Updated by okurz 4 months ago
- Related to action #175695: salt states sporadically fail in applying the security sensor repo with "xml.parsers.expat.ExpatError: syntax error: line 1, column 0", didn't we remove the security sensor repo? added
Updated by okurz 3 months ago
- Status changed from Blocked to In Progress
- Target version changed from future to Ready
https://jira.suse.com/browse/SENS-111 is still unresolved however in the meantime package versions in openSUSE Tumbleweed and Leap seem to be in a better state, e.g. after https://build.opensuse.org/request/show/1238560 having 0.7.0.4 included.
From zypper se --match-exact --verbose -t package 'velociraptor-client'
Loading repository data...
Reading installed packages...
S | Name | Type | Version | Arch | Repository
---+---------------------+---------+------------------------------------+--------+----------------------------------------
v | velociraptor-client | package | 0.6.7.4~git63.4a1ed09d-bp156.2.21 | x86_64 | openSUSE-Leap-15.6-1
name: velociraptor-client
v | velociraptor-client | package | 0.7.0.4.git142.862ef23-bp156.3.3.1 | x86_64 | Update repository of openSUSE Backports
name: velociraptor-client
i+ | velociraptor-client | package | 0.7.0.4.git142.862ef23-lp156.11.3 | x86_64 | Server Monitoring Software
name: velociraptor-client
so we can switch back to remove the additional repository again.
Updated by okurz 3 months ago
Merged. Now I did
sudo salt --no-color --state-output=changes \* cmd.run 'zypper rr security-sensor && zypper ref && zypper -n dup --download-only' queue=True | grep -v 'Result.*Clean'
and after checking the output for sanity a complete dup on all. I checked the status of services and they run ok but – probably as expected – many don't connect already for months and nobody cared.
I asked in #proj-linux-security-sensor https://suse.slack.com/archives/C02NJAA1PEC/p1739542859298009
Because the security sensor project was seen as critical and important may I ask what is the current view on it? I found that our servers in NUE2 FC Basement can not reach the velociraptor server anymore, probably since network changes for common criteria certification prevented access after 2024-11-04 but nobody seemed to have realized that or was alerted. What is the process around ensuring that servers are monitored and the requirements for the security sensor project are fulfilled? Honestly my assumption is that some parts of the original requirements like "> 95% of SUSE servers must be covered" are not followed up with and simply forgotten but I hope I am wrong
but if there is no follow-up then I am also ok with it