Project

General

Profile

Actions
Copy link

action #159060

open

coordination #155182: [epic] Participate in alpha-testing of new version of velociraptor-client

Rollback/switch to officially installed velociraptor-client repo and server size:S

Added by okurz 8 months ago. Updated 1 day ago.

Status:
Blocked
Priority:
Low
Assignee:
okurz
Target version:
future
Start date:
2024-02-08
Due date:
% Done:

0%

Estimated time:
Tags:
infra

Description

Motivation

#155179 is done. Crosscheck with the velociraptor-client development team and rollback our salt managed machines to use official packages and the non-development server again

Acceptance Criteria

  • AC1: salt-controlled machines use the stable velociraptor repo

Suggestions

Related issues 1 (0 open — 1 closed)

Copied from QA (public) - action #155179: Participate in alpha-testing of new version of velociraptor-clientResolvedokurz2024-02-08

Actions
Actions
Copy link
 #1

Updated by okurz 8 months ago

  • Copied from action #155179: Participate in alpha-testing of new version of velociraptor-client added
Actions
Copy link
 #2

Updated by okurz 8 months ago

  • Subject changed from Participate in alpha-testing of new version of velociraptor-client to Rollback to officially installed velociraptor-client repo and server
  • Status changed from New to Feedback
  • Target version changed from Tools - Next to future

https://suse.slack.com/archives/C02NJAA1PEC/p1713272166072159

(Oliver Kurz) Hi, can we switch the LSG QE machines back from https://download.opensuse.org/repositories/security:/sensor/15.5/ to official Leap packages and the server target URL accordingly?
(Marcela Mašláňová) All machines will need to install from IBS. We are just waiting for the official announcement
(Oliver Kurz) that sounds like a different problem but I will wait for such announcement then

Actions
Copy link
 #4

Updated by okurz 8 months ago

  • Subject changed from Rollback to officially installed velociraptor-client repo and server to Rollback/switch to officially installed velociraptor-client repo and server
  • Target version changed from future to Ready

announcement was sent by Jeff Mahoney in https://suse.slack.com/archives/C02NJAA1PEC/p1713535789090039 pointing to https://confluence.suse.com/display/CS/Sensor+-+Linux+Endpoint+Protection+Agent with deployment instructions on https://gitlab.suse.de/linux-security-sensor/suse-client-deployment and https://confluence.suse.com/display/CS/Deploying+the+Sensor+Client

https://suse.slack.com/archives/C02NJAA1PEC/p1713537499171589

(Oliver Kurz) @Jeff Mahoney 1. "It includes virtual machines and systems that are subject to frequent redeployment, like those in […] test systems." So do the 1..10k automated openQA tests need to install velociraptor-client and report to the server during openQA test runs? […] 3. "any Linux servers maintained by BCL" and "must switch to the IBS release" is not possible for the openqa.opensuse.org infrastructure. I don't know how to fulfill such requirements

Actions
Copy link
 #5

Updated by okurz 8 months ago

I checked the config mentioned on the documentation pages and could confirm that we are just expected to go back to https://sec-velociraptor.prg.suse.com:8000 so created https://gitlab.suse.de/openqa/salt-pillars-openqa/-/merge_requests/783 (merged)

Unfortunately we need to switch to an internal repo but ok:

https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/1160

Actions
Copy link
 #6

Updated by livdywan 8 months ago

https://gitlab.suse.de/openqa/salt-states-openqa/-/jobs/2513164

          ID: security-sensor.repo
    Function: pkgrepo.managed
      Result: False
     Comment: Failed to configure repo 'security-sensor.repo': refresh_db() got multiple values for keyword argument 'root'
     Started: 18:31:20.487570
    Duration: 1268.534 ms
     Changes:
Actions
Copy link
 #7

Updated by okurz 8 months ago

https://suse.slack.com/archives/C02NJAA1PEC/p1713696437484159

(Oliver Kurz) https://download.suse.de/ibs/SUSE:/Velociraptor/15.5/ only offers x86_64. I thought we should use that repo unconditionally but what about aarch64, s390x, ppc64le? Until we found a better place to install aarch64/ppc64le/s390x Leap 15.5 packages from I will revert to the official Leap repository content. Soon we will upgrade to Leap 15.6 anyway

Actions
Copy link
 #8

Updated by okurz 8 months ago

  • Due date set to 2024-05-05
Actions
Copy link
 #9

Updated by livdywan 8 months ago

  • Subject changed from Rollback/switch to officially installed velociraptor-client repo and server to Rollback/switch to officially installed velociraptor-client repo and server size:S
  • Description updated (diff)
Actions
Copy link
 #10

Updated by okurz 8 months ago

  • Due date changed from 2024-05-05 to 2024-05-12
Actions
Copy link
 #11

Updated by okurz 8 months ago

  • Due date deleted (2024-05-12)
  • Status changed from Feedback to Blocked
  • Target version changed from Ready to future
Actions
Copy link
 #13

Updated by okurz about 1 month ago

This is related to #169546 now.

In
https://suse.slack.com/archives/C02NJAA1PEC/p1731488334523429
I asked

Hi, https://jira.suse.com/browse/SENS-111 is unresolved since about 7 months and now we come to more problems due to the common criteria related network changes. With that machines which are not in a CC-compliant location can not access repositories on download.suse.de anymore. Can we switch to an official repository like Leap update channel again? Or second best an OBS repository

Actions
Copy link
 #15

Updated by okurz 21 days ago · Edited

  • Status changed from Blocked to In Progress

No news yet but since #170368 machines like tumblesle are again connecting to the salt master openqa.suse.de and in consequence show up more problematic regarding not being able to reach repositories like download.suse.de as we do not have and should not have wireguard tunnels there. With that I will disable IBS repositories and effectively downgrade the velociraptor-client to a version possibly incompatible with the current data collecting server. Let's see if someone brings this up as a problem and expedites the work to upstream the new version of the security sensor.

Based on sudo salt '*.qe.nue2.suse.org' cmd.run 'zypper ref security-sensor.repo' this affects the following machines: tumblesle, baremetal-support, jenkins, backup-vm, osiris, schort-server, qamaster, unreal6, backup-qam

Actions
Copy link
 #16

Updated by okurz 21 days ago

  • Target version changed from future to Ready
Actions
Copy link
 #17

Updated by okurz 21 days ago

  • Status changed from In Progress to Blocked

I called

sudo salt -L 'backup-qam.qe.nue2.suse.org,osiris-1.qe.nue2.suse.org,unreal6.qe.nue2.suse.org,qamaster.qe.nue2.suse.org,schort-server.qe.nue2.suse.org,jenkins.qe.nue2.suse.org,tumblesle.qe.nue2.suse.org' cmd.run 'echo "needs_wireguard: False" >> /etc/salt/grains'

as we shouldn't need wireguard on those, maybe on unreal6, maybe backup-qam.

Then I disabled the security sensor repo for all NUE2 machines not currently having wireguard

sudo salt '*.qe.nue2.suse.org' cmd.run 'zypper mr -d --name "Server Monitoring Software - disabled due to https://progress.opensuse.org/issues/159060" security-sensor.repo'

and then

sudo salt '*.qe.nue2.suse.org' cmd.run 'zypper --gpg-auto-import-keys ref'

completed fine again.

Back to blocking on https://jira.suse.com/browse/SENS-111

Actions
Copy link
 #18

Updated by okurz 14 days ago

  • Status changed from Blocked to Workable
  • Assignee deleted (okurz)
  • Priority changed from Low to High

Apparently what I did was not enough or seems to be at least partially reverted. From tumblesle I found:

Repository 'Server Monitoring Software' is invalid.
[security-sensor.repo|http://download.suse.de/ibs/SUSE:/Velociraptor/15.6] Failed to retrieve new repository metadata.
History:
 - [|] Error trying to read from 'http://download.suse.de/ibs/SUSE:/Velociraptor/15.6'
 - Timeout exceeded when accessing 'http://download.suse.de/ibs/SUSE:/Velociraptor/15.6/content'.
Warning: Skipping repository 'Server Monitoring Software' because of the above error.

we should ensure that http://download.suse.de/ibs/SUSE:/Velociraptor/15.6 is at best removed or at least disabled and not re-enabled by salt or something.

Actions
Copy link
 #19

Updated by mkittler 13 days ago

  • Description updated (diff)
  • Status changed from Workable to In Progress
  • Assignee set to mkittler
Actions
Copy link
 #20

Updated by mkittler 13 days ago · Edited

Judging by https://suse.slack.com/archives/C02NJAA1PEC/p1713696437484159 and https://jira.suse.com/browse/SENS-111 IBS is still the way to go.

It looks like we also use that consistently. Considering we don't have Wireguard on all NUE2 hosts (e.g. we don't have it on tumblesle) we might indeed want to exclude them using a command like sudo salt '*.qe.nue2.suse.org' cmd.run 'zypper mr -d --name "Server Monitoring Software - disabled due to https://progress.opensuse.org/issues/159060" security-sensor.repo' but we also need to ensure Salt knows about it.

Actions
Copy link
 #22

Updated by openqa_review 13 days ago

  • Due date set to 2024-12-28

Setting due date based on mean cycle time of SUSE QE Tools

Actions
Copy link
 #23

Updated by jbaier_cz 11 days ago

From https://gitlab.suse.de/openqa/salt-states-openqa/-/jobs/3541473 it also seems, that the new package has different name and conflicts with the current one (which needs to be uninstalled).

backup-vm.qe.nue2.suse.org:
----------
          ID: security-sensor.repo
    Function: pkg.latest
        Name: velociraptor-client
      Result: False
     Comment: An error was encountered while installing package(s): Zypper command failure: Running scope as unit: run-rf7ce53766060495a91eb212eb63b2e5a.scopeLoading repository data...
              Reading installed packages...
              Resolving package dependencies...

              Problem: 1: the installed velociraptor-0.6.7.4~git63.4a1ed09d-bp156.2.21.x86_64 conflicts with 'velociraptor-client' provided by the to be installed velociraptor-client-0.6.7.4~git63.4a1ed09d-bp156.2.21.x86_64
               Solution 1: deinstallation of velociraptor-0.6.7.4~git63.4a1ed09d-bp156.2.21.x86_64
               Solution 2: do not install velociraptor-client-0.6.7.4~git63.4a1ed09d-bp156.2.21.x86_64

              Choose from above solutions by number or cancel [1/2/c/d/?] (c): c
     Started: 10:54:59.136414
    Duration: 7939.167 ms
     Changes:   
Summary for backup-vm.qe.nue2.suse.org
Actions
Copy link
 #24

Updated by mkittler 11 days ago

The MR has been merged. I skimmed through the output of sudo salt 'monitor.qe.nue2.suse.org' cmd.run 'zypper lr -d | grep -i veloci' and it looked correct.

I'm looking into the package conflict.

Actions
Copy link
 #25

Updated by mkittler 10 days ago

  • Status changed from In Progress to Feedback

The package conflict is because the hosts without the IBS repo will now automatically vendor-change to the version of velociraptor-client. I think this is actually what we want and the only problem is that on the backup VM velociraptor (not -client) is also installed. It looks like we don't need it so I uninstalled it resolving the conflict. I applied the salt states again on NBG2 hosts and it seems that the backup VM was the only affected host.

This was the vendor switch:

martchus@backup-vm:~> sudo zypper in velociraptor-client
Loading repository data...
Reading installed packages...
Resolving package dependencies...

Problem: 1: the installed velociraptor-0.6.7.4~git63.4a1ed09d-bp156.2.21.x86_64 conflicts with 'velociraptor-client' provided by the to be installed velociraptor-client-0.6.7.4~git63.4a1ed09d-bp156.2.21.x86_64
 Solution 1: deinstallation of velociraptor-0.6.7.4~git63.4a1ed09d-bp156.2.21.x86_64
 Solution 2: do not install velociraptor-client-0.6.7.4~git63.4a1ed09d-bp156.2.21.x86_64

Choose from above solutions by number or cancel [1/2/c/d/?] (c): ^C
Trying to exit gracefully...
martchus@backup-vm:~> sudo zypper in velociraptor-client
Loading repository data...
Reading installed packages...
Resolving package dependencies...

Problem: 1: the installed velociraptor-0.6.7.4~git63.4a1ed09d-bp156.2.21.x86_64 conflicts with 'velociraptor-client' provided by the to be installed velociraptor-client-0.6.7.4~git63.4a1ed09d-bp156.2.21.x86_64
 Solution 1: deinstallation of velociraptor-0.6.7.4~git63.4a1ed09d-bp156.2.21.x86_64
 Solution 2: do not install velociraptor-client-0.6.7.4~git63.4a1ed09d-bp156.2.21.x86_64

Choose from above solutions by number or cancel [1/2/c/d/?] (c): 1

Resolving dependencies...
Resolving package dependencies...

The following NEW package is going to be installed:
  velociraptor-client

The following package is going to be REMOVED:
  velociraptor

1 new package to install, 1 to remove.

Package download size:    15.0 MiB

Package install size change:
              |      55.9 MiB  required by packages that will be installed
    -1.5 MiB  |  -   57.4 MiB  released by packages that will be removed

Backend:  classic_rpmtrans
Continue? [y/n/v/...? shows all options] (y):     
Retrieving: velociraptor-client-0.6.7.4~git63.4a1ed09d-bp156.2.21.x86_64 (Main Repository (OSS))                                                             (1/1),  15.0 MiB    
Retrieving: velociraptor-client-0.6.7.4~git63.4a1ed09d-bp156.2.21.x86_64.rpm .................................................................................[done (41.4 MiB/s)]

Checking for file conflicts: ..............................................................................................................................................[done]
Removed "/etc/systemd/system/multi-user.target.wants/velociraptor-client.service".
warning: /etc/velociraptor/client.config saved as /etc/velociraptor/client.config.rpmsave
(1/2) Removing: velociraptor-0.6.7.4~git63.4a1ed09d-bp156.2.21.x86_64 .....................................................................................................[done]
Updating /etc/sysconfig/velociraptor-client ...
(2/2) Installing: velociraptor-client-0.6.7.4~git63.4a1ed09d-bp156.2.21.x86_64 ............................................................................................[done]

I'm mentioning this because it means we actually didn't lose much by that. The upstream version is still 0.6.7.4~git63.4a1ed09d - only the downstream version changed.

Does this mean we can actually just use the Leap provided version at this point everywhere? (The installed version is still from 2 years ago btw, see https://build.opensuse.org/package/show/openSUSE:Leap:15.6/velociraptor. Just for comparison: The current version in Factory would only be 6 month old, see https://build.opensuse.org/package/show/openSUSE:Factory/velociraptor.)

I'm now also wondering why the title of this ticket says "… client repo and server". We don't install velociraptor (the server) via salt (where we only install velociraptor-client) and I'd expect the client is also sufficient for us.

Actions
Copy link
 #26

Updated by jbaier_cz 10 days ago

I spotted new related failure in the salt deployment, not sure if that's from zypper or salt-zypper module or what:

worker40.oqa.prg2.suse.org:
----------
          ID: security-sensor.repo
    Function: pkg.latest
        Name: velociraptor-client
      Result: False
     Comment: An exception occurred in this state: Traceback (most recent call last):
                File "/usr/lib/python3.6/site-packages/salt/state.py", line 2402, in call
                  *cdata["args"], **cdata["kwargs"]
                File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 149, in __call__
                  return self.loader.run(run_func, *args, **kwargs)
                File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1234, in run
                  return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
                File "/usr/lib/python3.6/site-packages/contextvars/__init__.py", line 38, in run
                  return callable(*args, **kwargs)
                File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1249, in _run_as
                  ret = _func_or_method(*args, **kwargs)
                File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1285, in wrapper
                  return f(*args, **kwargs)
                File "/usr/lib/python3.6/site-packages/salt/states/pkg.py", line 2659, in latest
                  *desired_pkgs, fromrepo=fromrepo, refresh=refresh, **kwargs
                File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 149, in __call__
                  return self.loader.run(run_func, *args, **kwargs)
                File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1234, in run
                  return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
                File "/usr/lib/python3.6/site-packages/contextvars/__init__.py", line 38, in run
                  return callable(*args, **kwargs)
                File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1249, in _run_as
                  ret = _func_or_method(*args, **kwargs)
                File "/usr/lib/python3.6/site-packages/salt/modules/zypperpkg.py", line 828, in latest_version
                  package_info = info_available(*names, **kwargs)
                File "/usr/lib/python3.6/site-packages/salt/modules/zypperpkg.py", line 752, in info_available
                  "info", "-t", "package", *batch[:batch_size]
                File "/usr/lib/python3.6/site-packages/salt/modules/zypperpkg.py", line 439, in __call
                  salt.utils.stringutils.to_str(self.__call_result["stdout"])
                File "/usr/lib64/python3.6/xml/dom/minidom.py", line 1968, in parseString
                  return expatbuilder.parseString(string)
                File "/usr/lib64/python3.6/xml/dom/expatbuilder.py", line 925, in parseString
                  return builder.parseString(string)
                File "/usr/lib64/python3.6/xml/dom/expatbuilder.py", line 223, in parseString
                  parser.Parse(string, True)
              xml.parsers.expat.ExpatError: syntax error: line 1, column 0
     Started: 13:14:41.455083
    Duration: 2986.675 ms
     Changes:   
----------
Actions
Copy link
 #27

Updated by mkittler 10 days ago

As discussed on Slack, this is not related to my changes and not reproducible. We also don't know what it tries to parse so there's no much to go with. So I'm not looking into this issue any further unless it happens again.

Actions
Copy link
 #28

Updated by mkittler 10 days ago

  • Status changed from Feedback to Blocked
Actions
Copy link
 #29

Updated by livdywan 9 days ago

  • Due date changed from 2024-12-28 to 2025-01-17

Let's assume a realistic due date with the holidays coming up

Actions
Copy link
 #30

Updated by okurz 1 day ago

  • Due date deleted (2025-01-17)
  • Assignee changed from mkittler to okurz
  • Priority changed from High to Low
  • Target version changed from Ready to future
Actions
Copy link

Also available in: Atom PDF