Project

General

Profile

Actions

action #155179

closed

coordination #155182: [epic] Participate in alpha-testing of new version of velociraptor-client

Participate in alpha-testing of new version of velociraptor-client

Added by okurz 3 months ago. Updated 12 days ago.

Status:
Resolved
Priority:
Low
Assignee:
Target version:
Start date:
2024-02-08
Due date:
% Done:

0%

Estimated time:

Description

Motivation

rsimai approached me asking if LSG QE can participate in alpha-testing a new version of velociraptor client. Given that we have the deployment maintained in salt this should be low effort for us. So we would change the repo that we install within our OSD infrastructure, let it be deployed and if there are any problems, e.g. observed from our monitoring, then we will report them.

Rollback steps

  • DONE ssh grenache-1.oqa.prg2.suse.org 'systemctl unmask --now openqa-worker-auto-restart@7'
  • DONE ssh openqaworker-arm-1.qe.nue2.suse.org 'sudo userdel --remove sensor-test'

Related issues 3 (1 open2 closed)

Related to openQA Project - action #130648: Upgrade all other LSG QE salt controlled machines to openSUSE Leap 15.5Resolvedokurz2023-06-09

Actions
Related to QA - action #157753: Bring back automatic recovery for openqaworker-arm-1 size:MResolvedybonatakis

Actions
Copied to QA - action #159060: Rollback/switch to officially installed velociraptor-client repo and server size:SFeedbackokurz2024-02-082024-05-05

Actions
Actions #2

Updated by okurz 3 months ago

  • Parent task set to #155182
Actions #3

Updated by okurz 3 months ago

  • Status changed from New to Feedback
  • Priority changed from Normal to Low

For now I will just need to wait until rsimai comes back to us with more specific information.

Actions #4

Updated by okurz about 2 months ago

  • Related to action #130648: Upgrade all other LSG QE salt controlled machines to openSUSE Leap 15.5 added
Actions #5

Updated by okurz about 2 months ago

  • Status changed from Feedback to Blocked
  • Target version changed from Tools - Next to Ready

There was some movement and I was looking into which package versions are installed on our systems. I realized that we have one system still on Leap 15.4, osiris, upgrading now. Done. Inconsistencies come from #130648 when I removed the security:sensor repo to not be added anymore on more recent Leap 15.5 but we still kept the repo for newer versions.

Blocking on https://suse.slack.com/archives/C02NJAA1PEC/p1709382114636189?thread_ts=1709283079.201359&cid=C02NJAA1PEC

I realized that https://build.opensuse.org/project/repository_state/security:sensor/15.5 has only x86_64 enabled. Assuming that you would like us to test packages from that project can those architectures be enabled?

Actions #6

Updated by okurz about 2 months ago

  • Status changed from Blocked to In Progress

https://download.opensuse.org/repositories/security:/sensor/15.5/ now has all four main architectures enabled but seemingly no s390x builds. Anyway, will enable that for testing now.

Actions #7

Updated by okurz about 2 months ago

https://gitlab.suse.de/openqa/salt-pillars-openqa/-/merge_requests/736 to use the new test server directly.

https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/1120 to use the new package version.

Full system access blocked on https://sd.suse.com/servicedesk/customer/portal/1/SD-149901 but we can try with the machines that can already access.

Merged both MRs, monitoring.

Actions #8

Updated by livdywan about 2 months ago

Full system access blocked on https://sd.suse.com/servicedesk/customer/portal/1/SD-149901 but we can try with the machines that can already access.

Please remember to share the ticket with OSD Admins.

Actions #9

Updated by okurz about 2 months ago ยท Edited

  • Due date set to 2024-03-20
  • Status changed from In Progress to Feedback

Follow-up MRs merged because I messed up.

sudo salt \* cmd.run 'zypper -n in --allow-vendor-change velociraptor-client'

to force the update. All hosts within the LSG QE salt managed infrastructure except s390x and openqa-piworker now run velociraptor-client-0.7.0.4.git47.0f8a4de1-lp155.38.1 including the updated config pointing to https://10.156.231.185:8000/. I will monitor if there is any considerable performance impact on our infrastructure.

Actions #10

Updated by okurz about 2 months ago

  • Target version changed from Ready to Tools - Next
Actions #11

Updated by okurz about 2 months ago

  • Due date deleted (2024-03-20)

Evaluating our monitoring data from https://monitor.qa.suse.de/ on a sample size of 10 machines I could not find any significant impact on performance. So until further notice I would continue to run the development version on our hosts.

Actions #12

Updated by okurz about 1 month ago

  • Description updated (diff)

I was asked by email if we can help to test on ppc64le. For this I did ssh grenache-1.oqa.prg2.suse.org 'systemctl mask --now openqa-worker-auto-restart@7' which is for grenache-8 and on grenache (novalink) did pvmctl lpar power-on -i name=grenache-8

I answered Marcela Maslanova by email and stated that they can use grenache-8 for days/weeks and should let us know if we can return the machine back to the openQA testing pool. Added according rollback step.

Actions #13

Updated by okurz about 1 month ago

  • Description updated (diff)

As requested by email I now did

sudo salt -C 'G@osarch:s390x' cmd.run 'zypper ar -f -p 85 https://download.opensuse.org/repositories/security:/sensor/SLE_15_SP5/security:sensor.repo && zypper -n in velociraptor-client-0.7.0.4.git47.0f8a4de1-150500.38.2.s390x'

as https://download.opensuse.org/repositories/security:/sensor/15.5/ does not supply s390x although it would be preferred to use that repo instead and stay consistent.

The output in the system journal was a very verbose log dump ending with

Mar 26 21:36:08 s390zl12 velociraptor-client[83250]: libbpf: prog 'trace_vfs_ioctl': failed to load: -22
Mar 26 21:36:08 s390zl12 velociraptor-client[83250]: libbpf: failed to load object 'chattrsnoop

Complete journal of the old and new version:

http://w3.nue.suse.com/~okurz/velociraptor-client-log-s390zl12-2024-03-26T21:38:18+01:00.log

sent by email

Actions #14

Updated by okurz about 1 month ago

  • Related to action #157753: Bring back automatic recovery for openqaworker-arm-1 size:M added
Actions #15

Updated by okurz about 1 month ago

  • Description updated (diff)

I was asked about ARM testing ressources as well. For this I created an account "sensor-test" on openqaworker-arm-1 and I will add SSH keys when I get them for access.

Actions #16

Updated by okurz 19 days ago

openqaworker-arm-1 is not needed (anymore) by the security sensor development team.

Actions #17

Updated by okurz 18 days ago

  • Description updated (diff)
Actions #18

Updated by okurz 12 days ago

  • Description updated (diff)
  • Status changed from Feedback to Resolved
  • Target version changed from Tools - Next to Ready

rollback task done, checked successful start of openQA jobs on the formerly disabled grenache openQA worker instance, task resolved.

Actions #19

Updated by okurz 12 days ago

  • Copied to action #159060: Rollback/switch to officially installed velociraptor-client repo and server size:S added
Actions

Also available in: Atom PDF