action #159456: [qe-core] implement a test 'report live patching state for openssl' - openQA Tests - openSUSE Project Management Tool

Actions

Copy link

action #159456

open

[qe-core] implement a test 'report live patching state for openssl'

Added by zluo 7 months ago. Updated 6 months ago.

Status:

Feedback

Priority:

Normal

Assignee:

zluo

Category:

Spike/Research

Target version:

QA - QE-Core: Ready

Start date:

Due date:

% Done:

Estimated time:

Difficulty:

Sprint:

QE-Core: May Sprint 24 (May 07 - Jun 04)

Tags:

qe-core-april-sprint, qe-core-may-sprint

Description

see https://jira.suse.com/browse/PED-7092 for requirements and also check https://progress.opensuse.org/issues/124251

find out how to test live patching for openssl
provide information about required setup and steps how to process and verify the test results

Related issues 1 (1 open — 0 closed)

Actions

Copy link

Updated by zluo 7 months ago

Copied from action #124251: [qe-core]Implement userspace livepatching tests for openssl added

Actions

Copy link

Updated by zluo 7 months ago

Description updated (diff)

Actions

Copy link

Updated by zluo 7 months ago

Subject changed from [qe-core] implement a test 'report livepatching state for openssl' to [qe-core] implement a test 'report live patching state for openssl'
Description updated (diff)
Category set to Spike/Research
Assignee set to zluo
Target version set to QE-Core: Ready
Start date deleted (~~2023-02-09~~)

Actions

Copy link

Updated by szarate 7 months ago

Sprint set to QE-Core: April Sprint 24 (Apr 10 - May 08)

Actions

Copy link

Updated by szarate 7 months ago

Tags set to qe-core-april-sprint, qe-core-may-sprint

Actions

Copy link

Updated by zluo 7 months ago · Edited

Bevor start with ulp, read and understand how live patching is working:

Activating Kernel Live Patching from the command line
SUSEConnect -p sle-module-live-patching/15.4/x86_64 -r LIVE_PATCHING_REGISTRATION_CODE
zypper install -t pattern lp_sles

Performing Kernel Live Patching
zypper se --details kernel-livepatch-*
klp -v patches

Checking expiration date of the live patch
Make sure that the lifecycle-data-sle-module-live-patching is installed, then run the zypper lifecycle command
-

then check https://documentation.suse.com/sles/15-SP4/html/SLES-all/cha-ulp.html

run zypper in libpulp0 libpulp-tools
and install openssl-livepatches packages

d4-45:~ # zypper install openssl-1_1-livepatches
Dienst 'Basesystem_Module_15_SP6_x86_64' wird aktualisiert.
Dienst 'Containers_Module_15_SP6_x86_64' wird aktualisiert.
Dienst 'Desktop_Applications_Module_15_SP6_x86_64' wird aktualisiert.
Dienst 'Development_Tools_Module_15_SP6_x86_64' wird aktualisiert.
Dienst 'Legacy_Module_15_SP6_x86_64' wird aktualisiert.
Dienst 'Public_Cloud_Module_15_SP6_x86_64' wird aktualisiert.
Dienst 'Python_3_Module_15_SP6_x86_64' wird aktualisiert.
Dienst 'SAP_Business_One_Module_15_SP6_x86_64' wird aktualisiert.
Dienst 'SUSE_Linux_Enterprise_Live_Patching_15_SP6_x86_64' wird aktualisiert.
Dienst 'SUSE_Linux_Enterprise_Server_15_SP6_x86_64' wird aktualisiert.
Dienst 'SUSE_Package_Hub_15_SP6_x86_64' wird aktualisiert.
Dienst 'Server_Applications_Module_15_SP6_x86_64' wird aktualisiert.
Dienst 'Transactional_Server_Module_15_SP6_x86_64' wird aktualisiert.
Dienst 'Web_and_Scripting_Module_15_SP6_x86_64' wird aktualisiert.
Repository-Daten werden geladen...
Installierte Pakete werden gelesen...
Paketabhängigkeiten werden aufgelöst...

Das folgende NEUE Paket wird installiert:
  openssl-1_1-livepatches

1 neues Paket zu installieren.
Gesamtgröße des Downloads: 24,9 KiB. Bereits im Cache gespeichert: 0 B. Nach der Operation werden zusätzlich 244,3 KiB
belegt.
Continue? [j/n/v/...? zeigt alle Optionen] (j): j
Abrufen: openssl-1_1-livepatches-0.2-150600.9.2.x86_64 (SLE-Module-Live-Patching15-SP6-Pool)        (1/1),  24,9 KiB    
Abrufen: openssl-1_1-livepatches-0.2-150600.9.2.x86_64.rpm .....................................................[fertig]

Überprüfung auf Dateikonflikte läuft: ..........................................................................[fertig]
Executing ulp_post_hook(). About to execute rpm-helper...
openssl-1_1-livepatches
ulp trigger executed.
Done executing rpm-helper.
(1/1) Installieren: openssl-1_1-livepatches-0.2-150600.9.2.x86_64 ..............................................[fertig]

to continue with Using libpulp
To enable live patching on an application, we need to preload the libpulp.so.0 library when starting the application, for example:


LD_PRELOAD=/usr/lib64/libpulp.so.0 openssl

and check:

d4-45:~ # l /usr/lib64/openssl-1_1-livepatches/0.2
insgesamt 276
drwxr-xr-x 1 root root  2028 24. Apr 21:29 ./
drwxr-xr-x 1 root root     6 24. Apr 21:29 ../
-rwxr-xr-x 1 root root 10872  8. Mär 19:42 libcrypto_1.1.1l-150400.5.14.0_livepatch2.so*
-rwxr-xr-x 1 root root 10872  8. Mär 19:42 libcrypto_1.1.1l-150400.7.10.5_livepatch2.so*
-rwxr-xr-x 1 root root 10872  8. Mär 19:42 libcrypto_1.1.1l-150400.7.13.1_livepatch2.so*
-rwxr-xr-x 1 root root 10872  8. Mär 19:42 libcrypto_1.1.1l-150400.7.16.1_livepatch2.so*
-rwxr-xr-x 1 root root 10872  8. Mär 19:42 libcrypto_1.1.1l-150400.7.19.1_livepatch2.so*
-rwxr-xr-x 1 root root 10872  8. Mär 19:42 libcrypto_1.1.1l-150400.7.22.1_livepatch2.so*
-rwxr-xr-x 1 root root 10872  8. Mär 19:42 libcrypto_1.1.1l-150400.7.25.1_livepatch2.so*
-rwxr-xr-x 1 root root 10872  8. Mär 19:42 libcrypto_1.1.1l-150400.7.28.1_livepatch2.so*
-rwxr-xr-x 1 root root 10936  8. Mär 19:42 libcrypto_1.1.1l-150400.7.31.2_livepatch2.so*
-rwxr-xr-x 1 root root 10872  8. Mär 19:42 libcrypto_1.1.1l-150400.7.34.1_livepatch2.so*
-rwxr-xr-x 1 root root 10872  8. Mär 19:42 libcrypto_1.1.1l-150400.7.37.1_livepatch2.so*
-rwxr-xr-x 1 root root 10872  8. Mär 19:42 libcrypto_1.1.1l-150400.7.42.1_livepatch2.so*
-rwxr-xr-x 1 root root 10872  8. Mär 19:42 libcrypto_1.1.1l-150400.7.45.1_livepatch2.so*
-rwxr-xr-x 1 root root 10872  8. Mär 19:42 libcrypto_1.1.1l-150400.7.48.1_livepatch2.so*
-rwxr-xr-x 1 root root 10872  8. Mär 19:42 libcrypto_1.1.1l-150400.7.53.1_livepatch2.so*
-rwxr-xr-x 1 root root 10872  8. Mär 19:42 libcrypto_1.1.1l-150400.7.57.1_livepatch2.so*
-rwxr-xr-x 1 root root 10872  8. Mär 19:42 libcrypto_1.1.1l-150400.7.7.1_livepatch2.so*
-rwxr-xr-x 1 root root 10872  8. Mär 19:42 libcrypto_1.1.1l-150500.15.4.0_livepatch2.so*
-rwxr-xr-x 1 root root 10872  8. Mär 19:42 libcrypto_1.1.1l-150500.17.12.1_livepatch2.so*
-rwxr-xr-x 1 root root 10872  8. Mär 19:42 libcrypto_1.1.1l-150500.17.15.1_livepatch2.so*
-rwxr-xr-x 1 root root 10872  8. Mär 19:42 libcrypto_1.1.1l-150500.17.19.1_livepatch2.so*
-rwxr-xr-x 1 root root 10872  8. Mär 19:42 libcrypto_1.1.1l-150500.17.6.1_livepatch2.so*
-rwxr-xr-x 1 root root 10872  8. Mär 19:42 libcrypto_1.1.1l-150500.17.9.1_livepatch2.so*

find out if a libray is patchable:


ulp livepatchable /usr/lib64/openssl-1_1-livepatches/0.2/libcrypto_1.1.1l-150400.5.14.0_livepatch2.so
ulp: file '/usr/lib64/openssl-1_1-livepatches/0.2/libcrypto_1.1.1l-150400.5.14.0_livepatch2.so' is livepatchable.

Actions

Copy link

Updated by zluo 7 months ago

Checking if a .so file is a live patch container


d4-45:~ # ulp livepatchable /usr/lib64/openssl-1_1-livepatches/0.2/libcrypto_1.1.1l-150400.5.14.0_livepatch2.so
ulp: file '/usr/lib64/openssl-1_1-livepatches/0.2/libcrypto_1.1.1l-150400.5.14.0_livepatch2.so' is livepatchable.
d4-45:~ # readelf -S /usr/lib64/openssl-1_1-livepatches/0.2/libcrypto_1.1.1l-150400.5.14.0_livepatch2.so | grep .ulp
  [25] .ulp              PROGBITS         0000000000000000  00002020
  [26] .ulp.comments     PROGBITS         0000000000000000  00002113
  [27] .ulp.rev          PROGBITS         0000000000000000  00002143

Actions

Copy link

Updated by zluo 7 months ago · Edited

install and configure apache2 with SSL. The reason is that we need a test environment with SSL which is working with apache2 server.

this part is little tricky. In general I had some problems, but now apache2 has been configured with SSL:
Generate a Self-Signed Certificate for apache2:

openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout sles15sp6.key -out sles15sp6.crt
configure a vhost conf and enable ssl for apache2 then and check that ssl is working

https://10.168.194.45/index.html

Actions

Copy link

Updated by zluo 7 months ago · Edited

now we need to add in /usr/lib/systemd/system/apache2.service to inject the LD_PRELOAD=libpulp.so:

Environment="LD_PRELOAD=libpulp.so.0"

run systemctl daemon-reload and systemctl restart apache2.service
check apache2 is working

check libpulp.so is loaded:

d4-45:~ # lsof |grep http.*pulp
lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/1000/gvfs
      Output information may be incomplete.
lsof: WARNING: can't stat() fuse.portal file system /run/user/1000/doc
      Output information may be incomplete.
httpd-pre 27417                      root mem       REG               0,33   2140768     193230 /usr/lib64/libpulp.so.0.0.0
httpd-pre 27432                    wwwrun mem       REG               0,33   2140768     193230 /usr/lib64/libpulp.so.0.0.0
httpd-pre 27433                    wwwrun mem       REG               0,33   2140768     193230 /usr/lib64/libpulp.so.0.0.0
httpd-pre 27434                    wwwrun mem       REG               0,33   2140768     193230 /usr/lib64/libpulp.so.0.0.0
httpd-pre 27435                    wwwrun mem       REG               0,33   2140768     193230 /usr/lib64/libpulp.so.0.0.0
httpd-pre 27436                    wwwrun mem       REG               0,33   2140768     193230 /usr/lib64/libpulp.so.0.0.0
httpd-pre 27439                    wwwrun mem       REG               0,33   2140768     193230 /usr/lib64/libpulp.so.0.0.0

Actions

Copy link

#10

Updated by zluo 7 months ago

finally we can try apply live patches:

d4-45:/usr/lib64/openssl-1_1-livepatches/0.2 # ulp trigger -p 27433 libcrypto_1.1.1l-150400.5.14.0_livepatch2.so 
ulp: Unable to get section data.
ulp: Processes patched: 0, Skipped: 1, Failed: 0.

but this is not working -- need to investigate it

Actions

Copy link

#11

Updated by MDoucha 7 months ago

zluo wrote in #note-9:

now we need to add in /usr/lib/systemd/system/apache2.service to inject the LD_PRELOAD=libpulp.so:
Environment="LD_PRELOAD=libpulp.so.0"

The recommended way to enable libpulp is to install the libpulp-load-default package.

zluo wrote in #note-10:

finally we can try apply live patches:

d4-45:/usr/lib64/openssl-1_1-livepatches/0.2 # ulp trigger -p 27433 libcrypto_1.1.1l-150400.5.14.0_livepatch2.so 
ulp: Unable to get section data.
ulp: Processes patched: 0, Skipped: 1, Failed: 0.

The Apache process probably has a different version of libcrypto.so loaded, or it's not loaded at all possibly due to vhost configuration. Check this using ulp patches -p $apache_pid.

Also check which libraries are targeted by the livepatch: ulp dump libcrypto_1.1.1l-150400.5.14.0_livepatch2.so

Actions

Copy link

#12

Updated by zluo 7 months ago

libpulp-load-default got installed and:


d4-45:/usr/lib64/openssl-1_1-livepatches/0.2 # ulp dump libcrypto_1.1.1l-150400.5.14.0_livepatch2.so
patch id: 8363b71355469b4a4c176d63eeb153749c7429fe
so filename: /usr/lib64/openssl-1_1-livepatches/0.2/libcrypto_1.1.1l-150400.5.14.0_livepatch2.so

* build id: 5ea903d6cb5a6287eb5079545cd7be149e19f5bf
* name: libcrypto.so.1.1
* units: 3

** old_fname: OpenSSL_version
** new_fname: OpenSSL_version_lp
** old_faddr: 0x11913e

** old_fname: DH_check_pub_key
** new_fname: DH_check_pub_key_lp
** old_faddr: 0x122c4e

** old_fname: DH_compute_key
** new_fname: DH_compute_key_lp
** old_faddr: 0x123dbe

static references: 0
* comments: 
 This patch fixes cve-2023-0286 cve-2023-5678
?


d4-45:/usr/lib64/openssl-1_1-livepatches/0.2 # ulp patches -p 27417
PID: 27417, name: httpd-prefork
  Livepatching status: enabled
  Livepatchable libraries:
    in httpd-prefork:
    in /lib64/libutil.so.1:
    in /lib64/libm.so.6:
    in /lib64/libpthread.so.0:
    in /lib64/librt.so.1:
    in /lib64/ld-linux-x86-64.so.2:
    in /lib64/libdl.so.2:
    in /lib64/libc.so.6:
    in /usr/lib64/libpulp.so.0 (version 0.3.1):

It looks now much better :)

Actions

Copy link

#13

Updated by zluo 7 months ago

but ulp trigger to apply patches doesn't work:


d4-45:/usr/lib64/openssl-1_1-livepatches/0.2 # ulp patches
PID: 23312, name: fwupd
  Livepatching status: enabled
  Livepatchable libraries:
    in fwupd:
    in /lib64/libresolv.so.2:
    in /lib64/libm.so.6:
    in /lib64/libpthread.so.0:
    in /lib64/ld-linux-x86-64.so.2:
    in /lib64/libdl.so.2:
    in /lib64/libc.so.6:
    in /usr/lib64/libpulp.so.0 (version 0.3.1):

PID: 25615, name: pickup
  Livepatching status: enabled
  Livepatchable libraries:
    in /lib64/libresolv.so.2:
    in /lib64/libm.so.6:
    in /lib64/libpthread.so.0:
    in /lib64/ld-linux-x86-64.so.2:
    in /lib64/libdl.so.2:
    in /lib64/libc.so.6:
    in /usr/lib64/libpulp.so.0 (version 0.3.1):

PID: 26772, name: sshd
  Livepatching status: enabled
  Livepatchable libraries:
    in /lib64/libnss_compat.so.2:
    in /lib64/libm.so.6:
    in /lib64/libpthread.so.0:
    in /lib64/libresolv.so.2:
    in /lib64/ld-linux-x86-64.so.2:
    in /lib64/libdl.so.2:
    in /lib64/libc.so.6:
    in /usr/lib64/libpulp.so.0 (version 0.3.1):

PID: 26777, name: sshd
  Livepatching status: enabled
  Livepatchable libraries:
    in /lib64/libnss_compat.so.2:
    in /lib64/libm.so.6:
    in /lib64/libpthread.so.0:
    in /lib64/libresolv.so.2:
    in /lib64/ld-linux-x86-64.so.2:
    in /lib64/libdl.so.2:
    in /lib64/libc.so.6:
    in /usr/lib64/libpulp.so.0 (version 0.3.1):

PID: 26788, name: bash
  Livepatching status: enabled
  Livepatchable libraries:
    in bash:
    in /lib64/ld-linux-x86-64.so.2:
    in /lib64/libc.so.6:
    in /lib64/libdl.so.2:
    in /usr/lib64/libpulp.so.0 (version 0.3.1):

PID: 26833, name: su
  Livepatching status: enabled
  Livepatchable libraries:
    in /lib64/libnss_compat.so.2:
    in /lib64/libm.so.6:
    in /lib64/libresolv.so.2:
    in /lib64/libpthread.so.0:
    in /lib64/ld-linux-x86-64.so.2:
    in /lib64/libdl.so.2:
    in /lib64/libc.so.6:
    in /usr/lib64/libpulp.so.0 (version 0.3.1):

PID: 26834, name: bash
  Livepatching status: enabled
  Livepatchable libraries:
    in bash:
    in /lib64/ld-linux-x86-64.so.2:
    in /lib64/libc.so.6:
    in /lib64/libdl.so.2:
    in /usr/lib64/libpulp.so.0 (version 0.3.1):

PID: 27895, name: sshd
  Livepatching status: enabled
  Livepatchable libraries:
    in /lib64/libnss_compat.so.2:
    in /lib64/libm.so.6:
    in /lib64/libpthread.so.0:
    in /lib64/libresolv.so.2:
    in /lib64/ld-linux-x86-64.so.2:
    in /lib64/libdl.so.2:
    in /lib64/libc.so.6:
    in /usr/lib64/libpulp.so.0 (version 0.3.1):

PID: 27899, name: sshd
  Livepatching status: enabled
  Livepatchable libraries:
    in /lib64/libnss_compat.so.2:
    in /lib64/libm.so.6:
    in /lib64/libpthread.so.0:
    in /lib64/libresolv.so.2:
    in /lib64/ld-linux-x86-64.so.2:
    in /lib64/libdl.so.2:
    in /lib64/libc.so.6:
    in /usr/lib64/libpulp.so.0 (version 0.3.1):

PID: 27911, name: bash
  Livepatching status: enabled
  Livepatchable libraries:
    in bash:
    in /lib64/ld-linux-x86-64.so.2:
    in /lib64/libc.so.6:
    in /lib64/libdl.so.2:
    in /usr/lib64/libpulp.so.0 (version 0.3.1):

PID: 27963, name: su
  Livepatching status: enabled
  Livepatchable libraries:
    in /lib64/libnss_compat.so.2:
    in /lib64/libm.so.6:
    in /lib64/libresolv.so.2:
    in /lib64/libpthread.so.0:
    in /lib64/ld-linux-x86-64.so.2:
    in /lib64/libdl.so.2:
    in /lib64/libc.so.6:
    in /usr/lib64/libpulp.so.0 (version 0.3.1):

PID: 27964, name: bash
  Livepatching status: enabled
  Livepatchable libraries:
    in bash:
    in /lib64/ld-linux-x86-64.so.2:
    in /lib64/libc.so.6:
    in /lib64/libdl.so.2:
    in /usr/lib64/libpulp.so.0 (version 0.3.1):

PID: 28117, name: httpd-prefork
  Livepatching status: enabled
  Livepatchable libraries:
    in httpd-prefork:
    in /lib64/libutil.so.1:
    in /lib64/libm.so.6:
    in /lib64/libpthread.so.0:
    in /lib64/librt.so.1:
    in /lib64/ld-linux-x86-64.so.2:
    in /lib64/libdl.so.2:
    in /lib64/libc.so.6:
    in /usr/lib64/libpulp.so.0 (version 0.3.1):

PID: 28132, name: httpd-prefork
  Livepatching status: enabled
  Livepatchable libraries:
    in httpd-prefork:
    in /lib64/libutil.so.1:
    in /lib64/libm.so.6:
    in /lib64/libpthread.so.0:
    in /lib64/librt.so.1:
    in /lib64/ld-linux-x86-64.so.2:
    in /lib64/libdl.so.2:
    in /lib64/libc.so.6:
    in /usr/lib64/libpulp.so.0 (version 0.3.1):

PID: 28133, name: httpd-prefork
  Livepatching status: enabled
  Livepatchable libraries:
    in httpd-prefork:
    in /lib64/libutil.so.1:
    in /lib64/libm.so.6:
    in /lib64/libpthread.so.0:
    in /lib64/librt.so.1:
    in /lib64/ld-linux-x86-64.so.2:
    in /lib64/libdl.so.2:
    in /lib64/libc.so.6:
    in /usr/lib64/libpulp.so.0 (version 0.3.1):

PID: 28134, name: httpd-prefork
  Livepatching status: enabled
  Livepatchable libraries:
    in httpd-prefork:
    in /lib64/libutil.so.1:
    in /lib64/libm.so.6:
    in /lib64/libpthread.so.0:
    in /lib64/librt.so.1:
    in /lib64/ld-linux-x86-64.so.2:
    in /lib64/libdl.so.2:
    in /lib64/libc.so.6:
    in /usr/lib64/libpulp.so.0 (version 0.3.1):

PID: 28135, name: httpd-prefork
  Livepatching status: enabled
  Livepatchable libraries:
    in httpd-prefork:
    in /lib64/libutil.so.1:
    in /lib64/libm.so.6:
    in /lib64/libpthread.so.0:
    in /lib64/librt.so.1:
    in /lib64/ld-linux-x86-64.so.2:
    in /lib64/libdl.so.2:
    in /lib64/libc.so.6:
    in /usr/lib64/libpulp.so.0 (version 0.3.1):

PID: 28136, name: httpd-prefork
  Livepatching status: enabled
  Livepatchable libraries:
    in httpd-prefork:
    in /lib64/libutil.so.1:
    in /lib64/libm.so.6:
    in /lib64/libpthread.so.0:
    in /lib64/librt.so.1:
    in /lib64/ld-linux-x86-64.so.2:
    in /lib64/libdl.so.2:
    in /lib64/libc.so.6:
    in /usr/lib64/libpulp.so.0 (version 0.3.1):

d4-45:/usr/lib64/openssl-1_1-livepatches/0.2 # ulp trigger -p 28136 libcrypto_1.1.1l-150400.5.14.0_livepatch2.so
ulp: Unable to get section data.
ulp: Processes patched: 0, Skipped: 1, Failed: 0.

Actions

Copy link

#14

Updated by zluo 7 months ago · Edited

d4-45:~ # lsof |grep http.*libcrypto.so
lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/1000/gvfs
      Output information may be incomplete.
lsof: WARNING: can't stat() fuse.portal file system /run/user/1000/doc
      Output information may be incomplete.
httpd-pre 28117                       root mem       REG               0,33   5133344      19546 /usr/lib64/libcrypto.so.3.1.4
httpd-pre 28132                     wwwrun mem       REG               0,33   5133344      19546 /usr/lib64/libcrypto.so.3.1.4
httpd-pre 28133                     wwwrun mem       REG               0,33   5133344      19546 /usr/lib64/libcrypto.so.3.1.4
httpd-pre 28134                     wwwrun mem       REG               0,33   5133344      19546 /usr/lib64/libcrypto.so.3.1.4
httpd-pre 28135                     wwwrun mem       REG               0,33   5133344      19546 /usr/lib64/libcrypto.so.3.1.4
httpd-pre 28136                     wwwrun mem       REG               0,33   5133344      19546 /usr/lib64/libcrypto.so.3.1.4

libcryto.so.3.1.4 has been actually loaded

but checked latest patch and it explains why this is not working (assume that older version cannot be applied to newer):


d4-45:/usr/lib64/openssl-1_1-livepatches/0.2 # ulp dump libcrypto_1.1.1l-150500.17.9.1_livepatch2.so
patch id: 8363b71355469b4a4c176d63eeb153745f798551
so filename: /usr/lib64/openssl-1_1-livepatches/0.2/libcrypto_1.1.1l-150500.17.9.1_livepatch2.so

* build id: 0cd6545dbbea3aaa2d97ed69b6befff4b621a26b
* name: libcrypto.so.1.1
* units: 3

** old_fname: OpenSSL_version
** new_fname: OpenSSL_version_lp
** old_faddr: 0x11a41e

** old_fname: DH_check_pub_key
** new_fname: DH_check_pub_key_lp
** old_faddr: 0x123f7e

** old_fname: DH_compute_key
** new_fname: DH_compute_key_lp
** old_faddr: 0x1250ee

static references: 0
* comments: 
 This patch fixes cve-2023-0286 cve-2023-5678
?

Actions

Copy link

#15

Updated by zluo 7 months ago

opened: https://bugzilla.suse.com/show_bug.cgi?id=1223412

will open a bug report for documentation about enabling libpulp by installing libpulp-load-default package.

Actions

Copy link

#16

Updated by MDoucha 7 months ago

zluo wrote in #note-14:

libcryto.so.3.1.4 has been actually loaded

but checked latest patch and it explains why this is not working (assume that older version cannot be applied to newer):

Yes, you need to downgrade libcrypto to the version included in the livepatch filename: e.g. openssl-1.1.1l-150400.7.28.1
Then restart apache and apply livepatch. Each livepatch.so targets a different openssl version so you should install and test all of them, one after another.

Actions

Copy link

#17

Updated by zluo 7 months ago

opened: https://bugzilla.suse.com/show_bug.cgi?id=1223412

will open a bug report for documentation about enabling libpulp by installing libpulp-load-default package.

MDoucha wrote in #note-16:

zluo wrote in #note-14:

libcryto.so.3.1.4 has been actually loaded

but checked latest patch and it explains why this is not working (assume that older version cannot be applied to newer):

Yes, you need to downgrade libcrypto to the version included in the livepatch filename: e.g. openssl-1.1.1l-150400.7.28.1
Then restart apache and apply livepatch. Each livepatch.so targets a different openssl version so you should install and test all of them, one after another.

Does it make sense to get older libcrypto downgraded on sles15 sp6? I mean all patches of libcrypto.so.1.* and it maybe has target for sles 15 sp5?

Actions

Copy link

#18

Updated by MDoucha 7 months ago

zluo wrote in #note-17:

Does it make sense to get older libcrypto downgraded on sles15 sp6? I mean all patches of libcrypto.so.1.* and it maybe has target for sles 15 sp5?

It will make sense once SLE-15SP6 starts receiving updates, because openssl RPMs will be supported and livepatched for 12 months after release. It doesn't make sense now because there's nothing to downgrade to, except maybe for switching between openssl1_1 and openssl3 (which IIUC should not be interchangeable as a dependency for Apache and other packages).

The openssl-1_1-livepatches package in SLE-15SP6 pool definitely targets only SLE-15SP4 and SLE-15SP5 openssl so it was likely added to SLE-15SP6 repos by mistake. There are only two openssl packages on SLE-15SP6 right now and neither is targeted by the livepatches:
libopenssl1_1-1.1.1w-150600.2.10
libopenssl3-3.1.4-150600.2.17

Actions

Copy link

#19

Updated by zluo 7 months ago

MDoucha wrote in #note-18:

zluo wrote in #note-17:

Does it make sense to get older libcrypto downgraded on sles15 sp6? I mean all patches of libcrypto.so.1.* and it maybe has target for sles 15 sp5?

It will make sense once SLE-15SP6 starts receiving updates, because openssl RPMs will be supported and livepatched for 12 months after release. It doesn't make sense now because there's nothing to downgrade to, except maybe for switching between openssl1_1 and openssl3 (which IIUC should not be interchangeable as a dependency for Apache and other packages).

The openssl-1_1-livepatches package in SLE-15SP6 pool definitely targets only SLE-15SP4 and SLE-15SP5 openssl so it was likely added to SLE-15SP6 repos by mistake. There are only two openssl packages on SLE-15SP6 right now and neither is targeted by the livepatches:
libopenssl1_1-1.1.1w-150600.2.10
libopenssl3-3.1.4-150600.2.17

okay, thanks for the explanations!
So my bug report is valid. Of course I can check this on sles 15 sp5 for the workflow and learning purpose :)

Actions

Copy link

#20

Updated by zluo 7 months ago

for update documentation: https://bugzilla.suse.com/show_bug.cgi?id=1223429

Actions

Copy link

#21

Updated by zluo 7 months ago

I installed sles 15 sp5 and configured apache2 with SSL support and installed all required packages:


d5-201:/usr/lib64/openssl-1_1-livepatches/0.1 # lsof |grep http.*pulp
lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/1000/gvfs
      Output information may be incomplete.
httpd-pre 31678                      root  mem       REG               0,48     39424     132611 /usr/lib64/libpulp.so.0.0.0
httpd-pre 31684                    wwwrun  mem       REG               0,48     39424     132611 /usr/lib64/libpulp.so.0.0.0
httpd-pre 31685                    wwwrun  mem       REG               0,48     39424     132611 /usr/lib64/libpulp.so.0.0.0
httpd-pre 31686                    wwwrun  mem       REG               0,48     39424     132611 /usr/lib64/libpulp.so.0.0.0
httpd-pre 31687                    wwwrun  mem       REG               0,48     39424     132611 /usr/lib64/libpulp.so.0.0.0
httpd-pre 31688                    wwwrun  mem       REG               0,48     39424     132611 /usr/lib64/libpulp.so.0.0.0


d5-201:/usr/lib64/openssl-1_1-livepatches/0.1 # lsof |grep http.*cryp
lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/1000/gvfs
      Output information may be incomplete.
httpd-pre 31678                      root  mem       REG               0,48   3389800      20263 /usr/lib64/libcrypto.so.1.1
httpd-pre 31678                      root  mem       REG               0,48    202744       9525 /usr/lib64/libcrypt.so.1.1.0
httpd-pre 31678                      root  mem       REG               0,48   1296440      10141 /usr/lib64/libgcrypt.so.20.3.4
httpd-pre 31684                    wwwrun  mem       REG               0,48   3389800      20263 /usr/lib64/libcrypto.so.1.1
httpd-pre 31684                    wwwrun  mem       REG               0,48    202744       9525 /usr/lib64/libcrypt.so.1.1.0
httpd-pre 31684                    wwwrun  mem       REG               0,48   1296440      10141 /usr/lib64/libgcrypt.so.20.3.4
httpd-pre 31685                    wwwrun  mem       REG               0,48   3389800      20263 /usr/lib64/libcrypto.so.1.1
httpd-pre 31685                    wwwrun  mem       REG               0,48    202744       9525 /usr/lib64/libcrypt.so.1.1.0
httpd-pre 31685                    wwwrun  mem       REG               0,48   1296440      10141 /usr/lib64/libgcrypt.so.20.3.4
httpd-pre 31686                    wwwrun  mem       REG               0,48   3389800      20263 /usr/lib64/libcrypto.so.1.1
httpd-pre 31686                    wwwrun  mem       REG               0,48    202744       9525 /usr/lib64/libcrypt.so.1.1.0
httpd-pre 31686                    wwwrun  mem       REG               0,48   1296440      10141 /usr/lib64/libgcrypt.so.20.3.4
httpd-pre 31687                    wwwrun  mem       REG               0,48   3389800      20263 /usr/lib64/libcrypto.so.1.1
httpd-pre 31687                    wwwrun  mem       REG               0,48    202744       9525 /usr/lib64/libcrypt.so.1.1.0
httpd-pre 31687                    wwwrun  mem       REG               0,48   1296440      10141 /usr/lib64/libgcrypt.so.20.3.4
httpd-pre 31688                    wwwrun  mem       REG               0,48   3389800      20263 /usr/lib64/libcrypto.so.1.1
httpd-pre 31688                    wwwrun  mem       REG               0,48    202744       9525 /usr/lib64/libcrypt.so.1.1.0
httpd-pre 31688                    wwwrun  mem       REG               0,48   1296440      10141 /usr/lib64/libgcrypt.so.20.3.4

--

d5-201:/usr/lib64/openssl-1_1-livepatches/0.1 # ulp trigger -p 31678 libcrypto_1.1.1l-150400.5.14.0_livepatch1.so
error: could not apply libcrypto_1.1.1l-150400.5.14.0_livepatch1.so to httpd-prefork (pid 31678): Build ID mismatch
note: run `ulp patches -b` to retrieve all build ids from patchable processes.
d5-201:/usr/lib64/openssl-1_1-livepatches/0.1 # ulp patches -b
PID: 30375, name: pickup
  Livepatchable libraries:
    in /usr/lib64/libcrypto.so.1.1 (a913351de6e5d8dd4830dce49ff9c17138855490):
    in /lib64/libresolv.so.2 (e42810d28240c9a071d143ac34efc1db577e5bfa):
    in /lib64/libm.so.6 (02848bab8c741aab67ab26460506dc26bb93cc6b):
    in /lib64/libpthread.so.0 (83611af746ef9652b7ecae972f50bec540256db0):
    in /lib64/ld-linux-x86-64.so.2 (306fa1f1f4692920c5a650484a28bc6ccdc99902):
    in /lib64/libdl.so.2 (3a7f65fd4552d07229d8985f6b5e20cae5016274):
    in /lib64/libc.so.6 (171a59c1c43a8f7b93c3dff765aae0b675fe10f6):
    in /usr/lib64/libpulp.so.0 (d35316d1f0dc4a95f77af4eb140a9950b0dd8a4f):

PID: 31678, name: httpd-prefork
  Livepatchable libraries:
    in httpd-prefork (9e38cbfaca060b6a37327aff830705bd95d4538b):
    in /usr/lib64/libcrypto.so.1.1 (a913351de6e5d8dd4830dce49ff9c17138855490):
    in /usr/lib64/libssl.so.1.1 (916c9760382644b0df996da763a64ac5fdc441dc):
    in /lib64/librt.so.1 (928a20e94e2b575919ada526ac5d5b5153aa4d3f):
    in /lib64/ld-linux-x86-64.so.2 (306fa1f1f4692920c5a650484a28bc6ccdc99902):
    in /lib64/libdl.so.2 (3a7f65fd4552d07229d8985f6b5e20cae5016274):
    in /lib64/libc.so.6 (171a59c1c43a8f7b93c3dff765aae0b675fe10f6):
    in /lib64/libpthread.so.0 (83611af746ef9652b7ecae972f50bec540256db0):
    in /usr/lib64/libpulp.so.0 (d35316d1f0dc4a95f77af4eb140a9950b0dd8a4f):

PID: 31684, name: httpd-prefork
  Livepatchable libraries:
    in httpd-prefork (9e38cbfaca060b6a37327aff830705bd95d4538b):
    in /usr/lib64/libcrypto.so.1.1 (a913351de6e5d8dd4830dce49ff9c17138855490):
    in /usr/lib64/libssl.so.1.1 (916c9760382644b0df996da763a64ac5fdc441dc):
    in /lib64/librt.so.1 (928a20e94e2b575919ada526ac5d5b5153aa4d3f):
    in /lib64/ld-linux-x86-64.so.2 (306fa1f1f4692920c5a650484a28bc6ccdc99902):
    in /lib64/libdl.so.2 (3a7f65fd4552d07229d8985f6b5e20cae5016274):
    in /lib64/libc.so.6 (171a59c1c43a8f7b93c3dff765aae0b675fe10f6):
    in /lib64/libpthread.so.0 (83611af746ef9652b7ecae972f50bec540256db0):
    in /usr/lib64/libpulp.so.0 (d35316d1f0dc4a95f77af4eb140a9950b0dd8a4f):

PID: 31685, name: httpd-prefork
  Livepatchable libraries:
    in httpd-prefork (9e38cbfaca060b6a37327aff830705bd95d4538b):
    in /usr/lib64/libcrypto.so.1.1 (a913351de6e5d8dd4830dce49ff9c17138855490):
    in /usr/lib64/libssl.so.1.1 (916c9760382644b0df996da763a64ac5fdc441dc):
    in /lib64/librt.so.1 (928a20e94e2b575919ada526ac5d5b5153aa4d3f):
    in /lib64/ld-linux-x86-64.so.2 (306fa1f1f4692920c5a650484a28bc6ccdc99902):
    in /lib64/libdl.so.2 (3a7f65fd4552d07229d8985f6b5e20cae5016274):
    in /lib64/libc.so.6 (171a59c1c43a8f7b93c3dff765aae0b675fe10f6):
    in /lib64/libpthread.so.0 (83611af746ef9652b7ecae972f50bec540256db0):
    in /usr/lib64/libpulp.so.0 (d35316d1f0dc4a95f77af4eb140a9950b0dd8a4f):

PID: 31686, name: httpd-prefork
  Livepatchable libraries:
    in httpd-prefork (9e38cbfaca060b6a37327aff830705bd95d4538b):
    in /usr/lib64/libcrypto.so.1.1 (a913351de6e5d8dd4830dce49ff9c17138855490):
    in /usr/lib64/libssl.so.1.1 (916c9760382644b0df996da763a64ac5fdc441dc):
    in /lib64/librt.so.1 (928a20e94e2b575919ada526ac5d5b5153aa4d3f):
    in /lib64/ld-linux-x86-64.so.2 (306fa1f1f4692920c5a650484a28bc6ccdc99902):
    in /lib64/libdl.so.2 (3a7f65fd4552d07229d8985f6b5e20cae5016274):
    in /lib64/libc.so.6 (171a59c1c43a8f7b93c3dff765aae0b675fe10f6):
    in /lib64/libpthread.so.0 (83611af746ef9652b7ecae972f50bec540256db0):
    in /usr/lib64/libpulp.so.0 (d35316d1f0dc4a95f77af4eb140a9950b0dd8a4f):

PID: 31687, name: httpd-prefork
  Livepatchable libraries:
    in httpd-prefork (9e38cbfaca060b6a37327aff830705bd95d4538b):
    in /usr/lib64/libcrypto.so.1.1 (a913351de6e5d8dd4830dce49ff9c17138855490):
    in /usr/lib64/libssl.so.1.1 (916c9760382644b0df996da763a64ac5fdc441dc):
    in /lib64/librt.so.1 (928a20e94e2b575919ada526ac5d5b5153aa4d3f):
    in /lib64/ld-linux-x86-64.so.2 (306fa1f1f4692920c5a650484a28bc6ccdc99902):
    in /lib64/libdl.so.2 (3a7f65fd4552d07229d8985f6b5e20cae5016274):
    in /lib64/libc.so.6 (171a59c1c43a8f7b93c3dff765aae0b675fe10f6):
    in /lib64/libpthread.so.0 (83611af746ef9652b7ecae972f50bec540256db0):
    in /usr/lib64/libpulp.so.0 (d35316d1f0dc4a95f77af4eb140a9950b0dd8a4f):

PID: 31688, name: httpd-prefork
  Livepatchable libraries:
    in httpd-prefork (9e38cbfaca060b6a37327aff830705bd95d4538b):
    in /usr/lib64/libcrypto.so.1.1 (a913351de6e5d8dd4830dce49ff9c17138855490):
    in /usr/lib64/libssl.so.1.1 (916c9760382644b0df996da763a64ac5fdc441dc):
    in /lib64/librt.so.1 (928a20e94e2b575919ada526ac5d5b5153aa4d3f):
    in /lib64/ld-linux-x86-64.so.2 (306fa1f1f4692920c5a650484a28bc6ccdc99902):
    in /lib64/libdl.so.2 (3a7f65fd4552d07229d8985f6b5e20cae5016274):
    in /lib64/libc.so.6 (171a59c1c43a8f7b93c3dff765aae0b675fe10f6):
    in /lib64/libpthread.so.0 (83611af746ef9652b7ecae972f50bec540256db0):
    in /usr/lib64/libpulp.so.0 (d35316d1f0dc4a95f77af4eb140a9950b0dd8a4f):

I have problem with live patches: Build ID mismatch

Actions

Copy link

#22

Updated by MDoucha 7 months ago

zluo wrote in #note-21:

I have problem with live patches: Build ID mismatch

Which libopenssl version is installed? The openssl-1_1-livepatches package targets only these versions on SLE-15SP5:

libopenssl1_1-1.1.1l-150500.15.4
libopenssl1_1-1.1.1l-150500.17.12.1
libopenssl1_1-1.1.1l-150500.17.15.1
libopenssl1_1-1.1.1l-150500.17.19.1
libopenssl1_1-1.1.1l-150500.17.6.1
libopenssl1_1-1.1.1l-150500.17.9.1

Applying the livepatch .so file to the wrong libopenssl version will result in build ID mismatch error.

Actions

Copy link

#23

Updated by zluo 7 months ago · Edited

MDoucha wrote in #note-22:

zluo wrote in #note-21:

I have problem with live patches: Build ID mismatch

Which libopenssl version is installed? The openssl-1_1-livepatches package targets only these versions on SLE-15SP5:

libopenssl1_1-1.1.1l-150500.15.4

libopenssl1_1-1.1.1l-150500.17.12.1

libopenssl1_1-1.1.1l-150500.17.15.1

libopenssl1_1-1.1.1l-150500.17.19.1

libopenssl1_1-1.1.1l-150500.17.6.1

libopenssl1_1-1.1.1l-150500.17.9.1

Applying the livepatch .so file to the wrong libopenssl version will result in build ID mismatch error.

well, I have installed following:

d5-201:/home/zaoliang # zypper info openssl-1_1
Repository-Daten werden geladen...
Installierte Pakete werden gelesen...


Informationen zu Paket openssl-1_1:
-----------------------------------
Repository         : sle-module-basesystem
Name               : openssl-1_1
Version            : 1.1.1l-150500.15.4
Arch               : x86_64
Anbieter           : SUSE LLC <https://www.suse.com/>
Support Level      : Stufe 3
Installierte Größe : 1,6 MiB
Installiert        : Ja (automatisch)
Status             : aktuell
Quellpaket         : openssl-1_1-1.1.1l-150500.15.4.src
Upstream-URL       : https://www.openssl.org/
Zusammenfassung    : Secure Sockets and Transport Layer Security
Beschreibung       : 
    OpenSSL is a software library to be used in applications that need to
    secure communications over computer networks against eavesdropping or
    need to ascertain the identity of the party at the other end.
    OpenSSL contains an implementation of the SSL and TLS protocols.

d5-201:/home/zaoliang # zypper info openssl-1_1-livepatches
Repository-Daten werden geladen...
Installierte Pakete werden gelesen...


Informationen zu Paket openssl-1_1-livepatches:
-----------------------------------------------
Repository         : sle-module-live-patching
Name               : openssl-1_1-livepatches
Version            : 0.1-150400.3.3.1
Arch               : x86_64
Anbieter           : SUSE LLC <https://www.suse.com/>
Support Level      : Stufe 3
Installierte Größe : 33,0 KiB
Installiert        : Ja
Status             : aktuell
Quellpaket         : openssl-1_1-livepatches-0.1-150400.3.3.1.src
Upstream-URL       : https://www.suse.com/products/live-patching
Zusammenfassung    : Livepatches for OpenSSL
Beschreibung       : 
    Live patching enables userland processes to be fixed without a restart cycle.
    This package provides live patches for the libraries provided by openssl.
    Applying a live patch requires libpulp-tools.

Actions

Copy link

#24

Updated by MDoucha 7 months ago

zluo wrote in #note-23:

well, I have installed following:

Informationen zu Paket openssl-1_1:
-----------------------------------
Repository         : sle-module-basesystem
Name               : openssl-1_1
Version            : 1.1.1l-150500.15.4

Then ulp trigger libcrypto_1.1.1l-150500.15.4.0_livepatch2.so is the only command that will work.

You've tried ulp trigger -p 31678 libcrypto_1.1.1l-150400.5.14.0_livepatch1.so which is the wrong version for another SLES release.

Actions

Copy link

#25

Updated by zluo 7 months ago

MDoucha wrote in #note-24:

zluo wrote in #note-23:
well, I have installed following:

--
Informationen zu Paket openssl-1_1:
-----------------------------------
Repository         : sle-module-basesystem
Name               : openssl-1_1
Version            : 1.1.1l-150500.15.4
Then ulp trigger libcrypto_1.1.1l-150500.15.4.0_livepatch2.so is the only command that will work.

You've tried ulp trigger -p 31678 libcrypto_1.1.1l-150400.5.14.0_livepatch1.so which is the wrong version for another SLES release.

yes, thanks!

after I zypper up livepatches, it works!

d5-201:/usr/lib64/openssl-1_1-livepatches/0.2 # l libcrypto_1.1.1l-150500.15.4.0_livepatch2.so
-rwxr-xr-x 1 root root 10872 20. Nov 11:30 libcrypto_1.1.1l-150500.15.4.0_livepatch2.so*
d5-201:/usr/lib64/openssl-1_1-livepatches/0.2 # ulp trigger -p 31678 libcrypto_1.1.1l-150500.15.4.0_livepatch2.so
ulp: Unable to get section data.
httpd-prefork (pid: 31678): SKIPPED Patch already applied
ulp: Processes patched: 0, Skipped: 1, Failed: 0.

Actions

Copy link

#26

Updated by zluo 7 months ago

Status changed from New to In Progress

Actions

Copy link

#27

Updated by szarate 6 months ago

Sprint changed from QE-Core: April Sprint 24 (Apr 10 - May 08) to QE-Core: May Sprint 25 (May 07 - Jun 04)

Actions

Copy link

#28

Updated by szarate 6 months ago

Status changed from In Progress to Feedback

Actions

Copy link

Also available in: Atom PDF

Project

General

Profile

QA » openQA Project » openQA Tests

Tags

Custom queries

action #159456

[qe-core] implement a test 'report live patching state for openssl'

Updated by zluo 7 months ago

Updated by zluo 7 months ago

Updated by zluo 7 months ago

Updated by szarate 7 months ago

Updated by szarate 7 months ago

Updated by zluo 7 months ago · Edited

Updated by zluo 7 months ago

Updated by zluo 7 months ago · Edited

Updated by zluo 7 months ago · Edited

Updated by zluo 7 months ago

Updated by MDoucha 7 months ago

Updated by zluo 7 months ago

Updated by zluo 7 months ago

Updated by zluo 7 months ago · Edited

Updated by zluo 7 months ago

Updated by MDoucha 7 months ago

Updated by zluo 7 months ago

Updated by MDoucha 7 months ago

Updated by zluo 7 months ago

Updated by zluo 7 months ago

Updated by zluo 7 months ago

Updated by MDoucha 7 months ago

Updated by zluo 7 months ago · Edited

Updated by MDoucha 7 months ago

Updated by zluo 7 months ago

Updated by zluo 7 months ago

Updated by szarate 6 months ago

Updated by szarate 6 months ago