Project

General

Profile

Actions

action #130594

closed

coordination #130582: [epic] Upgrade all our infrastructure, e.g. o3+osd workers+webui and production workloads, to openSUSE Leap 15.5

Upgrade osd webUI host to openSUSE Leap 15.5

Added by okurz 10 months ago. Updated 6 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Organisational
Target version:
Start date:
Due date:
% Done:

0%

Estimated time:
Tags:

Description

Motivation

  • Need to upgrade machines before EOL of Leap 15.4 and have a consistent environment

Acceptance criteria

  • AC1: osd webui host runs a clean upgraded openSUSE Leap 15.5 (no failed systemd services, no left over .rpm-new files, etc.)

Suggestions

Further details

  • If we loose access to the machine we need the help of EngineeringInfrastructure as only they have access to the VM

Related issues 3 (2 open1 closed)

Related to openQA Infrastructure - action #112850: Make pgstatstatements persistent on OSD (in salt)New

Actions
Copied from openQA Infrastructure - action #111872: Upgrade osd webUI host to openSUSE Leap 15.4Resolvedokurz

Actions
Copied to openQA Project - action #157981: Upgrade osd webUI host to openSUSE Leap 15.6New

Actions
Actions #1

Updated by okurz 10 months ago

  • Copied from action #111872: Upgrade osd webUI host to openSUSE Leap 15.4 added
Actions #2

Updated by okurz 10 months ago

  • Subject changed from Upgrade osd webUI host to openSUSE Leap 15.4 to Upgrade osd webUI host to openSUSE Leap 15.5
  • Category set to Organisational
  • Assignee deleted (okurz)
  • Target version changed from Ready to future
Actions #3

Updated by okurz 10 months ago

  • Description updated (diff)
Actions #4

Updated by okurz 7 months ago

  • Target version changed from future to Tools - Next
Actions #5

Updated by okurz 6 months ago

  • Status changed from New to Blocked
  • Assignee set to okurz
  • Target version changed from Tools - Next to Ready

I suggest to wait for #130648 with referenced MRs first.

Actions #6

Updated by okurz 6 months ago

  • Status changed from Blocked to New
Actions #7

Updated by okurz 6 months ago

  • Status changed from New to In Progress

From a dry-run I see

The following 17 packages are going to be downgraded:
  libpoppler117
    22.01.0-150400.3.11.2 -> 22.01.0-150400.3.3.1                         x86_64  repo-oss                                                      SUSE LLC <https://www.suse.com/>
  libsource-highlight4
    3.1.9-bp154.1.116 -> 3.1.8-150000.3.4.1                               x86_64  repo-oss                                                      SUSE LLC <https://www.suse.com/>
  perl-IO-Compress-Lzma
    2.206.0-lp154.13.1 -> 2.204-lp155.2.1                                 noarch  devel_openQA_Modules                                          obs://build.opensuse.org/devel:openQA
  perl-Minion
    10.25-lp154.2.6 -> 10.25-bp155.1.6                                    noarch  repo-oss                                                      obs://build.opensuse.org/devel:openQA -> openSUSE
  perl-Minion-Backend-SQLite
    5.0.7-lp154.2.3 -> 5.0.7-bp155.1.4                                    noarch  repo-oss                                                      obs://build.opensuse.org/devel:openQA -> openSUSE
  perl-Mojo-Pg
    4.27-lp154.2.20 -> 4.27-bp155.1.5                                     noarch  repo-oss                                                      obs://build.opensuse.org/devel:openQA -> openSUSE
  perl-YAML-PP
    0.036-lp154.2.1 -> 0.035-bp155.1.5                                    noarch  repo-oss                                                      obs://build.opensuse.org/devel:openQA -> openSUSE
  python3-constantly
    15.1.0-150000.3.4.1 -> 15.1.0-3.2.1                                   noarch  repo-oss                                                      SUSE LLC <https://www.suse.com/>
  python3-hyperlink
    17.2.1-150000.3.4.1 -> 17.2.1-3.2.1                                   noarch  repo-oss                                                      SUSE LLC <https://www.suse.com/>
  python3-packaging
    21.3-150200.3.3.1 -> 20.3-1.9                                         noarch  repo-oss                                                      SUSE LLC <https://www.suse.com/>
  python3-zope.interface
    4.4.2-150000.3.4.1 -> 4.4.2-3.2.1                                     x86_64  repo-oss                                                      SUSE LLC <https://www.suse.com/>
  system-user-velociraptor
    1.0.0-lp154.28.6 -> 1.0.0-bp155.26.1                                  x86_64  Server Monitoring Software                                    obs://build.opensuse.org/security
  telegraf
    1.26.3-bp154.2.6.1 -> 1.25.1-bp155.1.21                               x86_64  repo-oss                                                      openSUSE
  velociraptor-client
    0.6.7.5~git81.01be570-lp154.28.7 -> 0.6.7.4~git63.4a1ed09d-bp155.1.7  x86_64  repo-oss                                                      obs://build.opensuse.org/security -> openSUSE
  vim
    9.0.1894-150000.5.54.1 -> 9.0.1632-150500.20.3.1                      x86_64  Update repository with updates from SUSE Linux Enterprise 15  SUSE LLC <https://www.suse.com/>
  vim-data
    9.0.1894-150000.5.54.1 -> 9.0.1632-150500.20.3.1                      noarch  Update repository with updates from SUSE Linux Enterprise 15  SUSE LLC <https://www.suse.com/>
  vim-data-common
    9.0.1894-150000.5.54.1 -> 9.0.1632-150500.20.3.1                      noarch  Update repository with updates from SUSE Linux Enterprise 15  SUSE LLC <https://www.suse.com/>

in particular

  perl-IO-Compress-Lzma
    2.206.0-lp154.13.1 -> 2.204-lp155.2.1                                 noarch  devel_openQA_Modules                                          obs://build.opensuse.org/devel:openQA
…
  perl-YAML-PP
    0.036-lp154.2.1 -> 0.035-bp155.1.5                                    noarch  repo-oss                                                      obs://build.opensuse.org/devel:openQA -> openSUSE
…
  telegraf
    1.26.3-bp154.2.6.1 -> 1.25.1-bp155.1.21                               x86_64  repo-oss                                                      openSUSE

I assume those are ok.

 zypper --releasever=$new_version --gpg-auto-import-keys ref && test -f /etc/openqa/openqa.ini && sudo -u geekotest /opt/openqa-scripts/dump-psql && zypper -n --releasever=$new_version dup --auto-agree-with-licenses --replacefiles --download-in-advance

all done. I scheduled a reboot with rebootmgrctl reboot, not rebooting right now as too many users might actively work with the system.

For PostgreSQL we run v14, new default is 15. Preparing commands from older tickets but just equivalent to https://open.qa/docs/#_migrating_postgresql_database_on_opensuse:

oldver=14 newver=15
zypper in postgresql$newver-server postgresql$newver-contrib
sudo -u postgres /usr/lib/postgresql$newver/bin/initdb --encoding=UTF8 --locale=en_US.UTF-8 --lc-collate=C --lc-ctype=en_US.UTF-8 --lc-messages=C --lc-monetary=C --lc-numeric=C --lc-time=C -D /var/lib/pgsql/data.$newver
sudo -u postgres vimdiff /var/lib/pgsql/data.$oldver/postgresql.conf /var/lib/pgsql/data.$newver/postgresql.conf
sudo -u postgres /usr/lib/postgresql$newver/bin/pg_upgrade --check --link --old-bindir=/usr/lib/postgresql$oldver/bin --new-bindir=/usr/lib/postgresql$newver/bin --old-datadir=/var/lib/pgsql/data.$oldver --new-datadir=/var/lib/pgsql/data.$newver && systemctl stop openqa-webui openqa-scheduler openqa-livehandler openqa-gru postgresql && sudo -u postgres /usr/lib/postgresql$newver/bin/pg_upgrade --link --old-bindir=/usr/lib/postgresql$oldver/bin --new-bindir=/usr/lib/postgresql$newver/bin --old-datadir=/var/lib/pgsql/data.$oldver --new-datadir=/var/lib/pgsql/data.$newver && ln --force --no-dereference --relative --symbolic /var/lib/pgsql/data.$newver /var/lib/pgsql/data && systemctl start postgresql openqa-webui openqa-scheduler openqa-livehandler openqa-gru && sudo -u geekotest psql -c 'select version();' openqa

if all looks good then I should do

rm -rf /var/lib/pgsql/data.$oldver
zypper rm -u postgresql$oldver postgresql$oldver-contrib postgresql$oldver-server
Actions #8

Updated by okurz 6 months ago

  • Status changed from In Progress to Resolved

All done

Actions #9

Updated by okurz 6 months ago

  • Status changed from Resolved to In Progress

I found the webUI to be unavailable shortly after and a systemd failed services alert. postgresql died with

Oct 13 19:31:23 openqa systemd[1]: Started PostgreSQL database server.
Oct 13 20:50:30 openqa systemd[1]: Stopping PostgreSQL database server...
Oct 13 20:50:31 openqa systemd[1]: postgresql.service: Deactivated successfully.
Oct 13 20:50:31 openqa systemd[1]: Stopped PostgreSQL database server.
Oct 13 20:50:31 openqa systemd[1]: Starting PostgreSQL database server...
Oct 13 20:50:31 openqa postgresql-script[13721]: 2023-10-13 20:50:31.336 CEST   [13721]FATAL:  could not access file "pg_stat_statements": No such file or directory
Oct 13 20:50:31 openqa postgresql-script[13721]: 2023-10-13 20:50:31.336 CEST   [13721]LOG:  database system is shut down
Oct 13 20:50:31 openqa postgresql-script[13719]: pg_ctl: could not start server

so I guess salt pipelines caused postgresql to restart and then fail. As workaround I have removed the pg_stat_statements from postgres config and restarted and called systemctl start default.target. All back to normal.

Actions #10

Updated by okurz 6 months ago

  • Related to action #112850: Make pgstatstatements persistent on OSD (in salt) added
Actions #11

Updated by okurz 6 months ago

  • Status changed from In Progress to Resolved

All good again, commented in #112850-8

Actions #12

Updated by okurz 19 days ago

  • Copied to action #157981: Upgrade osd webUI host to openSUSE Leap 15.6 added
Actions

Also available in: Atom PDF