action #130594
closedcoordination #130582: [epic] Upgrade all our infrastructure, e.g. o3+osd workers+webui and production workloads, to openSUSE Leap 15.5
Upgrade osd webUI host to openSUSE Leap 15.5
0%
Description
Motivation¶
- Need to upgrade machines before EOL of Leap 15.4 and have a consistent environment
Acceptance criteria¶
- AC1: osd webui host runs a clean upgraded openSUSE Leap 15.5 (no failed systemd services, no left over .rpm-new files, etc.)
Suggestions¶
- read https://progress.opensuse.org/projects/openqav3/wiki#Distribution-upgrades
- Reserve some time when the instance is only executing a few or no openQA test jobs
- After upgrade reboot and check everything working as expected
Further details¶
- If we loose access to the machine we need the help of EngineeringInfrastructure as only they have access to the VM
Updated by okurz over 1 year ago
- Copied from action #111872: Upgrade osd webUI host to openSUSE Leap 15.4 added
Updated by okurz over 1 year ago
- Subject changed from Upgrade osd webUI host to openSUSE Leap 15.4 to Upgrade osd webUI host to openSUSE Leap 15.5
- Category set to Organisational
- Assignee deleted (
okurz) - Target version changed from Ready to future
Updated by okurz about 1 year ago
- Target version changed from future to Tools - Next
Updated by okurz 12 months ago
- Status changed from New to In Progress
From a dry-run I see
The following 17 packages are going to be downgraded:
libpoppler117
22.01.0-150400.3.11.2 -> 22.01.0-150400.3.3.1 x86_64 repo-oss SUSE LLC <https://www.suse.com/>
libsource-highlight4
3.1.9-bp154.1.116 -> 3.1.8-150000.3.4.1 x86_64 repo-oss SUSE LLC <https://www.suse.com/>
perl-IO-Compress-Lzma
2.206.0-lp154.13.1 -> 2.204-lp155.2.1 noarch devel_openQA_Modules obs://build.opensuse.org/devel:openQA
perl-Minion
10.25-lp154.2.6 -> 10.25-bp155.1.6 noarch repo-oss obs://build.opensuse.org/devel:openQA -> openSUSE
perl-Minion-Backend-SQLite
5.0.7-lp154.2.3 -> 5.0.7-bp155.1.4 noarch repo-oss obs://build.opensuse.org/devel:openQA -> openSUSE
perl-Mojo-Pg
4.27-lp154.2.20 -> 4.27-bp155.1.5 noarch repo-oss obs://build.opensuse.org/devel:openQA -> openSUSE
perl-YAML-PP
0.036-lp154.2.1 -> 0.035-bp155.1.5 noarch repo-oss obs://build.opensuse.org/devel:openQA -> openSUSE
python3-constantly
15.1.0-150000.3.4.1 -> 15.1.0-3.2.1 noarch repo-oss SUSE LLC <https://www.suse.com/>
python3-hyperlink
17.2.1-150000.3.4.1 -> 17.2.1-3.2.1 noarch repo-oss SUSE LLC <https://www.suse.com/>
python3-packaging
21.3-150200.3.3.1 -> 20.3-1.9 noarch repo-oss SUSE LLC <https://www.suse.com/>
python3-zope.interface
4.4.2-150000.3.4.1 -> 4.4.2-3.2.1 x86_64 repo-oss SUSE LLC <https://www.suse.com/>
system-user-velociraptor
1.0.0-lp154.28.6 -> 1.0.0-bp155.26.1 x86_64 Server Monitoring Software obs://build.opensuse.org/security
telegraf
1.26.3-bp154.2.6.1 -> 1.25.1-bp155.1.21 x86_64 repo-oss openSUSE
velociraptor-client
0.6.7.5~git81.01be570-lp154.28.7 -> 0.6.7.4~git63.4a1ed09d-bp155.1.7 x86_64 repo-oss obs://build.opensuse.org/security -> openSUSE
vim
9.0.1894-150000.5.54.1 -> 9.0.1632-150500.20.3.1 x86_64 Update repository with updates from SUSE Linux Enterprise 15 SUSE LLC <https://www.suse.com/>
vim-data
9.0.1894-150000.5.54.1 -> 9.0.1632-150500.20.3.1 noarch Update repository with updates from SUSE Linux Enterprise 15 SUSE LLC <https://www.suse.com/>
vim-data-common
9.0.1894-150000.5.54.1 -> 9.0.1632-150500.20.3.1 noarch Update repository with updates from SUSE Linux Enterprise 15 SUSE LLC <https://www.suse.com/>
in particular
perl-IO-Compress-Lzma
2.206.0-lp154.13.1 -> 2.204-lp155.2.1 noarch devel_openQA_Modules obs://build.opensuse.org/devel:openQA
…
perl-YAML-PP
0.036-lp154.2.1 -> 0.035-bp155.1.5 noarch repo-oss obs://build.opensuse.org/devel:openQA -> openSUSE
…
telegraf
1.26.3-bp154.2.6.1 -> 1.25.1-bp155.1.21 x86_64 repo-oss openSUSE
I assume those are ok.
zypper --releasever=$new_version --gpg-auto-import-keys ref && test -f /etc/openqa/openqa.ini && sudo -u geekotest /opt/openqa-scripts/dump-psql && zypper -n --releasever=$new_version dup --auto-agree-with-licenses --replacefiles --download-in-advance
all done. I scheduled a reboot with rebootmgrctl reboot
, not rebooting right now as too many users might actively work with the system.
For PostgreSQL we run v14, new default is 15. Preparing commands from older tickets but just equivalent to https://open.qa/docs/#_migrating_postgresql_database_on_opensuse:
oldver=14 newver=15
zypper in postgresql$newver-server postgresql$newver-contrib
sudo -u postgres /usr/lib/postgresql$newver/bin/initdb --encoding=UTF8 --locale=en_US.UTF-8 --lc-collate=C --lc-ctype=en_US.UTF-8 --lc-messages=C --lc-monetary=C --lc-numeric=C --lc-time=C -D /var/lib/pgsql/data.$newver
sudo -u postgres vimdiff /var/lib/pgsql/data.$oldver/postgresql.conf /var/lib/pgsql/data.$newver/postgresql.conf
sudo -u postgres /usr/lib/postgresql$newver/bin/pg_upgrade --check --link --old-bindir=/usr/lib/postgresql$oldver/bin --new-bindir=/usr/lib/postgresql$newver/bin --old-datadir=/var/lib/pgsql/data.$oldver --new-datadir=/var/lib/pgsql/data.$newver && systemctl stop openqa-webui openqa-scheduler openqa-livehandler openqa-gru postgresql && sudo -u postgres /usr/lib/postgresql$newver/bin/pg_upgrade --link --old-bindir=/usr/lib/postgresql$oldver/bin --new-bindir=/usr/lib/postgresql$newver/bin --old-datadir=/var/lib/pgsql/data.$oldver --new-datadir=/var/lib/pgsql/data.$newver && ln --force --no-dereference --relative --symbolic /var/lib/pgsql/data.$newver /var/lib/pgsql/data && systemctl start postgresql openqa-webui openqa-scheduler openqa-livehandler openqa-gru && sudo -u geekotest psql -c 'select version();' openqa
if all looks good then I should do
rm -rf /var/lib/pgsql/data.$oldver
zypper rm -u postgresql$oldver postgresql$oldver-contrib postgresql$oldver-server
Updated by okurz 12 months ago
- Status changed from Resolved to In Progress
I found the webUI to be unavailable shortly after and a systemd failed services alert. postgresql died with
Oct 13 19:31:23 openqa systemd[1]: Started PostgreSQL database server.
Oct 13 20:50:30 openqa systemd[1]: Stopping PostgreSQL database server...
Oct 13 20:50:31 openqa systemd[1]: postgresql.service: Deactivated successfully.
Oct 13 20:50:31 openqa systemd[1]: Stopped PostgreSQL database server.
Oct 13 20:50:31 openqa systemd[1]: Starting PostgreSQL database server...
Oct 13 20:50:31 openqa postgresql-script[13721]: 2023-10-13 20:50:31.336 CEST [13721]FATAL: could not access file "pg_stat_statements": No such file or directory
Oct 13 20:50:31 openqa postgresql-script[13721]: 2023-10-13 20:50:31.336 CEST [13721]LOG: database system is shut down
Oct 13 20:50:31 openqa postgresql-script[13719]: pg_ctl: could not start server
so I guess salt pipelines caused postgresql to restart and then fail. As workaround I have removed the pg_stat_statements from postgres config and restarted and called systemctl start default.target
. All back to normal.
Updated by okurz 12 months ago
- Related to action #112850: Make pgstatstatements persistent on OSD (in salt) added
Updated by okurz 6 months ago
- Copied to action #157981: Upgrade osd webUI host to openSUSE Leap 15.6 size:S added