QA (public) » openQA Project (public)

I suggest to wait for #130648 with referenced MRs first.

From a dry-run I see

The following 17 packages are going to be downgraded:
  libpoppler117
    22.01.0-150400.3.11.2 -> 22.01.0-150400.3.3.1                         x86_64  repo-oss                                                      SUSE LLC <https://www.suse.com/>
  libsource-highlight4
    3.1.9-bp154.1.116 -> 3.1.8-150000.3.4.1                               x86_64  repo-oss                                                      SUSE LLC <https://www.suse.com/>
  perl-IO-Compress-Lzma
    2.206.0-lp154.13.1 -> 2.204-lp155.2.1                                 noarch  devel_openQA_Modules                                          obs://build.opensuse.org/devel:openQA
  perl-Minion
    10.25-lp154.2.6 -> 10.25-bp155.1.6                                    noarch  repo-oss                                                      obs://build.opensuse.org/devel:openQA -> openSUSE
  perl-Minion-Backend-SQLite
    5.0.7-lp154.2.3 -> 5.0.7-bp155.1.4                                    noarch  repo-oss                                                      obs://build.opensuse.org/devel:openQA -> openSUSE
  perl-Mojo-Pg
    4.27-lp154.2.20 -> 4.27-bp155.1.5                                     noarch  repo-oss                                                      obs://build.opensuse.org/devel:openQA -> openSUSE
  perl-YAML-PP
    0.036-lp154.2.1 -> 0.035-bp155.1.5                                    noarch  repo-oss                                                      obs://build.opensuse.org/devel:openQA -> openSUSE
  python3-constantly
    15.1.0-150000.3.4.1 -> 15.1.0-3.2.1                                   noarch  repo-oss                                                      SUSE LLC <https://www.suse.com/>
  python3-hyperlink
    17.2.1-150000.3.4.1 -> 17.2.1-3.2.1                                   noarch  repo-oss                                                      SUSE LLC <https://www.suse.com/>
  python3-packaging
    21.3-150200.3.3.1 -> 20.3-1.9                                         noarch  repo-oss                                                      SUSE LLC <https://www.suse.com/>
  python3-zope.interface
    4.4.2-150000.3.4.1 -> 4.4.2-3.2.1                                     x86_64  repo-oss                                                      SUSE LLC <https://www.suse.com/>
  system-user-velociraptor
    1.0.0-lp154.28.6 -> 1.0.0-bp155.26.1                                  x86_64  Server Monitoring Software                                    obs://build.opensuse.org/security
  telegraf
    1.26.3-bp154.2.6.1 -> 1.25.1-bp155.1.21                               x86_64  repo-oss                                                      openSUSE
  velociraptor-client
    0.6.7.5~git81.01be570-lp154.28.7 -> 0.6.7.4~git63.4a1ed09d-bp155.1.7  x86_64  repo-oss                                                      obs://build.opensuse.org/security -> openSUSE
  vim
    9.0.1894-150000.5.54.1 -> 9.0.1632-150500.20.3.1                      x86_64  Update repository with updates from SUSE Linux Enterprise 15  SUSE LLC <https://www.suse.com/>
  vim-data
    9.0.1894-150000.5.54.1 -> 9.0.1632-150500.20.3.1                      noarch  Update repository with updates from SUSE Linux Enterprise 15  SUSE LLC <https://www.suse.com/>
  vim-data-common
    9.0.1894-150000.5.54.1 -> 9.0.1632-150500.20.3.1                      noarch  Update repository with updates from SUSE Linux Enterprise 15  SUSE LLC <https://www.suse.com/>

in particular

  perl-IO-Compress-Lzma
    2.206.0-lp154.13.1 -> 2.204-lp155.2.1                                 noarch  devel_openQA_Modules                                          obs://build.opensuse.org/devel:openQA
…
  perl-YAML-PP
    0.036-lp154.2.1 -> 0.035-bp155.1.5                                    noarch  repo-oss                                                      obs://build.opensuse.org/devel:openQA -> openSUSE
…
  telegraf
    1.26.3-bp154.2.6.1 -> 1.25.1-bp155.1.21                               x86_64  repo-oss                                                      openSUSE

I assume those are ok.

 zypper --releasever=$new_version --gpg-auto-import-keys ref && test -f /etc/openqa/openqa.ini && sudo -u geekotest /opt/openqa-scripts/dump-psql && zypper -n --releasever=$new_version dup --auto-agree-with-licenses --replacefiles --download-in-advance

all done. I scheduled a reboot with rebootmgrctl reboot, not rebooting right now as too many users might actively work with the system.

For PostgreSQL we run v14, new default is 15. Preparing commands from older tickets but just equivalent to https://open.qa/docs/#_migrating_postgresql_database_on_opensuse:

oldver=14 newver=15
zypper in postgresql$newver-server postgresql$newver-contrib
sudo -u postgres /usr/lib/postgresql$newver/bin/initdb --encoding=UTF8 --locale=en_US.UTF-8 --lc-collate=C --lc-ctype=en_US.UTF-8 --lc-messages=C --lc-monetary=C --lc-numeric=C --lc-time=C -D /var/lib/pgsql/data.$newver
sudo -u postgres vimdiff /var/lib/pgsql/data.$oldver/postgresql.conf /var/lib/pgsql/data.$newver/postgresql.conf
sudo -u postgres /usr/lib/postgresql$newver/bin/pg_upgrade --check --link --old-bindir=/usr/lib/postgresql$oldver/bin --new-bindir=/usr/lib/postgresql$newver/bin --old-datadir=/var/lib/pgsql/data.$oldver --new-datadir=/var/lib/pgsql/data.$newver && systemctl stop openqa-webui openqa-scheduler openqa-livehandler openqa-gru postgresql && sudo -u postgres /usr/lib/postgresql$newver/bin/pg_upgrade --link --old-bindir=/usr/lib/postgresql$oldver/bin --new-bindir=/usr/lib/postgresql$newver/bin --old-datadir=/var/lib/pgsql/data.$oldver --new-datadir=/var/lib/pgsql/data.$newver && ln --force --no-dereference --relative --symbolic /var/lib/pgsql/data.$newver /var/lib/pgsql/data && systemctl start postgresql openqa-webui openqa-scheduler openqa-livehandler openqa-gru && sudo -u geekotest psql -c 'select version();' openqa

if all looks good then I should do

rm -rf /var/lib/pgsql/data.$oldver
zypper rm -u postgresql$oldver postgresql$oldver-contrib postgresql$oldver-server

All done

I found the webUI to be unavailable shortly after and a systemd failed services alert. postgresql died with

Oct 13 19:31:23 openqa systemd[1]: Started PostgreSQL database server.
Oct 13 20:50:30 openqa systemd[1]: Stopping PostgreSQL database server...
Oct 13 20:50:31 openqa systemd[1]: postgresql.service: Deactivated successfully.
Oct 13 20:50:31 openqa systemd[1]: Stopped PostgreSQL database server.
Oct 13 20:50:31 openqa systemd[1]: Starting PostgreSQL database server...
Oct 13 20:50:31 openqa postgresql-script[13721]: 2023-10-13 20:50:31.336 CEST   [13721]FATAL:  could not access file "pg_stat_statements": No such file or directory
Oct 13 20:50:31 openqa postgresql-script[13721]: 2023-10-13 20:50:31.336 CEST   [13721]LOG:  database system is shut down
Oct 13 20:50:31 openqa postgresql-script[13719]: pg_ctl: could not start server

so I guess salt pipelines caused postgresql to restart and then fail. As workaround I have removed the pg_stat_statements from postgres config and restarted and called systemctl start default.target. All back to normal.

All good again, commented in #112850-8