Project

General

Profile

Actions

action #76927

closed

coordination #69478: [epic] Upgrade o3+osd workers+webui to openSUSE Leap 15.2

Upgrade postgresql database version on osd to default of Leap 15.2, like on o3 size:M

Added by okurz over 3 years ago. Updated over 2 years ago.

Status:
Resolved
Priority:
High
Assignee:
Category:
-
Target version:
Start date:
2020-11-03
Due date:
% Done:

0%

Estimated time:

Description

Motivation

See #76924

Acceptance criteria

  • AC1: osd runs openQA from a postgres12 database
  • AC2: no severe functional or performance related impact has been observed

Suggestions


Related issues 2 (0 open2 closed)

Copied from openQA Infrastructure - action #76924: Upgrade postgresql database version on o3 to default of Leap 15.2, i.e. postgres12 size:MResolvedmkittler2020-11-03

Actions
Copied to openQA Infrastructure - action #99204: Upgrade postgresql database version on osd to default of Leap 15.3, like on o3Resolvedosukup

Actions
Actions #1

Updated by okurz over 3 years ago

  • Copied from action #76924: Upgrade postgresql database version on o3 to default of Leap 15.2, i.e. postgres12 size:M added
Actions #2

Updated by okurz over 2 years ago

  • Description updated (diff)
  • Target version changed from future to Ready
Actions #3

Updated by mkittler over 2 years ago

  • Assignee set to mkittler

It makes sense to have both instances on the same version. Nevertheless I'll wait at least one day to see whether any problems on o3 are occurring. Then I'll do the migration like on o3.


We can also follow the process and estimate the ticket first. Considering one now has only to follow the instructions I've been creating for the o3 migration (https://github.com/Martchus/openQA-helper#postgresql-migration-on-opensuse) this is likely an S for anybody and definitely not more than an M.

Actions #4

Updated by mkittler over 2 years ago

Looks like we don't have the same structure with versioned data dirs and a symlink on OSD like on o3 (as shown in https://github.com/Martchus/openQA-helper#postgresql-migration-on-opensuse). I would create such a structure then before doing the migration. By the way, the partition is quite full on OSD but since the upgrade is using hard links it shouldn't be a problem (on o3 the disk usage increased only by 1 %).

Actions #5

Updated by livdywan over 2 years ago

  • Subject changed from Upgrade postgresql database version on osd to default of Leap 15.2, like on o3 to Upgrade postgresql database version on osd to default of Leap 15.2, like on o3 size:M
  • Status changed from New to Workable
Actions #6

Updated by mkittler over 2 years ago

  • Assignee deleted (mkittler)

I'm removing my assignment to give others the chance to learn how to do PostgreSQL migrations, see the added suggestion.

Actions #7

Updated by okurz over 2 years ago

  • Priority changed from Low to High

as osukup rightly reminded us of the pending EOL of Leap 15.2 this becomes more important so that we are properly setup for upgrade to 15.3

Actions #8

Updated by okurz over 2 years ago

  • Copied to action #99204: Upgrade postgresql database version on osd to default of Leap 15.3, like on o3 added
Actions #10

Updated by mkittler over 2 years ago

  • Description updated (diff)
Actions #11

Updated by osukup over 2 years ago

  • Assignee set to osukup
Actions #12

Updated by osukup over 2 years ago

now at step 6 of @mkittler HOWTO, next steps needs shutdown of openqa and posgresql, which ins't best idea in working hours :D

Actions #13

Updated by okurz over 2 years ago

if you are quick then nobody will notice as the workers will continue to run jobs and reconnect as soon as possible. That's one of the benefits of the hard-link approach. To be quick commands should be prepared in a script and not executed interactively while the database is shutdown as this would take too long.

Actions #14

Updated by osukup over 2 years ago

  • Status changed from Workable to In Progress
Actions #15

Updated by openqa_review over 2 years ago

  • Due date set to 2021-10-16

Setting due date based on mean cycle time of SUSE QE Tools

Actions #16

Updated by osukup over 2 years ago

  • Status changed from In Progress to Feedback

database migrated, added missing service to guide

packages for postgresql10 and orig database still in place

Actions #17

Updated by okurz over 2 years ago

there was an alert about long I/O times on the OSD partition /srv which happened at about the same time when you wrote that you finished the migration https://stats.openqa-monitor.qa.suse.de/d/WebuiDb/webui-summary?tab=alert&viewPanel=94&orgId=1&from=1633183630669&to=1633208648249 . I assume that the migration itself caused the alert and that there are no further negative effects.

Actions #18

Updated by osukup over 2 years ago

time +- same, migration was shorter but part of WebUi wasnt working -> fixed by manual restart services including openqa-websockets

Actions #19

Updated by osukup over 2 years ago

  • Status changed from Feedback to Resolved

osd with database looks fine, so closing as resolved

only question -> can we remove old data.10 folder ?

Actions #20

Updated by okurz over 2 years ago

  • Status changed from Resolved to Feedback

something looks weird now. /srv/PSQL10 has two subfolders data.10 and data.12. Can you fix that? Not sure what is best but a data.12 in PSQL10 does not look right :)

Actions #21

Updated by osukup over 2 years ago

what is /srv/PSQL10 folder ? database lives in /var/lib/pgsql/ maybe some backup scripts?

something looks weird now. /srv/PSQL10 has two subfolders data.10 and data.12. Can you fix that? Not sure what is best but a data.12 in PSQL10 does not look right :)

data.12 is current database

data.10 is old pgsql10 database ... which can be removed

Actions #22

Updated by okurz over 2 years ago

osukup wrote:

what is /srv/PSQL10 folder ? database lives in /var/lib/pgsql/ maybe some backup scripts?

No, this is manual and a bind mount, see

# grep PSQL10 /etc/fstab 
/srv/PSQL10 /var/lib/pgsql none bind 0 0

Needs to updated in https://gitlab.suse.de/openqa/salt-states-openqa/-/blob/master/fstab as well. Better we rename to not have the version in /etc/fstab, e.g. move to /srv/PSQL

something looks weird now. /srv/PSQL10 has two subfolders data.10 and data.12. Can you fix that? Not sure what is best but a data.12 in PSQL10 does not look right :)

data.12 is current database

data.10 is old pgsql10 database ... which can be removed

keep it as backup for some time. As long as we have the space we can also keep it until we go to the next version :)

EDIT: oh, you removed it already.

Actions #23

Updated by livdywan over 2 years ago

Discussed briefly in the Midweekly. Probably best to just propose an MR to rename the folder and then wrap this up.

Actions #25

Updated by osukup over 2 years ago

merged + dir renamed

Actions #26

Updated by osukup over 2 years ago

  • Status changed from Feedback to Resolved
Actions #27

Updated by okurz over 2 years ago

  • Due date deleted (2021-10-16)
Actions

Also available in: Atom PDF