Project

General

Profile

action #76927

coordination #69478: [epic] Upgrade o3+osd workers+webui to openSUSE Leap 15.2

Upgrade postgresql database version on osd to default of Leap 15.2, like on o3 size:M

Added by okurz about 1 year ago. Updated about 1 month ago.

Status:
Resolved
Priority:
High
Assignee:
Target version:
Start date:
2020-11-03
Due date:
% Done:

0%

Estimated time:

Description

Motivation

See #76924

Acceptance criteria

  • AC1: osd runs openQA from a postgres12 database
  • AC2: no severe functional or performance related impact has been observed

Suggestions


Related issues

Copied from openQA Infrastructure - action #76924: Upgrade postgresql database version on o3 to default of Leap 15.2, i.e. postgres12 size:MResolved2020-11-03

Copied to openQA Infrastructure - action #99204: Upgrade postgresql database version on osd to default of Leap 15.3, like on o3 size:MNew

History

#1 Updated by okurz about 1 year ago

  • Copied from action #76924: Upgrade postgresql database version on o3 to default of Leap 15.2, i.e. postgres12 size:M added

#2 Updated by okurz 3 months ago

  • Description updated (diff)
  • Target version changed from future to Ready

#3 Updated by mkittler 3 months ago

  • Assignee set to mkittler

It makes sense to have both instances on the same version. Nevertheless I'll wait at least one day to see whether any problems on o3 are occurring. Then I'll do the migration like on o3.


We can also follow the process and estimate the ticket first. Considering one now has only to follow the instructions I've been creating for the o3 migration (https://github.com/Martchus/openQA-helper#postgresql-migration-on-opensuse) this is likely an S for anybody and definitely not more than an M.

#4 Updated by mkittler 3 months ago

Looks like we don't have the same structure with versioned data dirs and a symlink on OSD like on o3 (as shown in https://github.com/Martchus/openQA-helper#postgresql-migration-on-opensuse). I would create such a structure then before doing the migration. By the way, the partition is quite full on OSD but since the upgrade is using hard links it shouldn't be a problem (on o3 the disk usage increased only by 1 %).

#5 Updated by cdywan 3 months ago

  • Subject changed from Upgrade postgresql database version on osd to default of Leap 15.2, like on o3 to Upgrade postgresql database version on osd to default of Leap 15.2, like on o3 size:M
  • Status changed from New to Workable

#6 Updated by mkittler 3 months ago

  • Assignee deleted (mkittler)

I'm removing my assignment to give others the chance to learn how to do PostgreSQL migrations, see the added suggestion.

#7 Updated by okurz 2 months ago

  • Priority changed from Low to High

as osukup rightly reminded us of the pending EOL of Leap 15.2 this becomes more important so that we are properly setup for upgrade to 15.3

#8 Updated by okurz 2 months ago

  • Copied to action #99204: Upgrade postgresql database version on osd to default of Leap 15.3, like on o3 size:M added

#10 Updated by mkittler 2 months ago

  • Description updated (diff)

#11 Updated by osukup 2 months ago

  • Assignee set to osukup

#12 Updated by osukup 2 months ago

now at step 6 of mkittler HOWTO, next steps needs shutdown of openqa and posgresql, which ins't best idea in working hours :D

#13 Updated by okurz 2 months ago

if you are quick then nobody will notice as the workers will continue to run jobs and reconnect as soon as possible. That's one of the benefits of the hard-link approach. To be quick commands should be prepared in a script and not executed interactively while the database is shutdown as this would take too long.

#14 Updated by osukup 2 months ago

  • Status changed from Workable to In Progress

#15 Updated by openqa_review 2 months ago

  • Due date set to 2021-10-16

Setting due date based on mean cycle time of SUSE QE Tools

#16 Updated by osukup 2 months ago

  • Status changed from In Progress to Feedback

database migrated, added missing service to guide

packages for postgresql10 and orig database still in place

#17 Updated by okurz about 2 months ago

there was an alert about long I/O times on the OSD partition /srv which happened at about the same time when you wrote that you finished the migration https://stats.openqa-monitor.qa.suse.de/d/WebuiDb/webui-summary?tab=alert&viewPanel=94&orgId=1&from=1633183630669&to=1633208648249 . I assume that the migration itself caused the alert and that there are no further negative effects.

#18 Updated by osukup about 2 months ago

time +- same, migration was shorter but part of WebUi wasnt working -> fixed by manual restart services including openqa-websockets

#19 Updated by osukup about 2 months ago

  • Status changed from Feedback to Resolved

osd with database looks fine, so closing as resolved

only question -> can we remove old data.10 folder ?

#20 Updated by okurz about 2 months ago

  • Status changed from Resolved to Feedback

something looks weird now. /srv/PSQL10 has two subfolders data.10 and data.12. Can you fix that? Not sure what is best but a data.12 in PSQL10 does not look right :)

#21 Updated by osukup about 2 months ago

what is /srv/PSQL10 folder ? database lives in /var/lib/pgsql/ maybe some backup scripts?

something looks weird now. /srv/PSQL10 has two subfolders data.10 and data.12. Can you fix that? Not sure what is best but a data.12 in PSQL10 does not look right :)

data.12 is current database

data.10 is old pgsql10 database ... which can be removed

#22 Updated by okurz about 2 months ago

osukup wrote:

what is /srv/PSQL10 folder ? database lives in /var/lib/pgsql/ maybe some backup scripts?

No, this is manual and a bind mount, see

# grep PSQL10 /etc/fstab 
/srv/PSQL10 /var/lib/pgsql none bind 0 0

Needs to updated in https://gitlab.suse.de/openqa/salt-states-openqa/-/blob/master/fstab as well. Better we rename to not have the version in /etc/fstab, e.g. move to /srv/PSQL

something looks weird now. /srv/PSQL10 has two subfolders data.10 and data.12. Can you fix that? Not sure what is best but a data.12 in PSQL10 does not look right :)

data.12 is current database

data.10 is old pgsql10 database ... which can be removed

keep it as backup for some time. As long as we have the space we can also keep it until we go to the next version :)

EDIT: oh, you removed it already.

#23 Updated by cdywan about 2 months ago

Discussed briefly in the Midweekly. Probably best to just propose an MR to rename the folder and then wrap this up.

#25 Updated by osukup about 2 months ago

merged + dir renamed

#26 Updated by osukup about 2 months ago

  • Status changed from Feedback to Resolved

#27 Updated by okurz about 1 month ago

  • Due date deleted (2021-10-16)

Also available in: Atom PDF