action #76927
closedcoordination #69478: [epic] Upgrade o3+osd workers+webui to openSUSE Leap 15.2
Upgrade postgresql database version on osd to default of Leap 15.2, like on o3 size:M
Added by okurz over 3 years ago. Updated over 2 years ago.
0%
Description
Motivation¶
See #76924
Acceptance criteria¶
- AC1: osd runs openQA from a postgres12 database
- AC2: no severe functional or performance related impact has been observed
Suggestions¶
- Same as in #76924 but done by a different person to improve our bus factor :)
- Follow the steps to upgrade which are no part of the official documentation: https://open.qa/docs/#_migrating_postgresql_database_on_opensuse
Updated by okurz over 3 years ago
- Copied from action #76924: Upgrade postgresql database version on o3 to default of Leap 15.2, i.e. postgres12 size:M added
Updated by okurz over 2 years ago
- Description updated (diff)
- Target version changed from future to Ready
Updated by mkittler over 2 years ago
- Assignee set to mkittler
It makes sense to have both instances on the same version. Nevertheless I'll wait at least one day to see whether any problems on o3 are occurring. Then I'll do the migration like on o3.
We can also follow the process and estimate the ticket first. Considering one now has only to follow the instructions I've been creating for the o3 migration (https://github.com/Martchus/openQA-helper#postgresql-migration-on-opensuse) this is likely an S for anybody and definitely not more than an M.
Updated by mkittler over 2 years ago
Looks like we don't have the same structure with versioned data dirs and a symlink on OSD like on o3 (as shown in https://github.com/Martchus/openQA-helper#postgresql-migration-on-opensuse). I would create such a structure then before doing the migration. By the way, the partition is quite full on OSD but since the upgrade is using hard links it shouldn't be a problem (on o3 the disk usage increased only by 1 %).
Updated by livdywan over 2 years ago
- Subject changed from Upgrade postgresql database version on osd to default of Leap 15.2, like on o3 to Upgrade postgresql database version on osd to default of Leap 15.2, like on o3 size:M
- Status changed from New to Workable
Updated by mkittler over 2 years ago
- Assignee deleted (
mkittler)
I'm removing my assignment to give others the chance to learn how to do PostgreSQL migrations, see the added suggestion.
Updated by okurz over 2 years ago
- Priority changed from Low to High
as osukup rightly reminded us of the pending EOL of Leap 15.2 this becomes more important so that we are properly setup for upgrade to 15.3
Updated by okurz over 2 years ago
- Copied to action #99204: Upgrade postgresql database version on osd to default of Leap 15.3, like on o3 added
Updated by osukup over 2 years ago
now at step 6 of @mkittler HOWTO, next steps needs shutdown of openqa and posgresql, which ins't best idea in working hours :D
Updated by okurz over 2 years ago
if you are quick then nobody will notice as the workers will continue to run jobs and reconnect as soon as possible. That's one of the benefits of the hard-link approach. To be quick commands should be prepared in a script and not executed interactively while the database is shutdown as this would take too long.
Updated by openqa_review over 2 years ago
- Due date set to 2021-10-16
Setting due date based on mean cycle time of SUSE QE Tools
Updated by osukup over 2 years ago
- Status changed from In Progress to Feedback
database migrated, added missing service to guide
packages for postgresql10 and orig database still in place
Updated by okurz over 2 years ago
there was an alert about long I/O times on the OSD partition /srv which happened at about the same time when you wrote that you finished the migration https://stats.openqa-monitor.qa.suse.de/d/WebuiDb/webui-summary?tab=alert&viewPanel=94&orgId=1&from=1633183630669&to=1633208648249 . I assume that the migration itself caused the alert and that there are no further negative effects.
Updated by osukup over 2 years ago
time +- same, migration was shorter but part of WebUi wasnt working -> fixed by manual restart services including openqa-websockets
Updated by osukup over 2 years ago
- Status changed from Feedback to Resolved
osd with database looks fine, so closing as resolved
only question -> can we remove old data.10 folder ?
Updated by okurz over 2 years ago
- Status changed from Resolved to Feedback
something looks weird now. /srv/PSQL10 has two subfolders data.10 and data.12. Can you fix that? Not sure what is best but a data.12 in PSQL10 does not look right :)
Updated by osukup over 2 years ago
what is /srv/PSQL10 folder ? database lives in /var/lib/pgsql/ maybe some backup scripts?
something looks weird now. /srv/PSQL10 has two subfolders data.10 and data.12. Can you fix that? Not sure what is best but a data.12 in PSQL10 does not look right :)
data.12 is current database
data.10 is old pgsql10 database ... which can be removed
Updated by okurz over 2 years ago
osukup wrote:
what is
/srv/PSQL10folder ? database lives in/var/lib/pgsql/maybe some backup scripts?
No, this is manual and a bind mount, see
# grep PSQL10 /etc/fstab
/srv/PSQL10 /var/lib/pgsql none bind 0 0
Needs to updated in https://gitlab.suse.de/openqa/salt-states-openqa/-/blob/master/fstab as well. Better we rename to not have the version in /etc/fstab, e.g. move to /srv/PSQL
something looks weird now. /srv/PSQL10 has two subfolders data.10 and data.12. Can you fix that? Not sure what is best but a data.12 in PSQL10 does not look right :)
data.12 is current database
data.10 is old pgsql10 database ... which can be removed
keep it as backup for some time. As long as we have the space we can also keep it until we go to the next version :)
EDIT: oh, you removed it already.
Updated by livdywan over 2 years ago
Discussed briefly in the Midweekly. Probably best to just propose an MR to rename the folder and then wrap this up.