Project

General

Profile

Actions

tickets #155032

closed

MySQL/Galera backups incomplete

Added by crameleon 3 months ago. Updated 8 days ago.

Status:
Resolved
Priority:
High
Assignee:
Category:
Core services and virtual infrastructure
Target version:
-
Start date:
2024-02-07
Due date:
% Done:

100%

Estimated time:

Description

Last year, the configuration and databases got copied to mybackup.i.o.o - example date:

mybackup (DB backup):~ # ls /backup/20231013/
conference_osem.sql.xz  my.cnf.d              webforums2.sql.xz    wiki_es.sql.xz         wiki_old_de.sql.xz  wiki_zh.sql.xz
etherpad.sql.xz         mysql.sql.xz          wiki_cn.sql.xz       wiki_fr.sql.xz         wiki_old_en.sql.xz  wiki_zh_tw.sql.xz
limesurvey.sql.xz       redmine.sql.xz        wiki_cs.sql.xz       wiki_hu.sql.xz         wiki_pl.sql.xz      wsrep.sql.xz
matomo.sql.xz           redmine_test.sql.xz   wiki_de.sql.xz       wiki_it.sql.xz         wiki_pt.sql.xz      xbttracker.sql.xz
members.sql.xz          suse_hackweek.sql.xz  wiki_el.sql.xz       wiki_ja.sql.xz         wiki_ru.sql.xz
mirrorcache.sql.xz      sys.sql.xz            wiki_en.sql.xz       wiki_languages.sql.xz  wiki_sv.sql.xz
my.cnf                  webforums.sql.xz      wiki_en_test.sql.xz  wiki_nl.sql.xz         wiki_tr.sql.xz

At some point, the backup directories started to only contain the configuration files:

mybackup (DB backup):~ # ls /backup/20240206/
my.cnf  my.cnf.d

This might be a late regression of the migration.
We should resolve this soon.

Actions #1

Updated by crameleon 3 months ago

  • Subject changed from MySQL backup incomplete to MySQL/Galera backups incomplete
  • Description updated (diff)
  • Category set to Core services and virtual infrastructure
  • Priority changed from Normal to High
  • Private changed from Yes to No
Actions #2

Updated by crameleon 3 months ago

Based on this it probably just needs to be reconfigured to use the right domain name of one of the Galera nodes instead of this old hardcoded IP address:

mybackup (DB backup):~ # systemctl status mysql-backupscript.service
× mysql-backupscript.service - Non-interactive MySQL backup
     Loaded: loaded (/usr/lib/systemd/system/mysql-backupscript.service; static)
     Active: failed (Result: exit-code) since Tue 2024-02-06 03:00:08 UTC; 22h ago
TriggeredBy: ● mysql-backupscript.timer
   Main PID: 22883 (code=exited, status=1/FAILURE)

Feb 06 03:00:08 mybackup systemd[1]: Starting Non-interactive MySQL backup...
Feb 06 03:00:08 mybackup mysql-backupscript[22909]: Network is unreachable
Feb 06 03:00:08 mybackup mysql-backupscript[22909]: Error: Could not connect to host 192.168.47.7 on port 5667
Feb 06 03:00:08 mybackup systemd[1]: mysql-backupscript.service: Main process exited, code=exited, status=1/FAILURE
Feb 06 03:00:08 mybackup systemd[1]: mysql-backupscript.service: Failed with result 'exit-code'.
Feb 06 03:00:08 mybackup systemd[1]: Failed to start Non-interactive MySQL backup.
Actions #3

Updated by crameleon 28 days ago

  • Status changed from New to In Progress
  • Assignee set to crameleon
  • % Done changed from 0 to 20

The output above just hints at the nsca target address. The setup is well explained https://progress.opensuse.org/projects/opensuse-admin-wiki/wiki/Galera_Cluster. It means the local MariaDB on mybackup.i.o.o needs to be re-added as a secondary node. Starting the resync.

Actions #4

Updated by crameleon 28 days ago

  • % Done changed from 20 to 100

Done, first backup succeeded, timer for automatic backups re-enabled.

Actions #5

Updated by crameleon 8 days ago

  • Status changed from In Progress to Resolved
Actions

Also available in: Atom PDF