Actions
action #175689
closedcoordination #161414: [epic] Improved salt based infrastructure management
monitor.qe.nue2.suse.org "502 Bad Gateway" from nginx on 2025-01-17, missing grafana server files?
Status:
Resolved
Priority:
Low
Assignee:
Category:
Regressions/Crashes
Target version:
Start date:
2025-01-17
Due date:
% Done:
0%
Estimated time:
Tags:
Description
Updated by okurz 21 days ago
- Copied from action #175686: OSD webUI ended up with "502 Bad Gateway" from nginx on 2025-01-17, needed manual restart of openqa-webui added
Updated by okurz 21 days ago
- Copied to action #175695: salt states sporadically fail in applying the security sensor repo with "xml.parsers.expat.ExpatError: syntax error: line 1, column 0", didn't we remove the security sensor repo? added
Updated by nicksinger 21 days ago
- Priority changed from Urgent to High
systemctl status grafana-server showed missing assets in /usr/share/grafana/public/build - the folder is indeed empty. Zypper log shows:
2025-01-17 00:39:23 <1> monitor(8585) [zypper] main.cc(main):143 ===== 'zypper' '--non-interactive' '--no-refresh' 'info' '-t' 'package' 'grafana' =====
2025-01-17 01:38:02 <1> monitor(27287) [zypper] main.cc(main):143 ===== 'zypper' '--non-interactive' '--no-refresh' 'info' '-t' 'package' 'grafana' =====
2025-01-17 02:38:46 <1> monitor(13268) [zypper] main.cc(main):143 ===== 'zypper' '--non-interactive' '--no-refresh' 'info' '-t' 'package' 'grafana' =====
2025-01-17 03:38:06 <1> monitor(31520) [zypper] main.cc(main):143 ===== 'zypper' '--non-interactive' '--no-refresh' 'info' '-t' 'package' 'grafana' =====
2025-01-17 03:56:01 <1> monitor(5176) [zypper++] Summary.cc(readPool):281 <install> U_Ts_(1285)grafana-11.3.0-lp156.6.1.x86_64(monitoring-software.repo)
2025-01-17 03:56:01 <1> monitor(5176) [zypper++] Summary.cc(readPool):281 <uninstall> I_TsU(143534)grafana-11.3.0-lp156.5.3.x86_64(@System)
2025-01-17 03:56:02 <2> monitor(5176) [PLUGIN] PluginScript.cc(~PluginDumpStderr):86 ! DEBUG:solvables: {bubblewrap, bubblewrap-zsh-completion, glibc, glibc-32bit, glibc-devel, glibc-extra, glibc-lang, glibc-locale, glibc-locale-base, grafana, libproxy1, libpxbackend-1_0, nscd, xen-libs}
2025-01-17 03:57:55 <1> monitor(5176) [zypp] PackageProvider.cc(providePackage):424 provide Package (1285)grafana-11.3.0-lp156.6.1.x86_64(monitoring-software.repo)
2025-01-17 03:57:55 <1> monitor(5176) [zypp++] DeltaCandidates.cc(deltaRpms):83 package: (1285)grafana-11.3.0-lp156.6.1.x86_64(monitoring-software.repo)
2025-01-17 03:57:55 <1> monitor(5176) [zypp::media] RepoProvideFile.cc(provideFile):253 [1]./x86_64/grafana-11.3.0-lp156.6.1.x86_64.rpm{55.2 MiB|sha256-111ee4e75ecce50d4b780d480027e97d28daf9953391dcb1eecd1ca7cf11e4a0|}
2025-01-17 03:57:55 <1> monitor(5176) [zypp::fetcher++] MediaSetAccess.cc(provide):275 Going to try to provide file ./x86_64/grafana-11.3.0-lp156.6.1.x86_64.rpm from media number 1
2025-01-17 03:57:55 <1> monitor(5176) [zypp::media++] MediaMultiCurl.cc(doGetFileCopy):1510 dest: /var/adm/mount/AP_0x0aXHdn/x86_64/grafana-11.3.0-lp156.6.1.x86_64.rpm
2025-01-17 03:57:55 <1> monitor(5176) [zypp::media++] MediaMultiCurl.cc(doGetFileCopy):1511 temp: /var/adm/mount/AP_0x0aXHdn/x86_64/grafana-11.3.0-lp156.6.1.x86_64.rpm.new.zypp.JUsTa2
2025-01-17 03:57:55 <1> monitor(5176) [zypp::media++] MediaCurl.cc(doGetFileCopyFile):1204 ./x86_64/grafana-11.3.0-lp156.6.1.x86_64.rpm
2025-01-17 03:57:55 <1> monitor(5176) [zypp::media++] MediaCurl.cc(doGetFileCopyFile):1214 URL: http://download.opensuse.org/repositories/devel:/openQA:/monitoring/15.6/x86_64/grafana-11.3.0-lp156.6.1.x86_64.rpm
2025-01-17 03:57:55 <1> monitor(5176) [zypp-curl++] curlhelper.cc(log_redirects_curl):159 redirecting to location: http://downloadcontentcdn.opensuse.org/repositories/devel:/openQA:/monitoring/15.6/x86_64/grafana-11.3.0-lp156.6.1.x86_64.rpm
2025-01-17 03:58:00 <1> monitor(5176) [zypp-core] PathInfo.cc(rename):753 rename /var/adm/mount/AP_0x0aXHdn/x86_64/grafana-11.3.0-lp156.6.1.x86_64.rpm.new.zypp.JUsTa2 -> /var/adm/mount/AP_0x0aXHdn/x86_64/grafana-11.3.0-lp156.6.1.x86_64.rpm
2025-01-17 03:58:00 <1> monitor(5176) [zypp::media++] MediaMultiCurl.cc(doGetFileCopy):1713 done: /var/adm/mount/AP_0x0aXHdn/x86_64/grafana-11.3.0-lp156.6.1.x86_64.rpm{- 0644 0/0 size 57915504}
2025-01-17 03:58:00 <1> monitor(5176) [zypp::media++] MediaHandler.cc(provideFile):977 provideFile([1]./x86_64/grafana-11.3.0-lp156.6.1.x86_64.rpm{55.2 MiB|sha256-111ee4e75ecce50d4b780d480027e97d28daf9953391dcb1eecd1ca7cf11e4a0|})
2025-01-17 03:58:00 <1> monitor(5176) [zypp::fetcher] Fetcher.cc(validate):387 Checking job [/var/adm/mount/AP_0x0aXHdn/x86_64/grafana-11.3.0-lp156.6.1.x86_64.rpm] (2 checkers )
2025-01-17 03:58:00 <1> monitor(5176) [zypp-core] PathInfo.cc(hardlinkCopy):921 hardlinkCopy /var/adm/mount/AP_0x0aXHdn/x86_64/grafana-11.3.0-lp156.6.1.x86_64.rpm -> /var/cache/zypp/packages/monitoring-software.repo/x86_64/grafana-11.3.0-lp156.6.1.x86_64.rpm
2025-01-17 03:58:00 <1> monitor(5176) [zypp::fetcher++] MediaSetAccess.cc(releaseFile):91 Going to release file ./x86_64/grafana-11.3.0-lp156.6.1.x86_64.rpm from media number 1
2025-01-17 03:58:00 <1> monitor(5176) [zypp-core] PathInfo.cc(unlink):711 unlink /var/adm/mount/AP_0x0aXHdn/x86_64/grafana-11.3.0-lp156.6.1.x86_64.rpm
2025-01-17 03:58:00 <1> monitor(5176) [zypp::media] RepoProvideFile.cc(provideFile):319 provideFile at /var/cache/zypp/packages/monitoring-software.repo/x86_64/grafana-11.3.0-lp156.6.1.x86_64.rpm
2025-01-17 03:58:00 <1> monitor(5176) [zypp] PackageProvider.cc(providePackage):525 provided Package (1285)grafana-11.3.0-lp156.6.1.x86_64(monitoring-software.repo) at /var/cache/zypp/packages/monitoring-software.repo/x86_64/grafana-11.3.0-lp156.6.1.x86_64.rpm
2025-01-17 03:58:19 <1> monitor(5176) [zypp] RpmHeader.cc(readPackage):259 ReferenceCounted(@0x55e837542650<=1){0x55e837845480}{grafana-11.3.0-lp156.6.1} from /var/cache/zypp/packages/monitoring-software.repo/x86_64/grafana-11.3.0-lp156.6.1.x86_64.rpm
2025-01-17 03:58:19 <1> monitor(5176) [librpmDb] RpmDb.cc(doInstallPackage):1675 RpmDb::installPackage(/var/cache/zypp/packages/monitoring-software.repo/x86_64/grafana-11.3.0-lp156.6.1.x86_64.rpm,0x0000000c)
2025-01-17 03:58:19 <1> monitor(5176) [zypp::exec++] forkspawnengine.cc(start):274 Executing[C] 'rpm' '--root' '/' '--dbpath' '/usr/lib/sysimage/rpm' '--define' '_dump_posttrans 1' '-U' '--percent' '--noglob' '--force' '--nodeps' '--' '/var/cache/zypp/packages/monitoring-software.repo/x86_64/grafana-11.3.0-lp156.6.1.x86_64.rpm'
2025-01-17 03:58:19 <1> monitor(5176) [Progress++] progressdata.cc(report):75 {#15|Installing: grafana-11.3.0-lp156.6.1.x86_64} START
2025-01-17 03:59:30 <1> monitor(5176) [Progress++] progressdata.cc(report):95 {#15|Installing: grafana-11.3.0-lp156.6.1.x86_64} END
2025-01-17 03:59:30 <1> monitor(5176) [zypp-core] PathInfo.cc(unlink):711 unlink /var/cache/zypp/packages/monitoring-software.repo/x86_64/grafana-11.3.0-lp156.6.1.x86_64.rpm
2025-01-17 04:37:59 <1> monitor(17840) [zypper] main.cc(main):143 ===== 'zypper' '--non-interactive' '--no-refresh' 'info' '-t' 'package' 'grafana' =====
2025-01-17 05:41:03 <1> monitor(4278) [zypper] main.cc(main):143 ===== 'zypper' '--non-interactive' '--no-refresh' 'info' '-t' 'package' 'grafana' =====
2025-01-17 06:41:09 <1> monitor(21784) [zypper] main.cc(main):143 ===== 'zypper' '--non-interactive' '--no-refresh' 'info' '-t' 'package' 'grafana' =====
2025-01-17 07:38:12 <1> monitor(6795) [zypper] main.cc(main):143 ===== 'zypper' '--non-interactive' '--no-refresh' 'info' '-t' 'package' 'grafana' =====
2025-01-17 08:37:54 <1> monitor(25076) [zypper] main.cc(main):143 ===== 'zypper' '--non-interactive' '--no-refresh' 'info' '-t' 'package' 'grafana' =====
2025-01-17 09:38:22 <1> monitor(12122) [zypper] main.cc(main):143 ===== 'zypper' '--non-interactive' '--no-refresh' 'info' '-t' 'package' 'grafana' =====
2025-01-17 09:47:34 <1> monitor(16009) [zypper] main.cc(main):143 ===== 'zypper' '--non-interactive' '--no-refresh' 'info' '-t' 'package' 'grafana' =====
2025-01-17 09:52:54 <1> monitor(18881) [zypper] main.cc(main):143 ===== 'zypper' '--non-interactive' '--no-refresh' 'info' '-t' 'package' 'grafana' =====
2025-01-17 09:53:36 <1> monitor(19944) [zypper] main.cc(main):143 ===== 'zypper' '--non-interactive' '--no-refresh' 'info' '-t' 'package' 'grafana' =====
2025-01-17 10:20:07 <1> monitor(28657) [zypper] main.cc(main):143 ===== 'zypper' 'se' 'grafana' =====
2025-01-17 10:20:43 <1> monitor(28916) [zypper] main.cc(main):143 ===== 'zypper' 'info' 'grafana' =====
So we have a broken package in https://build.opensuse.org/package/revisions/server:monitoring/grafana I will report (@mkittler maybe even to you? :) ).
As a workaround I downloaded the distribution package from https://grafana.com/grafana/download/11.3.2?edition=oss matching our version, extract it and copy the "build"-folder with the according web assets. After a restart of grafana the webui came back, now the package needs to be fixed.
Updated by nicksinger 21 days ago
- Due date set to 2025-01-20
- Status changed from New to In Progress
- Priority changed from High to Low
@mkittler helped with linking the package to an older version with:
osc linkpac --force --revision 145 server:monitoring grafana devel:openqa:monitoring
we discussed in jitsi that we might want to "revert" this link back to factory once the fix ends up there. I will check this next week.
Updated by nicksinger 18 days ago
- Status changed from Blocked to In Progress
Request accepted, will linkpac as described and check if the package is still missing files.
Updated by nicksinger 18 days ago
- Status changed from In Progress to Resolved
# osc linkpac --force openSUSE:Factory grafana devel:openQA:monitoring
forced overwrite of existing _link file
Creating _link... Done.
https://build.opensuse.org/project/show/devel:openQA:monitoring shows "grafana -> Changed: less than a minute ago"
verified with:
root@monitor:~ # zypper in -f grafana
root@monitor:~ # rpm -ql grafana | grep "/usr/share/grafana/public/build" | wc -l
661
so all good again.
Actions