Project

General

Profile

Actions
Copy link

action #151013

closed

o3 yielding "502 Bad Gateway" from nginx 2023-11-19, why was the config overwritten? size:M

Added by okurz 5 months ago. Updated 5 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
tinita
Category:
-
Target version:
openQA Project - Ready
Start date:
2023-11-19
Due date:
% Done:

0%

Estimated time:
Tags:
o3, nginx

Description

Observation

o3 yields "502 Bad Gateway" from nginx since 2023-11-19. Was notified by Bernhard Wiedemann. I tried restarting openqa-webui and nginx, no change. services run fine, curl http://localhost:9526 locally fine. I found that in /etc/nginx/vhosts.d the config file "openqa-upstreams.inc" was overwritten, likely by an update. Due to #133358 we had there server 127.0.0.1:9526; instead of server [::1]:9526. Also openqa-locations.inc with local changes, e.g. preventing some special asset download as well as enabling the optional faster image downloads directly over nginx. I reverted bot changes by moving back "*.inc.rpmsave" files and restarted nginx. So the service runs fine now but we should investigate why now the config files were automatically overwritten when they should not be.

Acceptance criteria

  • AC1: OS upgrades do not overwrite custom changes in /etc/nginx/ that we want to keep
  • AC2: Config file updates are still available in the system, e.g. as ".rpmnew" files

Suggestions

  • Read #151013-4 to find out what changed
  • Understand when nginx was updated:
$ sudo journalctl -u nginx
Nov 03 10:02:46 new-ariel systemd[1]: Started The nginx HTTP and reverse proxy server.
Nov 19 03:30:01 new-ariel systemd[1]: Stopping The nginx HTTP and reverse proxy server...
Nov 19 03:30:02 new-ariel systemd[1]: nginx.service: Deactivated successfully.
Nov 19 03:30:02 new-ariel systemd[1]: Stopped The nginx HTTP and reverse proxy server.
-- Boot 0cf3554b0359404db30d52ee5bc61e48 --
Nov 19 03:30:33 new-ariel systemd[1]: Starting The nginx HTTP and reverse proxy server..

so likely the problem was triggered on Nov 19 03:30 due to the nginx restart.

  • Check spec files of both openQA and nginx how the config files are handled. Maybe we are simply missing a "noreplace"
  • Read https://www.cl.cam.ac.uk/~jw35/docs/rpm_config.html
  • As necessary add a "simulated change" to the upstream nginx openQA config file and see if o3 file is overwritten again
  • Optionally investigate why this was never a problem in the past months
  • Optionally read history from tickets and pull requests bringing in the nginx config into our openQA
  • Research if there is a better way to handle local config overrides
  • Prevent the situation from happening again in the future
  • As necessary update our openQA documentation how to manage local changes to nginx

Related issues 1 (0 open — 1 closed)

Related to openQA Infrastructure - action #150908: o3 "Unable to fetch build results" and "Internal server error" on some pages size:MResolvedtinita2023-11-15

Actions
Actions
Copy link
 #1

Updated by okurz 5 months ago

  • Subject changed from o3 yields "502 Bad Gateway" from nginx since 2023-11-19 to o3 yields "502 Bad Gateway" from nginx since 2023-11-19, why was config overwritten?
Actions
Copy link
 #2

Updated by livdywan 5 months ago

  • Subject changed from o3 yields "502 Bad Gateway" from nginx since 2023-11-19, why was config overwritten? to o3 yielding "502 Bad Gateway" from nginx 2023-11-19, why was the config overwritten?

Apparently o3 is available again. Just to clarify since the ticket was suggesting it was still broken.

Actions
Copy link
 #3

Updated by tinita 5 months ago

What was the timestamp of the changed files, so we can see which update could be responsible?

Actions
Copy link
 #4

Updated by tinita 5 months ago 

% tail -30 /var/log/zypp/history
2023-11-15 12:30:48|patch  |openSUSE-SLE-15.5-2023-4310|1|noarch|repo-sle-update|moderate|recommended|needed|applied|
2023-11-15 12:30:48|patch  |openSUSE-SLE-15.5-2023-4430|1|noarch|repo-sle-update|important|security|needed|applied|
2023-11-15 12:30:48|patch  |openSUSE-SLE-15.5-2023-4439|1|noarch|repo-sle-update|moderate|security|needed|applied|
2023-11-15 12:30:48|patch  |openSUSE-SLE-15.5-2023-4370|1|noarch|repo-sle-update|moderate|security|needed|applied|
2023-11-15 12:30:48|patch  |openSUSE-SLE-15.5-2023-4385|1|noarch|repo-sle-update|moderate|recommended|needed|applied|
2023-11-16 15:09:33|command|root@new-ariel|'zypper' '-n' '--no-refresh' 'dup' '--auto-agree-with-licenses' '--download-in-advance'|
2023-11-16 15:09:34|install|openQA-common|4.6.1700144494.90cbb53-lp155.6167.1|x86_64||devel_openQA|785344bd255b487185b26fb2ccf91513f2772db695baabe54d668f157ebd81c2|
2023-11-16 15:09:34|install|openQA-client|4.6.1700144494.90cbb53-lp155.6167.1|x86_64||devel_openQA|2487a6abfef49b993d303996236e87aa631f8a9afe81dd475dbb2247ec69e351|
2023-11-16 15:09:39|install|openQA|4.6.1700144494.90cbb53-lp155.6167.1|x86_64||devel_openQA|ce10bb54271685fcfdebdb2cde78da60f402c2472045b39eb00d35e6f72f2f03|
2023-11-16 15:09:40|install|openQA-local-db|4.6.1700144494.90cbb53-lp155.6167.1|x86_64||devel_openQA|b5dd2e2cff0b72df647d14f5e6d11a9d644f037a077f5e3400a9dd82089474e1|
2023-11-16 15:09:40|install|openQA-munin|4.6.1700144494.90cbb53-lp155.6167.1|x86_64||devel_openQA|e0284c04e246148006c2ae999ed9022461a46b0dcaa8b83ba11da05442ccc32a|
2023-11-16 15:09:41|install|openQA-continuous-update|4.6.1700144494.90cbb53-lp155.6167.1|x86_64||devel_openQA|334321d032181ea7850e21f6ebef60be8610dd480ca21ecda9bd7e5476860022|
2023-11-16 15:09:42|install|openQA-auto-update|4.6.1700144494.90cbb53-lp155.6167.1|x86_64||devel_openQA|c6e0bb3290f74a0cefdab4837386bd81b15f7065695cb0987d78826ab77871a3|
2023-11-16 15:30:31|command|root@new-ariel|'zypper' '-n' '--no-refresh' 'dup' '--auto-agree-with-licenses' '--download-in-advance'|
2023-11-16 15:30:31|install|libatomic1|13.2.1+git7813-150000.1.6.1|x86_64||repo-sle-update|5304aa121f77b3e863a4245524c91d050818566513850f8bb684ad6f7990c329|
2023-11-16 15:30:31|install|libgcc_s1|13.2.1+git7813-150000.1.6.1|x86_64||repo-sle-update|a2c5137517792f9ce0984fd6c6d5ef83465dbe7453cc095484a249ae2260264d|
2023-11-16 15:30:31|install|libgomp1|13.2.1+git7813-150000.1.6.1|x86_64||repo-sle-update|cdf6bd5e0bc42ea655efd85d8efc9536b04224975997ee8bf535847ca1a89c90|
2023-11-16 15:30:31|install|libitm1|13.2.1+git7813-150000.1.6.1|x86_64||repo-sle-update|74f54402ce34731883e49be516f5574fd18bd4476c8c264073cf7e1aecf393ec|
2023-11-16 15:30:32|install|libstdc++6|13.2.1+git7813-150000.1.6.1|x86_64||repo-sle-update|a9200cb66263a1c7af89bdb24c54799056fdeeff8dc77d6307a361a1822663e4|
2023-11-16 15:30:32|install|liblsan0|13.2.1+git7813-150000.1.6.1|x86_64||repo-sle-update|6bf88711b48ae3144f0895baf965363f215ae3a244e8f34ff04a6e7cc753336e|
2023-11-16 15:30:32|patch  |openSUSE-SLE-15.5-2023-4458|1|noarch|repo-sle-update|important|security|needed|applied|
2023-11-17 00:00:03|command|root@new-ariel|'/usr/bin/zypper' '--non-interactive' '--terse' 'up' '--no-confirm' '-t' 'patch' '--skip-interactive'|
2023-11-17 00:00:03|install|python3-urllib3|1.25.10-150300.4.9.1|noarch||repo-sle-update|a9f1c027a66e6d078c11044aa427269aad7a612e5848d79d3161778c82a68ba5|
2023-11-17 00:00:03|patch  |openSUSE-SLE-15.5-2023-4467|1|noarch|repo-sle-update|moderate|security|needed|applied|
2023-11-18 00:01:04|command|root@new-ariel|'/usr/bin/zypper' '--non-interactive' '--terse' 'up' '--no-confirm' '-t' 'patch' '--skip-interactive'|
2023-11-18 00:01:06|install|grub2|2.06-150500.29.11.1|x86_64||repo-sle-update|9cbfdb1e0000e0808b9fbae0586799ab054b448328a9ad8cdd6120f0ab4f7ee2|
2023-11-18 00:01:06|install|grub2-i386-pc|2.06-150500.29.11.1|noarch||repo-sle-update|0851d06be61f41d26473998dfb224543b9109b0ea71fd1be90e59b1b20ee242a|
2023-11-18 00:01:06|install|grub2-snapper-plugin|2.06-150500.29.11.1|noarch||repo-sle-update|dbd3906c3033991bbff9ea60a6c371d91b7e587910ee025a12eb9cc9d484f5f7|
2023-11-18 00:01:06|install|grub2-i386-pc-extras|2.06-150500.29.11.1|noarch||repo-sle-update|c5b79c56de1aacfca5f6ee3db7eb0bd38ea2dea8d1655e9b63b37a57430bd693|
2023-11-18 00:01:10|patch  |openSUSE-SLE-15.5-2023-4478|1|noarch|repo-sle-update|moderate|recommended|needed|applied|
Actions
Copy link
 #5

Updated by okurz 5 months ago

  • Subject changed from o3 yielding "502 Bad Gateway" from nginx 2023-11-19, why was the config overwritten? to o3 yielding "502 Bad Gateway" from nginx 2023-11-19, why was the config overwritten? size:M
  • Description updated (diff)
  • Status changed from New to Workable
Actions
Copy link
 #6

Updated by tinita 5 months ago ยท Edited

  • Status changed from Workable to In Progress
  • Assignee set to tinita

The overwritten files are declared as noreplace:

%config(noreplace) %{_sysconfdir}/nginx/vhosts.d/openqa-locations.inc
%config(noreplace) %{_sysconfdir}/nginx/vhosts.d/openqa-upstreams.inc

But having a closer look at /var/log/zypp/history I found the moment when the files were moved::

2023-11-14 13:09:18|command|root@new-ariel|'zypper' '-n' 'dup' '--replacefiles' '--auto-agree-with-licenses' '--download-in-advance'|
2023-11-14 13:09:19|install|openQA-common|4.6.1699113915.9e4ee2f-lp155.6161.1|x86_64||devel_openQA|aece3cf4f117e7001f22b1bdb0f523dfed2b119d87e4c7c808d9edb7e8cf24dc|
2023-11-14 13:09:19|install|openQA-client|4.6.1699113915.9e4ee2f-lp155.6161.1|x86_64||devel_openQA|09c2243f62a548de971fca8c01e5994945b1ef85ff134317324c32c1d48b0f5f|
2023-11-14 13:09:25|install|openQA|4.6.1699113915.9e4ee2f-lp155.6161.1|x86_64||devel_openQA|207dd52467eabdf3265bc4d1a0062ad3786596270360dcb15f76965e89896832|
2023-11-14 13:09:26|install|openQA-local-db|4.6.1699113915.9e4ee2f-lp155.6161.1|x86_64||devel_openQA|7cbfc5b8a2844f4aa27d4b3611a86b82d99ef1896941a5da047afc2c980759f2|
2023-11-14 13:09:26|install|openQA-munin|4.6.1699113915.9e4ee2f-lp155.6161.1|x86_64||devel_openQA|725cbb93a877ac73ba81433de156a930cae70aba47c156fe0626072bcfde0a72|
2023-11-14 13:09:27|install|openQA-continuous-update|4.6.1699113915.9e4ee2f-lp155.6161.1|x86_64||devel_openQA|1331fcd48b9064baf5eea66635da54bace60d41042889f61090a3a94831b0575|
2023-11-14 13:09:28|install|openQA-auto-update|4.6.1699113915.9e4ee2f-lp155.6161.1|x86_64||devel_openQA|a11a6cb12d7ea185a30b51ee065fc16447e99468536c31a0fa10dd404be54083|
2023-11-14 13:14:42|command|root@new-ariel|'zypper' '-n' 'dup' '--replacefiles' '--auto-agree-with-licenses' '--download-in-advance'|
# ------------ Downgrade to very old version from 2022 happening here:
2023-11-14 13:14:43|install|openQA-common|4.6.1667810206.2bf912d-bp155.1.94|noarch||repo-oss|eb3a4cc0f2a59cb0223f9666f50be3729fe70fcc9a83fbb5714c10a86e9d365b|
2023-11-14 13:14:43|install|openQA-client|4.6.1667810206.2bf912d-bp155.1.94|noarch||repo-oss|e992ff1f555fb3b07f13f7186b76cbe08062ddc31bdde65d08a4187bff640dcd|
# 2023-11-14 13:14:48 openQA-4.6.1667810206.2bf912d-bp155.1.94.noarch.rpm installed ok
# Additional rpm output:
# warning: /etc/apparmor.d/usr.share.openqa.script.openqa saved as /etc/apparmor.d/usr.share.openqa.script.openqa.rpmsave
# warning: /etc/openqa/openqa.ini created as /etc/openqa/openqa.ini.rpmnew
# warning: /etc/nginx/vhosts.d/openqa-upstreams.inc saved as /etc/nginx/vhosts.d/openqa-upstreams.inc.rpmsave
# warning: /etc/nginx/vhosts.d/openqa-locations.inc saved as /etc/nginx/vhosts.d/openqa-locations.inc.rpmsave

So the openqa-upstreams.inc and openqa-locations.inc have been overwritten indeed (or rather moved to rpmsave and with the later upgrade the unchanged files from the package were added again)
I think it happened because of the downgrade to that old version that didn't even have those files in our package.

Actions
Copy link
 #7

Updated by tinita 5 months ago

  • Status changed from In Progress to Feedback

I think we might be fine, as long as we don't get such an accidental downgrade anymore.
I can still do a a check by committing a little change to one of the files and check that it does not overwrite.

Actions
Copy link
 #8

Updated by tinita 5 months ago

  • Related to action #150908: o3 "Unable to fetch build results" and "Internal server error" on some pages size:M added
Actions
Copy link
 #9

Updated by tinita 5 months ago

  • Status changed from Feedback to Resolved

Nobody requested that I do this extra check, so I'll resolve it :)

Actions
Copy link

Also available in: Atom PDF