action #131024
closedcoordination #110833: [saga][epic] Scale up: openQA can handle a schedule of 100k jobs with 1k worker instances
coordination #108209: [epic] Reduce load on OSD
Ensure both nginx+apache are properly covered in packages+testing+documentation size:S
Description
Motivation¶
By default we use apache, for o3 we use nginx. Before we recommend to use nginx more we should ensure that nginx is properly tested as part of our various tests. Maybe nginx is already tested as part of the container setup including our config? See https://github.com/os-autoinst/openQA/blob/master/container/webui/docker-compose.yaml#L144 and the use of the related config(s) files
Acceptance criteria¶
- AC1: Our apache+nginx config within github.com/os-autoinst/openQA/ are tested as part of automated tests
- AC2: Both apache+nginx configs are deployed from openSUSE packages
- AC3: Both apache+nginx are covered in openQA installation documentation
Suggestions¶
- Crosscheck the docker-compose based CI test that uses nginx
- Take a look into Makefile and dist/rpm/openQA.spec for mentions of "apache"
- Extend installation and packaging instructions to cover both apache and nginx. Likely we can just install both apache and nginx config unconditionally as they don't conflict. Do not create just "openQA-apache" and "openQA-nginx" because "openQA" already contains /etc/apache2/ so just extend that with nginx files?
- In dist/rpm/openQA.spec the "single-instance" package requires apache. We do not necessarily need to supply both nginx and apache for that so we should likely be ok to just keep apache in there as is
- We want to support both apache+nginx but it's ok if they are only supported/tested/used for specific use cases
- Provide nginx config using an openQA subpackage (that one could recommend nginx then)
Rollback steps¶
- Remove zypper lock on o3
Related¶
- #130477 for generalizing systemd files regarding the webserver
Updated by okurz 11 months ago
- Copied from action #130477: [O3]http connection to O3 repo is broken sporadically in virtualization tests, likely due to systemd dependencies on apache/nginx size:M added
Updated by okurz 10 months ago
- Description updated (diff)
https://github.com/os-autoinst/openQA/pull/5231 broke openqa.opensuse.org with
Jul 02 03:30:43 ariel systemd[1]: Starting The nginx HTTP and reverse proxy server...
Jul 02 03:30:43 ariel nginx[1806]: nginx: [emerg] duplicate upstream "webui" in /etc/nginx/vhosts.d/openqa.conf:3
Jul 02 03:30:43 ariel nginx[1806]: nginx: configuration file /etc/nginx/nginx.conf test failed
And when removing the file conf.d/openqa.conf this yields 403 Forbidden. For now I reverted to the old file conf.d and removed the vhosts file. To prevent an automatic upgrade to break this again I added a zypper lock zypper al -m "https://progress.opensuse.org/issues/131024" openQA*
Updated by tinita 10 months ago
- Related to action #132200: openQA is not accessible added
Updated by tinita 10 months ago
Also https://openqa.opensuse.org/snapshot-changes/opensuse/Tumbleweed/ was not working.
I needed to start the service explicitly: systemctl start factory-package-news-web.service
Updated by okurz 10 months ago
- Related to action #132218: Conduct lessons learned for "openQA is not accessible" on 2023-07-02 added
Updated by okurz 10 months ago
As brainstormed in #132218 please adhere to the following suggestions:
- Come up with a way to structure the config so that there is a file from the package which no admin does not need to manually change anyway and have the instance specific configuration in another layer. Research RPM config handling and test locally, e.g. with a local container, build package locally with
osc buildand install manually within the test environment with manual rpm calls -> to be covered in #131024 - Explicitly test on o3 before merge or just afterwards closely monitor -> to be covered in #131024
- Find out if we should reload nginx (and apache) on config file updates from package installations.
Updated by openqa_review 10 months ago
- Due date set to 2023-07-20
Setting due date based on mean cycle time of SUSE QE Tools
Updated by dheidler 10 months ago
add nginx option to bootstrap test: https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/17373
Updated by dheidler 10 months ago
schedule bootstrap test: https://github.com/os-autoinst/opensuse-jobgroups/pull/347
Updated by livdywan 10 months ago
- Blocks action #129487: high response times on osd - Limit the number of concurrent job upload handling on webUI side. Can we use a semaphore or lock using the database? size:M added
Updated by dheidler 10 months ago
- Status changed from In Progress to Blocked
Waiting for changes to reach Factory until we can:
schedule bootstrap test: https://github.com/os-autoinst/opensuse-jobgroups/pull/347
Updated by okurz 10 months ago
that would be https://github.com/os-autoinst/openQA/pull/5240
Updated by jbaier_cz 10 months ago
I am afraid that the configure-web-proxy --proxy=nginx does some stuff which makes nginx not happy: see https://openqa.opensuse.org/tests/3422958#step/openqa_webui/19
Updated by dheidler 10 months ago
get nginx logs in openQA-in-openQA test: https://github.com/os-autoinst/os-autoinst-distri-openQA/pull/133
Updated by okurz 10 months ago
https://github.com/os-autoinst/os-autoinst-distri-openQA/pull/133 merged. What else is missing?
Updated by dheidler 9 months ago
With https://openqa.opensuse.org/tests/3440059#step/openqa_webui/17 looking good only https://github.com/os-autoinst/os-autoinst-distri-openQA/pull/134 would be missing.
Updated by dheidler 9 months ago
Ah - seems that https://openqa.opensuse.org/tests/3439538#step/openqa_bootstrap/14 still fails due to fetchneedles using a group that I would guess is installed by apache.
Updated by dheidler 9 months ago
fix fetchneedles for installations without apache2:
https://github.com/os-autoinst/openQA/pull/5252
Updated by openqa_review 9 months ago
- Due date set to 2023-08-08
Setting due date based on mean cycle time of SUSE QE Tools
Updated by okurz 9 months ago
https://github.com/os-autoinst/os-autoinst-distri-openQA/actions/runs/5647049076/job/15296319523?pr=134 blocking https://github.com/os-autoinst/os-autoinst-distri-openQA/pull/134 in https://openqa.opensuse.org/tests/3449140#step/dashboard/1 due to https://progress.opensuse.org/issues/?query_id=230 . Please look into what workaround is necessary to mitigate that for now
Updated by dheidler 9 months ago
- Status changed from In Progress to Feedback
The referenced issue is blocked by the wheels epic which is also blocked on subtasks.
Which would mean that this ticked would be blocked for an indefinite time frame.
As the fail is not related to the changes and the changed sections actually pass in the testsuite,
we will merge this for now.
Updated by okurz 9 months ago
so in the end https://github.com/os-autoinst/os-autoinst-distri-openQA/pull/134 was merged and I also resolved #133301 meaning that now https://openqa.opensuse.org/group_overview/24 looks stable enough and includes nginx tests. IMHO you can resolve unless we see problems over the night.
Updated by livdywan 9 months ago
- Status changed from Feedback to Resolved
okurz wrote:
so in the end https://github.com/os-autoinst/os-autoinst-distri-openQA/pull/134 was merged and I also resolved #133301 meaning that now https://openqa.opensuse.org/group_overview/24 looks stable enough and includes nginx tests. IMHO you can resolve unless we see problems over the night.
👍🏾