action #131024
closed
coordination #110833: [saga][epic] Scale up: openQA can handle a schedule of 100k jobs with 1k worker instances
coordination #108209: [epic] Reduce load on OSD
Ensure both nginx+apache are properly covered in packages+testing+documentation size:S
Added by okurz 11 months ago.
Updated 9 months ago.
Category:
Feature requests
Description
Motivation¶
By default we use apache, for o3 we use nginx. Before we recommend to use nginx more we should ensure that nginx is properly tested as part of our various tests. Maybe nginx is already tested as part of the container setup including our config? See https://github.com/os-autoinst/openQA/blob/master/container/webui/docker-compose.yaml#L144 and the use of the related config(s) files
Acceptance criteria¶
- AC1: Our apache+nginx config within github.com/os-autoinst/openQA/ are tested as part of automated tests
- AC2: Both apache+nginx configs are deployed from openSUSE packages
- AC3: Both apache+nginx are covered in openQA installation documentation
Suggestions¶
- Crosscheck the docker-compose based CI test that uses nginx
- Take a look into Makefile and dist/rpm/openQA.spec for mentions of "apache"
- Extend installation and packaging instructions to cover both apache and nginx. Likely we can just install both apache and nginx config unconditionally as they don't conflict. Do not create just "openQA-apache" and "openQA-nginx" because "openQA" already contains /etc/apache2/ so just extend that with nginx files?
- In dist/rpm/openQA.spec the "single-instance" package requires apache. We do not necessarily need to supply both nginx and apache for that so we should likely be ok to just keep apache in there as is
- We want to support both apache+nginx but it's ok if they are only supported/tested/used for specific use cases
- Provide nginx config using an openQA subpackage (that one could recommend nginx then)
Rollback steps¶
Related¶
- #130477 for generalizing systemd files regarding the webserver
- Copied from action #130477: [O3]http connection to O3 repo is broken sporadically in virtualization tests, likely due to systemd dependencies on apache/nginx size:M added
- Description updated (diff)
- Status changed from New to Blocked
- Assignee set to okurz
- Status changed from Blocked to New
- Assignee deleted (
okurz)
As stated by mkittler they see that we should go ahead with this ticket regardless of the state in #130477 so setting back to "New"
- Subject changed from Ensure both nginx+apache are properly covered in packages+testing+documentation to Ensure both nginx+apache are properly covered in packages+testing+documentation size:S
- Description updated (diff)
- Status changed from New to Workable
- Status changed from Workable to In Progress
Nginx is covered by the docker-compose test.
The config is a bit different from the nginx config used in the rpm package, though.
- Status changed from In Progress to Feedback
- Description updated (diff)
https://github.com/os-autoinst/openQA/pull/5231 broke openqa.opensuse.org with
Jul 02 03:30:43 ariel systemd[1]: Starting The nginx HTTP and reverse proxy server...
Jul 02 03:30:43 ariel nginx[1806]: nginx: [emerg] duplicate upstream "webui" in /etc/nginx/vhosts.d/openqa.conf:3
Jul 02 03:30:43 ariel nginx[1806]: nginx: configuration file /etc/nginx/nginx.conf test failed
And when removing the file conf.d/openqa.conf this yields 403 Forbidden. For now I reverted to the old file conf.d and removed the vhosts file. To prevent an automatic upgrade to break this again I added a zypper lock zypper al -m "https://progress.opensuse.org/issues/131024" openQA*
- Related to action #132218: Conduct lessons learned for "openQA is not accessible" on 2023-07-02 added
As brainstormed in #132218 please adhere to the following suggestions:
- Come up with a way to structure the config so that there is a file from the package which no admin does not need to manually change anyway and have the instance specific configuration in another layer. Research RPM config handling and test locally, e.g. with a local container, build package locally with
osc build and install manually within the test environment with manual rpm calls -> to be covered in #131024
- Explicitly test on o3 before merge or just afterwards closely monitor -> to be covered in #131024
- Find out if we should reload nginx (and apache) on config file updates from package installations.
- Status changed from Feedback to In Progress
Dominik:
- PR is back to draft
- preparing the setup and bootstrap scripts to allow testing the nginx config as well
- Due date set to 2023-07-20
Setting due date based on mean cycle time of SUSE QE Tools
- Blocks action #129487: high response times on osd - Limit the number of concurrent job upload handling on webUI side. Can we use a semaphore or lock using the database? size:M added
on o3:
- package log removed
- updated openQA rpm
- switched to new nginx config
- Status changed from In Progress to Blocked
Please use "Blocked" only when you have an URL that people can follow to see what we are waiting for. I think you can reference the build throttled Jenkins build directly
- Status changed from Blocked to Workable
- Priority changed from Normal to High
- Status changed from Workable to Feedback
Pushed some fixes for the bootstrap script.
Waiting for them to hit factory.
- Due date deleted (
2023-07-20)
- Status changed from Feedback to Blocked
- Status changed from Blocked to In Progress
o3 is generally up, please try now again.
- Due date set to 2023-08-08
Setting due date based on mean cycle time of SUSE QE Tools
I think you meant to reference #106922
- Status changed from In Progress to Feedback
The referenced issue is blocked by the wheels epic which is also blocked on subtasks.
Which would mean that this ticked would be blocked for an indefinite time frame.
As the fail is not related to the changes and the changed sections actually pass in the testsuite,
we will merge this for now.
As the fail is not related to the changes and the changed sections actually pass in the testsuite,
we will merge this for now.
See #133301
- Status changed from Feedback to Resolved
- Due date deleted (
2023-08-08)
Also available in: Atom
PDF