action #131024
closedcoordination #110833: [saga][epic] Scale up: openQA can handle a schedule of 100k jobs with 1k worker instances
coordination #108209: [epic] Reduce load on OSD
Ensure both nginx+apache are properly covered in packages+testing+documentation size:S
Description
Motivation¶
By default we use apache, for o3 we use nginx. Before we recommend to use nginx more we should ensure that nginx is properly tested as part of our various tests. Maybe nginx is already tested as part of the container setup including our config? See https://github.com/os-autoinst/openQA/blob/master/container/webui/docker-compose.yaml#L144 and the use of the related config(s) files
Acceptance criteria¶
- AC1: Our apache+nginx config within github.com/os-autoinst/openQA/ are tested as part of automated tests
- AC2: Both apache+nginx configs are deployed from openSUSE packages
- AC3: Both apache+nginx are covered in openQA installation documentation
Suggestions¶
- Crosscheck the docker-compose based CI test that uses nginx
- Take a look into Makefile and dist/rpm/openQA.spec for mentions of "apache"
- Extend installation and packaging instructions to cover both apache and nginx. Likely we can just install both apache and nginx config unconditionally as they don't conflict. Do not create just "openQA-apache" and "openQA-nginx" because "openQA" already contains /etc/apache2/ so just extend that with nginx files?
- In dist/rpm/openQA.spec the "single-instance" package requires apache. We do not necessarily need to supply both nginx and apache for that so we should likely be ok to just keep apache in there as is
- We want to support both apache+nginx but it's ok if they are only supported/tested/used for specific use cases
- Provide nginx config using an openQA subpackage (that one could recommend nginx then)
Rollback steps¶
- Remove zypper lock on o3
Related¶
- #130477 for generalizing systemd files regarding the webserver
Updated by okurz over 1 year ago
- Copied from action #130477: [O3]http connection to O3 repo is broken sporadically in virtualization tests, likely due to systemd dependencies on apache/nginx size:M added
Updated by okurz over 1 year ago
- Status changed from Blocked to New
- Assignee deleted (
okurz)
As stated by mkittler they see that we should go ahead with this ticket regardless of the state in #130477 so setting back to "New"
Updated by dheidler over 1 year ago
- Subject changed from Ensure both nginx+apache are properly covered in packages+testing+documentation to Ensure both nginx+apache are properly covered in packages+testing+documentation size:S
- Description updated (diff)
- Status changed from New to Workable
Updated by dheidler over 1 year ago
Updated by dheidler over 1 year ago
Updated by dheidler over 1 year ago
Nginx is covered by the docker-compose test.
The config is a bit different from the nginx config used in the rpm package, though.
Updated by dheidler over 1 year ago
- Status changed from In Progress to Feedback
Updated by okurz over 1 year ago
- Description updated (diff)
https://github.com/os-autoinst/openQA/pull/5231 broke openqa.opensuse.org with
Jul 02 03:30:43 ariel systemd[1]: Starting The nginx HTTP and reverse proxy server...
Jul 02 03:30:43 ariel nginx[1806]: nginx: [emerg] duplicate upstream "webui" in /etc/nginx/vhosts.d/openqa.conf:3
Jul 02 03:30:43 ariel nginx[1806]: nginx: configuration file /etc/nginx/nginx.conf test failed
And when removing the file conf.d/openqa.conf this yields 403 Forbidden. For now I reverted to the old file conf.d and removed the vhosts file. To prevent an automatic upgrade to break this again I added a zypper lock zypper al -m "https://progress.opensuse.org/issues/131024" openQA*
Updated by tinita over 1 year ago
- Related to action #132200: openQA is not accessible added
Updated by tinita over 1 year ago
Also https://openqa.opensuse.org/snapshot-changes/opensuse/Tumbleweed/ was not working.
I needed to start the service explicitly: systemctl start factory-package-news-web.service
Updated by okurz over 1 year ago
- Related to action #132218: Conduct lessons learned for "openQA is not accessible" on 2023-07-02 added
Updated by okurz over 1 year ago
As brainstormed in #132218 please adhere to the following suggestions:
- Come up with a way to structure the config so that there is a file from the package which no admin does not need to manually change anyway and have the instance specific configuration in another layer. Research RPM config handling and test locally, e.g. with a local container, build package locally with
osc buildand install manually within the test environment with manual rpm calls -> to be covered in #131024 - Explicitly test on o3 before merge or just afterwards closely monitor -> to be covered in #131024
- Find out if we should reload nginx (and apache) on config file updates from package installations.
Updated by livdywan over 1 year ago
- Status changed from Feedback to In Progress
Dominik:
- PR is back to draft
- preparing the setup and bootstrap scripts to allow testing the nginx config as well
Updated by livdywan over 1 year ago
Updated by dheidler over 1 year ago
Updated by dheidler over 1 year ago
Updated by openqa_review over 1 year ago
- Due date set to 2023-07-20
Setting due date based on mean cycle time of SUSE QE Tools
Updated by dheidler over 1 year ago
add nginx option to bootstrap test: https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/17373
Updated by dheidler over 1 year ago
schedule bootstrap test: https://github.com/os-autoinst/opensuse-jobgroups/pull/347
Updated by dheidler over 1 year ago
Updated by dheidler over 1 year ago
Updated by livdywan over 1 year ago
- Blocks action #129487: high response times on osd - Limit the number of concurrent job upload handling on webUI side. Can we use a semaphore or lock using the database? size:M added
Updated by dheidler over 1 year ago
on o3:
- package log removed
- updated openQA rpm
- switched to new nginx config
Updated by dheidler over 1 year ago
- Status changed from In Progress to Blocked
Waiting for changes to reach Factory until we can:
schedule bootstrap test: https://github.com/os-autoinst/opensuse-jobgroups/pull/347
Updated by okurz over 1 year ago
Please use "Blocked" only when you have an URL that people can follow to see what we are waiting for. I think you can reference the build throttled Jenkins build directly
Updated by okurz over 1 year ago
- Status changed from Blocked to Workable
- Priority changed from Normal to High
Updated by dheidler over 1 year ago
- Status changed from Workable to Feedback
Pushed some fixes for the bootstrap script.
Waiting for them to hit factory.
Updated by okurz over 1 year ago
that would be https://github.com/os-autoinst/openQA/pull/5240
Updated by jbaier_cz over 1 year ago
I am afraid that the configure-web-proxy --proxy=nginx does some stuff which makes nginx not happy: see https://openqa.opensuse.org/tests/3422958#step/openqa_webui/19
Updated by dheidler over 1 year ago
Updated by dheidler over 1 year ago
get nginx logs in openQA-in-openQA test: https://github.com/os-autoinst/os-autoinst-distri-openQA/pull/133
Updated by okurz over 1 year ago
https://github.com/os-autoinst/os-autoinst-distri-openQA/pull/133 merged. What else is missing?
Updated by dheidler over 1 year ago
With https://openqa.opensuse.org/tests/3440059#step/openqa_webui/17 looking good only https://github.com/os-autoinst/os-autoinst-distri-openQA/pull/134 would be missing.
Updated by dheidler over 1 year ago
Ah - seems that https://openqa.opensuse.org/tests/3439538#step/openqa_bootstrap/14 still fails due to fetchneedles using a group that I would guess is installed by apache.
Updated by dheidler over 1 year ago
fix fetchneedles for installations without apache2:
https://github.com/os-autoinst/openQA/pull/5252
Updated by okurz over 1 year ago
- Due date deleted (
2023-07-20) - Status changed from Feedback to Blocked
https://github.com/os-autoinst/openQA/pull/5252 merged. I suggest to block this by #132143 for verification
Updated by okurz over 1 year ago
- Status changed from Blocked to In Progress
o3 is generally up, please try now again.
Updated by openqa_review over 1 year ago
- Due date set to 2023-08-08
Setting due date based on mean cycle time of SUSE QE Tools
Updated by okurz over 1 year ago
https://github.com/os-autoinst/os-autoinst-distri-openQA/actions/runs/5647049076/job/15296319523?pr=134 blocking https://github.com/os-autoinst/os-autoinst-distri-openQA/pull/134 in https://openqa.opensuse.org/tests/3449140#step/dashboard/1 due to https://progress.opensuse.org/issues/?query_id=230 . Please look into what workaround is necessary to mitigate that for now
Updated by dheidler over 1 year ago
- Status changed from In Progress to Feedback
The referenced issue is blocked by the wheels epic which is also blocked on subtasks.
Which would mean that this ticked would be blocked for an indefinite time frame.
As the fail is not related to the changes and the changed sections actually pass in the testsuite,
we will merge this for now.
Updated by livdywan over 1 year ago
As the fail is not related to the changes and the changed sections actually pass in the testsuite,
we will merge this for now.
See #133301
Updated by okurz over 1 year ago
so in the end https://github.com/os-autoinst/os-autoinst-distri-openQA/pull/134 was merged and I also resolved #133301 meaning that now https://openqa.opensuse.org/group_overview/24 looks stable enough and includes nginx tests. IMHO you can resolve unless we see problems over the night.
Updated by livdywan over 1 year ago
- Status changed from Feedback to Resolved
okurz wrote:
so in the end https://github.com/os-autoinst/os-autoinst-distri-openQA/pull/134 was merged and I also resolved #133301 meaning that now https://openqa.opensuse.org/group_overview/24 looks stable enough and includes nginx tests. IMHO you can resolve unless we see problems over the night.
👍🏾