action #42233
closedProduction-grade jenkins.qa.suse.de for QA SLE (EMEA)
Added by okurz over 5 years ago. Updated almost 4 years ago.
0%
Description
User Story¶
As QA SLE (EMEA) we want to have a production-grade jenkins server to be able to use a CI server for usual purposes in a reliable manner.
Acceptance criteria¶
- AC1: jenkins.qa.suse.de is a properly maintained jenkins instance with configuration in code, backup, instructions, maintainers
Suggestions¶
- Ensure https://jenkins.qa.suse.de allows login with redirects
- Requires valid CA certificate
- Backup
- Something like salt and/or container recipes to be able to recreate it?
- update plan
Updated by okurz over 5 years ago
- Copied from action #10516: [functional][u] Use jenkins as extension for openQA added
Updated by nicksinger over 5 years ago
I've changed proxy_pass in the nginx.conf (see comment in file) and would recommend you to pull out this location-block into a separate file in ./vhosts.d/. It keeps the config more clean and handy :)
If you want a quick way of versioning your configs before you started salting the setup, I can recommend https://news.ycombinator.com/item?id=11071754 as way to go. I use it on qanet and it's definitely better then nothing.
Updated by okurz over 5 years ago
- Status changed from In Progress to Workable
- Assignee deleted (
okurz)
awesome, let's pick it up again after some time of thinking, not working on this right now :)
Updated by coolo over 5 years ago
- Project changed from openQA Tests to openQA Infrastructure
- Category deleted (
Infrastructure)
Updated by okurz over 5 years ago
One observation, one problem: http://jenkins.qa.suse.de is sometimes rely slow to respond or nginx even throws an error. And second we could request an SSL certificate and provide https://jenkins.qa.suse.de
Updated by okurz over 5 years ago
- Status changed from Workable to Feedback
- Assignee set to okurz
Looked into the slow responses with nsinger.
First, there are quite some error messages about HPC, e.g. visible on http://jenkins.qa.suse.de/log/all . I ping asmorodskyi in #testing if he can have a look.
In /etc/nginx/nginx.conf we defined better logging with:
http {
…
log_format mine '$remote_addr - $remote_user [$time_local] "$request" "$request_time" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
…
server {
…
access_log /var/log/nginx/access_mine.log mine;
In one shell in a screen session I follow logs with multitail -f /var/log/nginx/{access,error}.log /var/log/jenkins/jenkins.log
and extra sudo tail -f /var/log/nginx/access_mine.log | grep 'GET / ' which reveals
10.163.8.207 - - [27/Nov/2018:11:33:55 +0100] "GET / HTTP/1.1" "3.697" 200 8497 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0" "-"
10.163.8.207 - - [27/Nov/2018:11:34:09 +0100] "GET / HTTP/1.1" "2.565" 200 8499 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0" "-"
10.163.8.207 - - [27/Nov/2018:11:34:22 +0100] "GET / HTTP/1.1" "2.677" 200 8498 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0" "-"
10.163.8.207 - - [27/Nov/2018:11:34:36 +0100] "GET / HTTP/1.1" "2.693" 200 8499 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0" "-"
10.163.8.207 - - [27/Nov/2018:11:34:49 +0100] "GET / HTTP/1.1" "2.583" 200 8499 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0" "-"
where the number before "200" seems to be according to http://nginx.org/en/docs/http/ngx_http_log_module.html#access_log the $request_time which is more or less which we observe by simple counting, e.g. taking around 4 seconds to load the main dashboard. This is not fast but also not super-slow. We have also observed the intial request to take 1.5m (!).
Next step: Observe again if this logs reveals the long times or if I need to add more logging to jenkins.
Updated by okurz over 5 years ago
10.163.8.207 - - [27/Nov/2018:15:49:56 +0100] "GET / HTTP/1.1" "63.730" 200 6375 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0" "-"
10.163.8.207 - - [27/Nov/2018:15:50:14 +0100] "GET / HTTP/1.1" "6.832" 200 6376 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0" "-"
10.163.8.207 - - [27/Nov/2018:15:50:30 +0100] "GET / HTTP/1.1" "4.880" 200 6375 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0" "-"
10.163.8.207 - - [27/Nov/2018:15:50:47 +0100] "GET / HTTP/1.1" "4.875" 200 6375 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0" "-"
so if I try to load the page after some time the response time as recorded by nginx is 64s and then goes down to 4-6s again. While at the same time jenkins provides no error message or similar in /var/log/jenkins/jenkins.log.
Updated by okurz over 5 years ago
As a workaround I am trying the following in /etc/crontab:
# Regurlarly access both the jenkins internal webpage as well as over the
# nginx proxy as a workaround to make sure it responds fast, hopefully.
# See https://progress.opensuse.org/issues/42233
*/30 * * * * root curl -s http://localhost:8080 &>/dev/null ; curl -s http://localhost &>/dev/null
Updated by okurz about 5 years ago
- Target version changed from Milestone 23 to future
Updated by okurz almost 5 years ago
- Assignee changed from okurz to mgriessmeier
Move to new QSF-u PO after I moved to the "tools"-team. I mainly checked the subject line so in individual instances you might not agree to take it over completely into QSF-u. Feel free to discuss with me or reassign to me or someone else in this case. Thanks.
Updated by SLindoMansilla about 4 years ago
- Subject changed from [functional][u] Production-grade jenkins.qa.suse.de for QA SLE (EMEA) to [functional] Production-grade jenkins.qa.suse.de for QA SLE (EMEA)
- Assignee changed from mgriessmeier to okurz
QSF-U doesn't care about the jenkins instance.
Please take over or close :)
Updated by okurz about 4 years ago
- Subject changed from [functional] Production-grade jenkins.qa.suse.de for QA SLE (EMEA) to Production-grade jenkins.qa.suse.de for QA SLE (EMEA)
Updated by okurz almost 4 years ago
- Status changed from Feedback to Resolved
https://gitlab.suse.de/okurz/jenkins-server-salt/ has an autoyast profile, salt setup config including the main jenkins configuration file. The repo can also be moved to https://gitlab.suse.de/qa-sle/jenkins-server-salt/ . For this however I am currently missing the necessary permissions. https://gitlab.suse.de/qa-sle/backup-server-salt/-/commit/72d1d98ed7914fbc12e816da51d52fc8d14d8b2d has the necessary backup config and an initial backup was successfully conducted from jenkins.qa.suse.de to backup.qa.suse.de:/home/rsnapshot/
Updated by okurz almost 4 years ago
moved gitlab project to https://gitlab.suse.de/qa-sle/jenkins-server-salt