action #42233

[functional][u] Production-grade jenkins.qa.suse.de for QA SLE (EMEA)

Added by okurz over 1 year ago. Updated 9 months ago.

Status:FeedbackStart date:01/02/2016
Priority:LowDue date:
Assignee:mgriessmeier% Done:

0%

Category:-
Target version:QA - future
Duration:

Description

User Story

As QA SLE (EMEA) we want to have a production-grade jenkins server to be able to use a CI server for usual purposes in a reliable manner.

Acceptance criteria

  • AC1: jenkins.qa.suse.de is a properly maintained jenkins instance with configuration in code, backup, instructions, maintainers

Suggestions

  • Ensure https://jenkins.qa.suse.de allows login with redirects
    • Requires valid CA certificate
  • Backup
  • Something like salt and/or container recipes to be able to recreate it?
  • update plan

Related issues

Copied from openQA Infrastructure - action #10516: [functional][u] Use jenkins as extension for openQA New 01/02/2016

History

#1 Updated by okurz over 1 year ago

  • Copied from action #10516: [functional][u] Use jenkins as extension for openQA added

#2 Updated by okurz over 1 year ago

  • Assignee set to okurz

#3 Updated by nicksinger over 1 year ago

  • Description updated (diff)

#4 Updated by nicksinger over 1 year ago

I've changed proxy_pass in the nginx.conf (see comment in file) and would recommend you to pull out this location-block into a separate file in ./vhosts.d/. It keeps the config more clean and handy :)

If you want a quick way of versioning your configs before you started salting the setup, I can recommend https://news.ycombinator.com/item?id=11071754 as way to go. I use it on qanet and it's definitely better then nothing.

#5 Updated by okurz over 1 year ago

  • Status changed from In Progress to Workable
  • Assignee deleted (okurz)

awesome, let's pick it up again after some time of thinking, not working on this right now :)

#6 Updated by coolo over 1 year ago

  • Project changed from openQA Tests to openQA Infrastructure
  • Category deleted (Infrastructure)

#7 Updated by okurz over 1 year ago

One observation, one problem: http://jenkins.qa.suse.de is sometimes rely slow to respond or nginx even throws an error. And second we could request an SSL certificate and provide https://jenkins.qa.suse.de

#8 Updated by okurz about 1 year ago

  • Status changed from Workable to Feedback
  • Assignee set to okurz

Looked into the slow responses with nsinger.

First, there are quite some error messages about HPC, e.g. visible on http://jenkins.qa.suse.de/log/all . I ping asmorodskyi in #testing if he can have a look.

In /etc/nginx/nginx.conf we defined better logging with:

http { 
    …
    log_format  mine  '$remote_addr - $remote_user [$time_local] "$request" "$request_time" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';
    …
    server {
        …
        access_log  /var/log/nginx/access_mine.log  mine;

In one shell in a screen session I follow logs with multitail -f /var/log/nginx/{access,error}.log /var/log/jenkins/jenkins.log

and extra sudo tail -f /var/log/nginx/access_mine.log | grep 'GET / ' which reveals

10.163.8.207 - - [27/Nov/2018:11:33:55 +0100] "GET / HTTP/1.1" "3.697" 200 8497 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0" "-"
10.163.8.207 - - [27/Nov/2018:11:34:09 +0100] "GET / HTTP/1.1" "2.565" 200 8499 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0" "-"
10.163.8.207 - - [27/Nov/2018:11:34:22 +0100] "GET / HTTP/1.1" "2.677" 200 8498 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0" "-"
10.163.8.207 - - [27/Nov/2018:11:34:36 +0100] "GET / HTTP/1.1" "2.693" 200 8499 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0" "-"
10.163.8.207 - - [27/Nov/2018:11:34:49 +0100] "GET / HTTP/1.1" "2.583" 200 8499 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0" "-"

where the number before "200" seems to be according to http://nginx.org/en/docs/http/ngx_http_log_module.html#access_log the $request_time which is more or less which we observe by simple counting, e.g. taking around 4 seconds to load the main dashboard. This is not fast but also not super-slow. We have also observed the intial request to take 1.5m (!).

Next step: Observe again if this logs reveals the long times or if I need to add more logging to jenkins.

#9 Updated by okurz about 1 year ago

10.163.8.207 - - [27/Nov/2018:15:49:56 +0100] "GET / HTTP/1.1" "63.730" 200 6375 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0" "-"
10.163.8.207 - - [27/Nov/2018:15:50:14 +0100] "GET / HTTP/1.1" "6.832" 200 6376 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0" "-"
10.163.8.207 - - [27/Nov/2018:15:50:30 +0100] "GET / HTTP/1.1" "4.880" 200 6375 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0" "-"
10.163.8.207 - - [27/Nov/2018:15:50:47 +0100] "GET / HTTP/1.1" "4.875" 200 6375 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0" "-"

so if I try to load the page after some time the response time as recorded by nginx is 64s and then goes down to 4-6s again. While at the same time jenkins provides no error message or similar in /var/log/jenkins/jenkins.log.

#10 Updated by okurz about 1 year ago

As a workaround I am trying the following in /etc/crontab:

# Regurlarly access both the jenkins internal webpage as well as over the
# nginx proxy as a workaround to make sure it responds fast, hopefully.
# See https://progress.opensuse.org/issues/42233
*/30 * * * *    root curl -s http://localhost:8080 &>/dev/null ; curl -s http://localhost &>/dev/null

#11 Updated by okurz 12 months ago

  • Target version changed from Milestone 23 to future

#12 Updated by okurz 9 months ago

  • Assignee changed from okurz to mgriessmeier

Move to new QSF-u PO after I moved to the "tools"-team. I mainly checked the subject line so in individual instances you might not agree to take it over completely into QSF-u. Feel free to discuss with me or reassign to me or someone else in this case. Thanks.

Also available in: Atom PDF