Project

General

Profile

Actions
Copy link

action #42233

closed

Production-grade jenkins.qa.suse.de for QA SLE (EMEA)

Added by okurz about 6 years ago. Updated over 4 years ago.

Status:
Resolved
Priority:
Low
Assignee:
okurz
Category:
-
Target version:
QA (public) - future
Start date:
2016-02-01
Due date:
% Done:

0%

Estimated time:

Description

User Story

As QA SLE (EMEA) we want to have a production-grade jenkins server to be able to use a CI server for usual purposes in a reliable manner.

Acceptance criteria

  • AC1: jenkins.qa.suse.de is a properly maintained jenkins instance with configuration in code, backup, instructions, maintainers

Suggestions

  • Ensure https://jenkins.qa.suse.de allows login with redirects
    • Requires valid CA certificate
  • Backup
  • Something like salt and/or container recipes to be able to recreate it?
  • update plan

Related issues 1 (0 open — 1 closed)

Copied from openQA Infrastructure (public) - action #10516: [functional][u] Use jenkins as extension for openQAResolvedokurz2016-02-01

Actions
Actions
Copy link
 #1

Updated by okurz about 6 years ago

  • Copied from action #10516: [functional][u] Use jenkins as extension for openQA added
Actions
Copy link
 #2

Updated by okurz about 6 years ago

  • Assignee set to okurz
Actions
Copy link
 #3

Updated by nicksinger about 6 years ago

  • Description updated (diff)
Actions
Copy link
 #4

Updated by nicksinger about 6 years ago

I've changed proxy_pass in the nginx.conf (see comment in file) and would recommend you to pull out this location-block into a separate file in ./vhosts.d/. It keeps the config more clean and handy :)

If you want a quick way of versioning your configs before you started salting the setup, I can recommend https://news.ycombinator.com/item?id=11071754 as way to go. I use it on qanet and it's definitely better then nothing.

Actions
Copy link
 #5

Updated by okurz about 6 years ago

  • Status changed from In Progress to Workable
  • Assignee deleted (okurz)

awesome, let's pick it up again after some time of thinking, not working on this right now :)

Actions
Copy link
 #6

Updated by coolo about 6 years ago

  • Project changed from openQA Tests (public) to openQA Infrastructure (public)
  • Category deleted (Infrastructure)
Actions
Copy link
 #7

Updated by okurz about 6 years ago

One observation, one problem: http://jenkins.qa.suse.de is sometimes rely slow to respond or nginx even throws an error. And second we could request an SSL certificate and provide https://jenkins.qa.suse.de

Actions
Copy link
 #8

Updated by okurz about 6 years ago

  • Status changed from Workable to Feedback
  • Assignee set to okurz

Looked into the slow responses with nsinger.

First, there are quite some error messages about HPC, e.g. visible on http://jenkins.qa.suse.de/log/all . I ping asmorodskyi in #testing if he can have a look.

In /etc/nginx/nginx.conf we defined better logging with:

http { 
    …
    log_format  mine  '$remote_addr - $remote_user [$time_local] "$request" "$request_time" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';
    …
    server {
        …
        access_log  /var/log/nginx/access_mine.log  mine;

In one shell in a screen session I follow logs with multitail -f /var/log/nginx/{access,error}.log /var/log/jenkins/jenkins.log

and extra sudo tail -f /var/log/nginx/access_mine.log | grep 'GET / ' which reveals

10.163.8.207 - - [27/Nov/2018:11:33:55 +0100] "GET / HTTP/1.1" "3.697" 200 8497 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0" "-"
10.163.8.207 - - [27/Nov/2018:11:34:09 +0100] "GET / HTTP/1.1" "2.565" 200 8499 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0" "-"
10.163.8.207 - - [27/Nov/2018:11:34:22 +0100] "GET / HTTP/1.1" "2.677" 200 8498 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0" "-"
10.163.8.207 - - [27/Nov/2018:11:34:36 +0100] "GET / HTTP/1.1" "2.693" 200 8499 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0" "-"
10.163.8.207 - - [27/Nov/2018:11:34:49 +0100] "GET / HTTP/1.1" "2.583" 200 8499 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0" "-"

where the number before "200" seems to be according to http://nginx.org/en/docs/http/ngx_http_log_module.html#access_log the $request_time which is more or less which we observe by simple counting, e.g. taking around 4 seconds to load the main dashboard. This is not fast but also not super-slow. We have also observed the intial request to take 1.5m (!).

Next step: Observe again if this logs reveals the long times or if I need to add more logging to jenkins.

Actions
Copy link
 #9

Updated by okurz about 6 years ago 

10.163.8.207 - - [27/Nov/2018:15:49:56 +0100] "GET / HTTP/1.1" "63.730" 200 6375 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0" "-"
10.163.8.207 - - [27/Nov/2018:15:50:14 +0100] "GET / HTTP/1.1" "6.832" 200 6376 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0" "-"
10.163.8.207 - - [27/Nov/2018:15:50:30 +0100] "GET / HTTP/1.1" "4.880" 200 6375 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0" "-"
10.163.8.207 - - [27/Nov/2018:15:50:47 +0100] "GET / HTTP/1.1" "4.875" 200 6375 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0" "-"

so if I try to load the page after some time the response time as recorded by nginx is 64s and then goes down to 4-6s again. While at the same time jenkins provides no error message or similar in /var/log/jenkins/jenkins.log.

Actions
Copy link
 #10

Updated by okurz about 6 years ago

As a workaround I am trying the following in /etc/crontab:

# Regurlarly access both the jenkins internal webpage as well as over the
# nginx proxy as a workaround to make sure it responds fast, hopefully.
# See https://progress.opensuse.org/issues/42233
*/30 * * * *    root curl -s http://localhost:8080 &>/dev/null ; curl -s http://localhost &>/dev/null
Actions
Copy link
 #11

Updated by okurz almost 6 years ago

  • Target version changed from Milestone 23 to future
Actions
Copy link
 #12

Updated by okurz over 5 years ago

  • Assignee changed from okurz to mgriessmeier

Move to new QSF-u PO after I moved to the "tools"-team. I mainly checked the subject line so in individual instances you might not agree to take it over completely into QSF-u. Feel free to discuss with me or reassign to me or someone else in this case. Thanks.

Actions
Copy link
 #13

Updated by SLindoMansilla over 4 years ago

  • Subject changed from [functional][u] Production-grade jenkins.qa.suse.de for QA SLE (EMEA) to [functional] Production-grade jenkins.qa.suse.de for QA SLE (EMEA)
  • Assignee changed from mgriessmeier to okurz

QSF-U doesn't care about the jenkins instance.
Please take over or close :)

Actions
Copy link
 #14

Updated by okurz over 4 years ago

  • Subject changed from [functional] Production-grade jenkins.qa.suse.de for QA SLE (EMEA) to Production-grade jenkins.qa.suse.de for QA SLE (EMEA)
Actions
Copy link
 #15

Updated by okurz over 4 years ago

  • Status changed from Feedback to Resolved

https://gitlab.suse.de/okurz/jenkins-server-salt/ has an autoyast profile, salt setup config including the main jenkins configuration file. The repo can also be moved to https://gitlab.suse.de/qa-sle/jenkins-server-salt/ . For this however I am currently missing the necessary permissions. https://gitlab.suse.de/qa-sle/backup-server-salt/-/commit/72d1d98ed7914fbc12e816da51d52fc8d14d8b2d has the necessary backup config and an initial backup was successfully conducted from jenkins.qa.suse.de to backup.qa.suse.de:/home/rsnapshot/

Actions
Copy link
 #16

Updated by okurz over 4 years ago

Actions
Copy link

Also available in: Atom PDF