action #159654: high response times on osd - nginx properly monitored in grafana size:S - openQA Project (public) - openSUSE Project Management Tool

Actions

Copy link

action #159654

closed

coordination #110833: [saga][epic] Scale up: openQA can handle a schedule of 100k jobs with 1k worker instances

coordination #108209: [epic] Reduce load on OSD

high response times on osd - nginx properly monitored in grafana size:S

Added by okurz 8 months ago. Updated 7 months ago.

Status:

Resolved

Priority:

Normal

Assignee:

jbaier_cz

Category:

Feature requests

Target version:

Ready

Start date:

2024-04-26

Due date:

% Done:

Estimated time:

Tags:

infra

Description

Motivation¶

Apache in prefork mode uses a lot of resources to provide mediocre performance. We have nginx on OSD deployed with #159651. Now let's make sure we have it properly monitored as the web proxy is critical for the overall performance and user experience

Acceptance criteria¶

AC1: Nginx on OSD is properly monitored in grafana
AC2: No alerts about apache being down

Suggestions¶

Follow #159651 for the actual nginx deployment
Add changes to salt-states-openqa including monitoring: we have multiple panels regarding apache that need to be adapted for nginx as applicable
Ensure that we have no alerts regarding "oh no, apache is down" ;)

Out of scope¶

No need for any additional metrics, just feature-parity with what we have regarding apache, e.g. response sizes, response codes, response times

Files

20240611_14h33m35s_grim.png (83.4 KB) 20240611_14h33m35s_grim.png

jbaier_cz, 2024-06-11 12:33

Related issues 2 (0 open — 2 closed)

Actions

Copy link

Updated by okurz 8 months ago

Copied from action #159651: high response times on osd - nginx with enabled rate limiting features size:S added

Actions

Copy link

Updated by jbaier_cz 8 months ago

Subject changed from high response times on osd - nginx properly monitored in grafana to high response times on osd - nginx properly monitored in grafana size:S
Description updated (diff)
Status changed from New to Workable

Actions

Copy link

Updated by okurz 8 months ago

Target version changed from Tools - Next to Ready

Actions

Copy link

Updated by jbaier_cz 7 months ago

Assignee set to jbaier_cz

Actions

Copy link

Updated by jbaier_cz 7 months ago

First draft https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/1198; I would also like to include nginx monitoring setup with https://github.com/influxdata/telegraf/tree/master/plugins/inputs/nginx to replace the already removed [[inputs.apache]]

Actions

Copy link

Updated by okurz 7 months ago

Related to action #160877: [alert] Scripts CI pipeline failing due to osd yielding 502 size:M added

Actions

Copy link

Updated by okurz 7 months ago

Due date set to 2024-06-18

Actions

Copy link

Updated by jbaier_cz 7 months ago

Status changed from Workable to In Progress

Actions

Copy link

Updated by jbaier_cz 7 months ago

Status changed from In Progress to Workable

Added 2 more commits which drops unrelated apache worker monitoring panels.

Actions

Copy link

#10

Updated by jbaier_cz 7 months ago

Last update contains two additional panels to monitor some stats provided by the ngx_http_stub_status_module. It might need some additional tweaks but at least we can see the data somewhere.

Actions

Copy link

#11

Updated by jbaier_cz 7 months ago

Status changed from Workable to Feedback

Now I need to wait for merge/deploy to see if the JSON changes make some sense.

Actions

Copy link

#12

Updated by okurz 7 months ago

https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/1198 merged. https://monitor.qa.suse.de/d/WebuiDb/webui-summary?orgId=1 shows some panels with data but "response codes" and "respone times" panels show "No data" as of now. Might need some time to populate or actual fixing.

Actions

Copy link

#13

Updated by jbaier_cz 7 months ago

Yes, the nginx_log collection is part of the same MR so there are likely no data in the database right now. I will monitor the monitoring and create a fix if necessary.

Actions

Copy link

#14

Updated by tinita 7 months ago

show field keys in the influx prompt shows apache, apache_log and nginx, but no nginx_log table.
But I don't know how to fix.

Actions

Copy link

#15

Updated by jbaier_cz 7 months ago

It could also mean that I made some mistake in the telegraf configuration and

[[inputs.tail]]
  files = ["/var/log/nginx/access.log"]
  interval = "30s"
  from_beginning = false
  name_override = "nginx_log"
  ## For parsing logstash-style "grok" patterns:
  data_format = "grok"
  grok_patterns = ["%{CUSTOM_LOG}"]
  grok_custom_pattern_files = []
  grok_custom_patterns = '''
      CUSTOM_LOG %{COMBINED_LOG_FORMAT} rt=%{NUMBER:response_time_s:float} urt=%"{NUMBER:upstream_response_time_s:float}"
  '''

is not working as expected

Actions

Copy link

#16