Project

General

Profile

Actions

action #103380

open

coordination #161414: [epic] Improved salt based infrastructure management

Configure retention/downsampling policy for specific monitoring data stored within InfluxDB

Added by mkittler almost 3 years ago. Updated about 1 month ago.

Status:
Blocked
Priority:
Low
Assignee:
Category:
Feature requests
Target version:
Start date:
2021-12-01
Due date:
% Done:

0%

Estimated time:

Description

Motivation

See #94492 - as part of this ticket a retention period for the entire telegraf database has been set (to remove data older than two years). It would still be desirable to setup more fine grained retention policies and down-sampling for specific monitoring data. This was not done because it seems InfluxDB only let's one create retention policies per database (and we store all of our monitoring data in the single database telegraf). Additionally, downsampling needs to be taken care of per field so setting it up for the entire database would be a lot of work (and one needed to extend it every time a new field is added).

Suggestions

  • Read comments in #94492 (but with a grain of salt since I might have missed some possibilities).
  • Likely we needed to move data we want to retain shorter than two years or downsample into another database (and ensure we use that database for reading/writing data).
  • Check if we have global retention enabled
  • Look into if maybe influxdb (v.1) vs. influxdb2 (separate package) is of any help here

Related issues 2 (1 open1 closed)

Related to openQA Infrastructure - action #94492: Configure retention/downsampling policy for monitoring data stored within InfluxDB size:MResolvedmkittler2021-06-22

Actions
Related to openQA Infrastructure - action #167722: Efficient use of monitoring data within influxdb on monitor.qe.nue2.suse.org size:MWorkablenicksinger2024-10-022024-11-29

Actions
Actions #1

Updated by mkittler almost 3 years ago

  • Related to action #94492: Configure retention/downsampling policy for monitoring data stored within InfluxDB size:M added
Actions #2

Updated by okurz almost 3 years ago

  • Priority changed from Normal to Low
  • Target version set to Ready
Actions #3

Updated by okurz almost 3 years ago

  • Target version changed from Ready to future

Discussed in SUSE QE Tools weekly meeting and we decided that our current approach with a global retention period might be good enough for the following years. Likely one could improve by starting with a fresh database or multiple ones and properly configure specific downsampling policies for each measurement.

Actions #5

Updated by okurz about 1 month ago

  • Related to action #167722: Efficient use of monitoring data within influxdb on monitor.qe.nue2.suse.org size:M added
Actions #6

Updated by okurz about 1 month ago

  • Description updated (diff)
  • Category set to Feature requests
  • Status changed from New to Blocked
  • Assignee set to okurz
  • Target version changed from future to Ready
  • Parent task set to #161414
Actions #7

Updated by okurz about 1 month ago

  • Description updated (diff)
Actions

Also available in: Atom PDF