https://progress.opensuse.org/https://progress.opensuse.org/themes/openSUSE/favicon/favicon.ico?15829177842021-12-06T10:21:51ZopenSUSE Project Management ToolopenQA Infrastructure - action #103539: Update expired SSL certificate on monitor.qa.suse.de with dehydrated and salt, same as on OSD size:Mhttps://progress.opensuse.org/issues/103539?journal_id=4705802021-12-06T10:21:51Zlivdywanliv.dywan@suse.com
<ul><li><strong>Subject</strong> changed from <i>Update SSL certificates on monitor.qa.suse.de with dehydrated and salt, same as on OSD</i> to <i>Update expired SSL certificate on monitor.qa.suse.de with dehydrated and salt, same as on OSD size:M</i></li><li><strong>Description</strong> updated (<a title="View differences" href="/journals/470580/diff?detail_id=445149">diff</a>)</li><li><strong>Status</strong> changed from <i>New</i> to <i>Workable</i></li></ul> openQA Infrastructure - action #103539: Update expired SSL certificate on monitor.qa.suse.de with dehydrated and salt, same as on OSD size:Mhttps://progress.opensuse.org/issues/103539?journal_id=4705952021-12-06T10:40:33Znicksingernsinger@suse.com
<ul><li><strong>Assignee</strong> set to <i>nicksinger</i></li></ul> openQA Infrastructure - action #103539: Update expired SSL certificate on monitor.qa.suse.de with dehydrated and salt, same as on OSD size:Mhttps://progress.opensuse.org/issues/103539?journal_id=4706222021-12-06T11:08:50Znicksingernsinger@suse.com
<ul><li><strong>Status</strong> changed from <i>Workable</i> to <i>In Progress</i></li></ul><p><a href="https://gitlab.suse.de/openqa/salt-pillars-openqa/-/merge_requests/370" class="external">https://gitlab.suse.de/openqa/salt-pillars-openqa/-/merge_requests/370</a></p>
openQA Infrastructure - action #103539: Update expired SSL certificate on monitor.qa.suse.de with dehydrated and salt, same as on OSD size:Mhttps://progress.opensuse.org/issues/103539?journal_id=4706882021-12-06T13:04:41Znicksingernsinger@suse.com
<ul><li><strong>Status</strong> changed from <i>In Progress</i> to <i>Feedback</i></li></ul><p><a href="https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/617" class="external">https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/617</a><br>
Changes are ready to be merged. For testing I already run the commands on OSD so even before anyone merges this we already have a valid certificate on <a href="https://monitor.qa.suse.de" class="external">https://monitor.qa.suse.de</a> again :)</p>
openQA Infrastructure - action #103539: Update expired SSL certificate on monitor.qa.suse.de with dehydrated and salt, same as on OSD size:Mhttps://progress.opensuse.org/issues/103539?journal_id=4708142021-12-07T09:16:18Zokurzokurz@suse.com
<ul><li><strong>Related to</strong> <i><a class="issue tracker-4 status-3 priority-4 priority-default closed" href="/issues/103527">action #103527</a>: osd-deployment pipelines fail and alerts are not handled size:M</i> added</li></ul> openQA Infrastructure - action #103539: Update expired SSL certificate on monitor.qa.suse.de with dehydrated and salt, same as on OSD size:Mhttps://progress.opensuse.org/issues/103539?journal_id=4732202021-12-14T13:14:39Zlivdywanliv.dywan@suse.com
<ul></ul><p>monitor.qa.suse.de and stats.openqa-monitor.qa.suse.de seems to work - any remaining steps wrt making sure renewal happens automatically?</p>
openQA Infrastructure - action #103539: Update expired SSL certificate on monitor.qa.suse.de with dehydrated and salt, same as on OSD size:Mhttps://progress.opensuse.org/issues/103539?journal_id=4732232021-12-14T13:15:32Zlivdywanliv.dywan@suse.com
<ul><li><strong>Priority</strong> changed from <i>Urgent</i> to <i>Normal</i></li></ul><p>Lowering priority since, as far as I can tell, the sites work without exceptions in the browser</p>
openQA Infrastructure - action #103539: Update expired SSL certificate on monitor.qa.suse.de with dehydrated and salt, same as on OSD size:Mhttps://progress.opensuse.org/issues/103539?journal_id=4732262021-12-14T13:30:47Zokurzokurz@suse.com
<ul></ul><p>I suggest to add monitoring, same as we already have for OSD</p>
openQA Infrastructure - action #103539: Update expired SSL certificate on monitor.qa.suse.de with dehydrated and salt, same as on OSD size:Mhttps://progress.opensuse.org/issues/103539?journal_id=4757852021-12-23T09:45:35Znicksingernsinger@suse.com
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/475785/diff?detail_id=450102">diff</a>)</li><li><strong>Due date</strong> set to <i>2022-01-07</i></li><li><strong>Status</strong> changed from <i>Feedback</i> to <i>Workable</i></li></ul><p>Monitoring is left to do. Will carry this over into the new year. If anybody is bored feel free to take over :) Otherwise I will add the monitoring in the new year. Our current certificate is valid until the 20th of Jan so we have headroom to implement this monitoring before the current cert expires</p>
openQA Infrastructure - action #103539: Update expired SSL certificate on monitor.qa.suse.de with dehydrated and salt, same as on OSD size:Mhttps://progress.opensuse.org/issues/103539?journal_id=4758242021-12-23T10:49:18Zokurzokurz@suse.com
<ul></ul><p><a href="https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/634" class="external">https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/634</a> should be a start to bring in the according telegraf config. For this we should read out the domain list from pillars but I realized that pillars have not been updated on monitor.qa since 2021-04 so that should be the first thing to check. Then in grafana copy the certificate related panel from the webui dashboard into <a href="https://monitor.qa.suse.de/d/EML0bpuGk/monitoring" class="external">https://monitor.qa.suse.de/d/EML0bpuGk/monitoring</a></p>
openQA Infrastructure - action #103539: Update expired SSL certificate on monitor.qa.suse.de with dehydrated and salt, same as on OSD size:Mhttps://progress.opensuse.org/issues/103539?journal_id=4778372022-01-06T14:46:50Zlivdywanliv.dywan@suse.com
<ul><li><strong>Due date</strong> changed from <i>2022-01-07</i> to <i>2022-01-14</i></li><li><strong>Priority</strong> changed from <i>Normal</i> to <i>High</i></li></ul><p>Seems like a good thing we already set the <em>due date</em> prematurely. <a class="user active user-mention" href="https://progress.opensuse.org/users/24624">@nicksinger</a> I recall your saying you were already looking into it. If you can, please update here at the next opportunity - or if there's something others can help you with, you're welcome to ask.</p>
openQA Infrastructure - action #103539: Update expired SSL certificate on monitor.qa.suse.de with dehydrated and salt, same as on OSD size:Mhttps://progress.opensuse.org/issues/103539?journal_id=4799442022-01-13T11:14:44Znicksingernsinger@suse.com
<ul><li><strong>Status</strong> changed from <i>Workable</i> to <i>In Progress</i></li></ul><p>okurz wrote:</p>
<blockquote>
<p><a href="https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/634" class="external">https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/634</a> should be a start to bring in the according telegraf config. For this we should read out the domain list from pillars but I realized that pillars have not been updated on monitor.qa since 2021-04 so that should be the first thing to check. Then in grafana copy the certificate related panel from the webui dashboard into <a href="https://monitor.qa.suse.de/d/EML0bpuGk/monitoring" class="external">https://monitor.qa.suse.de/d/EML0bpuGk/monitoring</a></p>
</blockquote>
<p>not sure how you determined that the pillar cache was not updated that long. According to <a href="https://salt-users.narkive.com/vJmNliU0/why-must-saltutil-refresh-pillar-be-run#post2" class="external">https://salt-users.narkive.com/vJmNliU0/why-must-saltutil-refresh-pillar-be-run#post2</a> this should happen with every highstate. I can't imagine we didn't do changes to that host in that time frame. Also checking with <code>salt-call pillar.get "dehydrated"</code> I see the most recent pillar data (which according to the post on salt-users should be from the cache).</p>
openQA Infrastructure - action #103539: Update expired SSL certificate on monitor.qa.suse.de with dehydrated and salt, same as on OSD size:Mhttps://progress.opensuse.org/issues/103539?journal_id=4799562022-01-13T12:29:13Znicksingernsinger@suse.com
<ul></ul><p>I've opened <a href="https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/639" class="external">https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/639</a> as a first draft how I would imagine the monitoring. It should be more generic and work for every host we eventually add in the future. I still need to figure out how to properly loop over the SANs of the hosts.txt entries per host to have a proper check for each of them (this is what happens in <a href="https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/639/diffs#85e7a4e39662e846e9a7e8c1660d41d28a0389c3_0_8" class="external">https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/639/diffs#85e7a4e39662e846e9a7e8c1660d41d28a0389c3_0_8</a>) - it might be even possible to shrink the certificates.conf down to a single [[inputs.x509_cert]] section</p>
openQA Infrastructure - action #103539: Update expired SSL certificate on monitor.qa.suse.de with dehydrated and salt, same as on OSD size:Mhttps://progress.opensuse.org/issues/103539?journal_id=4807082022-01-17T16:30:33Zlivdywanliv.dywan@suse.com
<ul><li><strong>Due date</strong> changed from <i>2022-01-14</i> to <i>2022-01-18</i></li></ul><p><a href="https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/639" class="external">https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/639</a> is still in progress, I'm starting to get worried that we'll see alerts for broken certificates afterall... bumping the due date to <em>tomorrow</em> in any case</p>
<p><a class="user active user-mention" href="https://progress.opensuse.org/users/24624">@nicksinger</a> Please prepare manual steps to do the update tomorrow</p>
openQA Infrastructure - action #103539: Update expired SSL certificate on monitor.qa.suse.de with dehydrated and salt, same as on OSD size:Mhttps://progress.opensuse.org/issues/103539?journal_id=4807242022-01-17T17:10:00Zlivdywanliv.dywan@suse.com
<ul></ul><p>nicksinger wrote:</p>
<blockquote>
<p>Monitoring is left to do. Will carry this over into the new year. If anybody is bored feel free to take over :) Otherwise I will add the monitoring in the new year. Our current certificate is valid until the 20th of Jan so we have headroom to implement this monitoring before the current cert expires</p>
</blockquote>
<p>Apparently we're at Feb 14 now, which suggests automated renewal worked at least once.</p>
openQA Infrastructure - action #103539: Update expired SSL certificate on monitor.qa.suse.de with dehydrated and salt, same as on OSD size:Mhttps://progress.opensuse.org/issues/103539?journal_id=4808442022-01-18T09:05:44Znicksingernsinger@suse.com
<ul></ul><p>cdywan wrote:</p>
<blockquote>
<p><a href="https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/639" class="external">https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/639</a> is still in progress, I'm starting to get worried that we'll see alerts for broken certificates afterall... bumping the due date to <em>tomorrow</em> in any case</p>
<p><a class="user active user-mention" href="https://progress.opensuse.org/users/24624">@nicksinger</a> Please prepare manual steps to do the update tomorrow</p>
</blockquote>
<p>Just to make it clear here in the ticket too:</p>
<p><strong>AC4</strong>: The certificates are automatically refreshed<br>
is already done. Therefore we don't need to worry about an expiring certificate and we won't see any alerts because this is why this ticket is still open - creating monitoring/alerts for it. I polished up <a href="https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/639" class="external">https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/639</a> and think it can be merged now to receive the metrics. A dashboard for grafana is still left to do but I'd like to cover it in a separate MR</p>
openQA Infrastructure - action #103539: Update expired SSL certificate on monitor.qa.suse.de with dehydrated and salt, same as on OSD size:Mhttps://progress.opensuse.org/issues/103539?journal_id=4809822022-01-18T13:24:15Znicksingernsinger@suse.com
<ul></ul><p>I came up with a new dashboard including alerts and proper instructions here: <a href="https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/640" class="external">https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/640</a><br>
It might not work after merging because I couldn't really test the template for the dashboard.</p>
openQA Infrastructure - action #103539: Update expired SSL certificate on monitor.qa.suse.de with dehydrated and salt, same as on OSD size:Mhttps://progress.opensuse.org/issues/103539?journal_id=4810482022-01-18T18:23:25Zlivdywanliv.dywan@suse.com
<ul><li><strong>File</strong> <a href="/attachments/12409">grafik.png</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/12409/grafik.png">grafik.png</a> added</li><li><strong>Status</strong> changed from <i>In Progress</i> to <i>Feedback</i></li></ul><p>nicksinger wrote:</p>
<blockquote>
<p>I came up with a new dashboard including alerts and proper instructions here: <a href="https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/640" class="external">https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/640</a><br>
It might not work after merging because I couldn't really test the template for the dashboard.</p>
</blockquote>
<p>The MR got merged. Looks very nice</p>
openQA Infrastructure - action #103539: Update expired SSL certificate on monitor.qa.suse.de with dehydrated and salt, same as on OSD size:Mhttps://progress.opensuse.org/issues/103539?journal_id=4810512022-01-18T19:22:33Zokurzokurz@suse.com
<ul><li><strong>Due date</strong> deleted (<del><i>2022-01-18</i></del>)</li><li><strong>Status</strong> changed from <i>Feedback</i> to <i>Resolved</i></li></ul><p>I agree. All looks good</p>