https://progress.opensuse.org/https://progress.opensuse.org/themes/openSUSE/favicon/favicon.ico?15829177842022-10-19T09:53:31ZopenSUSE Project Management ToolopenQA Infrastructure - action #119059: Use qa-power8 for ppc tests in o3 - network connected?https://progress.opensuse.org/issues/119059?journal_id=5649342022-10-19T09:53:31Zokurzokurz@suse.com
<ul></ul><p><a href="https://gitlab.suse.de/qa-sle/qanet-configs/-/merge_requests/43" class="external">https://gitlab.suse.de/qa-sle/qanet-configs/-/merge_requests/43</a> for proper DHCP/DNS</p>
openQA Infrastructure - action #119059: Use qa-power8 for ppc tests in o3 - network connected?https://progress.opensuse.org/issues/119059?journal_id=5649372022-10-19T09:58:44Zokurzokurz@suse.com
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/564937/diff?detail_id=532024">diff</a>)</li></ul> openQA Infrastructure - action #119059: Use qa-power8 for ppc tests in o3 - network connected?https://progress.opensuse.org/issues/119059?journal_id=5649432022-10-19T10:14:18Zokurzokurz@suse.com
<ul><li><strong>Tags</strong> set to <i>next-office-day</i></li></ul><p>According to BMC the machine is on. According to racktables <a href="https://racktables.nue.suse.com/index.php?page=object&object_id=2350" class="external">https://racktables.nue.suse.com/index.php?page=object&object_id=2350</a> there should be an IPv4 address 10.162.6.170 (ps64mm1070.qa.suse.de) but there is no response on ping. According to racktables there might not be any ethernet connection. The BMC menu "network access" mentions an IPv4 address 10.163.41.238 which does respond on ping and also serves ssh. I did not manage to login over ssh with any password known to me.</p>
<p><code>nmap -O</code> for OS detection says:</p>
<pre><code>$ sudo nmap -AO 10.163.41.238
Starting Nmap 7.92 ( https://nmap.org ) at 2022-10-19 12:11 CEST
Nmap scan report for 10.163.41.238
Host is up (0.000075s latency).
Not shown: 997 closed tcp ports (reset)
PORT STATE SERVICE VERSION
22/tcp open ssh OpenSSH 8.4 (protocol 2.0)
| ssh-hostkey:
| 3072 8f:8f:9c:6d:ae:30:0d:e1:65:1a:c5:0f:2e:21:65:16 (RSA)
| 256 9c:d3:c1:7e:ce:e7:8c:93:0c:af:b6:24:60:95:59:88 (ECDSA)
|_ 256 d0:31:38:c4:41:f7:78:14:da:be:a8:f3:b0:ea:ea:55 (ED25519)
25/tcp open smtp Postfix smtpd
|_smtp-commands: localhost, PIPELINING, SIZE, ETRN, ENHANCEDSTATUSCODES, 8BITMIME, DSN, SMTPUTF8, CHUNKING
53/tcp open tcpwrapped
Device type: general purpose
Running: Linux 2.6.X
OS CPE: cpe:/o:linux:linux_kernel:2.6.32
OS details: Linux 2.6.32
Network Distance: 0 hops
Service Info: Host: localhost
OS and Service detection performed. Please report any incorrect results at https://nmap.org/submit/ .
Nmap done: 1 IP address (1 host up) scanned in 21.56 seconds
</code></pre>
<p>nsinger said that the linux is likely 4.X</p>
openQA Infrastructure - action #119059: Use qa-power8 for ppc tests in o3 - network connected?https://progress.opensuse.org/issues/119059?journal_id=5649462022-10-19T10:24:45Zokurzokurz@suse.com
<ul><li><strong>Assignee</strong> deleted (<del><i>okurz</i></del>)</li></ul><p>So next step could be to learn how to install something, e.g. PXE boot or so.</p>
openQA Infrastructure - action #119059: Use qa-power8 for ppc tests in o3 - network connected?https://progress.opensuse.org/issues/119059?journal_id=5649492022-10-19T10:39:59Zokurzokurz@suse.com
<ul></ul><p>ok, we learned that the reported IPv4 address 10.163.41.238 is actually my own so not helping.</p>
openQA Infrastructure - action #119059: Use qa-power8 for ppc tests in o3 - network connected?https://progress.opensuse.org/issues/119059?journal_id=5652492022-10-20T09:33:40Ztinitatina.mueller+trick-redmine@suse.com
<ul><li><strong>Subject</strong> changed from <i>Use qa-power8 for ppc tests in o3</i> to <i>Use qa-power8 for ppc tests in o3 size:M</i></li><li><strong>Description</strong> updated (<a title="View differences" href="/journals/565249/diff?detail_id=532264">diff</a>)</li><li><strong>Status</strong> changed from <i>New</i> to <i>Workable</i></li></ul> openQA Infrastructure - action #119059: Use qa-power8 for ppc tests in o3 - network connected?https://progress.opensuse.org/issues/119059?journal_id=5654352022-10-20T14:01:34Zokurzokurz@suse.com
<ul></ul><p>I asked in <a href="https://suse.slack.com/archives/C02CANHLANP/p1666274256887309" class="external">#eng-testing</a> as well as <a href="https://suse.slack.com/archives/C029APBKLGK/p1666274251509269" class="external">#help-it-ama</a></p>
<blockquote>
<p>(Oliver Kurz) anyone being in or going to Nbg Maxtorhof SRV2 now/soon who could ensure that there is a network connection for QA-Power8 ? Or do you know of a better channel where to coordinate such work?</p>
</blockquote>
<p>If somebody in the meantime or if there is no answer stops by one can still ensure that there is a cable connection.</p>
openQA Infrastructure - action #119059: Use qa-power8 for ppc tests in o3 - network connected?https://progress.opensuse.org/issues/119059?journal_id=5660232022-10-23T08:12:27Zokurzokurz@suse.com
<ul></ul><p>acarvajal found that there is a physical connection but he did not remember on which switch port the cable is connected.</p>
openQA Infrastructure - action #119059: Use qa-power8 for ppc tests in o3 - network connected?https://progress.opensuse.org/issues/119059?journal_id=5660292022-10-23T17:23:54Zokurzokurz@suse.com
<ul><li><strong>Tags</strong> deleted (<del><i>next-office-day</i></del>)</li></ul><blockquote>
<p>no, sorry. IIRC it was the switch at the top of the rack, the row of ports below and either the 5th or 6th port from the left when seeing it from the back</p>
</blockquote>
<p>With that and IPMI SoL access that should be enough to make the machine useable again without needing physical access</p>
openQA Infrastructure - action #119059: Use qa-power8 for ppc tests in o3 - network connected?https://progress.opensuse.org/issues/119059?journal_id=5671392022-10-25T15:43:07Zmkittlermarius.kittler@suse.com
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/567139/diff?detail_id=533866">diff</a>)</li></ul><p>This comment is actually irrelevant, it is about the o3 worker (which <a class="issue tracker-4 status-3 priority-3 priority-lowest closed child" title="action: Recover o3 worker kerosene formerly known as power8, restore IPMI access size:M (Resolved)" href="https://progress.opensuse.org/issues/116078">#116078</a> is about).</p>
<hr>
<blockquote>
<p>With that</p>
</blockquote>
<p>Ok, so the 3rd suggestion is not relevant after all.</p>
<blockquote>
<p>and IPMI SoL access</p>
</blockquote>
<p>I still cannot connect via IPMI:</p>
<pre><code>ipmitool -I lanplus -C 3 -H openqaworker-power8-ipmi.suse.de -U … -P … power status
Error in open session response message : insufficient resources for session
Error: Unable to establish IPMI v2 / RMCP+ session
</code></pre>
<p>Broken IPMI access <em>is</em> the big problem here. Powering the machine on and off and many more things I've already tried via the BMC didn't help.</p>
openQA Infrastructure - action #119059: Use qa-power8 for ppc tests in o3 - network connected?https://progress.opensuse.org/issues/119059?journal_id=5672142022-10-25T20:50:00Zokurzokurz@suse.com
<ul></ul><p>We can also try make use of the HMC control servers, e.g. "powerhmc1.arch.suse.de" or something</p>
openQA Infrastructure - action #119059: Use qa-power8 for ppc tests in o3 - network connected?https://progress.opensuse.org/issues/119059?journal_id=5699382022-11-02T10:44:07Zokurzokurz@suse.com
<ul></ul><p>We added the machine to both powerhmc1.arch.suse.de and powerhmc3.arch.suse.de and in both cases we can configure the "partitions", i.e. virtual machines, on the PowerVM managed machines. The mode between OPAL – for bare-metal installations – and PowerVM can be switched in ASM, in this case <a href="https://qa-power8.qa.suse.de" class="external">https://qa-power8.qa.suse.de</a> in the menu "Firmware Configuration". For this no HMC connection must be configured. As necessary remove the connection in the HMC and in the ASM menu "Hardware Management Consoles". The problem is that ipmi seems to be not configured to allow remote access so we would need an OS installation on the host first to call ipmitool locally to allow remote access. As alternative we should try to use one of the VMs or create a new VM and try to enable remote access ipmitool.</p>
<p>So the suggestion is: (Re-)install one of the LPAR VMs and try to call ipmitool to allow remote access.</p>
openQA Infrastructure - action #119059: Use qa-power8 for ppc tests in o3 - network connected?https://progress.opensuse.org/issues/119059?journal_id=5699412022-11-02T10:46:09Zokurzokurz@suse.com
<ul><li><strong>Copied to</strong> <i><a class="issue tracker-4 status-1 priority-4 priority-default child" href="/issues/119755">action #119755</a>: Use a PowerVM machine to serve both PowerVM LPARs for testing as well as one VM running qemu tests</i> added</li></ul> openQA Infrastructure - action #119059: Use qa-power8 for ppc tests in o3 - network connected?https://progress.opensuse.org/issues/119059?journal_id=5722092022-11-08T15:25:11Zmkittlermarius.kittler@suse.com
<ul></ul><p>What are the ASM credentials for <a href="https://qa-power8.qa.suse.de" class="external">https://qa-power8.qa.suse.de</a>? I tried multiple combinations of standard ones we use elsewhere but nothing worked.</p>
openQA Infrastructure - action #119059: Use qa-power8 for ppc tests in o3 - network connected?https://progress.opensuse.org/issues/119059?journal_id=5752832022-11-16T11:12:18Zokurzokurz@suse.com
<ul><li><strong>Status</strong> changed from <i>Workable</i> to <i>In Progress</i></li><li><strong>Assignee</strong> set to <i>okurz</i></li></ul><p>mkittler wrote:</p>
<blockquote>
<p>What are the ASM credentials for <a href="https://qa-power8.qa.suse.de" class="external">https://qa-power8.qa.suse.de</a>? I tried multiple combinations of standard ones we use elsewhere but nothing worked.</p>
</blockquote>
<p>provided in chat.</p>
<p>I am now trying myself. First over <a href="https://powerhmc3.arch.suse.de/" class="external">https://powerhmc3.arch.suse.de/</a> deleted "doc_vm" and "kdump" to free some ressources, then creating new LPAR.</p>
<p>Following a "template" I created an LPAR named "aixlinux1" using a wizard and selected "Network boot" pointing to 10.162.0.1, qanet.qa.suse.de, using an IPv4 address 10.162.6.80 as from <a href="https://gitlab.suse.de/qa-sle/qanet-configs/-/blob/master/etc/dhcpd.conf#L439" class="external">https://gitlab.suse.de/qa-sle/qanet-configs/-/blob/master/etc/dhcpd.conf#L439</a> assuming that this one is currently unused. The wizard eventually showed an error "REST0293 Error while trying to retrieve Network Adapters: lpar_netboot: The network boot ended in an error."</p>
<p>Then I asked around and got a hint in <a href="https://suse.slack.com/archives/C02CANHLANP/p1668603577713149?thread_ts=1668602055.990499&cid=C02CANHLANP" class="external">https://suse.slack.com/archives/C02CANHLANP/p1668603577713149?thread_ts=1668602055.990499&cid=C02CANHLANP</a> from szarate to check instructions from<br>
<a href="https://confluence.suse.com/display/qasle/Power8+and+Power9+%28PowerVM%29+on+SLES" class="external">https://confluence.suse.com/display/qasle/Power8+and+Power9+%28PowerVM%29+on+SLES</a><br>
From there I also learned that one can connect to an HMC over ssh. I could login over ssh as <code>okurz@powerhmc3.arch.suse.de</code> but I could not add my ssh key. Anyway I considered the terminal access over this path more convenient with just <code>mkvterm -m qa-power8 -p aixlinux1</code> on powerhmc3.arch.suse.de. With that me with the help of szarate tried multiple approaches, reconfiguring the network devices and such but did not see any DHCP or BOOTP requests on qanet.qa.suse.de . So maybe the ethernet ports are really not connected to a switch.<br>
Also tried to hardcode network settings including gateway 10.162.63.254 based on the template we have seen on "redcurrant".</p>
<p>szarate and me come to the conclusion that likely the network is really not connected, not the BMC ones but the ethernet for the system. So this needs to be checked on "next-office-day" at Maxtorhof.</p>
openQA Infrastructure - action #119059: Use qa-power8 for ppc tests in o3 - network connected?https://progress.opensuse.org/issues/119059?journal_id=5753612022-11-16T14:07:24Zokurzokurz@suse.com
<ul><li><strong>Tags</strong> set to <i>next-office-day</i></li><li><strong>Status</strong> changed from <i>In Progress</i> to <i>Workable</i></li><li><strong>Assignee</strong> deleted (<del><i>okurz</i></del>)</li></ul> openQA Infrastructure - action #119059: Use qa-power8 for ppc tests in o3 - network connected?https://progress.opensuse.org/issues/119059?journal_id=5753642022-11-16T14:16:09Zokurzokurz@suse.com
<ul><li><strong>Project</strong> changed from <i>46</i> to <i>openQA Infrastructure</i></li></ul> openQA Infrastructure - action #119059: Use qa-power8 for ppc tests in o3 - network connected?https://progress.opensuse.org/issues/119059?journal_id=5753942022-11-16T15:14:04Zokurzokurz@suse.com
<ul><li><strong>Project</strong> changed from <i>openQA Infrastructure</i> to <i>46</i></li><li><strong>Subject</strong> changed from <i>Use qa-power8 for ppc tests in o3 size:M</i> to <i>[next-office-day] Use qa-power8 for ppc tests in o3 size:M</i></li></ul> openQA Infrastructure - action #119059: Use qa-power8 for ppc tests in o3 - network connected?https://progress.opensuse.org/issues/119059?journal_id=5753972022-11-16T15:14:14Zokurzokurz@suse.com
<ul><li><strong>Subject</strong> changed from <i>[next-office-day] Use qa-power8 for ppc tests in o3 size:M</i> to <i>[next-office-day] Use qa-power8 for ppc tests in o3 - network connected? size:M</i></li></ul> openQA Infrastructure - action #119059: Use qa-power8 for ppc tests in o3 - network connected?https://progress.opensuse.org/issues/119059?journal_id=5764902022-11-21T10:12:50Zokurzokurz@suse.com
<ul><li><strong>Status</strong> changed from <i>Workable</i> to <i>In Progress</i></li><li><strong>Assignee</strong> set to <i>okurz</i></li></ul> openQA Infrastructure - action #119059: Use qa-power8 for ppc tests in o3 - network connected?https://progress.opensuse.org/issues/119059?journal_id=5764992022-11-21T10:26:22Zokurzokurz@suse.com
<ul></ul><p>I checked the physical network connections. Updated racktables with entries for the two quad-ethernet access cards. There was already one connection to C10-1 (top) connector, gi30. Updated racktables accordingly. gi30 on qanet15nue.qa.suse.de is in VLAN12 so should be reachable in QA network. Connected a second connection C12-2 (second card on the right, second connector from top) to gi8, added to VLAN12 as well. And connected C10-4 (bottom) to gi2, VLAN12.</p>
openQA Infrastructure - action #119059: Use qa-power8 for ppc tests in o3 - network connected?https://progress.opensuse.org/issues/119059?journal_id=5765072022-11-21T10:33:45Zokurzokurz@suse.com
<ul><li><strong>Tags</strong> deleted (<del><i>next-office-day</i></del>)</li><li><strong>Subject</strong> changed from <i>[next-office-day] Use qa-power8 for ppc tests in o3 - network connected? size:M</i> to <i>Use qa-power8 for ppc tests in o3 - network connected? size:M</i></li><li><strong>Due date</strong> set to <i>2022-12-02</i></li><li><strong>Status</strong> changed from <i>In Progress</i> to <i>Feedback</i></li></ul><p><a href="https://suse.slack.com/archives/C02CANHLANP/p1669026762157029?thread_ts=1668602055.990499&cid=C02CANHLANP" class="external">https://suse.slack.com/archives/C02CANHLANP/p1669026762157029?thread_ts=1668602055.990499&cid=C02CANHLANP</a></p>
<blockquote>
<p>(Oliver Kurz) I have checked the machine QA-Power8 aka ps64mm1070 physically and I can confirm that it has ethernet connections. Moreover I connected two additional network cables now, see <a href="https://racktables.nue.suse.com/index.php?page=object&tab=ports&object_id=2350" class="external">https://racktables.nue.suse.com/index.php?page=object&tab=ports&object_id=2350</a> . @Santiago Zarate @Zaoliang Luo can you help with setting up the network for LPARs?</p>
</blockquote>
openQA Infrastructure - action #119059: Use qa-power8 for ppc tests in o3 - network connected?https://progress.opensuse.org/issues/119059?journal_id=5777982022-11-24T14:47:33Zokurzokurz@suse.com
<ul></ul><p>no response yet, asked again</p>
openQA Infrastructure - action #119059: Use qa-power8 for ppc tests in o3 - network connected?https://progress.opensuse.org/issues/119059?journal_id=5784132022-11-28T08:37:46Zokurzokurz@suse.com
<ul><li><strong>Status</strong> changed from <i>Feedback</i> to <i>In Progress</i></li></ul><p>I will look into setting up LPARs with network on qa-power8</p>
openQA Infrastructure - action #119059: Use qa-power8 for ppc tests in o3 - network connected?https://progress.opensuse.org/issues/119059?journal_id=5787132022-11-28T13:51:53Zokurzokurz@suse.com
<ul></ul><p><a href="https://www.youtube.com/@nigelargriffiths" class="external">https://www.youtube.com/@nigelargriffiths</a> could be helpful. as a reference for "training". In the meantime mgriessmeier and me have experimented and compared the setup on qa-power8 with other machines, e.g. "redcurrant" and "blackcurrant". with the views of the "virtual network topology" it looked like on other machines there is a proper network connection from the physical network all down to the LPARs, on qa-power8 there isn't. We managed to install the VIOS instance. First we tried the internal network connection (192.168.X.X) and a VIOS version which failed to deploy, then we selected the external network connection (10.X.X.X) and select a VIOS version with "<u>SP</u>" in the name. This eventually got stuck. Then we connected over ssh to powerhmc1, text console to the vios instance. That showed a progress of 0% and an error message that no volume group could be found with a waiting time of 105 minutes. We just tried to do something useful in the system, typed "q", "quit", "exit" and then suddenly the installation continued and progressed successfully till the end. Then in HMC the vios instance had a "RMC" connection. Then in the LPAR we could select the "bridged" network, configure an entry in the dhcp config on qanet including PXE, and then we could boot over network from a manual grub prompt using the existing outdated PXE boot entries as template with linux+initrd specified from relative paths to what we could find on qanet. We verified that we could access the YaST installer over ssh but did not progress with the installation. I suggest to redo all the above steps, could even be done on power8.openqanet.opensuse.org, and document again with all knowledge holes filled.</p>
openQA Infrastructure - action #119059: Use qa-power8 for ppc tests in o3 - network connected?https://progress.opensuse.org/issues/119059?journal_id=5788152022-11-28T21:24:46Zokurzokurz@suse.com
<ul><li><strong>Status</strong> changed from <i>In Progress</i> to <i>Workable</i></li><li><strong>Assignee</strong> deleted (<del><i>okurz</i></del>)</li></ul> openQA Infrastructure - action #119059: Use qa-power8 for ppc tests in o3 - network connected?https://progress.opensuse.org/issues/119059?journal_id=5794992022-11-30T08:20:01Zlivdywanliv.dywan@suse.com
<ul><li><strong>Due date</strong> deleted (<del><i>2022-12-02</i></del>)</li></ul><p>No due date for unassigned tickets</p>
openQA Infrastructure - action #119059: Use qa-power8 for ppc tests in o3 - network connected?https://progress.opensuse.org/issues/119059?journal_id=5826042022-12-08T06:50:02Zokurzokurz@suse.com
<ul><li><strong>Project</strong> changed from <i>46</i> to <i>openQA Infrastructure</i></li></ul> openQA Infrastructure - action #119059: Use qa-power8 for ppc tests in o3 - network connected?https://progress.opensuse.org/issues/119059?journal_id=5868972022-12-21T13:18:37Zokurzokurz@suse.com
<ul><li><strong>Related to</strong> <i><a class="issue tracker-4 status-3 priority-5 priority-high3 closed behind-schedule" href="/issues/122302">action #122302</a>: Support SD-105827 "PowerPC often fails to boot from network with 'error: time out opening'"</i> added</li></ul> openQA Infrastructure - action #119059: Use qa-power8 for ppc tests in o3 - network connected?https://progress.opensuse.org/issues/119059?journal_id=5913102023-01-09T12:06:56Zlivdywanliv.dywan@suse.com
<ul></ul><p>Next steps:</p>
<ul>
<li>Connect via SSH</li>
<li>Install an operating system</li>
<li>Make the machine usable as a worker</li>
</ul>
<p>We could do that in our mob session on Thursday since this should be a good learning opportunity for those not familiar with the setup, or another ad-hoc slot where there's more people involved.</p>
<p>We'll probably need a follow-up after that to get the network setup correctly after the machine's installed. See #119008 for reference.</p>
openQA Infrastructure - action #119059: Use qa-power8 for ppc tests in o3 - network connected?https://progress.opensuse.org/issues/119059?journal_id=5948382023-01-18T16:11:24Zmkittlermarius.kittler@suse.com
<ul></ul><p>I looked into this ticket again, so here a short summary:</p>
<ul>
<li>I was able to login on <a href="https://qa-power8.qa.suse.de" class="external">https://qa-power8.qa.suse.de</a>. Looks like the machine is configured to be in PowerVM mode (which should also explain why IPMI access is not working).</li>
<li>The system also appears to be already powered on. Not sure how to connect via SSH, though (and thus also not sure how to proceed with this ticket).</li>
<li>I have access to <a href="https://powerhmc3.arch.suse.de" class="external">https://powerhmc3.arch.suse.de</a> (not <a href="https://powerhmc1.arch.suse.de" class="external">https://powerhmc1.arch.suse.de</a>). It shows the host qa-power8 and I can even open the terminal for the test VM that's present. I'm still not sure how to proceed from there to install an OS. Maybe it helps to connect to the HMC via SSH. What that meant with "Connect via SSH" in the previous comment?</li>
<li>I've read <a class="issue tracker-4 status-6 priority-3 priority-lowest closed child" title="action: Use qa-power8 for ppc tests in o3 - network connected? (Rejected)" href="https://progress.opensuse.org/issues/119059#note-16">#119059#note-16</a>. So this has already been attempted but the network didn't work. After <a class="issue tracker-4 status-6 priority-3 priority-lowest closed child" title="action: Use qa-power8 for ppc tests in o3 - network connected? (Rejected)" href="https://progress.opensuse.org/issues/119059#note-23">#119059#note-23</a> and <a class="issue tracker-4 status-6 priority-3 priority-lowest closed child" title="action: Use qa-power8 for ppc tests in o3 - network connected? (Rejected)" href="https://progress.opensuse.org/issues/119059#note-26">#119059#note-26</a> this shouldn't be a problem anymore.</li>
<li>I could theoretically try to follow-up on <a class="issue tracker-4 status-6 priority-3 priority-lowest closed child" title="action: Use qa-power8 for ppc tests in o3 - network connected? (Rejected)" href="https://progress.opensuse.org/issues/119059#note-26">#119059#note-26</a> but the details (which are left to be documented) aren't clear to me. The comment also mentions hmc1 but I have only access to hmc3.</li>
</ul>
<hr>
<blockquote>
<p>We could do that in our mob session on Thursday since this should be a good learning opportunity for those not familiar with the setup, or another ad-hoc slot where there's more people involved.</p>
</blockquote>
<p>This would be a good idea. I feel a bit lost here.</p>
openQA Infrastructure - action #119059: Use qa-power8 for ppc tests in o3 - network connected?https://progress.opensuse.org/issues/119059?journal_id=5948442023-01-18T17:26:19Zokurzokurz@suse.com
<ul></ul><p>So first: It shouldn't technically matter if we use hmc1 or hmc3 but in the future hmc3 is to be used as we have full control over it and everybody has their own account. And I recommend you try to connect to the serial terminal by connecting to the HMC over ssh and then call the <code>mkvterm</code> commands to connect to the terminal. Regarding "ssh" to the final system I what I did was start an installation with Linuxrc parameter ssh=1 and then connect to the SUT, not the HMC </p>
openQA Infrastructure - action #119059: Use qa-power8 for ppc tests in o3 - network connected?https://progress.opensuse.org/issues/119059?journal_id=5971242023-01-26T13:36:48Zokurzokurz@suse.com
<ul><li><strong>Copied to</strong> <i><a class="issue tracker-4 status-1 priority-3 priority-lowest child" href="/issues/123712">action #123712</a>: PowerVM HMC within o3, e.g. on VM</i> added</li></ul> openQA Infrastructure - action #119059: Use qa-power8 for ppc tests in o3 - network connected?https://progress.opensuse.org/issues/119059?journal_id=5971302023-01-26T13:37:50Zokurzokurz@suse.com
<ul></ul><p>We took another go at this story and connected with ssh to powerhmc3.arch.suse.de, rebooted the machine in SMS and selected network boot ending up in the grub menu served by qanet. We could continue from here, install something like a current Leap 15.4 and similar as for grenache-1 we could try to use nested kvm on that. But to be able to use the machine within o3 we also need an HVM there. Consider this out of scope for the current ticket, see <a class="issue tracker-4 status-1 priority-3 priority-lowest child" title="action: PowerVM HMC within o3, e.g. on VM (New)" href="https://progress.opensuse.org/issues/123712">#123712</a> for that.</p>
<p>I disconnected qa-power8 from HMC within <a href="https://powerhmc3.arch.suse.de/dashboard/#resources/systems" class="external">https://powerhmc3.arch.suse.de/dashboard/#resources/systems</a> so that we can switch to OPAL mode in the ASM and try to have IPMI and then continue with an installation of bare-metal Leap and then use as openQA kvm worker and then move to o3 network.</p>
openQA Infrastructure - action #119059: Use qa-power8 for ppc tests in o3 - network connected?https://progress.opensuse.org/issues/119059?journal_id=5971692023-01-26T14:49:38Zmkittlermarius.kittler@suse.com
<ul></ul><blockquote>
<p>I disconnected qa-power8 from HMC within <a href="https://powerhmc3.arch.suse.de/dashboard/#resources/systems" class="external">https://powerhmc3.arch.suse.de/dashboard/#resources/systems</a> …</p>
</blockquote>
<p>Somehow that's not recognized by the ASM. It still says "Changing the firmware type is only allowed when the system is in powered off state and not managed by a management console." despite the system being powered off for quite a while now. That also means I could not enable IPMI so far. I hope I've been using the right ASM. (I have been using <a href="https://qa-power8.qa.suse.de" class="external">https://qa-power8.qa.suse.de</a> as mentioned in the ticket description.)</p>
openQA Infrastructure - action #119059: Use qa-power8 for ppc tests in o3 - network connected?https://progress.opensuse.org/issues/119059?journal_id=5971962023-01-26T15:39:42Zokurzokurz@suse.com
<ul></ul><p>Well, then I suggest to reconnect with the HMC, ensure it's powered off and then disconnect again</p>
openQA Infrastructure - action #119059: Use qa-power8 for ppc tests in o3 - network connected?https://progress.opensuse.org/issues/119059?journal_id=6006482023-02-10T11:50:56Zmkittlermarius.kittler@suse.com
<ul></ul><p>I connected again via <a href="https://powerhmc3.arch.suse.de" class="external">https://powerhmc3.arch.suse.de</a>. Then I disconnected all connections to other HMCs that were still shown (there were 2 of them). Then I disconnected it from powerhmc3 itself. Unfortunately this didn't help. One can actually list the connected HMCs in the ASM and it says that the following are still connected: 10.161.26.36, 172.16.0.1<br>
There was actually a 3rd one which I've removed. The remaining two aren't removable, though. Hence I'm not sure how to proceed. We somehow need to remove those (stale) entries.</p>
openQA Infrastructure - action #119059: Use qa-power8 for ppc tests in o3 - network connected?https://progress.opensuse.org/issues/119059?journal_id=6023452023-02-15T10:20:07Zlivdywanliv.dywan@suse.com
<ul></ul><blockquote>
<p>The remaining two aren't removable, though. Hence I'm not sure how to proceed. We somehow need to remove those (stale) entries.</p>
</blockquote>
<p>Let's check it in the infra call. Maybe Nick knows?</p>
openQA Infrastructure - action #119059: Use qa-power8 for ppc tests in o3 - network connected?https://progress.opensuse.org/issues/119059?journal_id=6024952023-02-15T12:29:33Zokurzokurz@suse.com
<ul></ul><p><a href="https://powerhmc1.arch.suse.de/dashboard/" class="external">https://powerhmc1.arch.suse.de/dashboard/</a> is currently "Service Unavailable". The ASM shows that there is still a connection to powerhmc1.arch.suse.de by IPv4 address. mkittler could remove some connections with a checkbox but not for hmc1. mkittler has tried "soft-reset" of the service processor. We now tried the "reset", i.e. "hard reset". right now <a href="https://qa-power8.qa.suse.de" class="external">https://qa-power8.qa.suse.de</a> is not reachable anymore. Maybe after reset and with hmc1 being broken the connection vanishes? Nope, the connection still show up. But I could still login over ssh to powerhmc1.arch.suse.de . I logged in using ssh <a href="mailto:hscroot@powerhmc1.arch.suse.de">hscroot@powerhmc1.arch.suse.de</a> using credentials from <a href="https://gitlab.suse.de/openqa/salt-pillars-openqa/-/blob/master/openqa/workerconf.sls#L1198" class="external">https://gitlab.suse.de/openqa/salt-pillars-openqa/-/blob/master/openqa/workerconf.sls#L1198</a><br>
and did</p>
<pre><code>rmsysconn -m qa-power8 -o remove
</code></pre>
<p>After that in the ASM <a href="https://qa-power8.qa.suse.de/" class="external">https://qa-power8.qa.suse.de/</a> in the menu "Hardware Management Consoles" I could remove the one connection using the checkbox. But there is still a connection "V467f48*7044a59" with IPv4 address 172.16.0.1 which according to nsinger is the "PowerVM HMC network" but we don't know which HMC that is. I guess the address ending with a .1 still refers to powerhmc1. That would be my guess. Right now powerhmc1.arch.suse.de is rebooting. I suggest to try again at a later time to login over ssh to powerhmc1 and if that does not help then open a ticket <em>and</em> ask in more channels.</p>
openQA Infrastructure - action #119059: Use qa-power8 for ppc tests in o3 - network connected?https://progress.opensuse.org/issues/119059?journal_id=6052462023-02-22T09:57:07Zlivdywanliv.dywan@suse.com
<ul></ul><p>I think this is actually this ticket, hence adding the bugref to <a href="https://openqa.suse.de/tests/10528653:" class="external">https://openqa.suse.de/tests/10528653:</a></p>
<pre><code>[2023-02-17T19:05:46.030419+01:00] [debug] [pid:575072] Could not connect to hscroot@powerhmc1.arch.suse.de, Retrying after some seconds...
[2023-02-17T19:05:56.033590+01:00] [info] [pid:575072] ::: backend::baseclass::die_handler: Backend process died, backend errors are reported below in the following lines:
Error connecting to <hscroot@powerhmc1.arch.suse.de>: Connection timed out
</code></pre> openQA Infrastructure - action #119059: Use qa-power8 for ppc tests in o3 - network connected?https://progress.opensuse.org/issues/119059?journal_id=6054592023-02-22T15:30:25Zlivdywanliv.dywan@suse.com
<ul></ul><p>See also <a class="issue tracker-4 status-3 priority-4 priority-default closed" title="action: test fails in bootloader_start - The command server of powerhmc1 cannot be connected size:M (Resolved)" href="https://progress.opensuse.org/issues/124391">#124391</a> regarding powerhmc1.arch.suse.de being unavailable, and <a href="https://suse.slack.com/archives/C02CANHLANP/p1677065099482609" class="external">discussions in Slack</a></p>
openQA Infrastructure - action #119059: Use qa-power8 for ppc tests in o3 - network connected?https://progress.opensuse.org/issues/119059?journal_id=6059302023-02-23T10:39:57Zmkittlermarius.kittler@suse.com
<ul><li><strong>Subject</strong> changed from <i>Use qa-power8 for ppc tests in o3 - network connected? size:M</i> to <i>Use qa-power8 for ppc tests in o3 - network connected?</i></li></ul><p>In the last mob session <a class="user active user-mention" href="https://progress.opensuse.org/users/17668">@okurz</a> wanted to set this up as barematel machine. We still struggle removing one HMC connection preventing us to continue with that approach.</p>
<p>Alternatively, we could continue with the HMC mode. This would likely mean setting up an HMC within the o3 network (or making HMC 3 somehow accessible).</p>
openQA Infrastructure - action #119059: Use qa-power8 for ppc tests in o3 - network connected?https://progress.opensuse.org/issues/119059?journal_id=6059332023-02-23T10:43:45Zmkittlermarius.kittler@suse.com
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/605933/diff?detail_id=568697">diff</a>)</li></ul> openQA Infrastructure - action #119059: Use qa-power8 for ppc tests in o3 - network connected?https://progress.opensuse.org/issues/119059?journal_id=6059932023-02-23T12:34:00Znicksingernsinger@suse.com
<ul></ul><p>I logged into powerhmc2 over ssh and did a ´cat /proc/net/fib_trie´ to confirm that hmc2 is indeed 172.16.0.1. Afterwards I could do another ´rmsysconn -m qa-power8 -o remove´. The option to change firmware mode was still not enabled but the "Hardware Management Consoles" where previous connections where displayed now showed that no HMCs are connected anymore and gave a quick guide how to "Reset the server to a non-HMC managed configuration". For this you need to go to "System Service Aids", select "Factory Configuration" and use the option "Reset server firmware settings". After this go back to "Hardware Management Consoles" and it now shows a button "Reset the server to a non-HMC managed configuration" which you can click. The ASM confirms with "Operation was successful. This Server is no longer managed by an HMC". Now the "Firmware Configuration" can be changed from "PowerVM" to "OPAL"</p>
openQA Infrastructure - action #119059: Use qa-power8 for ppc tests in o3 - network connected?https://progress.opensuse.org/issues/119059?journal_id=6067402023-02-27T15:41:22Zmkittlermarius.kittler@suse.com
<ul></ul><p>Looks like you also already have changed the config to "OPAL" and you've also set the console type to IPMI. Unfortunately I still wasn't able to connect via IPMI, e.g. <code>ipmitool -I lanplus -C 3 -H qa-power8.qa.suse.de -U … -P … power status</code> didn't work. Looks like this host is in the same state as the other o3 worker power8. I've also tried to set the IPMI password again (to be identical to the ASM password) but it didn't work.</p>
openQA Infrastructure - action #119059: Use qa-power8 for ppc tests in o3 - network connected?https://progress.opensuse.org/issues/119059?journal_id=6076312023-03-01T10:56:42Zokurzokurz@suse.com
<ul><li><strong>Copied to</strong> <i><a class="issue tracker-4 status-3 priority-5 priority-high3 closed child" href="/issues/125216">action #125216</a>: Use qa-power8 for ppc tests in o3 - try one of the suggestions size:M</i> added</li></ul> openQA Infrastructure - action #119059: Use qa-power8 for ppc tests in o3 - network connected?https://progress.opensuse.org/issues/119059?journal_id=6076432023-03-01T10:58:15Zokurzokurz@suse.com
<ul><li><strong>Tags</strong> changed from <i>infra, mob</i> to <i>infra</i></li><li><strong>Description</strong> updated (<a title="View differences" href="/journals/607643/diff?detail_id=570389">diff</a>)</li><li><strong>Status</strong> changed from <i>Workable</i> to <i>Blocked</i></li><li><strong>Assignee</strong> set to <i>okurz</i></li></ul><p>With <a class="issue tracker-4 status-6 priority-3 priority-lowest closed child" title="action: Use qa-power8 for ppc tests in o3 - network connected? (Rejected)" href="https://progress.opensuse.org/issues/119059#note-45">#119059#note-45</a> we now know how to set a machine to OPAL even if it refuses.</p>
<p>We see some options, updated the description and created two subtasks.</p>
openQA Infrastructure - action #119059: Use qa-power8 for ppc tests in o3 - network connected?https://progress.opensuse.org/issues/119059?journal_id=6139942023-03-16T00:12:26Zopenqa_reviewopenqa-review@suse.de
<ul></ul><p>This is an autogenerated message for openQA integration by the openqa_review script:</p>
<p>This bug is still referenced in a failing openQA test: create_hdd_textmode_hmc_ntlm<br>
<a href="https://openqa.suse.de/tests/10695007#step/bootloader/1" class="external">https://openqa.suse.de/tests/10695007#step/bootloader/1</a></p>
<p>To prevent further reminder comments one of the following options should be followed:</p>
<ol>
<li>The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted</li>
<li>The openQA job group is moved to "Released" or "EOL" (End-of-Life)</li>
<li>The bugref in the openQA scenario is removed or replaced, e.g. <code>label:wontfix:boo1234</code></li>
</ol>
<p>Expect the next reminder at the earliest in 28 days if nothing changes in this ticket.</p>
openQA Infrastructure - action #119059: Use qa-power8 for ppc tests in o3 - network connected?https://progress.opensuse.org/issues/119059?journal_id=6190072023-03-31T07:26:17Zokurzokurz@suse.com
<ul></ul><p>blocked by <a class="issue tracker-4 status-3 priority-5 priority-high3 closed child" title="action: Use qa-power8 for ppc tests in o3 - try one of the suggestions size:M (Resolved)" href="https://progress.opensuse.org/issues/125216">#125216</a></p>
openQA Infrastructure - action #119059: Use qa-power8 for ppc tests in o3 - network connected?https://progress.opensuse.org/issues/119059?journal_id=6288592023-05-03T07:25:16Zlivdywanliv.dywan@suse.com
<ul></ul><p>okurz wrote:</p>
<blockquote>
<p>blocked by <a class="issue tracker-4 status-3 priority-5 priority-high3 closed child" title="action: Use qa-power8 for ppc tests in o3 - try one of the suggestions size:M (Resolved)" href="https://progress.opensuse.org/issues/125216">#125216</a></p>
</blockquote>
<p>This was unblocked more recently, and I'm getting started on the evaluation. Given the priority we might also re-consider e.g. pick the easiest to setup and maintain depending on how strongly stakeholders feel about the options. Just a thought I would consider anyway, but let's see what we find.</p>
openQA Infrastructure - action #119059: Use qa-power8 for ppc tests in o3 - network connected?https://progress.opensuse.org/issues/119059?journal_id=6342882023-05-17T12:33:57Zokurzokurz@suse.com
<ul><li><strong>Status</strong> changed from <i>Blocked</i> to <i>New</i></li><li><strong>Assignee</strong> deleted (<del><i>okurz</i></del>)</li><li><strong>Priority</strong> changed from <i>High</i> to <i>Low</i></li><li><strong>Target version</strong> changed from <i>Ready</i> to <i>future</i></li></ul><p><a class="issue tracker-4 status-3 priority-5 priority-high3 closed child" title="action: Use qa-power8 for ppc tests in o3 - try one of the suggestions size:M (Resolved)" href="https://progress.opensuse.org/issues/125216#note-11">#125216-11</a></p>
openQA Infrastructure - action #119059: Use qa-power8 for ppc tests in o3 - network connected?https://progress.opensuse.org/issues/119059?journal_id=6343032023-05-17T12:36:28Zokurzokurz@suse.com
<ul><li><strong>Parent task</strong> changed from <i>#109743</i> to <i>#129499</i></li></ul> openQA Infrastructure - action #119059: Use qa-power8 for ppc tests in o3 - network connected?https://progress.opensuse.org/issues/119059?journal_id=7683072024-02-27T13:02:29Zokurzokurz@suse.com
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>Rejected</i></li><li><strong>Assignee</strong> set to <i>okurz</i></li></ul><p>The machine is in NUE3 as cold redundancy, <a href="https://racktables.nue.suse.com/index.php?page=object&object_id=2350" class="external">https://racktables.nue.suse.com/index.php?page=object&object_id=2350</a>, not usable in the planned form anymore.</p>