action #25800
closed[sle][y][functional] test for yast2-rmt
Added by riafarov over 6 years ago. Updated about 5 years ago.
0%
Description
Motivation¶
According to PRD, smt will be dropped, hence no sense in testing it even some functionality is still there. It will be replaced with rmt, for which we need to create a new test.
Acceptance criteria¶
- AC1: A simple test for yast2-rmt exists in openQA
Suggestions¶
- Create a new test module "tests/console/yast2_rmt.pm" to be scheduled within the yast2_ncurses scenario based on an existing test module (copy and adapt)
- Make it very very simple to install the package "yast2-rmt"
- Start the configuration part and cover the real basics
Further details¶
RMT repo can be found here: https://github.com/SUSE/rmt
Updated by okurz over 6 years ago
- Parent task deleted (
#20580)
new test added recently, deleting parent ticket because it's not about "initial setup" of sle15 testing.
Updated by okurz about 6 years ago
- Target version changed from Milestone 14 to Milestone 16
Since then IIRC "migration" has done something about RMT, recheck the sit later.
Updated by okurz about 6 years ago
- Subject changed from [sle][functional] Create new test for RMT, when available to [sle][y][functional] Create new test for RMT, when available
- Target version changed from Milestone 16 to Milestone 18
Updated by okurz almost 6 years ago
- Target version changed from Milestone 18 to Milestone 18
Updated by okurz almost 6 years ago
- Related to action #28465: [sle][rmt] Implement disconnect RMT on SLES15([SLE][migration][sle15sp1]) added
Updated by okurz almost 6 years ago
- Target version changed from Milestone 18 to future
Updated by skotov over 5 years ago
Please include testing of YaST plugin for RMT into tests https://github.com/SUSE/yast2-rmt
Updated by okurz over 5 years ago
- Subject changed from [sle][y][functional] Create new test for RMT, when available to [sle][y][functional] test for yast2-rmt
As discussed in #42764#note-5 QA SLE migration takes care about RMT in general, we can focus on yast2-rmt
Updated by okurz over 5 years ago
- Target version changed from future to Milestone 22
Updated by okurz over 5 years ago
- Description updated (diff)
- Due date set to 2019-01-29
- Status changed from New to Workable
Due to the fact that we have many yast module tests covered in openQA already IMHO it makes sense to cover this as well in openQA in at best a very simple way.
I hope that within the next sprint we could cover this in a very very simple and limited test, i.e. a "smoke test". This approach has proven to be quite helpful in the past because many modules uncovered already bugs which seem trivial.
Updated by riafarov over 5 years ago
- Estimated time set to 5.00 h
We don't want to introduce MM test yet for a full blown test, but rather plan it as a future enhancement.
We could test run SUSEconnect using localhost.
Updated by JRivrain over 5 years ago
Manual testing is not convincing so far. The RMT server does not appear to work properly :
susetest:/home/bernhard # SUSEConnect --list-extensions --url http://127.0.0.1 --debug
cmd options: '{:list_extensions=>true, :url=>"http://127.0.0.1", :write_config=>true, :debug=>true, :language=>"POSIX"}'
Merged options: #<SUSE::Connect::Config insecure=false, url="http://127.0.0.1", list_extensions=true, write_config=true, debug=true, language="POSIX">
Reading credentials from /etc/zypp/credentials.d/SCCcredentials
Read credentials: SCC_6964d32abca145b3b7fa01bbf627ab1f : 3ca2676e2ab84d5f
Reading credentials from /etc/zypp/credentials.d/SCCcredentials
Read credentials: SCC_6964d32abca145b3b7fa01bbf627ab1f : 3ca2676e2ab84d5f
Reading credentials from /etc/zypp/credentials.d/SCCcredentials
Read credentials: SCC_6964d32abca145b3b7fa01bbf627ab1f : 3ca2676e2ab84d5f
opening connection to 127.0.0.1:80...
opened
<- "GET /connect/systems/activations HTTP/1.1\r\nAccept-Encoding: identity\r\nAccept: application/json,application/vnd.scc.suse.com.v4+json\r\nUser-Agent: SUSEConnect/0.3.16\r\nAuthorization: Basic U0NDXzY5NjRkMzJhYmNhMTQ1YjNiN2ZhMDFiYmY2MjdhYjFmOjNjYTI2NzZlMmFiODRkNWY=\r\nContent-Type: application/json\r\nAccept-Language: POSIX\r\nConnection: close\r\nHost: 127.0.0.1\r\n\r\n"
-> "HTTP/1.1 404 Not Found\r\n"
-> "Server: nginx/1.14.0\r\n"
-> "Date: Wed, 16 Jan 2019 16:22:51 GMT\r\n"
-> "Content-Type: text/html\r\n"
-> "Content-Length: 169\r\n"
-> "Connection: close\r\n"
-> "\r\n"
reading 169 bytes...
-> "<html>\r\n<head><title>404 Not Found</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>404 Not Found</h1></center>\r\n<hr><center>nginx/1.14.0</center>\r\n</body>\r\n</html>\r\n"
read 169 bytes
Conn close
Merged options: #<SUSE::Connect::Config insecure=false, url="http://127.0.0.1", list_extensions=true, write_config=true, debug=true, language="POSIX">
opening connection to 127.0.0.1:80...
opened
<- "GET /connect/repositories/installer HTTP/1.1\r\nAccept-Encoding: identity\r\nAccept: application/json,application/vnd.scc.suse.com.v4+json\r\nUser-Agent: SUSEConnect/0.3.16\r\nContent-Type: application/json\r\nAccept-Language: POSIX\r\nConnection: close\r\nHost: 127.0.0.1\r\n\r\n"
-> "HTTP/1.1 404 Not Found\r\n"
-> "Server: nginx/1.14.0\r\n"
-> "Date: Wed, 16 Jan 2019 16:22:51 GMT\r\n"
-> "Content-Type: text/html\r\n"
-> "Content-Length: 169\r\n"
-> "Connection: close\r\n"
-> "\r\n"
reading 169 bytes...
-> "<html>\r\n<head><title>404 Not Found</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>404 Not Found</h1></center>\r\n<hr><center>nginx/1.14.0</center>\r\n</body>\r\n</html>\r\n"
read 169 bytes
Conn close
Your Registration Proxy server doesn't support this function. Please update it and try again.
I synced it with rmt-cli sync, but it does not work any better after. I may be missing some configuration step, but then can we consider that the yast module was successful if we need extra steps ?
Updated by JRivrain over 5 years ago
I have 2 problems :
- In HTTP : The paths requested by SUSEConnect or yast2-registration does not seem to exist on the server (/connect/systems/activations or, http://localhost/center/regsvc?command=listproducts for example)
- In HTTPS : I cannot get SUSEconnect nor yast registration tool to to work with the SSL certificate generated during the install with yast2-rmt. However, the URLS above, that do not work in HTTP, are actually accessible with a web browser ! I can even log in here http://localhost/center/regsvc?command=listproducts but when I provide the credentials (which am sure are valid) I get a json file saying "Invalid system credentials". Then cannot try again anyway. I assume this is not meant to be used with a web browser anyway.
In conclusion, we need to use ssl, but the registration tools "refuse" to use it. I will try to report a bug.
Updated by JRivrain over 5 years ago
I found a workaround, in /etc/SUSEConnect, set as insecure but use https as follows :
insecure: true
url: https://localhost
But now I get this error : "Registration server returned 'No product found' (422)" (see http://pastebin.nue.suse.com/27224/src)
I am communicating with SCC group #happy-customer on IRC. I hope they will have a hint on what to do next.
Updated by JRivrain over 5 years ago
The 422 error is because we did not have the proper subscriptions in our scc organization. Somebody from scc gave me access to another organization with all the subscriptions active. Hopefully it will work.
Remark : I am no sure it is a good idea to put this in the yast2-ncurses SUT, because we have the mirroring process takes at least 40 minutes.
Updated by JRivrain over 5 years ago
Proposed test :
# install, run the module
zypper in yast2-rmt rmt-server
yast2 rmt : use credentials from proxies tab here https://scc.suse.com/proxies?current_organization=279158
# Set-up the mirror
rmt-cli sync (up to 10 minutes)
rmt-cli products enable SLES/15.1/x86_64
rmt-cli mirror ( 10 minutes more )
# Set-up the client
SUSEConnect --cleanup
yes | /usr/share/rmt/public/tools/rmt-client-setup --host localhost --regcert https://localhost/rmt.crt
# Actual test
grep localhost /etc/SUSEConnect
SUSEConnect -s (status)
zypper lr -u |grep localhost
zypper in something ?
Updated by riafarov over 5 years ago
I totally agree with statement that we don't want test which runs 40 minutes, but could you please explore if we can inject lightweight mirrors? Also, I guess rmt-cli is the tool which YaST module uses, but testing that tool is out of scope. Let's maybe discuss later today.
Updated by okurz over 5 years ago
As discussed in the daily I suggest to limit the test to cover only yast related parts not relying on RMT itself. E.g. avoid relying on any syncing or external repos. In the most easy case just abort the yast UI after going over some steps. We could still test if e.g. the services are setup.
Updated by okurz over 5 years ago
@jivrain I think the ticket is actually "In Progress" by you, right?
Updated by JRivrain over 5 years ago
- Status changed from Workable to In Progress
Updated by JRivrain over 5 years ago
As discussed in IRC, we can also skip the registration phase, since it will create a new proxy for each test run. The SCC team says it is not a big deal, but also, using credentials in the test makes that we cannot test this on Opensuse because we would expose the credentials on www. And if we are not going to use rmt-cli or SUSEConnect, then we don't need it anyway.
So we definitely get a non-working RMT server since we are not registered, but we know if the UI works... more or less.
I am writing a test that just checks the UI and looks quickly what was created (systemd units, files, opened ports).
Updated by JRivrain over 5 years ago
- Status changed from In Progress to Feedback
Updated by oorlov over 5 years ago
- Due date changed from 2019-01-29 to 2019-02-12
Updated by JRivrain over 5 years ago
Not sure where to document this properly, but the "experiments" I did earlier could serve again in the future, e.g if we want to create a multi-machine test or more in-deep test.
This a way to install and test RMT-server. each step is important, and has to be done in that exact order
1) Install, run the module¶
# zypper in yast2-rmt rmt-server
# yast2 rmt (use credentials from proxies tab here https://scc.suse.com/proxies?current_organization=279158. You probably need to request access to SCC team)
- Every time we will do this, a new proxy will be created in https://scc.suse.com/proxies?current_organization=279158.
### 2) Set-up the rmt-server/mirror
# rmt-cli sync (up to 10 minutes)
# rmt-cli products enable SLES/15.1/x86_64 (Minimal addition, for speed. A better idea is to add everything that would be needed for our test client.)
- To see what we have active on our machine, we can type "SUSEConnect --status-text". Then compare with what is available on the server with "rmt-cli products --all". # rmt-cli mirror ( 10 minutes more, with only SLES/15.1/x86_64 enabled. A lot more with if we enable the desktop ) ### 3) Set-up the client # SUSEConnect --cleanup
- At that point, if we forget to cleanup, we may have a warning that suggests us to de-activate first (from the main SCC server) . Bad idea ! This would prevent all openqa clones of the same image to register or even use zypper. # yes | /usr/share/rmt/public/tools/rmt-client-setup --host localhost --regcert https://localhost/rmt.crt ### 4) Test it. # grep localhost /etc/SUSEConnect # SUSEConnect -s (or --status-text) # zypper lr -u |grep localhost # zypper in something
Additionally, it could be interesting to try to register a new (not already registered) machine,or add a custom repository.
Official doc : https://www.suse.com/documentation/sles-15/singlehtml/book_rmt/book_rmt.html
Updated by okurz over 5 years ago
JRivrain wrote:
Not sure where to document this properly, but the "experiments" I did earlier could serve again in the future, e.g if we want to create a multi-machine test or more in-deep test.
Sure, nice write-up. What was used so far regarding "document test procedures" are the following alternatives:
- Have it automated -> IMHO the only scalable good long-time sustainable way
- Document in "testopia" -> other teams are still using this, we do not and also do not want to anymore :)
- Just here in the ticket -> When we remember we can search for it and look it up again
- Provide a semi-automatic script as part of the product, e.g. a helper script in the package itself, that provides a benefit to real-world users but also makes testing easier
Our general approach is: Automate tests that we want to repeat, otherwise test something new every time with manual exploratory testing
Updated by riafarov over 5 years ago
https://openqa.suse.de/tests/2419377#step/yast2_rmt/5 fails for SLE 15 SP1
Updated by JRivrain over 5 years ago
Looks like the needles have not been uploaded to openqa ? That needle was merged already 3 days ago : https://github.com/os-autoinst/os-autoinst-needles-opensuse/pull/503/files#diff-cd8182150e9cab2d513bc094653d3638
EDIT : I got confused, this is for SLE. The needles have not been merged : https://gitlab.suse.de/openqa/os-autoinst-needles-sles/merge_requests/1051#note_156655
Updated by riafarov over 5 years ago
- Status changed from Feedback to Resolved
Updated by okurz over 5 years ago
good job! I would not even have expected that we can cover that much in a test without relying on RMT too much but looks convincing now :)
Updated by okurz over 5 years ago
FYI, https://openqa.suse.de/tests/2429843 failed in yast2_rmt. It seems the console font can change slightly so I will redo at least this one failed needle with a lower match level of 80% instead of the default 96%.
Updated by okurz over 5 years ago
- Status changed from Resolved to Feedback
ok this is becoming a bit too much work for me after creating some needles with a lower match ratio of 80% , now fails in https://openqa.suse.de/tests/2435890#step/yast2_rmt/10 on a needle that wants to cover the IP-adress? I am not sure if this is a wise idea but I will leave the choice to you guys.
Updated by riafarov over 5 years ago
okurz wrote:
ok this is becoming a bit too much work for me after creating some needles with a lower match ratio of 80% , now fails in https://openqa.suse.de/tests/2435890#step/yast2_rmt/10 on a needle that wants to cover the IP-adress? I am not sure if this is a wise idea but I will leave the choice to you guys.
I suggest that we disable this test for s390x
Updated by JRivrain over 5 years ago
Either this, or just create another set another set of needles for it, there seem to be a different font on s390. The second needle it's my (dumb) mistake, I did not consider we could have different network settings.
Updated by okurz over 5 years ago
I recommend to not maintain a second set of needles. Why not just reduce the match level to 80% in all needles and cover the one step specific about IP adress in a more generic way? I am fine when you want to exclude the test module in case of s390x. Maybe it is better though to exclude with something like the existing helper function has_ttys and a comment that this is about needle maintenance
Updated by JRivrain over 5 years ago
Oliver: sorry I missed your comment, I had already created new needles when I red it. I prefer not lowering match level to 80%, I remember seeing some crazy false positive with such kind of match level. I do not really mind to exclude it, but I don't see for what reason we should do so, it is just a matter of few more needles.
I can propose these PRs : https://github.com/os-autoinst/os-autoinst-needles-opensuse/pull/508 and https://gitlab.suse.de/openqa/os-autoinst-needles-sles/merge_requests/1064
Updated by okurz over 5 years ago
Well in cases like these I see a false positive unlikely. It can happen more when e.g. we want to distinguish in the partitioner sda1 and sdc1 which van really look alike depending on the "a" character in the used font. Please keep in mind that "just some more" needles is not a one-time task but we would need to adapt to any necessary needle change twice at least and that is likely done by different persons doing review in their domain so error-prone and much more work.
Please use either a single set of proper needles or exclude the test module for remote back ends with the differing font rendering. I assume the exclusion of the module in a PR takes about 20m, changing all needles the same.
Updated by JRivrain over 5 years ago
On ncurses, I remember seeing something matching that was not even remotely looking like the original, at a match level under 90 percent. So I'd rather exclude it than going very low. I will check what can be done though.
Updated by riafarov over 5 years ago
Test is still failing on s390x. So can it be handled with needling (e.g. with exclude area)? I would prefer that. If that will create big maintenance overhead, then I guess we can exclude it for s390x or make it smoke test for s390x and run full blown test for qemu.
Updated by JRivrain over 5 years ago
I would like to work on this but the shared worker is stuck 90% of the time ( http://amazing.suse.cz/admin/workers/13 ) I launched the job, it failed, then I could not re-launch it. I discussed with Matthias, he may have time to take a look tomorrow.
Updated by okurz about 5 years ago
You could use http://open.qa/docs/#_triggering_tests_based_on_an_any_remote_git_refspec_or_open_github_pull_request to run tests on s390x . I would still suggest to use a single needle set with lower match ratio. I know about false matches e.g. in case of full-screen needles on black screen terminals but not in cases like the yast ncurses UI with match areas that are way smaller. Keep in mind that the match ratio is depending on the match area sizes.
Updated by okurz about 5 years ago
@JRivrain depending on your current plan I suggest one of the following:
- use a single needle set with lower match ratio urgently (before sprint ends tomorrow) and close the ticket
- Create a follow-up ticket for the s390x case and exclude the test module on s390x from the schedule in this ticket
- Just exclude it without a plan to work on s390x as in #25800#note-38
Updated by riafarov about 5 years ago
- Status changed from Feedback to Resolved
- Assignee changed from riafarov to JRivrain