Project

General

Profile

action #48521

[sle][security][sle15sp1] Enhance utils->script_run_interactive() to report fail for any mismatch

Added by llzhao over 3 years ago. Updated over 3 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Enhancement to existing tests
Target version:
-
Start date:
2019-02-28
Due date:
% Done:

0%

Estimated time:
Difficulty:

Description

Enhance utils->script_run_interactive() to report fail for any mismatch,
to avoid test case fake "PASS".

related file:
".../lib/utils.pm"


Related issues

Related to openQA Tests - action #49100: [qam] test fails in usr_lib_dovecot_imap - test dies when logout ends with return code 1 even though action was completed successfullyResolved2019-03-12

History

#1 Updated by llzhao over 3 years ago

  • Assignee changed from llzhao to whdu

In some cases the interactive "# $command" fails but function "retrieve_mail_pop3()" returns "success".
This may induce test case fake "PASS".

Two negative testing examples for your reference:

  1. revise code ".../lib/utils.pm" as following on purpose
sub retrieve_mail_pop3 {
    ...
    script_run_interactive(
        "telnet localhost pop3",
        ...
            {
276                # prompt => qr/\+OK/m,                -> comment out this line
277                prompt => qr/\+error message/m,       -> using unmatching string on purpose           
                      string => "pass $pw\n",
            },

Run test case "usr_lib_dovecot_pop3" and check the log,
the command "# telnet" reports "-ERR Disconnected for inactivity. Connection closed by foreign host."
but function "retrieve_mail_pop3" returns "success".

FYI:
http://10.67.19.89/tests/629#step/usr_lib_dovecot_pop3/36

2 . revise code ".../lib/utils.pm" as following on purpose

...
256     script_run_interactive(
257         #"telnet localhost pop3",             -> comment out this line
258         "telnet localhost err-protocol",           -> using a error protocol on purpose   

Run test case "usr_lib_dovecot_pop3" and check the log,
the command "# telnet" reports "telnet: err-proptocol: bad port"
but function "retrieve_mail_pop3" returns "success" and this makes test case "PASS", it is "FAIL" indeed.

FYI:
http://10.67.19.89/tests/630#step/usr_lib_dovecot_pop3/36

#2 Updated by vsvecova over 3 years ago

  • Related to action #49100: [qam] test fails in usr_lib_dovecot_imap - test dies when logout ends with return code 1 even though action was completed successfully added

#3 Updated by whdu over 3 years ago

  • Status changed from In Progress to Resolved

Merged PR#6978

vsvecova: Telnet return code 1, which should be 0 because all operations succeeded. Need further investigation, or we could create a new poo.

Also available in: Atom PDF