Mutest User API

Mutest is a simple send/expect based testing framework.

This module implements the basic send/expect functionality for mutest. The test developer first creates a munet topology (Configuration) and then writes test scripts (“test cases”) which are composed of calls to the functions defined below (“steps”). In short these are:

Send/Expect functions:

Control/Utility functions:

Test scripts are located by the mutest command by their name. The name of a test script should take the form mutest_TESTNAME.py where TESTNAME is replaced with a user chosen name for the test case.

Here’s a simple example test script which first checks that a specific forwarding entry is in the FIB for the IP destination 10.0.1.1. Then it checks repeatedly for up to 10 seconds for a second forwarding entry in the FIB for the IP destination 10.0.2.1.

match_step("r1", 'vtysh -c "show ip fib 10.0.1.1"', "Routing entry for 10.0.1.0/24",
           "Check for FIB entry for 10.0.1.1")

wait_step("r1",
          'vtysh -c "show ip fib 10.0.2.1"',
          "Routing entry for 10.0.2.0/24",
          desc="Check for FIB entry for 10.0.2.1",
          timeout=10)

Notice that the call arguments can be specified by their correct position in the list or using keyword names, and they can also be specified over multiple lines if preferred.

All of the functions are documented and defined below.

exception munet.mutest.userapi.ScriptError

An unrecoverable script failure.

exception munet.mutest.userapi.CLIOnErrorError(desc='')

Enter CLI after error.

class munet.mutest.userapi.TestCaseInfo(tag, name, path)

Object to hold nestable TestCase Results.

Parameters:

  • tag (str)

  • name (str)

  • path (Path)

munet.mutest.userapi.section(desc)

Start a new section for steps, with a description.

This starts a new section of tests. The result is basically the same as doing a non-inline include. The current test number is used to form a new sub-set of test steps. So if the current test number is 2.3, a section will now number subsequent steps 2.3.1, 2.3.2, …

A subsequent section() or non-inline include() call ends the current section and advances the base test number.

Parameters:

desc (str) – the description for the new section.

munet.mutest.userapi.log(fmt, *args, **kwargs)

Log a message in the testcase output log.

munet.mutest.userapi.include(pathname, new_section=False)

Include a file as part of testcase.

Parameters:

  • pathname (str) – the file to include.

  • new_section – if a new section should be created, otherwise commands are executed inline.

munet.mutest.userapi.script_dir()

The pathname to the directory containing the current script file.

When an include() is called the script_dir is updated to be current with the includeded file, and is reverted to the previous value when the include completes.

Return type:

Path

munet.mutest.userapi.get_target(name)

Get the target object with the given name.

Parameters:

name (str)

Return type:

Commander

munet.mutest.userapi.pause()

Pause the mutest and allow a user to either enter pdb or the cli.

munet.mutest.userapi.step(target, cmd)

Execute a cmd on a target and return the output.

Parameters:

  • target (str) – the target to execute the cmd on.

  • cmd (str) – string to execute on the target.

Returns:

Returns the str output of the cmd.

Return type:

str

munet.mutest.userapi.step_json(target, cmd)

Execute a json cmd on a target and return the json object or array.

Parameters:

  • target (str) – the target to execute the cmd on.

  • cmd (str) – string to execute on the target.

Returns:

Returns the json object or array after parsing the cmd output.

If json parse fails, a warning is logged and an empty dict is used.

Return type:

list | dict

munet.mutest.userapi.test_step(expr_or_value, desc, target='')

Evaluates expr_or_value and posts a result base on it bool(expr).

If expr_or_value evaluates to a positive result (i.e., True, non-zero, non-None, non-empty string, non-empty list, etc..) then a PASS result is recorded, otherwise record a FAIL is recorded.

Parameters:

  • expr – an expression or value to evaluate

  • desc (str) – description of this test step.

  • target (str) – optional target to associate with this test in the result string.

  • expr_or_value (Any)

Returns:

A bool indicating the test PASS or FAIL result.

Return type:

bool

munet.mutest.userapi.match_step(target, cmd, match, desc='', expect_fail=False, flags=re.DOTALL, exact_match=False)

Execute a cmd on a target check result.

Execute cmd on target and check if the regexp in match matches or doesn’t match (according to the expect_fail value) the cmd output.

If the match regexp includes groups and if the match succeeds the group values will be returned in a list, otherwise the command output is returned.

Parameters:

  • target (str) – the target to execute the cmd on.

  • cmd (str) – string to execut on the target.

  • match (str) – regex to match against output.

  • desc (str) – description of test, if no description then step failure is not considered an error and no result is logged.

  • expect_fail (bool) – if True then succeed when the regexp doesn’t match.

  • flags (int) – python regex flags to modify matching behavior

  • exact_match (bool) – if True then match must be exactly matched somewhere in the output of cmd using str.find().

Returns:

Returns a 2-tuple. The first value is a bool indicating success. The second value will be a list from re.Match.groups() if non-empty, otherwise re.Match.group(0) if there was a match otherwise None.

Return type:

(<class ‘bool’>, Union[str, list])

munet.mutest.userapi.match_step_json(target, cmd, match, desc='', expect_fail=False, exact_match=False)

Execute a cmd on a target check result.

Execute cmd on target and check if the json object or array in match matches or doesn’t match (according to the expect_fail value) the json output from cmd.

Parameters:

  • target (str) – the target to execute the cmd on.

  • cmd (str) – string to execut on the target.

  • match (str | list | dict) – A json str, object (dict), or array (list) to compare against the json output from cmd.

  • desc (str) – description of test, if no description then step failure is not considered an error and no result is logged.

  • expect_fail (bool) – if True then succeed if the a json doesn’t match.

  • exact_match (bool) – if True then the json must exactly match.

Returns:

Returns a 2-tuple. The first value is a bool indicating success. The second value is a dict of the diff if there is a difference found in the json compare, otherwise the value is the json object (dict) or array (list) from the cmd.

If json parse fails, a warning is logged and an empty dict is used.

Return type:

(<class ‘bool’>, Union[list, dict])

munet.mutest.userapi.wait_step(target, cmd, match, desc='', timeout=10.0, interval=0.5, expect_fail=False, flags=re.DOTALL, exact_match=False)

Execute a cmd on a target repeatedly, looking for a result.

Execute cmd on target, every interval seconds for up to timeout seconds until the output of cmd does or doesn’t match (according to the expect_fail value) the match value.

Parameters:

  • target (str) – the target to execute the cmd on.

  • cmd (str) – string to execut on the target.

  • match (str | dict) – regexp to match against output.

  • timeout (float) – The number of seconds to repeat the cmd looking for a match (or non-match if expect_fail is True).

  • interval (float) – The number of seconds between running the cmd. If not specified the value is calculated from the timeout value so that on average the cmd will execute 10 times. The minimum calculated interval is .25s, shorter values can be passed explicitly.

  • desc (str) – description of test, if no description then step failure is not considered an error and no result is logged.

  • expect_fail (bool) – if True then succeed when the regexp doesn’t match.

  • flags (int) – python regex flags to modify matching behavior

  • exact_match (bool) – if True then match must be exactly matched somewhere in the output of cmd using str.find().

Returns:

Returns a 2-tuple. The first value is a bool indicating success. The second value will be a list from re.Match.groups() if non-empty, otherwise re.Match.group(0) if there was a match otherwise None.

Return type:

(<class ‘bool’>, Union[str, list])

munet.mutest.userapi.wait_step_json(target, cmd, match, desc='', timeout=10, interval=None, expect_fail=False, exact_match=False)

Execute a cmd repeatedly and wait for matching result.

Execute cmd on target, every interval seconds until the output of cmd matches or doesn’t match (according to the expect_fail value) match, for up to timeout seconds.

Parameters:

  • target (str) – the target to execute the cmd on.

  • cmd (str) – string to execut on the target.

  • match (str | list | dict) – A json object, json array, or str representation of json to compare against json output from cmd.

  • desc (str) – description of test, if no description then step failure is not considered an error and no result is logged.

  • timeout – The number of seconds to repeat the cmd looking for a match (or non-match if expect_fail is True).

  • interval – The number of seconds between running the cmd. If not specified the value is calculated from the timeout value so that on average the cmd will execute 10 times. The minimum calculated interval is .25s, shorter values can be passed explicitly.

  • expect_fail (bool) – if True then succeed if the a json doesn’t match.

  • exact_match (bool) – if True then the json must exactly match.

Returns:

Returns a 2-tuple. The first value is a bool indicating success. The second value is a dict of the diff if there is a difference found in the json compare, otherwise the value is a json object (dict) or array (list) from the cmd output.

If json parse fails, a warning is logged and an empty dict is used.

Return type:

(<class ‘bool’>, Union[list, dict])

munet.mutest.userapi.luInclude(filename, CallOnFail=None)

Backward compatible API, do not use in new tests.

munet.mutest.userapi.luLast(usenl=False)

Backward compatible API, do not use in new tests.

munet.mutest.userapi.luCommand(target, cmd, regexp='.', op='none', result='', ltime=10, returnJson=False, wait_time=0.5)

Backward compatible API, do not use in new tests.

Only non-json is verified to any degree of confidence by code inspection.

For non-json should return match.group() if match else return bool(op == “fail”).

For json if no diff return the json else diff return bool(op == “jsoncmp_fail”)

bug if no json from output (fail parse) could maybe generate diff, which could then return

class munet.mutest.userapi.TestCase(tag, name, path, targets, args, output_logger=None, result_logger=None, full_summary=False)

A mutest testcase.

This is normally meant to be used internally by the mutest command to implement the user API. See README-mutest.org for usage details on the user API.

Parameters:

  • tag (int) – identity of the test in a run. (x.x…)

  • name (str) – the name of the test case

  • path (Path) – the test file that is being executed.

  • targets (dict) – a dictionary of objects which implement cmd_nostatus(str)

  • output_logger (Logger) – a logger for output and other messages from the test.

  • result_logger (Logger) – a logger to output the results of test steps to.

  • full_summary (bool) – if True then print entire doctstring instead of only the first line in the results report

  • args (Namespace)

tag

identity of the test in a run

name

the name of the test

targets

dictionary of targets.

steps

total steps executed so far.

passed

number of passing steps.

failed

number of failing steps.

last

the last command output.

last_m

the last result of re.search during a matching step on the output with newlines converted to spaces.

Parameters:

  • tag (int)

  • name (str)

  • path (Path)

  • targets (dict)

  • args (Namespace)

  • output_logger (Logger)

  • result_logger (Logger)

  • full_summary (bool)

step(target, cmd)

See step().

Parameters:

  • target (str)

  • cmd (str)

Return type:

str

step_json(target, cmd)

See step_json().

Parameters:

  • target (str)

  • cmd (str)

Return type:

list | dict

match_step(target, cmd, match, desc='', expect_fail=False, flags=re.DOTALL, exact_match=False)

See match_step().

Parameters:

  • target (str)

  • cmd (str)

  • match (str)

  • desc (str)

  • expect_fail (bool)

  • flags (int)

  • exact_match (bool)

Return type:

(<class ‘bool’>, Union[str, list])

match_step_json(target, cmd, match, desc='', expect_fail=False, exact_match=False)

See match_step_json().

Parameters:

  • target (str)

  • cmd (str)

  • match (str | list | dict)

  • desc (str)

  • expect_fail (bool)

  • exact_match (bool)

Return type:

(<class ‘bool’>, Union[list, dict])

wait_step(target, cmd, match, desc='', timeout=10, interval=0.5, expect_fail=False, flags=re.DOTALL, exact_match=False)

See wait_step().

Parameters:

  • target (str)

  • cmd (str)

  • match (str | dict)

  • desc (str)

  • expect_fail (bool)

  • flags (int)

  • exact_match (bool)

Return type:

(<class ‘bool’>, Union[str, list])

wait_step_json(target, cmd, match, desc='', timeout=10, interval=None, expect_fail=False, exact_match=False)

See wait_step_json().

Parameters:

  • target (str)

  • cmd (str)

  • match (str | list | dict)

  • desc (str)

  • expect_fail (bool)

  • exact_match (bool)

Return type:

(<class ‘bool’>, Union[list, dict])