Command execution¶

The rok_common.cmdutils module provides some extra features on top of Python’s subprocess module that make command execution more flexible and platform-agnostic. In case you need to use Python to invoke and orchestrate CLI tools, we recommend using this library over the standard Python API for command execution.

Note

rok_common is available in any notebook (or related) image provided by Arrikto in your Kubeflow deployment. To test the examples in the following sections, create a new Notebook from the Kubeflow UI using the available jupyter-kale image.

Examples¶

Below follow some sample usages of the rok_common.cmdutils library:

Import the module¶

>>> from rok_common import cmdutils

Run a command and print to the console¶

>>> c = cmdutils.run("which echo")
/bin/echo
>>> c
<ExtCommand [MjY3bxTC7vY] `which echo', status=FINISHED (ret: 0), PID=18236, shell=False>

In case of an error, run() raises an exception:

>>> cmdutils.run("ls /does/not/compute")
ls: cannot access '/does/not/compute': No such file or directory
[vK8UUGJfmDE] Command failed with ret: 2. Stderr was not captured.
Traceback (most recent call last):
  ...
rok_common.cmdutils.CommandExecutionError: Command `<ExtCommand [vK8UUGJfmDE] `ls /does/not/compute', status=FINISHED (ret: 2), PID=20501, shell=False>' failed

Run a command and capture its output¶

>>> out = cmdutils.yld("which echo")
>>> out
'/bin/echo'

In case of an error, yld() raises an exception and reports the stderr as well:

>>> out = cmdutils.yld("ls /does/not/compute")
[xj8HgonPWQk] Command failed with ret: 2. Error log: ls: cannot access '/does/not/compute': No such file or directory

Traceback (most recent call last):
  ...
rok_common.cmdutils.CommandExecutionError: Command `<ExtCommand [xj8HgonPWQk] `ls /does/not/compute', status=FINISHED (ret: 2), PID=20243, shell=False>' failed. Error log: ls: cannot access '/does/not/compute': No such file or directory\n

Run a command and capture its output, while printing to the console¶

>>> import sys
>>> c = cmdutils.run("which echo", stdout=[cmdutils.STORE, sys.stdout],
...                  stderr=[cmdutils.STORE, sys.stderr])
/bin/echo
>>> c.out
'/bin/echo\n'

In case of an error, run() raises an exception, and both captures and reports the stderr in the error message as well:

>>> cmdutils.run("ls /does/not/compute",
...              stdout=[cmdutils.STORE, sys.stdout],
...              stderr=[cmdutils.STORE, sys.stderr])
ls: cannot access '/does/not/compute': No such file or directory
[xj8HgonPWQk] Command failed with ret: 2. Error log: ls: cannot access '/does/not/compute': No such file or directory
Traceback (most recent call last):
  ...
rok_common.cmdutils.CommandExecutionError: Command `<ExtCommand [xj8HgonPWQk] `ls /does/not/compute', status=FINISHED (ret: 2), PID=20243, shell=False>' failed. Error log: ls: cannot access '/does/not/compute': No such file or directory\n

Run a command and write to its stdin¶

>>> cmdutils.yld("sha256sum", input="test")
'9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08  -'

Run a command asynchronously¶

>>> c = cmdutils.run("sleep 2", wait=False)
>>> c.wait()

Run a command with a timeout¶

>>> cmdutils.yld("sleep infinity", timeout=1)
[h3EPxWt0B1A] Command did not terminate after 1.00 seconds
Traceback (most recent call last):
  ...
rok_common.cmdutils.CommandTimeoutError: Command `<ExtCommand [h3EPxWt0B1A] `sleep infinity', status=RUNNING, PID=22124, shell=False>' timed out after 1.00s

Run a command and preserve whitespace¶

>>> cmdutils.yld(["ls", "/tmp/file with spaces"])
'/tmp/file with spaces'

You should prefer the list form when the command contains user-controlled variables, or when whitespace should be treated literally. The same example in a string form fails:

>>> cmdutils.yld("ls /tmp/file with spaces")
[vRhtfmASMxw] Command failed with ret: 2. Error log: ls: cannot access
'/tmp/file': No such file or directory
ls: cannot access 'with': No such file or directory
ls: cannot access 'spaces': No such file or directory

Traceback (most recent call last):
  ...
rok_common.cmdutils.CommandExecutionError: Command `<ExtCommand [vRhtfmASMxw] `ls /tmp/file with spaces', status=FINISHED (ret: 2), PID=14434, shell=False>' failed. Error log: ls: cannot access '/tmp/file': No such file or directory\nls:cannot access 'with': No such file or directory\nls: cannot access 'spaces': No such file or directory\n

Note that you can make it work with a bit of extra care, but in general you should avoid doing so:

>>> cmdutils.yld("ls '/tmp/file with spaces'")
'/tmp/file with spaces'

Enable logging¶

We propose the INFO logging level for the cmdutils module:

>>> import logging
>>> logging.basicConfig(level=logging.INFO)
>>> cmdutils.yld("which echo")
INFO:rok_common.cmdutils:[8qmHl1CZfwQ] Started command with PID 22775: which echo
INFO:rok_common.cmdutils:[8qmHl1CZfwQ] Command exited with ret: 0
'/bin/echo'

In DEBUG, the logs are more verbose:

>>> import logging
>>> logging.basicConfig(level=logging.DEBUG)
>>> cmdutils.yld("which echo")
DEBUG:rok_common.cmdutils:[uheXgEB-8BM] Starting command: which echo
INFO:rok_common.cmdutils:[uheXgEB-8BM] Started command with PID 23086: which echo
DEBUG:rok_common.cmdutils:[uheXgEB-8BM] Waiting command
INFO:rok_common.cmdutils:[uheXgEB-8BM] Command exited with ret: 0
DEBUG:rok_common.cmdutils:[uheXgEB-8BM] Cleaning command
DEBUG:rok_common.cmdutils:[uheXgEB-8BM] Command execution finished
'/bin/echo'

Create a generic command that accepts parameters¶

>>> class Find(ExtCommand):
...
...     command = "find %(path)s -name %(name)s"
...
...     def __init__(self, path, name):
...            self.path = path
...            self.name = name
>>> extcom = Find(path="/usr", name="python*")
>>> extcom.start(wait=False)
>>> print extcom.status
RUNNING
>>> print extcom.out
/usr/bin/python
/usr/bin/python2
>>> extcom.wait()
>>> print extcom.out
/usr/bin/python
/usr/bin/python2
/usr/bin/python2.7
/usr/lib/python2.7
>>> print extcom.status
FINISHED
>>> print extcom.returncode
0