The Hitchhiker's Guide to CLIs in Python — Part 2: Python packages for writing CLIs

04 May 2020

Let's see how we can write a command-line interface using Python. There are several options to do this, in the standard library and on PyPI.

We'll use an example CLI called smol-pip, and see how we can implement it using these different options.


  $ smol-pip install --upgrade package_name

smol-pip has one subcommand called install that can be used to install a package from PyPI. It also has an --upgrade option that will upgrade the package if it's already installed. And the package_name required argument to identify the package.

Standard library

sys

We'll look at the standard library first. It has the sys module that defines the argv variable. sys.argv is a list where the first element contains the name of the CLI that was invoked, and the rest of them are the command-line options passed to the CLI.

Internally, sys.argv uses the getopt module to parse and create this list. The getopt module implements a parser for command-line options whose API designed to similar to the Unix getopt function. It follows the POSIX standard.

Let's look at smol-pip, written using the sys module.


  import sys

  help = "Pip Installs Packages."

  if __name__ == "__main__":
      arguments = sys.argv
      if arguments[1] in ["-h", "--help"]:
          print(help)
      elif arguments[1] in ["-v", "--version"]:
          print("0.1.0")
      else:
          print(arguments)
          # ['smol-pip', 'install', '--upgrade', 'Click']
          if arguments[1] == "install":
              # dispatch to install / upgrade code
          else:
              raise ValueError("Unknown subcommand!")

When the CLI is invoked, we get the list of arguments using sys.argv. Since the first element is the name of the CLI itself, we check the element at index 1. If it is -h or --help, we print the help. And do the same for the version. Finally, we check the subcommand that was invoked, and dispatch control to the relevant code.

optparse

Up until Python 3.2, the standard library also had the optparse module. It has since been deprecated. optparse could parse only options, and not positional arguments. Steven Bethard, talks about it in detail in PEP 389. This PEP proposed the deprecation of optparse in favor of the new and improved argparse module! It was approved by Guido on February 21, 2010.

argparse

argparse was written because both getopt and optparse support only options, and not arguments. argparse handles both, and as a result, it is able to generate better help messages. argparse also allows customization of characters that are used to identify options. For example, using a + instead of a -, or even forward slashes.

-pf
-file
+f
+rgb
/f
/file

argparse also added support for subcommands. This is a common pattern in CLIs. For example:

pip install
pip freeze
pip search

Let's look at smol-pip, written using the argparse module.


  import argparse

  parser = argparse.ArgumentParser(
      description="Pip Installs Packages."
  )
  parser.add_argument(
      "-v",
      "--version",
      action="version",
      version="0.1.0"
  )

We import argparse, initialize a parser object and add the description of our CLI. We also add a version option, which will be print the version of our CLI when it is invoked with -v or --version.


  subparsers = parser.add_subparsers(dest="subparser_name")
  install = subparsers.add_parser("install")
  install.add_argument(
      "-u",
      "--upgrade",
      action="store_true",
      help="Upgrade package to the newest available version.",
  )
  install.add_argument("package_name")

We then initialize a subparsers object, and add a subparser for the install subcommand. To which we then add an --upgrade option and a package_name argument. The action=store_true makes sure that --upgrade is treated like a boolean flag.


  if __name__ == "__main__":
      arguments = parser.parse_args()
      print(arguments)
      # Namespace(package_name='Click', upgrade=True)
      if arguments.subparser_name == "install":
          # dispatch to install / upgrade code
      else:
          raise ValueError("Unknown subcommand!")

When the CLI is invoked, we use the parser.parse_args() function to get a Namespace object. This object contains the parsed command-line options and arguments as its attributes. Finally, we check the subcommand that was invoked, and dispatch control to the relevant code.


  $ smol-pip --help
  usage: smol-pip [-h] [-v] {install} ...

  Pip Installs Packages.

  positional arguments:
    {install}

  optional arguments:
    -h, --help     show this help message and exit
    -v, --version  show program's version number and exit

argparse automatically generates a help for our CLI! This can be viewed by invoking the CLI with -h or --help.

Python Package Index

Now let's look at some packages on PyPI.

docopt

docopt was written by Vladimir Keleshev, and is kinda cool in the way it works. It takes a documentation-first approach. docopt just requires a POSIX-compliant help string as an input from which it'll infer subcommands, options and arguments.


  help = """Pip Installs Packages.

  Usage:
    smol-pip install PACKAGE_NAME
    smol-pip install --upgrade PACKAGE_NAME

  Options:
    -h --help     Show this screen.
    --version     Show version.
  """

So this time, we first create a help string that shows our CLI's description and usage.


  from docopt import docopt

  if __name__ == "__main__":
      arguments = docopt(help, version="0.1.0")
      print(arguments)
      # {'--upgrade': True,
      #  'PACKAGE_NAME': 'Click',
      #  'install': True}
      if arguments["install"]:
          # dispatch to install / upgrade code
      else:
          raise ValueError("Unknown subcommand!")

And when the CLI is invoked, we call docopt(), pass in the help string and a version. And docopt() returns a dictionary of parsed command-line options and arguments! Finally, we check the subcommand that was invoked, and dispatch control to the relevant code.

In all the previous examples, we had to write some boilerplate to dispatch control to the relevant install and upgrade code, along with parsing results. We would need some more boilerplate if we had to validate the parsed command-line options and arguments. This boilerplate can grow real big for large applications.

Also, there might be some common CLI features we might want to add. For example, progress bars and colored text, which would require some more code. Let's look at a package that can help us reduce boilerplate, and implement common CLI features.

click

click (short for Command Line Interface Creation Kit) was written by Armin Ronacher to support the Flask project. It is designed to nestable and composable. It supports arbitrary nesting of commands. For example, python setup.py sdist bdist_wheel, where the bdist_wheel subcommand is called after sdist, kinda like subcommand chain.

click automatically dispatches control to relevant code based on the invoked subcommand. It also supports callbacks that can be used to validate parsed command-line options and arguments. And it's POSIX-compliant.

Let's look at smol-pip, written using the click.


  import click

  @click.group("pip")
  @click.version_option("0.1.0")
  def cli(*args, **kwargs):
      """Pip Installs Packages."""
      pass

We import click and create a cli() function with a docstring. click follows a decorator based approach to writing CLIs.

We add a click.group() decorator to the cli() function to make it a command group. A command group can contain contain one or more subcommands. We also add a version_option, which will print the version of our CLI when it is invoked with -v or --version.


  @cli.command("install")
  @click.option(
      "-u",
      "--upgrade",
      is_flag=True,
      help="Upgrade package to the newest available version.",
  )
  @click.argument("package_name")
  def install(*args, **kwargs):
      """Install packages."""
      # install / upgrade package_name

We then create a install() function with a docstring. This function will contain the code required to install or upgrade a package. We use the cli.command() decorator to convert this function into a subcommand. Here cli is the command group we defined earlier.

We then use the click.option() decorator to add an option called --upgrade along with a help string. The is_flag=True makes sure that --upgrade is treated like a boolean flag. We also add a package_name argument using the click.argument() decorator.


  if __name__ == "__main__":
      cli()

When the CLI is invoked, click will automatically dispatch control to the relevant code, which is the install() function in this case. We will get the parsed command-line options and arguments as kwargs to the install() function, which we can then use to install or upgrade a package.


  $ smol-pip --help
  Usage: smol-pip [OPTIONS] COMMAND [ARGS]...

    Pip Installs Packages.

  Options:
    --version  Show the version and exit.
    --help     Show this message and exit.

  Commands:
    install  Install packages.

Like argparse, click automatically generates a help for our CLI based on the function docstrings and option help strings we added! Again, this can be viewed by invoking the CLI with -h or --help.

click promises that when multiple apps written using click are strung together, they will work seamlessly. This is helpful because multiple people can work on smaller parts of a large CLI and stitch all of them together at the end.

The click way of writing a CLI where we don't have to define parsers from the start, or focus on our help text from the start, is great for quick iterations!

Continue to Part 3 — Writing and packaging a CLI using Click

Vinayak Mehta