Day 56 — The littlest Jupyter console

27 October 2020 · recurse-center

This week I'm planning to build a terminal frontend for Jupyter (in Rust!) so today I started looking into how Jupyter works!

I found this doc which shows how messaging works in Jupyter. When we start a jupyter_console (like IPython), it starts a Python kernel in the "backend".

The kernel exposes some sockets, using which the jupyter_console "frontend" can communicate with the kernel (the "backend"). A single kernel can simultaneously be connected to multiple frontends.

These are the sockets that the kernel exposes:

Shell: This is the socket where a frontend sends requests for executing code, among other things.
IOPub: This is the socket where the kernel publishes "side effects" such as printing execution results to stdout, printing error tracebacks to stderr, etc.
stdin: This socket is connected to all frontends and lets the kernel request input from "the active frontend".
Control: This socket is used to send shutdown and restart messages to the kernel, and other kernel management messages.
Heartbeat: This socket is used for monitoring the kernel's heartbeat to check if the frontend and the kernel are connected or not.

After that I read through the jupyter_console and jupyter_client codebases to understand how they send messages to the kernel. (zeromq!)

I thought that it might be fun to implement and see what the littlest (read "barely functioning") jupyter_console would look like! And came up with something which has the following limitations (and more that I can't list down right now):

Assumes that there's only one frontend session active at any given time
Supports only backspaces for editing code! (no arrows keys or tabs :( )
Executes only one line of code (\n and \t can be added though!)
Only reads messages from the IOPub channel
Does not store code history!

import sys from jupyter_client import KernelManager try: manager = KernelManager() manager.start_kernel() client = manager.client() execution_state = "idle" execution_count = 1 while True: code = input(f"In [{execution_count}]: ") if not code.strip(): continue msg_id = client.execute(code) execution_state = "busy" # https://github.com/jupyter/jupyter_console/blob/c0bdeb1918f1235cf620ea10290a4c448aa56c68/jupyter_console/ptshell.py#L717 while execution_state != "idle" and client.is_alive(): # https://github.com/jupyter/jupyter_console/blob/c0bdeb1918f1235cf620ea10290a4c448aa56c68/jupyter_console/ptshell.py#L845 while client.iopub_channel.msg_ready(): msg = client.iopub_channel.get_msg() msg_type = msg["header"]["msg_type"] if msg_type == "status": execution_state = msg["content"]["execution_state"] elif msg_type == "stream": if msg["content"]["name"] == "stdout": print(msg["content"]["text"]) sys.stdout.flush() elif msg["content"]["name"] == "stderr": print(msg["content"]["text"], file=sys.stderr) sys.stderr.flush() elif msg_type == "execute_result": pass elif msg_type == "display_data": pass elif msg_type == "execute_input": execution_count = int(msg["content"]["execution_count"]) + 1 elif msg_type == "clear_output": pass elif msg_type == "error": for frame in msg["content"]["traceback"]: print(frame, file=sys.stderr) except KeyboardInterrupt: manager.shutdown_kernel() except Exception as e: print(e)

It works (barely)! I want to call it cutypr lol!


  In [1]: import os
  In [2]: print(os.getcwd())
  /home/vinayak/dev

  In [3]:

Now I need to understand how zeromq works; and implement, client.execute(code), client.iopub_channel.msg_ready(), and client.iopub_channel.get_msg() in Rust! And also figure out how to start a Python kernel from Rust.

Vinayak Mehta

Day 56 — The littlest Jupyter console