Day 56 — The littlest Jupyter console
27 October 2020 · recurse-centerThis week I'm planning to build a terminal frontend for Jupyter (in Rust!) so today I started looking into how Jupyter works!
I found this doc which shows how messaging works in Jupyter. When we start a jupyter_console (like IPython), it starts a Python kernel in the "backend".
The kernel exposes some sockets, using which the jupyter_console "frontend" can communicate with the kernel (the "backend"). A single kernel can simultaneously be connected to multiple frontends.
These are the sockets that the kernel exposes:
- Shell: This is the socket where a frontend sends requests for executing code, among other things.
- IOPub: This is the socket where the kernel publishes "side effects" such as printing execution results to
stdout, printing error tracebacks tostderr, etc. - stdin: This socket is connected to all frontends and lets the kernel request input from "the active frontend".
- Control: This socket is used to send shutdown and restart messages to the kernel, and other kernel management messages.
- Heartbeat: This socket is used for monitoring the kernel's heartbeat to check if the frontend and the kernel are connected or not.
After that I read through the jupyter_console and jupyter_client codebases to understand how they send messages to the kernel. (zeromq!)
I thought that it might be fun to implement and see what the littlest (read "barely functioning") jupyter_console would look like! And came up with something which has the following limitations (and more that I can't list down right now):
- Assumes that there's only one frontend session active at any given time
- Supports only backspaces for editing code! (no arrows keys or tabs :( )
- Executes only one line of code (
\nand\tcan be added though!) - Only reads messages from the IOPub channel
- Does not store code history!
| import sys | |
| from jupyter_client import KernelManager | |
| try: | |
| manager = KernelManager() | |
| manager.start_kernel() | |
| client = manager.client() | |
| execution_state = "idle" | |
| execution_count = 1 | |
| while True: | |
| code = input(f"In [{execution_count}]: ") | |
| if not code.strip(): | |
| continue | |
| msg_id = client.execute(code) | |
| execution_state = "busy" | |
| # https://github.com/jupyter/jupyter_console/blob/c0bdeb1918f1235cf620ea10290a4c448aa56c68/jupyter_console/ptshell.py#L717 | |
| while execution_state != "idle" and client.is_alive(): | |
| # https://github.com/jupyter/jupyter_console/blob/c0bdeb1918f1235cf620ea10290a4c448aa56c68/jupyter_console/ptshell.py#L845 | |
| while client.iopub_channel.msg_ready(): | |
| msg = client.iopub_channel.get_msg() | |
| msg_type = msg["header"]["msg_type"] | |
| if msg_type == "status": | |
| execution_state = msg["content"]["execution_state"] | |
| elif msg_type == "stream": | |
| if msg["content"]["name"] == "stdout": | |
| print(msg["content"]["text"]) | |
| sys.stdout.flush() | |
| elif msg["content"]["name"] == "stderr": | |
| print(msg["content"]["text"], file=sys.stderr) | |
| sys.stderr.flush() | |
| elif msg_type == "execute_result": | |
| pass | |
| elif msg_type == "display_data": | |
| pass | |
| elif msg_type == "execute_input": | |
| execution_count = int(msg["content"]["execution_count"]) + 1 | |
| elif msg_type == "clear_output": | |
| pass | |
| elif msg_type == "error": | |
| for frame in msg["content"]["traceback"]: | |
| print(frame, file=sys.stderr) | |
| except KeyboardInterrupt: | |
| manager.shutdown_kernel() | |
| except Exception as e: | |
| print(e) |
It works (barely)! I want to call it cutypr lol!
In [1]: import os
In [2]: print(os.getcwd())
/home/vinayak/dev
In [3]:
Now I need to understand how zeromq works; and implement, client.execute(code), client.iopub_channel.msg_ready(), and client.iopub_channel.get_msg() in Rust! And also figure out how to start a Python kernel from Rust.