HTTPS to a compute node, with a self-signed certificate
In the previous post, I showed the recipe you may be tempted to use to run JupyterLab on a compute node, and why it is a bad idea: the last mile, from the login node to the compute node, is unencrypted HTTP. In this post I’ll go through the easiest fix to understand, although a somewhat tedious one: make JupyterLab speak HTTPS itself, with a self-signed certificate.
As a reminder, the recipe was to run, in a job,
#!/bin/bash
#SBATCH --time=1:00:00
jupyter lab --no-browser --ip="$(hostname -i)" --port=8888then, on your laptop, tunnel to it through the login node,
ssh -N -L 8888:<compute node address>:8888 <login node>and open http://localhost:8888/lab?token=<token>, with the token printed in the job’s output. That last part is less reliable than it sounds: on some clusters, the job’s output only shows up on the login node after a delay, or only when the job ends.1 So below, you make the token yourself, before the job, and you never need to read the job’s output at all.
The one-time setup
HTTPS needs a certificate, and a key that goes with it. Normally a certificate is signed by a certificate authority that your browser trusts, which is how your browser knows it is talking to the right server. A compute node has no domain name for an authority to sign, so instead you sign your own.
You do this once, on the login node, and every job reuses it. In the same place, make the token too:
mkdir -p ~/.jupyter-tls
chmod 700 ~/.jupyter-tls
cd ~/.jupyter-tls
(
umask 077
openssl req -x509 -newkey rsa:2048 -nodes -days 365 \
-keyout key.pem -out cert.pem \
-subj "/CN=localhost" -addext "subjectAltName=DNS:localhost"
openssl rand -hex 24 > token
)umask 077 makes sure the files are created readable by you only. The certificate is for localhost because, through the tunnel, that is what your browser connects to, and it lasts a year, after which you run this again.2
Since your browser has no authority to vouch for this certificate, you are the one who vouches for it. You do this by its fingerprint, a hash of the certificate:
openssl x509 -in ~/.jupyter-tls/cert.pem -noout -fingerprint -sha256Note it down; you’ll need it once.3
The key and the token are the secrets here. Whoever has the key can pretend to be your JupyterLab, and whoever has the token can use it, whenever a job of yours is running. So treat ~/.jupyter-tls like your SSH keys: don’t copy it around, and if you think it has leaked, delete it and make a new one.
Modifying the original command
JupyterLab can serve HTTPS directly, given a certificate and a key, and it takes its token from JUPYTER_TOKEN if set. So the job becomes
#!/bin/bash
#SBATCH --time=1:00:00
export JUPYTER_TOKEN="$(cat ~/.jupyter-tls/token)"
jupyter lab --no-browser --ip="$(hostname -i)" --port=28417 \
--certfile="$HOME/.jupyter-tls/cert.pem" --keyfile="$HOME/.jupyter-tls/key.pem" \
--ServerApp.port_retries=0The port is now a random-looking one rather than Jupyter’s default, 8888, which is everyone’s first choice. The last option is so that, if the port is taken anyway, the job fails rather than quietly moving to another port, which you would only find out about from the job’s output.4
Without the job’s output, the one thing left to find out is which node the job is running on. squeue --me shows it, under NODELIST, once the job has started. The tunnel on your laptop is exactly the same ssh -L as before, with that node:
ssh -N -L 28417:<node>:28417 <login node>and you open https://localhost:28417/lab?token=<token>, with the token from ~/.jupyter-tls/token. Since the token does not change, neither does that URL.
The first time, your browser will warn you that the certificate is not trusted. That is expected: it is self-signed. Before you accept it, open the certificate from the warning page, and check that its SHA-256 fingerprint is the one you noted down. Then accept it, and your browser remembers it for localhost:28417, so the next job, with the same certificate, opens without a warning.5
Don’t skip that check, as it is the whole point. A man in the middle between the login node and the compute node can hand your browser a certificate of their own, and the browser would give you the same warning. The fingerprint is what tells you apart.6
Limitations
This fixes the eavesdropping: nobody on the cluster network can read your token, your code, or your data any more. It does not change the fact that JupyterLab is still listening on the compute node’s network address. Anyone on the cluster can still connect to the port, and the token is the only thing in the way. Now they cannot sniff it, but it is also the same token for every job, for a year, sitting in your home directory.
And it is tedious. A certificate and a token to make and keep safe, a fingerprint to check, and a warning to click through, more than once. A fixed port that someone else may have taken. And it only works this simply because JupyterLab can serve HTTPS itself.
Can the same trick work with RStudio in a container?
Technically, yes, but it becomes much more complicated. On an HPC you would most likely run the open-source RStudio Server, from a container, and the open-source edition cannot serve HTTPS: that is a feature of Posit Workbench, the commercial edition. So the certificate needs something else to serve it, a reverse proxy running next to RStudio on the compute node, which speaks HTTPS to your browser and plain HTTP to RStudio, so that the plain HTTP never leaves the node.
That proxy is one more program to get onto the cluster and configure, and not just any proxy will do. RStudio thinks it is serving plain HTTP, and redirects your browser accordingly, to http://, unless the proxy is one that understands HTTP and tells it otherwise. Then the container itself needs care: Rocker’s recipe for running RStudio Server with Apptainer already needs several directories bind-mounted, a password, and a handful of options. And on a node shared with other users’ jobs, they can reach RStudio’s plain HTTP port too, without going through the proxy, since loopback is shared by everyone on the node. So it can be done, but by now it is no longer the easy-to-understand option.
Conclusion
This is simple, but tedious. It is technically easy, and there is nothing HPC-specific about it: the same trick, a self-signed certificate and a check of its fingerprint, can be applied anywhere you run a service over a network you don’t trust. But it works this simply only for apps that can serve HTTPS themselves, and the RStudio you are likely to run on an HPC is not one of them. In the next post of this series, I’ll talk about a solution specific to HPC, or rather to Slurm, that is easy to deploy with minimal setup, and works the same whether it is JupyterLab or RStudio. Stay tuned.
Footnotes
Slurm writes the job’s output to
slurm-<job ID>.outfrom the compute node. Whether and when the login node sees it depends on the program buffering its output, on the shared filesystem caching it, and on whether the cluster writes it somewhere local first and copies it over at the end. Flushing the output, e.g. withPYTHONUNBUFFERED=1, fixes only the first.↩︎-nodesleaves the key unencrypted, so that JupyterLab can read it without asking you for a passphrase.↩︎Once per browser, that is. You’ll need it again when you connect from another browser or another laptop, when your browser forgets that you accepted the certificate (Firefox remembers it until you remove the exception, while some other browsers remember it only for a while, or until you clear your browsing data), and when you make a new certificate after a year, which comes with a new fingerprint to note down. Each time, it is the same check as the first.↩︎
It fails when something else on the node is already listening on that port: another user’s job, if your cluster shares nodes between jobs, or another job of yours on the same node. A random-looking port makes that unlikely, but not impossible. If it does fail, pick another number, below 32768 to stay out of the range Linux hands out for outgoing connections, and use it in both the job and the tunnel.↩︎
This is why I make the certificate once rather than once per job. With a new certificate every job, there is a new fingerprint to check every job, and a warning every job, and once that becomes routine, it is tempting to click through without checking. With one certificate, you check once, and from then on a warning means something: someone else’s certificate is on the other end, and that is when to stop.↩︎
And you can trust the fingerprint you are comparing with because you read it on the login node over
ssh, which is already encrypted.↩︎