How to use Conda environments as custom Jupyter kernels

Running Jupyter Notebooks with Remote Conda Environments via Visual Studio Code

Simplifying your development workflow across environments

Working with Jupyter notebooks across different environments can be tricky, especially when you want the convenience of your preferred local editor with the power of remote compute resources. At EMBL we provide a Visual Studio Code (VSCode) server instance on demand that can be helpful to handle this situation.

The Challenge

Imagine you have a Jupyter notebook stored on a remote server and want to run it using VSCode on your local laptop. VSCode offers a great alternative notebook interface to Jupyter Notebooks and Jupyter Lab. Imagine you need to use a Conda environment to run this notebook but it is installed in the remote server. At first glance, this might seem like it requires complex bridging between local and remote setups.

The Solution: VSCode Server on JupyterHub

The solution is to bring all components together in one place (the remote machine) rather than trying to bridge local and remote environments. By using VSCode server directly on JupyterHub, you can streamline your entire workflow.

Step-by-Step Implementation

Here’s how to set this up:

  1. Open a terminal and connect to the remote machine where the notebook and the conda environment are
  2. Activate the Conda env conda activate my-env
  3. Add the IPython kernel in your target Conda environment: conda install ipykernel
  4. Start VSCode in JupyterHub (not your local machine) by selecting “Visual Studio Coding environment” from the menu.
  5. When the editor is ready, open the folder containing your notebook.
  6. Open the notebook
  7. Select kernel by:
    • Clicking the “Select kernel” button in the top right corner
    • Selecting “Python Environments…”
    • Secting the Conda environment of your choice
  8. Verify the kernel is starting correctly (check the top right corner) by, e.g. trying to run a cell.

Considerations

An important discovery: you don’t need to install the full Jupyter server in your conda environment. The ipykernel package alone is sufficient to run notebooks with a custom kernel from your conda environment.

EMBL JupyterHub offers more than just Jupyter notebooks. In fact it is a comprehensive environment that includes:

  • Jupyter notebooks
  • RStudio
  • VSCode
  • OpenRefine
  • RawGraphs

This makes it an attractive option for developers who want access to multiple tools without complex local configurations.

VSCode is also available through BAND, the cloud-based platform provided by the Image Analysis support team. This provides additional flexibility for users who need image analysis-oriented environments eg. FIJI, Napari, CellPose etc.

Why This Approach Works

This solution elegantly sidesteps the complexity of bridging local and remote environments by consolidating everything in the remote environment. Benefits include:

  • Simplified setup: No need for complex SSH tunneling or remote kernel configurations
  • Resource access: Full access to remote compute resources and file systems. This means that you can access everything that is stored in group shares and easily gain graphical access to more resources than a regular laptop.
  • Tool availability: Access to VSCode-specific features while maintaining notebook functionality via the VSCode interface.

This approach demonstrates that sometimes the best solution isn’t the most obvious one—instead of forcing local and remote components to work together, bringing everything to a unified remote environment can provide a streamlined development experience.

Francesco