Run the SigProfilerExtractor Workbench on a server or cluster¶
Run the Workbench (SigProfilerExtractor's browser app) on a Linux server (a lab workstation, an HPC cluster, a cloud VM) and use it from the browser on your own computer: upload inputs, start runs, follow them, explore results and download them, the same way you use a Jupyter server.
- No admin rights needed: no
sudo, no Docker, no system Python. One script,spe-server, installs everything into a folder of yours. - It runs as you: each person starts their own copy, with their own link, files and runs.
- GPUs: it installs PyTorch's CUDA 12.8 build, which uses an NVIDIA GPU when there is one and the CPUs otherwise.
- On a server or a cluster:
startruns it on the machine you are on;submitruns it as a SLURM job on a cluster's compute node (which one?).
You open a link with a token in it. The token is the key to the app: anyone with the link can read your files on the server and run programs as you, so keep it private.
What you need¶
- A Linux server you can log in to with SSH (x86-64 or ARM).
bash,git,tar, andcurlorwget, which nearly every server has.- Internet access from the server the first time, to github.com, pypi.org and download.pytorch.org. After that it can be offline.
- About 8 GB free for the install (2 GB with
--torch cpu), plus room for results and any reference genomes (about 3 GB each). It asks for 20 GB free to start.
Which one: start or submit?¶
The launcher runs the app in one of two ways. Pick by where you are logged in:
./spe-server start |
./spe-server submit |
|
|---|---|---|
| Use it on | a machine you may run work on directly: a lab workstation, a cloud VM, an interactive session on a compute node | a cluster's login node, where SLURM (sbatch) runs the work |
| Where the app runs | on this machine, in the background | on a compute node SLURM picks, as a batch job |
| CPUs, memory, GPUs | this machine's, shared with anyone else on it (limit with --cpus) |
only what the job asks for (--cpus, --memory, --gpus), its own |
| How long it runs | until stop, or until the machine restarts |
until stop or the job's time limit (--time); a job may also wait in the queue first |
| The link | this machine's name | the compute node's name; the tunnel goes through the login node |
On a cluster, use submit from the login node. Login nodes are shared by
everyone for editing and submitting jobs; a fit running there slows them all,
and clusters often stop long or heavy processes on them. Run start on a
cluster only inside an interactive job on a compute node (for example
srun --pty bash), and then the app ends when that session ends.
Not sure? If sbatch exists (command -v sbatch) and the machine's name says
"login" or "head", use submit; otherwise use start. Either way, every other
command (url, status, logs, stop, ...) works the same, and the data
folder is the same, so you can switch between them: stop one, then run the
other.
1. Get the code¶
Log in to the server and clone the repository on a disk with room, such
as a lab or project share, not a small home folder. Your data goes in
spe-data next to the clone.
cd /path/with/room # e.g. /lab/share/web_servers
git clone -b workbench-server https://github.com/SigProfilerSuite/SigProfilerExtractor.git
cd SigProfilerExtractor/server
2. Start it¶
./spe-server start
The first start downloads uv, which installs Python 3.13, PyTorch and SigProfilerExtractor from this clone (several minutes, once). Then it starts the app in the background and prints:
SigProfilerExtractor Workbench is running on myserver.example.edu.
Open this link in your browser (keep it private: it is the key to the app):
http://myserver.example.edu:8765/?token=45abac...
Not reachable from your computer (firewall, off campus, a cloud VM)? Run this on your
computer, keep it open, and use the link with localhost in place of
myserver.example.edu:
ssh -N -L 8765:localhost:8765 you@myserver.example.edu
Your files: /path/with/room/spe-data
On a cluster's login node, run ./spe-server submit instead (which
one?): it installs the same way, then submits a
SLURM job that runs the app on a compute node, and prints the link once the job
has started (on a SLURM cluster has the options).
3. Open it in your browser¶
Try the link as printed first. On campus or on the VPN it often just works.
If the page does not load (it times out), the server's port is closed to your computer. Make an SSH tunnel:
- Open a terminal on your own computer: Terminal on a Mac, PowerShell on
Windows 10 or 11 (both have
ssh). - Paste the
ssh -N -L ...command the launcher printed and log in as you normally do. It then shows nothing; that is right. Leave it open while you use the app. - Open the link with
localhostin place of the server's name:http://localhost:8765/?token=45abac...
Print the link and the tunnel command again any time with ./spe-server url.
4. Use it¶
The Workbench is the same as on your own computer, with these differences:
- Choosing files on the server: the Choose and Browse buttons open the Workbench's own file browser, which shows the server's folders (a window from your computer's system cannot show them). It starts in your home folder; the buttons along its top jump to Home, Results, the Data folder and Uploads. Open folders by clicking them, click .. up one level to go up, then click a file, or Use this folder for a folder. Folders holding results are marked results, and inputs are marked calls, matrix or bedpe. You can also paste a path on the server.
- Inputs on your computer: click Upload beside Choose in the Input
card. For variant calls or BEDPE files, select all the files at once (they
go into one folder); for a matrix or a segmentation file, select the one
file. A bar shows the progress; the files land in a new folder in the data
folder's
uploads/(named likevcf_20261114-153012), and the Workbench then uses them as the input. Up to 8 GB per upload. For larger inputs, copy them to the server withscporrsyncand choose them there. - Results are saved on the server, in the data folder's
SigProfilerExtractor_results/unless you choose another output folder. Open a run's results and click Download all as .zip at the top: the server packs the folder first (it shows how many files and how much), then your browser downloads it. Single files have their own Download buttons. - GPU: tick Use the GPU in a run's Advanced settings.
Closing the browser stops nothing: runs keep going, and the same link finds them later, even from another computer. The app keeps running after you log out of the server.
5. Stop it¶
./spe-server stop
This stops the app and any run in progress; so does Quit in the
Workbench's top bar. Start it again with ./spe-server start; the link
stays the same. It does not come back by itself after the server restarts.
On a SLURM cluster¶
Clone on a disk the compute nodes share (a lab or project share), then, on the login node:
./spe-server submit --gpus 1 --cpus 32 --memory 128G --time 2-00:00:00 -- --partition=NAME
It installs on the login node first (compute nodes often have no internet),
submits a job that runs the Workbench, waits for the job to start, and prints
the link. Some login nodes cap the memory each program may use (ulimit -v
shows it; 500 MB on some), too little to install or load PyTorch. There it
says so and the job installs everything on its compute node first instead,
which adds several minutes to the first submit; ./spe-server logs follows
it. Before submitting it shows exactly what the job asks for:
spe-server: submitting a SLURM job that asks for:
CPUs 32 given; change with --cpus N
memory 128G given; change with --memory SIZE (e.g. 64G)
GPUs 1 given; change with --gpus N
time 2-00:00:00 given; change with --time T (e.g. 8:00:00, 2-00:00:00)
other --partition=NAME (given; options after --)
(each is remembered for the next submit)
Asking for resources¶
The job holds what it asks for from the moment it starts until it ends, whether a run is going or not. Every run you start in the Workbench runs inside it and can use only that.
| You want | Option | Becomes (sbatch) |
Default |
|---|---|---|---|
| CPUs | --cpus N |
--cpus-per-task N |
8 |
| Memory | --memory SIZE, e.g. 64G, 500M |
--mem SIZE (for the whole job) |
32G |
| GPUs | --gpus N |
--gres gpu:N |
0: none |
| Time limit | --time T: MM:SS, HH:MM:SS or D-HH:MM:SS |
--time T |
the partition's default, which can be short: set it |
| Anything else | after --, as sbatch takes it: --partition=NAME, --account=NAME, --qos=NAME, --constraint=... |
the same | none |
How much to ask for:
- CPUs: a run fits its NMF replicates in parallel, one per CPU (the Computer fields in Advanced settings use all of the job's CPUs unless you set fewer), so more CPUs make a run faster.
- GPUs: ask for one (
--gpus 1) to use Use the GPU in a run's Advanced settings; without one the run uses the CPUs. Most clusters put GPUs in their own partition, so add it after--. - Memory: grows with the number of samples, mutation contexts and CPUs. A
safe start is about 4 GB per CPU (32 CPUs, 128G). Too little, and SLURM
stops the job when it runs out (
OUT_OF_MEMORYinstatus). - Time: long enough for every run you plan, plus some margin. A small test (21 samples, 1 to 2 signatures, 3 replicates) took 2 minutes on 4 CPUs; a whole cohort with the defaults (1 to 25 signatures, 100 replicates) can take many hours. When the limit comes, the job ends and so does any run still going (the app is told a minute early, so it stops them cleanly).
- Partition, account, QOS: whatever your cluster requires. Many clusters
refuse a job without
--account, or put it in a default partition with a short time limit.
Your cluster's limits: sinfo -o "%P %l %c %m %G" lists each partition's
time limit, CPUs, memory and GPUs per node; sacctmgr show qos
format=name,maxwall the QOS limits; your cluster's documentation, which
accounts you have. Asking for more than a partition allows leaves the job
waiting forever or refused; asking for less makes it start sooner.
Changing them later: ./spe-server stop, then submit again with the
new options. Options you leave out keep their last value, so
./spe-server submit --time 1-00:00:00 changes only the time limit.
Using the job¶
The other commands then work on the job: url prints the link (also once a
waiting job starts), status shows the node and the time left or why it is
still waiting, logs follows its log, and stop cancels it, stopping any
run first. squeue -u $USER shows it as spe-server-<you>.
Compute nodes are usually reachable only from inside the cluster, so the printed tunnel goes through the login node:
ssh -N -L 8765:node12.cluster.edu:8765 you@login.cluster.edu
Run it on your computer, then open the link with localhost in place of the
node's name.
On the UAlbany DGX cluster¶
From dgx-head01:
./spe-server submit --gpus 1 --cpus 32 --memory 128G --time 2-00:00:00 \
-- --partition=dgx --qos=freetier
freetier allows up to 7 days but can be preempted by paidtier-1 jobs;
--qos=paidtier-1 is not preempted but allows at most 1 day.
Commands¶
| Command | Does |
|---|---|
./spe-server start |
Start it and print the link |
./spe-server submit |
Run it as a SLURM job instead |
./spe-server url |
Print the link and the tunnel command again |
./spe-server status |
Is it running? (a job: where, and the time left) |
./spe-server logs |
Follow its log (Ctrl-C to leave; the app keeps running) |
./spe-server stop |
Stop it, and any run in progress |
./spe-server new-token |
New link; the old one stops working |
./spe-server update |
Reinstall from the clone and restart (git pull first) |
./spe-server install |
Install only (start does it when needed) |
./spe-server --help |
All of this, in the terminal |
Options¶
None are needed. Each one you give is remembered for the next start or
submit on that server.
| Option | Default | Meaning |
|---|---|---|
--data DIR |
spe-data beside the clone |
Where the install, uploads, results, genomes and run records go |
--port N |
8765 | The port to try first; if it is taken, the next free one |
--cpus N |
all (submit: 8) |
CPUs the runs may use; all removes a limit |
--no-gpu |
GPUs used | Hide the GPUs from the app; --gpu undoes it |
--torch BUILD |
cu128 |
PyTorch build: cu128 (NVIDIA driver 525 or newer; also runs on CPUs) or cpu (6 GB smaller). Changing it reinstalls PyTorch |
--gpus N |
0 | submit only: GPUs the job asks for |
--memory SIZE |
32G | submit only: memory the job asks for |
--time T |
the partition's | submit only: the job's time limit, e.g. 2-00:00:00 |
Good to know¶
- Where your files go. Everything is in the data folder: the install in
.workbench/install/, uploads inuploads/, results inSigProfilerExtractor_results/, reference genomes inreferences/, run records in.cache/. In the clone, installing adds only files git ignores (SigProfilerExtractor/_version.py,*.egg-info); nothing large goes in your home folder. - One copy per data folder. Run records are kept per data folder, and a
run is tracked by its process on its own machine, so
startandsubmitrefuse while a copy for the same data folder runs anywhere, and say where. - Ports. If someone else on the server already has 8765, the app takes the next free one and the link shows it. That port is tried first next time, so a bookmarked link keeps working.
- CPUs. Runs use at most the CPUs the app has:
--cpus, a SLURM job's CPUs, or all. The Computer fields in Advanced settings are capped there, and a run asking for more is refused. On a shared server, leave room for others with--cpus. - Updating.
git pull, then./spe-server stopandstartto run the new code; changed requirements are installed on that start. Or./spe-server update, which reinstalls everything and restarts. The launcher was calledspe-workbenchbefore;spe-serverfinds the same data folder, install and running app or job. - Two people on one server each clone and start their own copy; they get different ports, tokens, files and runs.
Troubleshooting¶
"that disk has only N GB free": the data folder would be on a small disk,
such as a home folder. Give a folder on a larger one; it is remembered:
./spe-server start --data /lab/share/spe-data.
"could not download ..." or "could not install PyTorch": the server cannot
reach the internet. If it uses a proxy, set https_proxy before starting. On
a cluster, install on the login node: submit does that, or run
./spe-server install there.
"memory allocation of ... bytes failed" or "Aborted (core dumped)" while
installing: the login node caps each program's memory (ulimit -v). The
launcher now notices this and leaves the install to the job: git pull, then
./spe-server submit again. start and install refuse to run on such a
node, since PyTorch cannot load there.
The link does not load (it times out): use the SSH tunnel (step 3).
"This site can't provide a secure connection" or ERR_SSL_PROTOCOL_ERROR:
the browser switched the link to https://. The app speaks plain http://:
type the link with http:// in front, or use the tunnel and localhost.
The tunnel says "Address already in use": something on your own computer
has that port. Use another one on your side, for example
ssh -N -L 9000:localhost:8765 you@myserver.example.edu, then open
http://localhost:9000/?token=....
The page asks for a token: paste the part of the link after ?token=,
or open the whole link. After new-token, old links stop working.
The ready pill shows no GPU: on a cluster, ask for one (--gpus 1). On a
server, check that nvidia-smi lists it, and that --no-gpu is not set
(--gpu undoes it). A driver older than 525 cannot run the CUDA 12.8 build.
A SLURM job keeps waiting: ./spe-server status shows why (the
reason SLURM gives). Ask for less (--cpus, --memory, --gpus, --time)
or another partition.
The job ends at once: ./spe-server logs shows why. Often the cluster
requires an option, such as -- --partition=NAME --account=NAME, or the data
folder is not on a disk the compute nodes can see.
A run in a job stops partway: the job may have run out of memory or time,
or been preempted. Submit again with more --memory or --time, or a QOS
that is not preempted.
Anything else: ./spe-server logs usually says what went wrong.
On a cloud VM¶
- Disk: at least 30 GB for the install and a genome or two; more for data.
- Machine: CPUs matter most; an NVIDIA GPU speeds up runs with Use the
GPU. Without one,
--torch cpusaves 6 GB. - Reaching it: keep the firewall closed and use an SSH tunnel. On Google
Cloud:
gcloud compute ssh VM_NAME --zone ZONE -- -N -L 8765:localhost:8765, then openhttp://localhost:8765/?token=.... The app speaks plain HTTP, so do not open its port to the internet: the token would cross it unencrypted.
Security¶
The token grants everything the Workbench can do: read any file you can on
the server and run processes as you. Treat it like a password: do not paste
it in shared documents or chats. ./spe-server new-token replaces it. The
port is plain HTTP, so on a network you do not trust, use the SSH tunnel
instead of opening the port.
How it works¶
gateway/spe_gateway.py imports the Workbench's own server and handler. The
Workbench answers only 127.0.0.1 and has no login. The gateway listens on
the server, admits only requests that carry the token, then passes them to
the Workbench as if they came from the server itself, so its own checks still
run. It adds:
- Files: it tells the Workbench to draw its own file browser (no window
from the server's system can reach your screen), to put uploads in the data
folder's
uploads/, and to start browsing in your home folder. The browser, upload and .zip download are the Workbench's own (SigProfilerExtractor/web/files.py), so they work the same everywhere. - The ready pill in the top bar shows the CPUs and GPUs and live use
(
ready · 32 CPUs · 1 GPU (63% busy, 14/80 GB)). Click it for graphs of CPU load, memory, and each GPU's load and memory over the last 10 minutes. In a SLURM job it also shows the job, its time left and how full the node is. - CPU limits, as above.
Path in server/ |
What |
|---|---|
spe-server |
The launcher |
gateway/spe_gateway.py |
Token sign-in, free-port choice, file settings, resources |
gateway/static/ |
The sign-in page's styles, and the resources panel's script and styles |
gateway/entrypoint.sh |
Start-up script the launcher runs, and its settings |
In the data folder, .workbench/ holds the install, the token, the link, the
log and the last start's settings (launch.env).
The Workbench's own launcher, ../launch_web.sh, is separate and still works
as upstream intends, with its own environment (.venv-dashboard) and run
records (~/.cache). Stopping one does not stop the other.