How to start attachable interactive Slurm in background?
Feb 1, 11:03pm

To start an attachable interactive Slurm job in the background, you typically use the srun or salloc command with the --pty and --detach options, or you can use screen or tmux to manage your session. Here’s how you can do it:

1. Using srun with --pty and --detach

This method allows you to start an interactive job and detach from it, so you can reattach later.

bash
srun --pty --job-name=myjob --time=01:00:00 --mem=4G --nodes=1 --ntasks-per-node=1 --detach bash
  • --pty: Allocates a pseudo-terminal.
  • --detach: Detaches the job and runs it in the background.
  • --job-name=myjob: Gives your job a name (optional but helpful).
  • --time=01:00:00: Sets the maximum runtime.
  • --mem=4G: Requests 4GB of memory.
  • --nodes=1: Requests 1 node.
  • --ntasks-per-node=1: Requests 1 task per node.

To reattach: Find your job ID with squeue -u $USER, then reattach using:

bash
sattach <jobid>

2. Using salloc with screen or tmux

If your cluster doesn’t support --detach with srun, you can use salloc to allocate resources and then start a screen or tmux session inside it.

Step 1: Allocate resources

bash
salloc --job-name=myjob --time=01:00:00 --mem=4G --nodes=1 --ntasks-per-node=1

This will give you a shell on a compute node.

Step 2: Start a screen or tmux session

bash
screen -S mysession
# or
tmux new -s mysession

Now, run your interactive commands inside this session.

Step 3: Detach from the session

  • For screen: Press Ctrl+A then D.
  • For tmux: Press Ctrl+B then D.

Step 4: Reattach later

bash
screen -r mysession
# or
tmux attach -t mysession

3. Using sbatch with an interactive script

If you want to submit an interactive job as a batch script and reattach later, you can use a script like this:

interactive.sh

bash
#!/bin/bash
#SBATCH --job-name=myjob
#SBATCH --time=01:00:00
#SBATCH --mem=4G
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
screen -S mysession

Submit it with:

bash
sbatch interactive.sh

Then, reattach with:

bash
screen -r mysession

Key Notes

  • Reattaching: Always check your job ID with squeue -u $USER and use sattach or screen -r/tmux attach as needed.
  • Cluster-specific: Some clusters may have specific modules or configurations for interactive jobs. Check your cluster’s documentation.
Feb 1, 11:03pm

This is a shared conversation and may include unverified content that doesn't reflect Mistral AI's views. Login to resume chatting.

Vibe can make mistakes. Check answers. Learn more