HPC Cluster

Logging In

To login to the CSE HPC cluster, open an ssh connection to cse-hpc.cse.ohio-state.edu using PUTTY or another SSH client. You will be placed on one of the servers in the login pool and given a terminal that you can use to compile programs and submit jobs to run on the cluster. Please do not run intensive jobs on the login servers as these are a shared resource and not intended for running compute jobs.

Running jobs using SLURM

Interactive Jobs

Interactive jobs are used for testing, debugging, and developing your code before running it on the larger batch partition, allowing you to ensure your code will run correctly before using more computing resources.

To run an interactive job:

  1. Copy any files you need to your /data/$USER folder. This folder is the only one that is mounted across all cluster nodes. If you attempt to run a job out of your /home/$USER directory, it will fail. Your /data/$USER folder has a quota of 100GB so make sure to keep only the files you need for your currently running jobs in there.
  2. Change directory to your /data/$USER folder.
  3. Run ‘salloc –p interactive –t HH:MM:SS’ to request an interactive job. This will give you a shell on the allocated machine with a time limit of HH:MM:SS. The maximum runtime of any interactive job is 4 hours.
  4. Run your command in the shell. If you would like to run your command with MPI, you will need to run it using ‘srun –mpi=pmi2 /path/to/binary’.
  5. Exit the shell using the ‘exit’ command to end your allocation and release the node.
  6. Copy any results back to your /home/$USER folder from your /data/$USER folder.

Batch Jobs

Batch jobs are used for running actual tests or experiments with your code, after any debugging has been done locally or on the interactive node, and have access to greater resources than interactive jobs.

To run a batch job:

  1. Copy any files you need to your /data/$USER folder. This folder is the only one that is mounted across all cluster nodes. If you attempt to run a job out of your /home/$USER directory, it will fail. Your /data/$USER folder has a quota of 100GB so make sure to keep only the files you need for your currently running jobs in there.
  2. Change directory to your /data/$USER folder.
  3. Create a script file for your batch job. A simple example is shown below:
  4. Run your job using ‘sbatch /path/to/script/file’. You will be given a job number when the job is submitted.
  5. Wait for your job to run. You can check the status of the queue using the ‘squeue’ command.
  6. Once your job runs, you will be able to collect your results from the slurm-JOB_NUMBER.out file.
  7. Copy any results back to your /home/$USER folder from your /data/$USER folder.
#!/bin/bash
#SBATCH –p batch –N 10 –t 1-01:01
srun –mpi=pmi2 mpiprog

The script file runs in the directory where the job is submitted by default. This batch file runs in the batch partition, with 10 allocated nodes, and a time limit of 1 day, 1 hour, and 1 minute as specified by the #SBATCH line. The script also runs an MPI program using ‘srun’. The maximum number of nodes for any batch job is 42 nodes and the maximum time limit is 2 days.
 

Additional Information

If you would like to cancel any of your running or pending jobs, you can cancel them using ‘scancel JOB_NUMBER’.

Remember that the cluster will backfill smaller jobs into the spaces between larger jobs. You should only request the resources (i.e. number of nodes and runtime) you need for your job to increase the chance that your job will be scheduled.

More information for running commands under SLURM can be found in the man pages for each command and on the SLURM web page

Using LMod to load the MPI module

  1. To load the MPI module, run ‘module load mpi/mvapich2-2.3.2-x86_64’. This will add tools like ‘mpicc’ (MPI C Compiler) and other helpful tools to your path so you can use them.
  2. To unload the module, run ‘module unload mpi/mvapich2-2.3.2-x86_64’.

Support CSE

$50,000 for the 50th

Donate to the department to help us reach the goal of $50,000 in donations to celebrate over 50 years of teaching excellence

Click here to make your donation.

Newsletter