is a HPC cluster administered by RCS and built on a shared facility model: research groups buy compute nodes (and other hardware) which are added to the cluster; all members of any such group may use the system.
In July 2010, some of the older compute nodes within the cluster were replaced with newer, more computationally powerful nodes via the Revolving Green Fund. So while the system is primarily for those who have invested in compute nodes, limited access is now available for all computational researchers on campus.
RedQueen2 runs Scientific Linux v5.x (a build from source, with modifications, of RedHat Enterprise Linux, cf. CentOS).
The primary role of this system is to run batch (i.e., non-interactive) jobs in the traditional way. A limited number of jobs may also be run interactively (i.e., by using an application's GUI).
Should this documentation not address an issue with RedQueen2, users should email firstname.lastname@example.org stating that they have a problem on RedQueen2: please give your username on the system (but not your password) and describe the problem.
Help for Users New to Linux-Based HPC Clusters
This documentation assumes a basic knowledge of Linux-based HPC clusters, for example, how to access the system 'via' SSH and X-Windows (aka X11), and the concept of a batch system and queues. If you are new to these research computing and are not familiar with these things, please read this introductory material.
Currently (2010 October) the system comprises:
- 11 12-core compute nodes, each with 24GB RAM;
- two fileservers each with 12 600GB internal disks
There is no fast interconnect.
What is the system most suitable for?
The system can be used for any research-related computation including:
- Parallel jobs (including both MPI and OpenMP);
- long-running and/or memory-intensive serial jobs.
As there is no fast interconnect (e.g., no Myrinet or Infiniband), the system is not suitable for large-scale parallel work.
Are to be found under /software.
Getting Access to RedQueen2
To get an account on RedQueen2, email email@example.com .
Logging into RedQueen2
Log in using SSH, after starting the VPN if required.
Linux users will be able to login using OpenSSH, which comes with all popular distros, by typing
ssh -l <username> rq2.rcs.manchester.ac.uk # ...replace <username> with your username, cf. mpciish2...
at the command-line.
MS Windows users should download and install PuTTY, an SSH client.
It is likely that you will wish to upload files to RQ2, or download them from RQ2 to your desktop/laptop. Linux users can do this by using either scp or sftp, from the OpenSSH utilities suite (which comes will all popular distros), for example:
scp myfile.txt <username>@rq2.rcs.manchester.ac.uk: # ...to copy a file to RQ2 --- don't forget the ":" at the end... scp <username>@rq2.rcs.manchester.ac.uk:results.out results.copy # ...copy a file from RQ2...
MS Windows users should download and install WinSCP, a GUI-based file-transfer client.
Using X-Windows: GUI-Based Applications
Using SSH on its own will enable you to login to RQ2 and use the command-line. If you want to use GUI-based applications, such as gedit, a Notepad-like editor, or you want to use Matlab interactively, then you need to run an X11 server on your local desktop and enable X11-tunnelling in your SSH connection.
All popular Linux distros run an X11-based desktop (GNOME, KDE). The only remaining step is to enable X11-tunnelling when logging in:
ssh -X -l <username> rq2.rcs.manchester.ac.uk # ...that's an UPPERcase X...
MS Windows users will need to download and install an X11 server. The two obvious options are eXceed, for which the University has a site licence, and Xming, which is free to download and install. When connecting:
- Start the VPN, if necessary.
- Start eXceed or Xming, then
start PuTTY, being careful to enable X11 tunnelling — click on "SSH" on the left-hand-side, then ensure the X11 tunnelling box is "checked", before starting a connection.
Running Jobs on Redqueen2: SGE
All computational jobs lasting more than a few minutes must be submitted to the batch system, SGE. Both interactive (e.g., GUI-based) and non-interactive (i.e., batch) jobs can be run, though we 'strongly' discourage the use of interactive jobs as such jobs use more system resources than the equivalent batch jobs.
The documentation linked for each application listed below contains examples illustrating how to submit batch jobs to SGE.
SGE Configuration: Available Queues
Since this is a relatively small system, there is only one available queue
which can be used for serial, parallel and interactive work.
SGE Configuration: Available Parallel Environments
Three parallel environments are available: for Fluent jobs, please use fluent.pe; for OpenMPI-based jobs please use either orte.pe or orte-12.pe. The former allows any size of parallel job; with the latter, the size of a parallel jobs must be a multiple of 12 — this ensure only whole compute nodes are used.
User Filesystems and Backups
Home directories are backed up to the IT Services central system.
Applications, Compilers and Libraries
All applications are installed under /software.
The Intel compiler is installed in its default location, under /opt.
Libraries built locally, including MPI libraries, are installed under /usr/local.
Users' home-directories are in /home-fs1-b1, /home-fs1-g1, /home-fs2-b1 and /home-fs2-g1.
Scratch space for users is available in /scratch-fs1-f1, /scratch-fs1-h1, /scratch-fs2-f1 and /scratch-fs2-h1. Scratch space is NOT backed up in any way.
Using the Installed Software with the Batch System (SGE): Details and Examples
There are three versions of Matlab installed on RQ2: Matlab R2007b in /software/matlab_R2007b and Matlab R2009a in /software/matlab_R2009a and Matlab R2012a in /software-fs2/matlab/R2012a .
- There are a limited number of licences available for Matlab at the University; if this number is exceeded, Matlab will fail to start.
- There are a VERY limited number of licences available for some Matlab toolboxes; if this number is exceeded, Matlab will fail to start.
Both serial and parallel Matlab jobs may be submitted to the batch system; Fluent may also be run interactively — though the latter is strongly discouraged, except for short, test jobs:
Submitting serial and parallel Matlab jobs to the batch system;
running Matlab interactively.
- Using the NAG toolbox with Matlab.
Fluent is installed in /software/Fluent.Inc.
This installation uses the floating licences provided by trf.me.umist.ac.uk, so it is possible that at some time all available licences may be allocated and Fluent will refuse to start.
All Fluent jobs run on RedQueen2 must be submitted to the batch/queue system, SGE.
Both serial and parallel Fluent jobs may be submitted to the batch system; Fluent may also be run interactively — though the latter is strongly discouraged:
Documented on the School of MACE CFD Group's wiki pages.
Documented on the School of MACE CFD Group's wiki pages.
- G09 is available on the system.
- All users must complete the University's Gaussian Guidelines before access can be granted.