Computational Science Community Wiki

Differences between revisions 9 and 10
Revision 9 as of 2009-11-18 14:42:48
Size: 10833
Editor: SimonHood
Comment:
Revision 10 as of 2009-11-18 14:44:44
Size: 10817
Editor: SimonHood
Comment:
Deletions are marked like this. Additions are marked like this.
Line 9: Line 9:
 * the steps required to access RCS HPC systems <---> in particular  * the steps required to access RCS HPC systems &mdash; in particular
Line 23: Line 23:
All RCS HPC systems are used remotely <I>via</I> SSH. Users authenticate All RCS HPC systems are used remotely ''via'' SSH. Users authenticate
Line 26: Line 26:
jobs to the <I>batch system queues</I>. jobs to the ''batch system queues''.
Line 30: Line 30:
document <---> for those familiar with remotely accessing Linux-based HPC document &mdash; for those familiar with remotely accessing Linux-based HPC
Line 61: Line 61:
  All RCS-administered HPC systems are Linux <I>clusters</I> of many
  computers, usually called <I>nodes</I>. In most
  All RCS-administered HPC systems are Linux ''clusters'' of many
  computers, usually called ''nodes''. In most
Line 69: Line 69:
  all computational jobs must be run run on <I>compute nodes</I>, not on the   all computational jobs must be run run on ''compute nodes'', not on the
Line 85: Line 85:
  is running on the remote HPC system. Using a <I>virtual desktop</I> to   is running on the remote HPC system. Using a ''virtual desktop'' to
Line 169: Line 169:
be prompted for your password <---> enter that given to you by the system's be prompted for your password &mdash; enter that given to you by the system's
Line 185: Line 185:
<B>Host Name</B> box <---> in the above case {{{man2.nw-grid.ac.uk}}}; then
click <B>Open</B>.
'''Host Name''' box &mdash; in the above case {{{man2.nw-grid.ac.uk}}}; then
click '''Open'''.
Line 195: Line 195:
It should be safe to click <B>Yes</B>. (If this alert appears again, for It should be safe to click '''Yes'''. (If this alert appears again, for
Line 199: Line 199:
The next step is authentication. Enter your username at the prompt <---> this The next step is authentication. Enter your username at the prompt &mdash; this
Line 215: Line 215:
GUI-based editor, such as <B>gedit</B>, or start the Matlab GUI? Then you will
need to be running an <I>X11 Server</I> on your local desktop/laptop and also
to connect using PuTTY <I>with X11 tunnelling</I> enabled.
GUI-based editor, such as '''gedit''', or start the Matlab GUI? Then you will
need to be running an ''X11 Server'' on your local desktop/laptop and also
to connect using PuTTY ''with X11 tunnelling'' enabled.
Line 221: Line 221:
server <---> that on Linux is always running (assuming you are running server &mdash; that on Linux is always running (assuming you are running
Line 237: Line 237:
 * Start the X11 server <---> eXceed or Xming.</LI>

 * Start PuTTY <---> ensure the <B>Enable X11 forwarding</B> box is
 * Start the X11 server &mdash; eXceed or Xming.</LI>

 * Start PuTTY &mdash; ensure the '''Enable X11 forwarding''' box is
Line 284: Line 284:
system you which to upload/download files to/from in the <B>Host name</B> system you which to upload/download files to/from in the '''Host name'''
Line 291: Line 291:
The click <B>Login</B>. The click '''Login'''.
Line 299: Line 299:
It should be safe to click <B>Yes</B>. (If this alert appears a second time, for It should be safe to click '''Yes'''. (If this alert appears a second time, for

Introductory Notes for New Users of RCS HPC Systems

This Page

This page is intended for users new to RCS HPC/HTC systems. It describes:

  • the steps required to access RCS HPC systems — in particular

    • those steps required from a MS Windows desktop/laptop;
  • the nature of these HPC systems;
  • sharing resources with other users and running computational job on
    • them;
  • some experimental services which are new to these HPC systems.

Overview

All RCS HPC systems are used remotely via SSH. Users authenticate (i.e., login) using an SSH client; after successful authentication a command-line interface is presented. This can be used to submit computational jobs to the batch system queues.

The remainder of this section may be considered the short version of this document — for those familiar with remotely accessing Linux-based HPC systems and submitting jobs to batch systems. For those that are not, please read the remaining sections!

Getting an Account and Authentication

  • Email rcs@manchester.ac.uk briefly describing your computational requirements.

Connecting

  • All systems are accessed via SSH, SCP and/or SFTP.

Network/Firewall Issues

  • All systems are firewalled. Some systems are accessible from all University of Manchester IP addresses; others are not. Few are accessible from outside of the University of Manchester.

Using GUI-Based Applications

  • SSH, on its own, gives a command-line interface only. Should the use of GUI-based applications be required, for example the Notepad/Wordpad-like editor Gedit, or the Matlab graphical shell, then X-Windows (X11) may be tunnelled through the SSH connection (and an X-server will be required on the local desktop/laptop).

The Nature of the Systems

  • All RCS-administered HPC systems are Linux clusters of many computers, usually called nodes. In most cases these clusters exist on a completely private network; users directly access only one or two login/head nodes.

Running a Computational Job in the Batch System

  • Many people are likely to be using each cluster simultaneously;

    all computational jobs must be run run on compute nodes, not on the login/head node(s). Computational work is submitted to these compute nodes via the batch system queues.

Running Interactive Computational Jobs

  • The vast majority of computational work carried out on RCS HPC systems is done in batch mode, i.e., non-interactively. On rare occasion it is necessary to run jobs interactively. Experimental queues exist on two RCS systems, Man2 and Mace01, which facilitate this.

Virtual Desktops

  • Running GUI-based, interactive computations presents a problem: if the local desktop or laptop which on which the GUI is displayed is switched off, or looses network connectivity, the computation will be killed even though it

    is running on the remote HPC system. Using a virtual desktop to display the GUI eliminates this problem.

Troubleshooting and FAQ

Getting an Account; Authentication

Getting an Account

To get an account on any RCS-administered HPC system, email rcs@manchester.ac.uk, briefly describing the computational work that you wish to carry out, for example:

  • applications, compilers or libraries needed;
  • a rough estimate of diskspace required;
  • the nature of the computational jobs you hope to run, for example:
    • Do you wish to run a few long-running jobs, or a lot of short jobs?
    • Does your work require a particularly large amount of memory?
    • Is your code serial or parallel? (i.e., can it use more than one
      • CPU at once?)

Getting Your Username and Password

For each HPC system run by RCS, you will have a username and password to enable you to authenticate (login) and run computational jobs. These credentials are independent of your central IT Services username and password, though, simply for ease of administration, the username will usually be the same.

Once you have an account on an RCS-administered HPC system, the system-administrator will contact you to give you your username and password.

For security reasons, as soon as you have received your credentials for a system, you should login and change your password (using the passwd) command.

Connecting to RCS Linux-Based HPC Systems

Secure Shell (SSH)

Secure Shell (SSH) is a network protocol which is used to connect to remote computers, i.e., to authenticate (login) and interact with the remote system.

Macintosh OS-X systems and all popular Linux distributions include an SSH client called OpenSSH; MS Windows users must download and install one. The most popular is <XLNK="http://www.chiark.greenend.org.uk/~sgtatham/putty/">PuTTY</XLNK> which can be freely <XLNK="http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html">downloaded</XLNK> and installed.

Using OpenSSH on Linux and OS-X

At a command line, on Linux or OS-X, simply type

  ssh <username>@<remote.system.name>

for example

  ssh mabcdef2@machine.rcs.manchesterac.uk

The first time you connect to a particular system you will be prompted to confirm its authenticity, for example

  The authenticity of host 'man2.nw-grid.ac.uk (130.88.200.243)' can't be established.
  RSA key fingerprint is cf:48:69:ff:99:f0:a1:4a:80:0b:46:b5:40:c0:fc:4c.
  Are you sure you want to continue connecting (yes/no)? 

Unless you have any reason for doubt, enter yes and you will then be prompted for your password — enter that given to you by the system's administrator (not your central IT Services password).

Using PuTTY

From a MS Windows desktop/laptop, to authenticate (login) to a remote Linux system, install and start PuTTY <CENTER> <IMG SRC="_images/putty.jpeg" ALT="PuTTY Configuration"

  • CAPTION="Enter the name of the system to which you wish to connect and
    • click Open.">

</CENTER> and enter the name of the system to which you wish to connect in the Host Name box — in the above case man2.nw-grid.ac.uk; then click Open.

The first time you connect to any given system you will see a PuTTY aecurity alert <CENTER> <IMG SRC="_images/putty_sa.jpeg" ALT="PuTTY Security Alert"

  • CAPTION="The first time you connect to a system you will see a PuTTY
    • security alert">

</CENTER> It should be safe to click Yes. (If this alert appears again, for a particular system, it may be a good idea to email the system administrator.)

The next step is authentication. Enter your username at the prompt — this will usually be your central IT Services username: <CENTER> <IMG SRC="_images/putty_login.jpeg" ALT="PuTTY login prompt"

  • CAPTION="Enter your username at the prompt and then the password
    • given to your for this system, when asked.">

</CENTER> Then, when prompted, enter the password given to you by RCS.

GUI-Based Applications and X-Windows

Using PuTTY alone allows you to login and enter commands, for example, submit computational jobs to the batch system. But what if you want to start a GUI-based editor, such as gedit, or start the Matlab GUI? Then you will need to be running an X11 Server on your local desktop/laptop and also to connect using PuTTY with X11 tunnelling enabled.

Macintosh OS-X systems and all popular Linux distributions include an X11 server — that on Linux is always running (assuming you are running a GUI-based desktop such as GNOME or KDE). MS Windows users must download and install one. The most popular are Hummingbird eXceed and Xming; the University has a site licence for eXceed; Xming may be freely <XLNK="http://www.straightrunning.com/XmingNotes/">downloaded</XLNK> and installed.

<CENTER> <IMG SRC="_images/putty_x11.jpeg" ALT="PuTTY: Enable X11 Forwarding"

  • CAPTION="Ensure the Enable X11 Forwarding box is checked.">

</CENTER>

Once you have an X11 server installed, then in order:

  • Start the X11 server — eXceed or Xming.</LI>

  • Start PuTTY — ensure the Enable X11 forwarding box is

    • <Q>checked</Q> (see figure).</LI>

  • Login as to the remote Linux system as normal. You should then be
    • able to start GUI-based applications such as gedit and Matlab on the

      remote system and have them displayed on your local desktop/laptop.</LI>

File Transfer

It is likely that you will wish to upload files to the HPC system, or download them to your desktop/laptop. Linux users can do this by using the OpenSSH utilities suite (which comes will all popular distros). MS Windows users must download a suitable client; <XLNK="http://winscp.net/">WinSCP</XLNK>, which is freely <XLNK="http://winscp.net/eng/download.php">downloadable</XLNK>, is a popular choice.

Using SCP and SFTP

At a command line, on Linux or OS-X, to upload a file from your desktop/laptop, simply type

  scp <local.filename> <username>@<remote.system.name>:<remote.filename>

for example

  scp my_prog.f90 mabcdef2@machine.rcs.manchester.ac.uk:my_programme.f90

To download a file to your desktop/laptop, enter, for example,

  scp mabcdef2@machine.rcs.manchester.ac.uk:my_results.dat my_remote_results.dat

Using WinSCP

To download or upload files, start WinSCP and enter the name of the system you which to upload/download files to/from in the Host name box, and your username and password. <CENTER> <IMG SRC="_images/winscp_login.jpeg" ALT="WinSCP Login"

  • CAPTION="Enter the name of the remote system, your username
    • and password, and click Login">

</CENTER> The click Login.

The first time you login to any given system you will see a warning message. <CENTER> <IMG SRC="_images/winscp_warning.jpeg" ALT="WinSCP Warning"

  • CAPTION="The first time you login to any given system you will see a
    • warning message.">

</CENTER> It should be safe to click Yes. (If this alert appears a second time, for any given system, it may be a good idea to email the system administrator.)

Once logged in, a nice drag-n-drop interface is presented. <CENTER> <IMG SRC="_images/winscp.jpeg" ALT="WinSCP Drag-n-Drop Interface"

  • CAPTION="Once logged in, a nice drag-n-drop interface is presented.">

</CENTER>