Running ParaView In Parallel

From ScorecWiki

Contents

Overview

This article will give you an understanding of how to run ParaView on SCOREC's or other parallel computing resources.

Choosing and Connecting to Parallel Resources

To connect to a parallel system, you must first have access to it. To have access to a given cluster added to your SCOREC login you must email help@scorec.rpi.edu with a brief description of what you would like to do and access will likely be granted. Currently the best system to run paraview in parallel on is the medusa cluster.

Running ParaView in Parallel

SSH into the master node of the cluster where you will be running your parallel back end and do the following:

  • Create a shell script that executes the following command to start the parallel server, replacing PORT with an integer between 1025 and 65535:
#!/bin/bash 

/usr/local/mpich/latest/ch_p4/bin/mpirun -np NUMPROCS -machinefile MACHINES \
 /usr/local/paraview/latest/parallel/bin/pvserver --use-offscreen-rendering \
 --server-port=PORT
  • Submit this script to the queuing system; replace 32 with the number of processors that you would like to use and assume for this example the script is called pview.sh:
qsub -cwd -pe mpich 32 pview.sh
  • The master server process will report that it is ready to accept connections to wherever you have redirected STDOUT to (usually scriptname.oJOBID):
Listen on port: 31337
Waiting for client...
  • Now you need to figure out which node the master process is running on. To do this look at the first line in the parallel output file from SGE:
cat pview.sh.poJOBID

  • Start the paraview client on your workstation, telling it how to connect to the server that you just started:
/usr/local/paraview/latest/parallel/bin/pvclient --server-host=hostname.where.server.started.from \
 --server-port=PortNumberYouSpecifiedAbove
  • The server will acknowledge the client's connection by outputting the following line to wherever STDOUT is redirected:
Client connected.

That's it! You can separate the data reader from the client by using a special paraview server for that and you can further separate the renderer from the other backend processes but these steps are unnecessary at SCOREC due to the layout and configuration of the network infrastructure.

Running on CCNI opterons

A common install of ParaView is available at the CCNI.

There is an 35 minute movie at http://www.scorec.rpi.edu/~kjansen/CFD-08/CCNI-ParaView/CCNI-ParaView.html demonstrating the steps described below. There are three major steps involved in running Paraview on CCNI opterons:

  • Launching pvserver:

Submit job script to queue system, e.g., using "sbatch -n 4 job-script.sh" in case of SLURM, and record master (rank 0) node (as can be done from first host listed in file hosts.$SLURM_JOB_ID created within job-script below).

#!/bin/bash -x
#SBATCH -p opterons
#SBATCH -o slurm.o%j
#SBATCH -e slurm.e%j

srun hostname -s > /tmp//hosts.$SLURM_JOB_ID

if [ "x$SLURM_NPROCS" = "x" ]
then
  if [ "x$SLURM_NTASKS_PER_NODE" = "x" ]
  then
    SLURM_NTASKS_PER_NODE=1
  fi
  SLURM_NPROCS=`expr $SLURM_JOB_NUM_NODES \* $SLURM_NTASKS_PER_NODE`
fi

echo "Starting Job"
date

cp /tmp/hosts.$SLURM_JOB_ID hosts.$SLURM_JOB_ID

export JOBNAME=`squeue -j $SLURM_JOB_ID -o %j | grep -v NAME`
cp $JOBNAME $JOBNAME.$SLURM_JOB_ID

mpirun_ssh -hostfile /tmp/hosts.$SLURM_JOB_ID -np $SLURM_NPROCS /gpfs/small/PHASTA/shared/ParaView/install-paraview-olb-all/bin/pvserver --use-offscreen-rendering
# mpirun_ssh -hostfile /tmp/hosts.$SLURM_JOB_ID -np $SLURM_NPROCS /gpfs/small/PHASTA/shared/ParaView/install-paraview-olb/bin/pvserver --use-offscreen-rendering

echo "Job completed"
date

rm hosts.$SLURM_JOB_ID

rm /tmp/hosts.$SLURM_JOB_ID

You can use specific server port as mentioned in the previous section (i.e., option --server-port=PORT). Also note that this install of paraview is with optimizations (during compile time) and may not be stable in which case try to use /gpfs/small/PHASTA/shared/ParaView/install-paraview-olb/bin/pvserver.

  • SSH forwarding to master (rank 0) node:

You need to forward a port from local/client side to (server-) port (default is 11111) on master (rank 0) node of pvserver job (client runs on local side and server runs on remote-host which will be CCNI compute nodes in this case). In case you launch paraview client on landing pad nodes then you can skip this step. In case you launch paraview on other local machine (like lab. workstation or personal laptop) then ssh forwarding is required.

This can be done by doing "ssh -L <local-port>:<master-node>:<server-port> username@lp0[2-4].ccni.rpi.edu" (either lp02, lp03 or lp04), e.g., "ssh -L 11111:omb125:11111 PHASfake@lp03.ccni.rpi.edu". Note that ssh forwarding from a port on machine (running paraview client) within RPI network (like SCOREC workstation) will work straight but from personal desktop/laptop it may require VPN client.

  • Launching client and starting server:

In this step first launch the paraview client (e.g., /usr/local/paraview/latest/serial/bin/paraview on SCOREC or /gpfs/small/PHASTA/shared/ParaView/install-paraview-olb-all/bin/paraview on landing pads). Next start the server depending on where you run paraview client, if you do ssh forwarding from personal machine then use localhost and local-port (which is forwarded). Otherwise directly connect to server-port on master (rank 0) node that is possible from landing pads (and you may be able to use Infiniband by using -ib in server hostname, e.g., omb125-ib). For more details on starting paraview server see: http://paraview.org/Wiki/Starting_the_server.