Companion Website

Computational Physics

From Equations to Simulation

Chapter 15

Parallel Computing: An Orientation to HPC

These programs accompany Chapter 15 of Computational Physics: From Equations to Simulation. They give readers practical first steps with parallel reductions, grid calculations, and independent Monte Carlo sampling. More specialized examples introduce MPI communication, GPU data transfers, and cluster job submission.

Start with a serial calculation, check its result, then compare it with parallel execution. The programs have editable settings and run independently. Parallel facilities differ by language and may require a suitable compiler, toolbox, packages, or hardware. A parallel program may be slower than a well-optimized serial program.

Introductory programs

Choose an individual source file. On narrow screens, scroll horizontally to see every language column.

Parallel-computing examples in Fortran, C++, MATLAB, and Python
Program Fortran C++ MATLAB Python
Parallel reductionComputes a thresholded sum and compares serial and parallel execution. Demonstrates load balancing, safe reductions, and timing; Fortran/C++ include optional incorrect shared-accumulator demonstrations. Source Source Source Source
Two-dimensional heat diffusionAdvances a periodic heat equation with separate old and new temperature fields. Checks conservation and a known discrete solution. MATLAB distributes independent simulations rather than grid rows. Its optimized vectorized serial calculation may already use multiple CPU threads, so parfor can be slower. Source Source Source Source
Parallel Monte CarloEstimates pi using independent random-point batches. Verifies identical hit counts across worker counts within each language and measures strong and weak scaling. Source Source Source Source
MPI heat diffusion in stripsDistributes a heat grid among MPI processes, exchanges boundary values, and checks the reconstructed field and conserved mean temperature. Source Source — —
GPU heat diffusionCompares a CPU calculation with GPU-resident updates and an intentionally inefficient version that transfers the field each step. Without a compatible GPU, only the CPU validation runs. — — Source Source

A dash means no corresponding implementation is supplied. MPI and GPU examples are optional, not required for the first three programs. Random-number sequences are not identical across languages.

Running the programs

  • Fortran: Fortran 95-style source with OpenMP directives for the first three programs; use an OpenMP-enabled compiler such as GNU Fortran. The MPI example requires mpifort and a working MPI installation. An older Fortran compiler is not assumed to support either feature.
  • C++: C++11 with OpenMP for the first three programs. The MPI example requires mpic++. An ordinary C++11 installation alone does not provide OpenMP or MPI.
  • MATLAB: Base MATLAB runs the serial CPU calculations. Actual parfor execution requires Parallel Computing Toolbox and an available worker pool. GPU execution additionally requires a supported NVIDIA CUDA GPU; gpuArray does not use parpool. Without a supported GPU, only CPU validation runs. These scripts use local functions and therefore require MATLAB R2016b or newer; the parallel and GPU functions also depend on the installed toolbox and hardware.
  • Python: Python 3 and NumPy for reduction and Monte Carlo; both use separate worker processes through ProcessPoolExecutor (standard-library multiprocessing). The heat-diffusion CPU thread example additionally requires Numba. The supplied GPU example uses CuPy with a compatible NVIDIA CUDA GPU, driver, and CUDA environment; without them, the NumPy CPU check still runs. CuPy has other experimental GPU backends, but this example targets CUDA. Run multiprocessing examples as standalone .py files.

On macOS, Apple's default Clang toolchain does not normally provide OpenMP support without additional setup; Homebrew GCC or a properly configured libomp installation may be needed. On Windows, MPI users can use a compiler-compatible MPI installation such as Microsoft MPI (MS-MPI); the supplied commands assume Unix-style compiler wrappers and may need adaptation.

Apple-silicon Macs and most integrated GPUs cannot execute the supplied CUDA GPU examples; they can still run the CPU checks. A cloud notebook such as Google Colab may provide an NVIDIA GPU for trying the Python version, depending on availability and the required CuPy setup.

For example, after changing to the corresponding language folder, compile or run a first program:

gfortran -std=f95 -O3 -fopenmp ch15_parallel_reduction.f95 -o ch15_parallel_reduction
            ./ch15_parallel_reduction
            g++ -std=c++11 -O3 -fopenmp ch15_parallel_reduction.cpp -o ch15_parallel_reduction
            ./ch15_parallel_reduction
            python3 -m pip install numpy
            python3 ch15_parallel_reduction.py
          

For MATLAB, run ch15_parallel_reduction from the Command Window. The -O3 examples give the compiler more opportunity to vectorize stencil loops, helping make the heat-diffusion serial and parallel timing comparison fair; lower optimization levels may produce different results. Check the individual source comments for exact commands and settings.

First checks and performance measurements

What to verify before interpreting speedup
Example Correctness check Performance lesson
Reduction Right-endpoint integral error agrees with its predicted leading error; serial and parallel sums agree within rounding. Chunked scheduling can balance thresholded work. Race conditions give unreliable sums; extra threads may not help.
Heat diffusion The computed field agrees with the exact discrete Fourier-mode solution, and the mean temperature is conserved. Uniform grid updates suit static scheduling, but memory use and synchronization can limit speedup.
Monte Carlo Integer hit counts match across worker counts within one language; compare the estimate with pi in units of statistical standard error. Independent batches give a simple strong- and weak-scaling experiment; random streams belong to batches, not workers.
MPI heat diffusion Compare gathered field with the serial reference and check conservation; deliberately broken exchange is optional. Communication time includes waiting for neighbors; more processes increase communication relative to local work.
GPU heat diffusion Compare GPU and CPU temperature fields and both against the known discrete solution. Measure GPU-resident work separately from repeated CPU–GPU transfers; acceleration is not guaranteed.

Timing depends on the CPU or GPU, compiler, optimization settings, worker configuration, and workload. In particular, MATLAB vectorized built-in operations may already exploit internal multithreading; a parfor speedup below one is not necessarily an error. The programs print their own measurements rather than prescribing expected speedups. For a fair comparison, use equivalent numerical work and a suitable optimized serial reference. The Monte Carlo samples remain deterministic within each implementation, but different languages use different generators.

MPI and cluster submission

Compile the MPI heat-diffusion source with an MPI compiler wrapper, then launch the executable with mpiexec or the launcher recommended by your installation:

mpifort -O3 -std=f95 ch15_mpi_heat_strips.f95 -o ch15_mpi_heat_strips
            mpiexec -n 2 ./ch15_mpi_heat_strips
            mpic++ -std=c++11 -O3 ch15_mpi_heat_strips.cpp -o ch15_mpi_heat_strips
            mpiexec -n 2 ./ch15_mpi_heat_strips
          

Run the Fortran and C++ examples in their own directories, since both build an executable with the same name. More MPI processes than available execution slots may require explicit oversubscription settings; oversubscribed timing results are not good scaling benchmarks. A personal computer with MPI can run these examples without a cluster.

For a managed cluster, these two short submission templates use the C++ MPI executable and request one node with four MPI processes. Adjust resource requests, environment modules, and launcher to match your computing center:

The scripts assume a previously compiled ch15_mpi_heat_strips executable in the submission directory; neither script installs MPI or compiles the application. Scheduler access is not needed for the introductory programs.

Verification scope

The core Fortran and C++ programs have been compiled and checked with GNU compilers; the MPI examples have also passed multi-process tests with Open MPI, including uneven strip sizes. The Python CPU programs passed their numerical tests, and the MATLAB reduction, heat-diffusion, and Monte Carlo examples were run successfully in MATLAB. The CPU validation paths of both GPU examples have been checked, but their GPU operations have not been tested on compatible GPU hardware. The SLURM and PBS scripts have been syntax-checked, not submitted to a production scheduler.

The programs include their own numerical checks. Run them on your own system before interpreting timing results, and do not assume identical execution times or random sequences across languages or computers.

← Back to Chapter 15