Chapter 15
Parallel Computing: An Orientation to HPC
These programs accompany Chapter 15 of Computational Physics: From Equations to Simulation. They give readers practical first steps with parallel reductions, grid calculations, and independent Monte Carlo sampling. More specialized examples introduce MPI communication, GPU data transfers, and cluster job submission.
Start with a serial calculation, check its result, then compare it with parallel execution. The programs have editable settings and run independently. Parallel facilities differ by language and may require a suitable compiler, toolbox, packages, or hardware. A parallel program may be slower than a well-optimized serial program.
Introductory programs
Choose an individual source file. On narrow screens, scroll horizontally to see every language column.
| Program | Fortran | C++ | MATLAB | Python |
|---|---|---|---|---|
| Parallel reductionComputes a thresholded sum and compares serial and parallel execution. Demonstrates load balancing, safe reductions, and timing; Fortran/C++ include optional incorrect shared-accumulator demonstrations. | Source | Source | Source | Source |
Two-dimensional heat diffusionAdvances a periodic heat equation with separate old and new temperature fields. Checks conservation and a known discrete solution. MATLAB distributes independent simulations rather than grid rows. Its optimized vectorized serial calculation may already use multiple CPU threads, so parfor can be slower.
|
Source | Source | Source | Source |
| Parallel Monte CarloEstimates pi using independent random-point batches. Verifies identical hit counts across worker counts within each language and measures strong and weak scaling. | Source | Source | Source | Source |
| MPI heat diffusion in stripsDistributes a heat grid among MPI processes, exchanges boundary values, and checks the reconstructed field and conserved mean temperature. | Source | Source | — | — |
| GPU heat diffusionCompares a CPU calculation with GPU-resident updates and an intentionally inefficient version that transfers the field each step. Without a compatible GPU, only the CPU validation runs. | — | — | Source | Source |
A dash means no corresponding implementation is supplied. MPI and GPU examples are optional, not required for the first three programs. Random-number sequences are not identical across languages.
Running the programs
- Fortran: Fortran 95-style source with OpenMP directives for the first three programs; use an OpenMP-enabled compiler such as GNU Fortran. The MPI example requires
mpifortand a working MPI installation. An older Fortran compiler is not assumed to support either feature. - C++: C++11 with OpenMP for the first three programs. The MPI example requires
mpic++. An ordinary C++11 installation alone does not provide OpenMP or MPI. - MATLAB: Base MATLAB runs the serial CPU calculations. Actual
parforexecution requires Parallel Computing Toolbox and an available worker pool. GPU execution additionally requires a supported NVIDIA CUDA GPU;gpuArraydoes not useparpool. Without a supported GPU, only CPU validation runs. These scripts use local functions and therefore require MATLAB R2016b or newer; the parallel and GPU functions also depend on the installed toolbox and hardware. - Python: Python 3 and NumPy for reduction and Monte Carlo; both use separate worker processes through
ProcessPoolExecutor(standard-library multiprocessing). The heat-diffusion CPU thread example additionally requires Numba. The supplied GPU example uses CuPy with a compatible NVIDIA CUDA GPU, driver, and CUDA environment; without them, the NumPy CPU check still runs. CuPy has other experimental GPU backends, but this example targets CUDA. Run multiprocessing examples as standalone.pyfiles.
On macOS, Apple's default Clang toolchain does not normally provide OpenMP support without additional setup; Homebrew GCC or a properly configured libomp installation may be needed. On Windows, MPI users can use a compiler-compatible MPI installation such as Microsoft MPI (MS-MPI); the supplied commands assume Unix-style compiler wrappers and may need adaptation.
Apple-silicon Macs and most integrated GPUs cannot execute the supplied CUDA GPU examples; they can still run the CPU checks. A cloud notebook such as Google Colab may provide an NVIDIA GPU for trying the Python version, depending on availability and the required CuPy setup.
For example, after changing to the corresponding language folder, compile or run a first program:
gfortran -std=f95 -O3 -fopenmp ch15_parallel_reduction.f95 -o ch15_parallel_reduction
./ch15_parallel_reduction
g++ -std=c++11 -O3 -fopenmp ch15_parallel_reduction.cpp -o ch15_parallel_reduction
./ch15_parallel_reduction
python3 -m pip install numpy
python3 ch15_parallel_reduction.py
For MATLAB, run ch15_parallel_reduction from the Command Window. The -O3 examples give the compiler more opportunity to vectorize stencil loops, helping make the heat-diffusion serial and parallel timing comparison fair; lower optimization levels may produce different results. Check the individual source comments for exact commands and settings.
First checks and performance measurements
| Example | Correctness check | Performance lesson |
|---|---|---|
| Reduction | Right-endpoint integral error agrees with its predicted leading error; serial and parallel sums agree within rounding. | Chunked scheduling can balance thresholded work. Race conditions give unreliable sums; extra threads may not help. |
| Heat diffusion | The computed field agrees with the exact discrete Fourier-mode solution, and the mean temperature is conserved. | Uniform grid updates suit static scheduling, but memory use and synchronization can limit speedup. |
| Monte Carlo | Integer hit counts match across worker counts within one language; compare the estimate with pi in units of statistical standard error. | Independent batches give a simple strong- and weak-scaling experiment; random streams belong to batches, not workers. |
| MPI heat diffusion | Compare gathered field with the serial reference and check conservation; deliberately broken exchange is optional. | Communication time includes waiting for neighbors; more processes increase communication relative to local work. |
| GPU heat diffusion | Compare GPU and CPU temperature fields and both against the known discrete solution. | Measure GPU-resident work separately from repeated CPU–GPU transfers; acceleration is not guaranteed. |
Timing depends on the CPU or GPU, compiler, optimization settings, worker configuration, and workload. In particular, MATLAB vectorized built-in operations may already exploit internal multithreading; a parfor speedup below one is not necessarily an error. The programs print their own measurements rather than prescribing expected speedups. For a fair comparison, use equivalent numerical work and a suitable optimized serial reference. The Monte Carlo samples remain deterministic within each implementation, but different languages use different generators.
MPI and cluster submission
Compile the MPI heat-diffusion source with an MPI compiler wrapper, then launch the executable with mpiexec or the launcher recommended by your installation:
mpifort -O3 -std=f95 ch15_mpi_heat_strips.f95 -o ch15_mpi_heat_strips
mpiexec -n 2 ./ch15_mpi_heat_strips
mpic++ -std=c++11 -O3 ch15_mpi_heat_strips.cpp -o ch15_mpi_heat_strips
mpiexec -n 2 ./ch15_mpi_heat_strips
Run the Fortran and C++ examples in their own directories, since both build an executable with the same name. More MPI processes than available execution slots may require explicit oversubscription settings; oversubscribed timing results are not good scaling benchmarks. A personal computer with MPI can run these examples without a cluster.
For a managed cluster, these two short submission templates use the C++ MPI executable and request one node with four MPI processes. Adjust resource requests, environment modules, and launcher to match your computing center:
- SLURM job submission script (.sh) — submit with
sbatch ch15_mpi_heat_slurm.sh. - PBS Pro / OpenPBS job submission script (.pbs) — submit with
qsub ch15_mpi_heat_pbs.pbs. Legacy Torque systems may use different resource syntax.
The scripts assume a previously compiled ch15_mpi_heat_strips executable in the submission directory; neither script installs MPI or compiles the application. Scheduler access is not needed for the introductory programs.
Verification scope
The core Fortran and C++ programs have been compiled and checked with GNU compilers; the MPI examples have also passed multi-process tests with Open MPI, including uneven strip sizes. The Python CPU programs passed their numerical tests, and the MATLAB reduction, heat-diffusion, and Monte Carlo examples were run successfully in MATLAB. The CPU validation paths of both GPU examples have been checked, but their GPU operations have not been tested on compatible GPU hardware. The SLURM and PBS scripts have been syntax-checked, not submitted to a production scheduler.
The programs include their own numerical checks. Run them on your own system before interpreting timing results, and do not assume identical execution times or random sequences across languages or computers.