Chapter 15
Parallel Computing: An Orientation to HPC
These resources accompany Chapter 15 of Computational Physics: From Equations to Simulation by Alexander Godunov. They help readers choose a practical first parallel example, identify software requirements, check numerical results, and interpret performance measurements. A multicore personal computer is sufficient for the introductory CPU programs; MPI, GPU, and cluster examples have additional requirements.
Where to begin
Begin with the parallel reduction example: it compares a known serial sum with a parallel reduction and introduces load balancing and race conditions. Next, try two-dimensional heat diffusion to see independent updates of a physical field. Parallel Monte Carlo adds independent random streams and strong and weak scaling. MPI heat diffusion and GPU heat diffusion are optional next steps; neither is required to understand the first three examples.
In every case, first run the serial calculation and its numerical checks. Then run the parallel version at the same problem size and requested accuracy. Do not interpret speedup until both calculations are correct.
Software and hardware requirements
On a narrow screen, scroll horizontally to view all language columns.
| Task | Fortran | C++ | MATLAB | Python |
|---|---|---|---|---|
| Introductory multicore examples | GNU Fortran or another compiler with OpenMP support; -fopenmp with gfortran |
C++11 compiler with OpenMP support; -fopenmp with GCC |
parfor requires Parallel Computing Toolbox and worker support; base MATLAB runs serial checks |
Python 3 and NumPy; ProcessPoolExecutor for reduction/Monte Carlo; NumPy and Numba for threaded heat diffusion |
| MPI domain decomposition | MPI implementation and mpifort; supplied source uses use mpi |
MPI implementation and mpic++; code uses the C MPI interface from C++11 |
Not supplied | Not supplied |
| Optional GPU example | Not supplied | Not supplied | Parallel Computing Toolbox with supported NVIDIA CUDA GPU for gpuArray |
NumPy for CPU check; CuPy, compatible CUDA installation/driver, and NVIDIA GPU for GPU calculations |
These are requirements for the supplied programs, not a complete list of parallel facilities supported by each language. MATLAB scripts use local functions, requiring R2016b or newer. A compiler or language installation alone does not guarantee that OpenMP, MPI, a toolbox, or GPU support is present.
On macOS, the Apple-provided Clang toolchain normally needs additional
OpenMP setup (for example, an appropriate GCC installation or libomp).
On Windows, install a compatible MPI implementation such as MS-MPI and
adapt the Unix-style example commands. The supplied CUDA examples do not
run on Apple-silicon GPUs. Without supported GPU hardware, the GPU programs
can still perform their CPU validation. A cloud service such as
Google Colab
may offer a suitable GPU for Python, subject to availability and setup.
Checks with known answers
Parallel output must satisfy the same numerical requirements as serial output. The companion programs provide checks that do not depend on observing a speedup.
| Program | What to check | What the check establishes |
|---|---|---|
| Parallel reduction | The serial and parallel sums agree within rounding; the right-endpoint integral error agrees with its predicted leading error. | Work is combined correctly. Different reduction orders may change the last digits. |
| Heat diffusion | Compare with the exact discrete Fourier-mode decay, check the analytical PDE error, and monitor mean temperature. | Separates programming errors from discretization error; detects incorrect in-place updates. |
| Parallel Monte Carlo | Check exactly matching integer hit counts across worker counts within one language; compare the estimate of pi with its estimated standard error. | Confirms reproducible batch assignment and reduction; agreement of hit counts does not establish random-generator quality. |
| MPI heat diffusion | Compare the gathered field with the serial solution, including uneven strip counts; check conservation of mean temperature. | Tests halo exchange, domain indexing, and reconstruction. Conservation alone is not a complete field check. |
| GPU heat diffusion | Compare CPU/GPU temperature fields and the known discrete solution, allowing for floating-point differences. | Distinguishes GPU implementation errors from differences in arithmetic order and hardware. |
Vectorization: before using more cores
Vectorization (SIMD) lets one processor instruction operate on several data values. It is different from running several CPU threads or MPI processes. Before adding parallel workers, check whether a compiler has already vectorized the serial numerical loops.
- Inspect the compiler report. With GNU Fortran or GCC C++, add
-fopt-info-vecto the compile command to see successful vectorizations;-fopt-info-vec-missedreports missed opportunities. A compiler flag is not proof that a particular loop was vectorized. - Optimization settings matter. In testing the companion heat stencil with GCC, the one-thread OpenMP loop lost vectorization at
-O2while the serial loop retained it. With-O3, GCC was able to vectorize both. This is compiler- and code-dependent, not a universal guarantee. - Floating-point reductions need care. SIMD sums may require regrouping additions and therefore changing the last digits. Compilers can decline to vectorize them under strict floating-point rules. OpenMP SIMD reductions or relaxed floating-point options may help, but reassociation changes numerical behavior; do not enable
-ffast-mathautomatically. - Keep Python effects distinct. NumPy vectorization often speeds up a Python loop primarily by moving work out of the interpreter. That gain is not, by itself, evidence of SIMD instructions or multicore parallel execution.
Measuring parallel performance
For P workers, strong scaling holds the total problem size
fixed and compares times using S(P) = T_serial / T_parallel(P).
Parallel efficiency is S(P)/P. Weak scaling increases total
work approximately in proportion to P; compare elapsed time
with a one-worker run at the same work per worker. State which baseline
is used: one-worker parallel execution and optimized serial execution
need not have the same cost.
- Measure elapsed time, not summed CPU time. Fortran
cpu_timecan include CPU time accumulated over multiple threads and is not a suitable elapsed-time speedup clock. Suitable clocks includeomp_get_wtime, C++steady_clock, MATLABtic/toc, and Pythonperf_counter. MPI timing should report the slowest rank, since the job finishes when all ranks finish. - Repeat measurements. Use a problem large enough for meaningful timing and report a median, along with the machine, compiler or interpreter, optimization flags, and worker count.
- Distinguish setup and execution. The companion examples exclude pool startup from the timed region but include ordinary task dispatch or communication. GPU operations may finish asynchronously: for MATLAB use
wait(gpuDevice)before ending atic/tocmeasurement, and for CuPy usecupy.cuda.Device().synchronize()before reading the final time. Count data transfers if they are part of the measured workflow. - Compare equivalent work. Keep grid size, time steps, sample count, and numerical tolerances fixed during strong-scaling comparisons. Do not compare optimized parallel execution to an unnecessarily slow serial loop.
- Avoid oversubscribing Python workers. Libraries called within a process pool can start their own threads (for example, threaded BLAS). In suitable cases set
OMP_NUM_THREADS=1before launching workers, along with the relevant BLAS-library limit such asOPENBLAS_NUM_THREADS=1orMKL_NUM_THREADS=1. Numba uses a separateNUMBA_NUM_THREADSsetting. - Relate measurements to Amdahl's law. For
P > 1and observed speedupS, the Karp–Flatt estimatee = (1/S − 1/P)/(1 − 1/P)is an effective serial fraction: overhead and changing hardware performance contribute, so do not interpret it as just serial source code. - Account for real hardware. Logical cores are not necessarily physical cores. Memory bandwidth, simultaneous multithreading, performance/efficiency cores, and changes in laptop clock frequency can prevent ideal scaling.
The reduction program illustrates how equal iteration counts can conceal
unequal work when a threshold skips half the expensive operations.
The heat stencil has much more uniform per-point work, but may become
limited by memory movement. MATLAB's optimized vectorized serial calculation may already use several
CPU cores, whereas each process-based parfor worker normally
uses one computational thread by default. Speedup below one can therefore
partly reflect the multithreaded serial baseline; a separate test with
maxNumCompThreads(1), or MATLAB started with
-singleCompThread, can help compare like with like.
Additional workers can also compete for memory and processor resources. MPI halo timing includes time waiting for neighboring
ranks, not merely bytes transferred. Repeated GPU-to-CPU copies can cost
more than the GPU computation itself.
Reproducibility and common programming errors
- Shared accumulators: Several threads updating one sum without synchronization cause a data race. Use OpenMP
reduction, or another controlled combination of private partial results.atomiccan be correct but slower. - Stencil updates: Read from the previous time-step field and write into a separate next-step field. An in-place update changes the numerical scheme and may violate conservation.
- Random streams: Associate streams with fixed batch identities, not with worker numbers, and combine batch results in a stable order. Identical seeds are not a promise of identical sequences across languages.
- Monte Carlo uncertainty: An estimate of pi fluctuates from run to run; the difference from the reference should be compared with the estimated standard error. In the supplied weak-scaling tests, larger runs reuse the earlier batches, so those estimates are correlated.
- MPI boundaries: Exchange the correct ghost rows and tags before each update. A broken halo exchange can cause both field disagreement and conservation drift.
From a laptop to a cluster
The MPI heat example can run on a personal computer with an MPI
implementation: a cluster is not necessary. First test with one or two
processes, then increase the process count while checking numerical
agreement. Oversubscribing more processes than available execution slots
may help test correctness but usually gives misleading timing results.
In a local, unscheduled Open MPI run, the default slots commonly
correspond to physical cores rather than hardware threads. A request
for more ranks than slots may fail with a not-enough-slots message.
Depending on the Open MPI version and host settings,
--use-hwthread-cpus or --oversubscribe may
permit the run; neither makes oversubscribed timings representative.
The Codes page includes
individual SLURM and PBS job-submission scripts for the MPI heat example.
A job script requests resources and launches a previously compiled
executable; it does not install a compiler or MPI. Adapt the queue,
account, modules, resource syntax, and launcher to the local computing
center. The supplied PBS script uses PBS Pro/OpenPBS-style
select resources rather than older Torque-style
nodes:ppn notation.
Selected documentation
The references below are official documentation or established instructional materials. Start with the shorter tutorials before consulting full API specifications.
- LLNL OpenMP tutorial — threads, work sharing, data scoping, reductions, and synchronization.
- LLNL MPI tutorial — processes, point-to-point communication, and collective operations.
- OpenMP reference guides — concise directive and runtime references.
- Open MPI documentation (version 5.0 documentation) — installation, process launch, and MPI interface details; check version 4.1 documentation or local manuals if that is the installed version.
- MathWorks
parforandgpuArray— MATLAB worker and GPU programming. - Python
ProcessPoolExecutorand Numba parallel execution — separate processes and compiled CPU-threaded loops. - NumPy parallel random-number generation — spawning reproducible streams.
- CuPy installation — CUDA compatibility, packages, and drivers.
- SLURM
sbatchreference and Open MPI under SLURM — cluster-job requests and launches; exact commands depend on the installed Open MPI version and site configuration.
Further reading
- B. Chapman, G. Jost, and R. van der Pas, Using OpenMP, MIT Press, 2007. Shared-memory parallel programming.
- W. Gropp, E. Lusk, and A. Skjellum, Using MPI, third edition, MIT Press, 2014. Message passing and MPI programming.
- G. Hager and G. Wellein, Introduction to High Performance Computing for Scientists and Engineers, CRC Press, 2011. Hardware, performance, and scientific computing.
- P. Pacheco and M. Malensek, An Introduction to Parallel Programming, second edition, Morgan Kaufmann, 2021. Practical OpenMP, MPI, and GPU programming.