Companion Website

Computational Physics

From Equations to Simulation

Chapter 15

Parallel Computing: An Orientation to HPC

These resources accompany Chapter 15 of Computational Physics: From Equations to Simulation by Alexander Godunov. They help readers choose a practical first parallel example, identify software requirements, check numerical results, and interpret performance measurements. A multicore personal computer is sufficient for the introductory CPU programs; MPI, GPU, and cluster examples have additional requirements.

Where to begin

Begin with the parallel reduction example: it compares a known serial sum with a parallel reduction and introduces load balancing and race conditions. Next, try two-dimensional heat diffusion to see independent updates of a physical field. Parallel Monte Carlo adds independent random streams and strong and weak scaling. MPI heat diffusion and GPU heat diffusion are optional next steps; neither is required to understand the first three examples.

In every case, first run the serial calculation and its numerical checks. Then run the parallel version at the same problem size and requested accuracy. Do not interpret speedup until both calculations are correct.

Software and hardware requirements

On a narrow screen, scroll horizontally to view all language columns.

Practical parallel facilities by language
TaskFortranC++MATLABPython
Introductory multicore examples GNU Fortran or another compiler with OpenMP support; -fopenmp with gfortran C++11 compiler with OpenMP support; -fopenmp with GCC parfor requires Parallel Computing Toolbox and worker support; base MATLAB runs serial checks Python 3 and NumPy; ProcessPoolExecutor for reduction/Monte Carlo; NumPy and Numba for threaded heat diffusion
MPI domain decomposition MPI implementation and mpifort; supplied source uses use mpi MPI implementation and mpic++; code uses the C MPI interface from C++11 Not supplied Not supplied
Optional GPU example Not supplied Not supplied Parallel Computing Toolbox with supported NVIDIA CUDA GPU for gpuArray NumPy for CPU check; CuPy, compatible CUDA installation/driver, and NVIDIA GPU for GPU calculations

These are requirements for the supplied programs, not a complete list of parallel facilities supported by each language. MATLAB scripts use local functions, requiring R2016b or newer. A compiler or language installation alone does not guarantee that OpenMP, MPI, a toolbox, or GPU support is present.

On macOS, the Apple-provided Clang toolchain normally needs additional OpenMP setup (for example, an appropriate GCC installation or libomp). On Windows, install a compatible MPI implementation such as MS-MPI and adapt the Unix-style example commands. The supplied CUDA examples do not run on Apple-silicon GPUs. Without supported GPU hardware, the GPU programs can still perform their CPU validation. A cloud service such as Google Colab may offer a suitable GPU for Python, subject to availability and setup.

Checks with known answers

Parallel output must satisfy the same numerical requirements as serial output. The companion programs provide checks that do not depend on observing a speedup.

Useful numerical checks for each companion example
ProgramWhat to checkWhat the check establishes
Parallel reduction The serial and parallel sums agree within rounding; the right-endpoint integral error agrees with its predicted leading error. Work is combined correctly. Different reduction orders may change the last digits.
Heat diffusion Compare with the exact discrete Fourier-mode decay, check the analytical PDE error, and monitor mean temperature. Separates programming errors from discretization error; detects incorrect in-place updates.
Parallel Monte Carlo Check exactly matching integer hit counts across worker counts within one language; compare the estimate of pi with its estimated standard error. Confirms reproducible batch assignment and reduction; agreement of hit counts does not establish random-generator quality.
MPI heat diffusion Compare the gathered field with the serial solution, including uneven strip counts; check conservation of mean temperature. Tests halo exchange, domain indexing, and reconstruction. Conservation alone is not a complete field check.
GPU heat diffusion Compare CPU/GPU temperature fields and the known discrete solution, allowing for floating-point differences. Distinguishes GPU implementation errors from differences in arithmetic order and hardware.

Vectorization: before using more cores

Vectorization (SIMD) lets one processor instruction operate on several data values. It is different from running several CPU threads or MPI processes. Before adding parallel workers, check whether a compiler has already vectorized the serial numerical loops.

  • Inspect the compiler report. With GNU Fortran or GCC C++, add -fopt-info-vec to the compile command to see successful vectorizations; -fopt-info-vec-missed reports missed opportunities. A compiler flag is not proof that a particular loop was vectorized.
  • Optimization settings matter. In testing the companion heat stencil with GCC, the one-thread OpenMP loop lost vectorization at -O2 while the serial loop retained it. With -O3, GCC was able to vectorize both. This is compiler- and code-dependent, not a universal guarantee.
  • Floating-point reductions need care. SIMD sums may require regrouping additions and therefore changing the last digits. Compilers can decline to vectorize them under strict floating-point rules. OpenMP SIMD reductions or relaxed floating-point options may help, but reassociation changes numerical behavior; do not enable -ffast-math automatically.
  • Keep Python effects distinct. NumPy vectorization often speeds up a Python loop primarily by moving work out of the interpreter. That gain is not, by itself, evidence of SIMD instructions or multicore parallel execution.

Measuring parallel performance

For P workers, strong scaling holds the total problem size fixed and compares times using S(P) = T_serial / T_parallel(P). Parallel efficiency is S(P)/P. Weak scaling increases total work approximately in proportion to P; compare elapsed time with a one-worker run at the same work per worker. State which baseline is used: one-worker parallel execution and optimized serial execution need not have the same cost.

  • Measure elapsed time, not summed CPU time. Fortran cpu_time can include CPU time accumulated over multiple threads and is not a suitable elapsed-time speedup clock. Suitable clocks include omp_get_wtime, C++ steady_clock, MATLAB tic/toc, and Python perf_counter. MPI timing should report the slowest rank, since the job finishes when all ranks finish.
  • Repeat measurements. Use a problem large enough for meaningful timing and report a median, along with the machine, compiler or interpreter, optimization flags, and worker count.
  • Distinguish setup and execution. The companion examples exclude pool startup from the timed region but include ordinary task dispatch or communication. GPU operations may finish asynchronously: for MATLAB use wait(gpuDevice) before ending a tic/toc measurement, and for CuPy use cupy.cuda.Device().synchronize() before reading the final time. Count data transfers if they are part of the measured workflow.
  • Compare equivalent work. Keep grid size, time steps, sample count, and numerical tolerances fixed during strong-scaling comparisons. Do not compare optimized parallel execution to an unnecessarily slow serial loop.
  • Avoid oversubscribing Python workers. Libraries called within a process pool can start their own threads (for example, threaded BLAS). In suitable cases set OMP_NUM_THREADS=1 before launching workers, along with the relevant BLAS-library limit such as OPENBLAS_NUM_THREADS=1 or MKL_NUM_THREADS=1. Numba uses a separate NUMBA_NUM_THREADS setting.
  • Relate measurements to Amdahl's law. For P > 1 and observed speedup S, the Karp–Flatt estimate e = (1/S − 1/P)/(1 − 1/P) is an effective serial fraction: overhead and changing hardware performance contribute, so do not interpret it as just serial source code.
  • Account for real hardware. Logical cores are not necessarily physical cores. Memory bandwidth, simultaneous multithreading, performance/efficiency cores, and changes in laptop clock frequency can prevent ideal scaling.

The reduction program illustrates how equal iteration counts can conceal unequal work when a threshold skips half the expensive operations. The heat stencil has much more uniform per-point work, but may become limited by memory movement. MATLAB's optimized vectorized serial calculation may already use several CPU cores, whereas each process-based parfor worker normally uses one computational thread by default. Speedup below one can therefore partly reflect the multithreaded serial baseline; a separate test with maxNumCompThreads(1), or MATLAB started with -singleCompThread, can help compare like with like. Additional workers can also compete for memory and processor resources. MPI halo timing includes time waiting for neighboring ranks, not merely bytes transferred. Repeated GPU-to-CPU copies can cost more than the GPU computation itself.

Reproducibility and common programming errors

  • Shared accumulators: Several threads updating one sum without synchronization cause a data race. Use OpenMP reduction, or another controlled combination of private partial results. atomic can be correct but slower.
  • Stencil updates: Read from the previous time-step field and write into a separate next-step field. An in-place update changes the numerical scheme and may violate conservation.
  • Random streams: Associate streams with fixed batch identities, not with worker numbers, and combine batch results in a stable order. Identical seeds are not a promise of identical sequences across languages.
  • Monte Carlo uncertainty: An estimate of pi fluctuates from run to run; the difference from the reference should be compared with the estimated standard error. In the supplied weak-scaling tests, larger runs reuse the earlier batches, so those estimates are correlated.
  • MPI boundaries: Exchange the correct ghost rows and tags before each update. A broken halo exchange can cause both field disagreement and conservation drift.

From a laptop to a cluster

The MPI heat example can run on a personal computer with an MPI implementation: a cluster is not necessary. First test with one or two processes, then increase the process count while checking numerical agreement. Oversubscribing more processes than available execution slots may help test correctness but usually gives misleading timing results. In a local, unscheduled Open MPI run, the default slots commonly correspond to physical cores rather than hardware threads. A request for more ranks than slots may fail with a not-enough-slots message. Depending on the Open MPI version and host settings, --use-hwthread-cpus or --oversubscribe may permit the run; neither makes oversubscribed timings representative.

The Codes page includes individual SLURM and PBS job-submission scripts for the MPI heat example. A job script requests resources and launches a previously compiled executable; it does not install a compiler or MPI. Adapt the queue, account, modules, resource syntax, and launcher to the local computing center. The supplied PBS script uses PBS Pro/OpenPBS-style select resources rather than older Torque-style nodes:ppn notation.

Selected documentation

The references below are official documentation or established instructional materials. Start with the shorter tutorials before consulting full API specifications.

Further reading

← Back to Chapter 15