Configuration
One rule: the environment says where the library runs, before the import, and everything else is a setting you can read and change at runtime through one object. On a CPU nothing needs to be set. On GPUs:
HIPPYMFEM_DEVICE=gpu HIPPYMFEM_HYPRE_DEVICE=1 \
mpirun -n 4 tools/mpirun_pinned.sh python my_script.py
HIPPYMFEM_DEVICE=gpu puts the element kernels on the card, and
HIPPYMFEM_HYPRE_DEVICE=1 (with a CUDA or HIP build of PyMFEM) puts MFEM and hypre there
too, configured when hippymfem is imported. tools/mpirun_pinned.sh gives each
rank one card before CUDA initializes. Every other setting below has a default chosen
from measurements, and most runs never touch one.
Set before the import
These are read by XLA or MPI before the library can act. hm.config shows them
read-only.
variable |
what it decides |
|---|---|
|
where the element kernels run: unset, |
|
MFEM and hypre on the device too (CUDA or HIP PyMFEM) |
|
JAX’s share of each card (the default follows the ranks per card and whether hypre shares it); How large a problem fits |
|
restrict each rank to one card before CUDA starts
(on by default; needs |
|
import petsc4py first, for the PETSc solvers (Linear solvers) |
|
|
|
the node’s card count; written by
|
Every other setting, one object
import hippymfem as hm
hm.config.hessian # "element": how element Hessians are differentiated
hm.config.hessian = "quadrature" # the same as HIPPYMFEM_HESSIAN=quadrature
hm.config.show() # every setting, its value and source: put it in a bug report
hippymfem.config names each setting as an attribute, with a one-line description
and the environment variable that seeds it (HIPPYMFEM_ and the name in capitals,
except where the table says otherwise). Assigning an attribute takes effect as the
environment variable would have, and a value the setting does not know is refused, in
the environment as well. Neither import hippymfem nor
hm.config.as_dict(load=False) loads JAX; reading a kernel setting imports the kernel
module, as using it would.
The settings a run may want to change:
|
default |
what it decides |
|---|---|---|
|
|
how element Hessian blocks are differentiated:
|
|
|
precision of the element kernels: |
|
empty |
path of a single-precision build of hypre: the Jacobian of a PDE problem and the CG solves with it then run in that library (Single precision) |
|
off |
device scatter in a fixed order: bit-identical repeats, more working memory |
|
0 |
elements per kernel launch; 0 plans the chunk from the free memory (How large a problem fits) |
|
0 |
GiB of card memory the chunk planner leaves to whatever else shares the card |
|
0.5 |
share of the node’s available memory, split between its ranks, for host element batches |
|
|
hypre’s matrix-vector kernel on a GPU: its own on several ranks of a CUDA build, the vendor’s on one (Several GPUs) |
|
|
megabytes of freed device memory a recycling pool
keeps for hypre; |
|
|
vectors that hypre has used are updated on the
device; |
|
0 |
threads of a sparsity-pattern build; 0 takes this rank’s share of the node’s cores |
The remaining ones select a reference implementation that the tests compare the default against, or tune a threshold. Their defaults are the measured best, and they are listed for completeness:
|
default |
what it decides |
|---|---|---|
|
|
how the parallel matrix is built: straight into
true-dof rows where the prolongation is boolean
( |
|
|
with hypre on a device: hand hypre its two
blocks, or |
|
|
the true-dof route also on one rank (with hypre on a device) |
|
on |
essential dofs eliminated in the scatter rather than by MFEM’s calls afterwards |
|
on |
one differentiation pass for all blocks of a linearization point |
|
|
with the kernels and hypre on one GPU, values cross between JAX’s memory and MFEM’s on the device instead of through the host |
|
off |
keep a pattern’s index arrays on the device |
|
off |
keep MFEM’s |
|
2048 |
elements per step of an element kernel on the host; 0 maps the whole batch at once |
|
16 |
doubles per tangent and quadrature point the chunk planner assumes |
|
8 |
tangents a chunk of an element vector (a residual, a gradient) is planned for; 0 plans it as a matrix, with one for every dof |
|
0.25 |
share of the device budget above which element geometry stays on the host and is streamed |
|
off |
keep the inverse Jacobians once an element where every element of a group is mapped affinely (the element arrays change at round-off) |
|
0 |
elements per slice when the quadrature geometry is built here; a positive value forces that path |
|
on |
keep a large assembly accumulator between
assemblies (above |
|
|
sparsity patterns without a global sort when
numba is importable and the pattern has
|
|
on |
the index tables of a large pattern from the dof tables and row by row, no map kept twice: half the host memory of a rank (How large a problem fits) |
|
|
where the sort route sorts: |
|
off |
print what each phase of a pattern build took |
|
on |
CG with a hypre preconditioner runs in hypre’s own PCG, the same iteration with one product fewer per V-cycle; off: MFEM’s CG |
|
-1 |
default BoomerAMG relaxation type of new solvers; -1 keeps MFEM’s |
|
-1 |
default BoomerAMG level cap; -1 keeps hypre’s |
|
0 |
reuse a solver’s AMG hierarchy across operators |
|
0 |
a p-multigrid in the single-precision solves of new solvers, for spaces of order two or more |
|
0 |
under the linear elements of a p-multigrid, a V-cycle over up to this many of the meshes that the mesh was refined from (geometric levels) |
|
0 |
with hypre on a device, the levels of those meshes of at most this many rows a rank live in host memory, with everything under them, where the gathered last matrix has no more rows than that in all (geometric levels) |
|
0 |
the multigrid cycles of a prior’s Hessian preconditioner go over up to this many of the meshes that its mesh was refined from, in the single-precision hypre, where the mesh was last cut in two (geometric levels) |
|
off |
the blocks of a linearization point are matrices
of the single-precision hypre where it is loaded:
two thirds of their memory, products to single
precision ( |
Some switches of the device assembly and of the single-precision solves are read from the
environment alone, when their module is imported. Each is on unless set to 0, except
those that name a choice and the BoomerAMG options, which are a string and empty by
default:
variable |
what it decides |
|---|---|
|
the scatter map of a fused assembly in its compact form, a third of the four-byte map (How large a problem fits) |
|
streamed element geometry goes to the card from pinned host memory instead of through JAX’s copy (How large a problem fits) |
|
how an assembly hands over the rows that other ranks
own where matrices are finished on the device:
|
|
a matrix for the single-precision hypre is
accumulated in single precision; |
|
the Jacobian of the single-precision hypre keeps what the rounding of its values took from each row’s sum, and its solver adds that in every product (Single precision) |
|
a p-multigrid keeps the product of the matrix and
its interpolation for the second residual of a
cycle; |
|
the two products of a p-multigrid come from the
Jacobian’s element matrices, in the pass that
assembles it; |
|
the relaxation on a geometric level of a p-multigrid
whose next mesh is three or more times coarser a
side: |
|
a probe’s clocks on the corrections from some of a
p-multigrid’s coarser meshes down (the indices of
the levels, as |
|
the first and the middle rank say where a p-multigrid with levels in host memory is, in its set-up and its first cycles |
|
single-precision matrices of one pattern share one copy of its column indices on the device |
|
the single-precision hypre allocates device memory from the recycling pool of the double-precision one |
|
BoomerAMG options of the single-precision solves by
name, such as |