Common
Parallel vectors, operator protocols, linear algebra, the RNG.
hippymfem.common.parvector
Distributed vectors.
hIPPyMFEM represents every distributed vector as a ParVector: a
contiguous numpy array of the locally owned entries, wrapped by an
mfem.HypreParVector that aliases the same memory. That gives three things
at once:
v.arrayis a plain numpy view, so kernels and MPI calls are direct;v.hypreis accepted by every hypre solver andHypreParMatrix;because
HypreParVectorderives frommfem::Vector, the same object is also accepted by MFEM’s ownCGSolver/GMRESSolverand byOperator.
Vectors on a finite element space live on true dofs. MFEM numbers true
dofs contiguously by rank, so the partition is the exclusive prefix sum of
GetTrueVSize(); this is computed by one allgather and matches the row
partition that ParBilinearForm.ParallelAssemble() produces. That
compatibility is asserted in the test suite rather than assumed.
- hippymfem.common.parvector.DEVICE_VECTORS = 'auto'
Where the arithmetic of a
ParVectorruns when MFEM is configured on a GPU."auto"and"1": a vector that was last handed to MFEM or hypre (throughParVector.hypre) is updated, copied and reduced on the device by MFEM, so that a result hypre left there is not copied to the host for an update and back for the next product; a vector last touched as a numpy array stays with numpy."0": always numpy on the host. The results agree to round-off (the device reductions sum in a different order).HIPPYMFEM_DEVICE_VECTORS.
- hippymfem.common.parvector.set_device_vectors(mode)[source]
"auto","1"or"0"; returns the previous setting.
- hippymfem.common.parvector.device_vectors()[source]
Whether vector arithmetic follows a vector onto the device (see
DEVICE_VECTORS): MFEM configured on a GPU and the setting not"0".
- hippymfem.common.parvector.bigint_array(values)[source]
An MFEM array of
HYPRE_BigIntholdingvalues(at least one entry), for the constructors that takeHYPRE_BigInt *and have no numpy typemap: itsGetData()is that pointer. Filled by index: an array made from a list can hand back a pointer into storage that does not outlive the call.
- hippymfem.common.parvector.bigint_view(pointer, n)[source]
The
nentries behind aHYPRE_BigInt *that MFEM returned, as numpy’s.
- hippymfem.common.parvector.partition(comm, local_size)[source]
Return
(offsets, global_size)for a contiguous by-rank partition.offsetshascomm.size + 1entries;offsets[r]is the global index of rankr’s first entry.
- hippymfem.common.parvector.allreduce(comm, value, op=mpi4py.MPI.SUM)[source]
One number reduced over
commby MPI’s own collective: a float, or an integer (a count, a flag) wherevalueis one.comm.allreduceof mpi4py does the same through pickled point-to-point messages, up a tree of the ranks and down again: two dozen hops one after the other on 4096 ranks, where the library’s collective is a fraction of that. Every inner product and norm of a Newton-CG iteration ends in one.Every rank’s value goes as a double, whatever its type: the type of a buffer is each rank’s own choice, and an integer on one rank reduced with doubles on the others reduced their bits (
0on rank 0 and1.5on rank 1 summed to 4609434218613702656 on rank 0). Integers up to 2^53 are exact; an integer comes back wherevaluewas one and the result is whole.
- hippymfem.common.parvector.allreduce_extreme(comm, loc, op)[source]
MPI.MAXorMPI.MINof floats overcomm, NaN wherever a rank holds one.MPI’s MAX and MIN compare, and a NaN compares false both ways: whether a rank’s NaN survives the reduction depends on the order the ranks’ values meet in, so
norm("linf")of a vector with a NaN on rank 1 of 2 came out finite. The values go through with NaN replaced by the op’s identity, next to a flag that the same op carries to every rank, in one reduction.locis a float or a 1-D array; the result has its shape.
- class hippymfem.common.parvector.Layout(comm, local_size)[source]
Bases:
objectThe global index layout of a distributed vector.
Building one costs a single
allgather, so it is built once and shared: a vector derived from an existing one (duplicate,copy, aMultiVectorcolumn, anything fromFunctionSpace.vector) reuses its parent’s layout and does not communicate, which keeps allocation cheap inside sampling and Krylov loops.It is deliberately not memoized on
(comm, local_size): local sizes differ between ranks, so one rank could hit the cache and skip theallgatherwhile another misses and performs it. The ranks would then desynchronize, and a later, unrelated collective would receive the wrong message. Constructing a layout from a local size always communicates; only reuse is free.of_range()makes one from what a matrix or a space already knows of this rank’s range, with one gather of the sizes by MPI’s own collective on a buffer.- comm
- local_size
- global_size
- col_starts
- classmethod of_range(comm, first, last, global_size)[source]
The layout of a vector whose entries on this rank are the global indices
firsttolast - 1ofglobal_size, as the caller knows them on every rank (the rows of a matrix, the true dofs of a space).The offsets of every rank are gathered here, by MPI’s own collective on a buffer (0.13 ms on 4096 ranks, once for a solver’s operator, where every rank makes the layout). Gathered at their first read instead, they hung a read made on one rank only, as a debugging print makes.
- property offsets
The first global index of every rank, and the global size at the end.
- class hippymfem.common.parvector.ParVector(comm, local_size=None, array=None, layout=None)[source]
Bases:
KeepAliveA distributed vector of real numbers.
- Parameters:
comm (mpi4py communicator)
local_size (int) – Number of locally owned entries. Ignored if
arrayis given.array (numpy.ndarray, optional) – Existing contiguous float64 buffer to adopt (not copied).
- classmethod from_fes(fes, comm=None, layout=None)[source]
A true-dof vector on the parallel finite element space
fes.
- classmethod from_array(comm, array, layout=None)[source]
Adopt
array(no copy) as the local part of a new vector.
- createVecLeft()
A new zero vector with the same layout; communicates nothing.
- createVecRight()
A new zero vector with the same layout; communicates nothing.
- property array
numpy view of the locally owned entries.
With MFEM configured on a GPU the current values may be only in device memory, so this makes the host copy current and valid for read and write (callers do both through this view) before handing it over. Without that the symptom is not an error but stale values, such as a matvec that returns zeros. With MFEM on the host the check is a single flag test.
- property hypre
The
mfem.HypreParVectoraliasing this vector’s memory.Asking for it marks the vector as handed to MFEM: with MFEM on a GPU its next update, copy or inner product runs there (
DEVICE_VECTORS).
- property offsets
Global offsets array,
comm.size + 1entries.
- property owner_range
(first, last)global indices owned by this rank.
- property local_size
- property global_size
- inner(y)[source]
Global Euclidean inner product.
Reads through
_host_readlike every other method: reading_arraydirectly after a hypre matvec on a device would return a stale value. Two vectors that are both with MFEM on a device are reduced there.
- dot(y)
Global Euclidean inner product.
Reads through
_host_readlike every other method: reading_arraydirectly after a hypre matvec on a device would return a stale value. Two vectors that are both with MFEM on a device are reduced there.
- allgather(out=None)[source]
The whole vector as a numpy array on every rank.
Uses
Allgathervwith the layout’s own counts, so it is one collective with no Python-level concatenation. This sits inside the per-solve path ofReplicatedLUSolver.
- scatter_from_zero(full)[source]
Inverse of
gather_to_zero();fullis read on rank 0 only.
- hippymfem.common.parvector.device_active()[source]
Whether MFEM is currently configured on a GPU, not merely built for one.
A PyMFEM built for a GPU keeps everything on the host until a GPU
mfem.Deviceis constructed, and until then none of the synchronization below is needed.Device::IsEnabled()answers this cheaply enough for the vector access path.
- hippymfem.common.parvector.host_sync(obj)[source]
Make an MFEM object’s data valid on the host before it is read as numpy.
When MFEM is configured on a device, an
mfem::Vectormay hold its current values only in device memory, andGetDataArray()silently hands back the host pointer regardless. Reading it gives stale or zero data rather than an error, and the failure surfaces far from the cause (a singular matrix, say).HostReadperforms the copy and marks the host copy valid; aSparseMatrixkeeps its three arrays separately and needs all three. On a host-only build this returns immediately.
- hippymfem.common.parvector.host_readwrite(obj)[source]
Make an MFEM object’s data valid on the host and invalidate the device copy.
Use this when the host side is about to write: marking the data host-read-only would let a later device operation read the pre-write values.
ParVectorgoes through this, because its numpy array and itsHypreParVectoralias the same memory and either side may write.
- hippymfem.common.parvector.host_write(obj)[source]
Give the host write access to an MFEM object’s data without copying it from the device: for a caller that overwrites every entry. The device copy is invalidated, as with
host_readwrite().
- hippymfem.common.parvector.to_numpy(v, copy=True)[source]
Local data of an MFEM
Vectoras a numpy array.mfem.Vector.GetDataArray()returns a view that does not hold a Python reference to the vector that owns the memory. Soform.ParallelAssemble().GetDataArray().sum() # WRONG
reads freed memory: nothing keeps the temporary
HypreParVectoralive pastGetDataArray(). The symptom is a plausible but slightly wrong number that changes from run to run, not a crash. Always go through this helper, which copies while the owner is still referenced by its argument.
hippymfem.common.operators
Operator and solver protocols, and adapters between them and MFEM.
Operators are duck-typed exactly as in hIPPYlib:
mult(x, y)y = A xmultTranspose(x, y)y = A^T x(optional)init_vector(x, dim)dim == 0-> a vector in the range ofA;dim == 1-> the domain. Accepts"noise"for priors, as hIPPYlib does.inner(x, y)y^T A x(optional; supplied byOperatorwhenmultexists)
Solvers expose solve(x, b) meaning x = A^{-1} b.
dim follows hIPPYlib’s convention throughout, including the quirk that the
same integer means “range” for operators and “solution” for solvers.
- class hippymfem.common.operators.Operator[source]
Bases:
KeepAliveBase class supplying
innerand vector generation frommult.
- hippymfem.common.operators.make_vector(obj, dim=0)[source]
Create a vector compatible with
obj.init_vector(x, dim).hIPPYlib’s
init_vectorfills a target instead of returning one, which is awkward when a fresh vector is wanted. This adapts the convention in one place, so callers never hand-roll a sink object (and never accidentally hand one thatinit_vector_like()does not recognise).
- hippymfem.common.operators.init_vector_like(x, template)[source]
hIPPYlib’s
init_vectoridiom: resizexin place to matchtemplate.xmay be aParVector(resized only if the layout differs), a_Slot(used byOperator.generate_vector()), or a list whose element 0 is set.
- hippymfem.common.operators.as_operator(obj, comm=None)[source]
objas anOperator: itself when it hasmult, else aMatrixOperatoraround the (hypre) matrix. The one place the library tells the two apart.
- class hippymfem.common.operators.MatrixOperator(A, comm=None, range_vec=None, domain_vec=None)[source]
Bases:
OperatorWrap an
mfem.HypreParMatrixin the hIPPYlib operator protocol.- transpmult(x, y)
- class hippymfem.common.operators.TransposeOperator(A)[source]
Bases:
OperatorA^Tas an operator, without forming the transpose.
- class hippymfem.common.operators.IdentityOperator(template)[source]
Bases:
Operator- multTranspose(x, y)
- class hippymfem.common.operators.SumOperator(A, B, alpha=1.0, beta=1.0)[source]
Bases:
Operatoralpha*A + beta*Bwith matching layouts.
- class hippymfem.common.operators.ProductOperator(A, B)[source]
Bases:
OperatorA @ B: appliesBthenA.
- class hippymfem.common.operators.DiagonalOperator(d)[source]
Bases:
OperatorMultiplication by a vector, entrywise.
- multTranspose(x, y)
- class hippymfem.common.operators.Solver2Operator(S, init_vector=None)[source]
Bases:
OperatorPresent a solver’s
solveas an operator’smult.
- class hippymfem.common.operators.Operator2Solver(op, init_vector=None)[source]
Bases:
objectPresent an operator’s
multas a solver’ssolve.
- class hippymfem.common.operators.MFEMOperator(*args, **kwargs)[source]
Bases:
PyOperatorBaseAdapt a hIPPyMFEM operator so MFEM’s own solvers can drive it.
MFEM hands us
mfem.Vectorobjects; we copy throughParVectorscratch space because MFEM does not guarantee its vectors carry a hypre partition. The copies go through the host synchronization helpers: with MFEM on a device the vector it hands over may be current only on the card, and the result we write has to be marked as the host copy being the valid one.
hippymfem.common.linalg
Matrix utilities: products, transposes, diagonals, and scipy/hypre bridges.
- hippymfem.common.linalg.alias(arr)[source]
A numpy view of
arr’s buffer that holds no reference toarr.PyMFEM’s array accessors make the owning MFEM object the numpy base. This is for a view that must outlive a scratch wrapper (see
_diag_block()) while aliasing memory owned by something else, such as hypre’s own matrix. The caller must keep that owner alive; nothing here can check it.
- hippymfem.common.linalg.take_ownership(mat)[source]
Hand a C++-allocated
HypreParMatrixto Python’s garbage collector.PyMFEM marks only
ParMultandTransposewith SWIG’s%newobject; the annotations forRAP,Add,ParAddand theEliminate*methods are commented out inmfem/_par/hypre.i. Their results arrive withthisown == Falseand are never freed, a matrix-sized leak per call that an optimizer, assembling and eliminating on every iteration, cannot afford.Each of those functions returns a freshly
new-edHypreParMatrixthat owns its hypre data and that nothing on the C++ side tracks, so transferring ownership to Python is safe.
- hippymfem.common.linalg.own(mat, *keepers)[source]
Tie the lifetime of
keeperstomat(appending;Noneis skipped).MFEM objects hold raw pointers into one another and PyMFEM does not always transfer ownership: a matrix assembled by a form belongs to the form, a
ParMultresult reads its factors, a matrix built on numpy arrays reads them, a grid function reads its space. Attaching the owners to the object that needs them makes the Python reference graph match the C++ one, so the usualA = assemble_matrix(...)just works and dropping the helpers is safe. This is the one place that does it; the owners are found under_hippymfem_owners.
- hippymfem.common.linalg.ParAdd(A, B, alpha=1.0, beta=1.0)[source]
alpha*A + beta*Bas a newHypreParMatrix.
- hippymfem.common.linalg.require_int32_count(n, what, comm=None)[source]
Raise when a rank’s count of nonzeros does not fit a 32-bit index.
hypre keeps the row pointers and local column indices of its CSR blocks as 32-bit integers, and the patterns cast to
int32for it; above2**31 - 1the cast would wrap without a word and assemble a wrong matrix. Withcommthe verdict is reduced, so that every rank raises together.
- hippymfem.common.linalg.set_diagonal_entries(A, idx, value, report_missing=False)[source]
Set
A[i, i] = valuefor the local row indices inidx, in place.valueis a scalar, or one value per entry ofidx. A row with no structural diagonal entry cannot be written and is skipped; withreport_missingthe count of such rows is returned instead ofA, for callers that depend on the entry being there.Needed because MFEM’s
DiagonalPolicyis honored only on the serialSparseMatrixpath: in parallel,EliminateRowsColsalways leaves 1.0 behind.
- hippymfem.common.linalg.get_diagonal(A, d=None, comm=None)[source]
Extract
diag(A)into aParVector.
- hippymfem.common.linalg.estimate_diagonal_inv2(Asolver, k, d)[source]
Probing estimate of
diag(A^{-1})fromkRademacher vectors.This is hIPPYlib’s estimator: with
zhaving independent +-1 entries,E[ z .* A^{-1} z ] = diag(A^{-1}).
- hippymfem.common.linalg.hypre_to_scipy(A)[source]
Local rows of
Aas a scipy CSR matrix with global column indices.PyMFEM’s
ToScipyCSRdoes the same but builds a freshmfem.SparseMatrixper call, which a CUDA-enabled build does not survive repeatedly (see_diag_block()). Here the merge goes through the shared wrapper and the arrays are copied out, so the result does not alias MFEM memory and stays valid after the next call.
- hippymfem.common.linalg.scipy_to_hypre(csr, comm=None, col_starts=None)[source]
Build a
HypreParMatrixfrom per-rank CSR blocks with global columns.csrholds this rank’s rows; the row partition is inferred from the block heights.col_startsmust be given whenever the matrix is not square with identical row and column partitions.
- hippymfem.common.linalg.to_dense(A, comm=None)[source]
Gather
Ainto a dense numpy array, replicated on every rank.For small matrices only; used by the test suite to compare against dense reference computations.
- hippymfem.common.linalg.operator_to_dense(op, domain_size, comm=None)[source]
Dense matrix of a matrix-free operator by applying it to unit vectors.
domain_sizeis the global domain dimension. Used only in tests.
- hippymfem.common.linalg.amg_method(amg_type='boomer')[source]
Name of the default algebraic multigrid preconditioner.
Kept for source compatibility with hIPPYlib drivers, where this selects between PETSc’s AMG implementations. Here hypre’s BoomerAMG is the only option, so the argument is accepted and ignored.
hippymfem.common.random
Partition-independent parallel random numbers.
Reproducibility across MPI decompositions is a test instrument, not a convenience: the test suite runs on several rank counts and checks results that must not depend on the partition, which is meaningless if the random input itself does.
So every entry of a random vector is tied to its global index. We use a counter-based generator (Philox) advanced to the global offset and draw exactly one 64-bit word per entry, mapping uniforms to normals with the inverse normal CDF rather than a ziggurat. Both properties are needed: a fixed word budget per entry, and a generator that can jump ahead in O(1).
- class hippymfem.common.random.Random(seed=1)[source]
Bases:
objectCounter-based parallel RNG.
- Parameters:
seed (int) – Stream key. Two
Randomobjects with the same seed produce the same global sequence regardless of how vectors are partitioned.
- tell()[source]
The position in the stream: the number of words drawn since
set_seed().
- seek(position)[source]
Move to a position of the stream (see
tell()); the draws that follow are the ones a generator that had drawnpositionwords would make.
- normal_blocks(sigma, global_offsets, n)[source]
(len(offsets), n)normals, rowiat global words[offsets[i], +n).The same numbers
normal_block()gives row by row, computed for every row at once (seehippymfem.common.philox); like it, does not advance the stream. This is what element-indexed noise draws: one call per sample instead of one generator per element.
- normal_block(sigma, global_offset, n)[source]
Normals for the global index range
[offset, offset+n).Does not advance the stream: the caller decides how much of the global stream this draw consumed, which lets element-indexed noise be drawn without a collective.
- normal_array(sigma, comm, local_size, global_offset)[source]
Normals for one locally-owned block, advancing the stream globally.
Kept for callers that have no cheaper way to know the global stream length; prefer
normal_block()plus oneadvance().
- hippymfem.common.random.parRandom = <hippymfem.common.random.Random object>
module-level instance, mirroring
hippylibx.parRandom
hippymfem.common.parameterList
A dictionary of (value, description) pairs that rejects unknown keys.
Ported from hIPPYlib so that parameter dictionaries in driver scripts are interchangeable between the two libraries.
hippymfem.common.keepalive
Reference retention for PyMFEM objects.
PyMFEM hands raw C++ pointers to MFEM. If the Python wrapper of, say, a
FiniteElementCollection is garbage collected while a
ParFiniteElementSpace still points at it, the next call into MFEM
segfaults.
Every class in hIPPyMFEM that hands an object to MFEM keeps a strong
reference to it through KeepAlive.
hippymfem.common.mfemconfig
What this PyMFEM was built with, read rather than probed.
MFEM’s own way of answering “do you have CUDA?” is to try to configure a device and
MFEM_ABORT if not, which ends the MPI job instead of returning false. The build
configuration is recorded in the installed config/_config.hpp, so it can be read
directly and without side effects.
- hippymfem.common.mfemconfig.mfem_config(refresh=False)[source]
A dict of the build flags that are defined, plus
version.
- hippymfem.common.mfemconfig.mfem_has(flag)[source]
mfem_has("MFEM_USE_CUDA"), without touching an MFEM device.
- hippymfem.common.mfemconfig.configure_device(kind='gpu', comm=None, quiet=False)[source]
Put MFEM (and therefore hypre) on a device, one per rank.
mfem.Device("cuda")with no device id puts every rank on device 0, which nothing reports and which shows up only as poor GPU scaling. This pickslocal_rank % n_devicesinstead, the rule the element kernels use, so a rank’s matrix and kernels share a card.Only useful with a PyMFEM built for a GPU: CUDA (
mfem_has("MFEM_USE_CUDA")) or HIP (MFEM_USE_HIP).kindmay name the vendor ("cuda","hip") or just"gpu"; each means the backend the build has, so a script runs unchanged on either vendor. Both assembly routes work with hypre on a device: the true-dof scatter builds the parallel matrix from hypre’s two blocks (hippymfem.fem.tdofassemble), and the triple-product fallback builds its local matrix with the constructor that copies into hypre’s own memory (hippymfem.fem.parmat.local_par_matrix()).This is the configuration that matters. The element kernels are a small share of a Newton step, so moving only them to the GPU gains little; moving the solves as well is worth more than an order of magnitude (see the GPU guide).
Returns the
mfem.Deviceit built, which must be kept alive as long as MFEM is used; this module holds a reference to it as well.
hippymfem.common.identitycache
A cache keyed on the identity of live objects that cannot answer for a dead one.
- class hippymfem.common.identitycache.IdentityCache[source]
Bases:
objectValues keyed on the identity of objects, plus hashable extras.
The assembler’s pattern caches key on
id()(the tables of a space, the pattern of a space pair on a set of element groups, the transpose of a prolongation). An address is reused once its object dies, so a stale entry could answer for the wrong object; holding every keyed object alive would prevent that but keep them all for the life of the process. Instead an entry is dropped as soon as any object it is keyed on is finalized (weakref.finalize), so objects are freed when their owners release them.objsare the identity-keyed objects (Noneis allowed and keyed as itself);extrais any hashable value that is part of the key by value, such as a quadrature degree or a device.
hippymfem.common.mpiutil
MPI helpers shared by modules that must not import one another.
- hippymfem.common.mpiutil.local_rank(comm=None)[source]
This process’s index among the ranks sharing its node.
A collective (it splits the communicator), so it must be called by every rank of
command never behind a rank-local condition. Returns 0 when MPI is not available. The import-time device pinning inhippymfem._jaxconfigcannot use it, MPI not being up yet, and reads the launcher’s variables instead.