Skip to main content

When the decorator is not the answer

from numba import config

# Same process-wide singleton as before: NRT allocation counting has to be switched on
# before *any* function is compiled. It is not needed until "Where the time actually
# went", but this is where it has to live.
config.NRT_STATS = True

import time

import numpy as np
import numba
from numba import jit, prange
from numba.core.runtime import rtsys

numba.__version__
'0.66.0'

The previous notebook argued for Numba: write the loop, get the register, beat the array notation. This one is the other half of the story — a real production kernel where I deleted my own Numba implementation and replaced it with C++ behind pybind11.

The case is hpc4cmb/toast#a38d1d6, “replace _fma by a compiled C++ function inplace_weighted_sum”, from 2021. I wrote the Numba version, I wrote the C++ that replaced it, and I wrote the commit message justifying the swap:

This not only eliminates Numba as an optional dependency, but also makes it faster: - In a test with n_weights = 68; n_out = 19 * 60 * 60 * 24, the function is faster by ~30% - In real world testing with the crosstalk operator, this speeds up matmul by ~10% - Performance gain likely due to guaranteed memory alignment in C++ code - Benchmarked on 10-core i9-7900X with DDR4 memory at 3000 MHz

Five years of tooling later, this notebook rebuilds that kernel at exactly those sizes and takes my own claims apart one at a time — because every one of them turns out to be about something other than “compiled languages are fast”. By the end we will have: measured what @jit did and did not fix, established that parallel=True makes this kernel slower, reproduced from Python a performance warning I had left for myself in a C++ comment, and put a number on that last bullet — the alignment hypothesis, which I flagged in 2021 as a guess and never checked — with a controlled experiment that never leaves Numba.

I think the commit was right, in the end, but for none of the reasons I gave at the time.

The kernel

TOAST processes time-ordered data from cosmic microwave background telescopes. Detectors on a focal plane pick up a little of each other’s signal — crosstalk — and both simulating it and removing it means applying a mixing matrix across detectors. For one output detector that is a weighted sum of every detector’s entire time stream:

\[ \mathrm{out}_j \mathrel{+}= \sum_{i} w_i \, a_{ij} \]

Here is what I wrote, verbatim from the commit that is about to delete it — docstring included, because my own docstring is the first piece of evidence:

def _fma(out, weights, *arrays):
    """Simple fused multiply–add, compiled to avoid Python memory implications.

    :param out: must be zero array in the same shape of each in `arrays`

    If not compiled, a lot of Python objects will be created,
    and as the Python garbage collector is inefficient,
    it would have larger memory footprints.
    """
    for weight, array in zip(weights, arrays):
        out += weight * array

Two things to notice before we run anything.

The signature is *arrays, not a matrix. Mathematically this is a matrix–vector product and should go to BLAS. It cannot: the \(a_i\) are separate allocations, handed out one detector at a time by TOAST’s data cache, and there is no \((68, m)\) block to pass. Stacking them into one would mean copying 893 MB to save 20 ms. That single constraint is why the function is variadic, and it propagates all the way through — the C++ replacement’s signature is double ** arrays for the same reason.

The justification I gave is memory, not speed. “A lot of Python objects will be created… larger memory footprints.” Numba was added here as an allocation fix. Hold on to that; we are going to check whether it worked.

m = 19 * 60 * 60 * 24  # a day of samples at 19 Hz
n = 68  # detectors

out = np.zeros(m)
weights = np.arange(1, 1 + n, dtype=np.float64)
arrays = [np.arange(1 + i * n, 1 + i * n + m, dtype=np.float64) for i in range(n)]

CELL = m * 8  # bytes in one time stream
READ = n * CELL  # bytes of array data one call must read
print(f"{n} arrays × {m:,} samples = {CELL / 1e6:.2f} MB each, {READ / 1e9:.3f} GB total")
print(f"flops per call: {2 * n * m / 1e6:.0f} M      arithmetic intensity: {2 * n * m / READ:.2f} flop/byte")
68 arrays × 1,641,600 samples = 13.13 MB each, 0.893 GB total
flops per call: 223 M      arithmetic intensity: 0.25 flop/byte

A quarter of a flop per byte. A modern desktop CPU needs somewhere in the region of 20 to 50 flops per byte to keep its floating-point units fed, so this kernel is roughly two orders of magnitude short: nothing in this notebook is compute-bound. Every number below is a statement about memory traffic, and the only question a compiler gets to answer is how much traffic it generates for the same arithmetic.

One consequence worth keeping in mind: out is 13 MB and fits in this CPU’s last-level cache, while the 893 MB of input does not. So the ideal call streams 893 MB in from DRAM once, and does its read–modify–write on out entirely in cache. That is the floor.

t_np = %timeit -o _fma(out, weights, *arrays)
30.7 ms ± 425 μs per loop (mean ± std. dev. of 7 runs, 10 loops each)

Adding the decorator

@jit is an ordinary function of a function, so — as in the loops notebook — we can compile the identical code object rather than a rewrite of it. Both variants below come from the source above, untouched. parallel=True is the one I shipped; the plain one is here so that we can tell the two decisions apart, which at the time I did not.

_fma_jit = jit(nopython=True, nogil=True)(_fma)
_fma_par = jit(nopython=True, nogil=True, parallel=True)(_fma)

print("same function object:", _fma_jit.py_func is _fma is _fma_par.py_func)
same function object: True
t_jit = %timeit -o _fma_jit(out, weights, *arrays)
31.3 ms ± 1.46 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
t_par = %timeit -o _fma_par(out, weights, *arrays)
24.7 ms ± 286 μs per loop (mean ± std. dev. of 7 runs, 1 loop each)

That is not the shape of result the loops notebook trained us to expect. Compiling the function bought essentially nothing — the JIT-compiled version ties with interpreted NumPy — and adding every core in the machine on top of that did not even manage a factor of two.

Both of those deserve an explanation, and they are different explanations. Take the allocations first.

Where the time actually went

Numba’s runtime (NRT) counts every array allocation a compiled function makes. It also counts the reference-counted handle wrapped around each array argument, and this function has 70 of those, so — exactly as in the loops notebook — we need a baseline that allocates nothing.

def nrt_allocs(fn, *args):
    """NRT allocations for one call, once the function is compiled."""
    fn(*args)
    before = rtsys.get_allocation_stats()
    fn(*args)
    after = rtsys.get_allocation_stats()
    return after.alloc - before.alloc


@jit(nopython=True, nogil=True)
def _wrappers_only(out, weights, *arrays):
    return out  # hand back an input — nothing new can be allocated


BASELINE = nrt_allocs(_wrappers_only, out, weights, *arrays)
print(f"wrapping {2 + n} array arguments, computing nothing: {BASELINE} allocations")
print(f"_fma_jit: {nrt_allocs(_fma_jit, out, weights, *arrays) - BASELINE} beyond the baseline")
wrapping 70 array arguments, computing nothing: 70 allocations
_fma_jit: 68 beyond the baseline

Sixty-eight. One full-size temporary per input array — 893 MB of allocation churn per call, the exact quantity the docstring was written to eliminate.

It is worth seeing why, because the source looks like it cannot allocate. out += weight * array is an in-place operator, and in-place is supposed to mean no temporary. But += is only in-place at the outermost level: weight * array is a subexpression, it has to be evaluated before the addition can happen, and its value is an array — so it needs an address, so it needs memory. NumPy allocates it, and Numba’s array-expression optimiser, faced with the same expression, does the same thing.

So @jit reproduced the very behaviour it was added to fix. And once you know that, the timing stops being surprising: both versions move identical bytes — 893 MB of input, plus a 13 MB temporary written and read back per iteration — so both run at the same speed. The decorator was never going to help, because the decorator did not change the memory traffic.

Writing the loop

The fix is the one from the previous notebook: say it elementwise, so that weight * array never becomes a value that needs an address. Here it is a scalar in a register, and the accumulation happens in place for real.

def _fma_loop_src(out, weights, *arrays):
    for i in range(len(weights)):
        w = weights[i]
        a = arrays[i]
        for j in range(out.shape[0]):
            out[j] += w * a[j]


_fma_loop = jit(nopython=True, nogil=True)(_fma_loop_src)

ref = np.zeros(m)
_fma(ref, weights, *arrays)
chk = np.zeros(m)
_fma_loop(chk, weights, *arrays)
print("matches:", np.allclose(chk, ref))
print(f"_fma_loop: {nrt_allocs(_fma_loop, out, weights, *arrays) - BASELINE} allocations beyond the baseline")
matches: True
_fma_loop: 0 allocations beyond the baseline
t_loop = %timeit -o _fma_loop(out, weights, *arrays)
17.3 ms ± 51.8 μs per loop (mean ± std. dev. of 7 runs, 100 loops each)

Zero allocations, and it goes from tying with interpreted NumPy to beating it comfortably. Note what that difference is: the 13 MB temporary, written and read back once per outer iteration, disappearing. No arithmetic changed, no vectorisation changed — LLVM emits <8 x double> operations for both — only the traffic.

This is the version I should have written in the first place. It is nine lines of Python, it allocates nothing, and it is the fair opponent for the C++ that replaced it. Everything from here compares against this, not against the version I actually benchmarked in 2021 — which matters, because the ~30% in the commit message was measured against a Numba version carrying 893 MB of temporaries it did not need.

parallel=True, and why it is a trap here

The Numba call I shipped was jit(..., parallel=True), and the C++ I replaced it with is # pragma omp parallel for simd. Twice, in two languages, I reached for every core in the machine. Let us find out what that is worth on a kernel with an arithmetic intensity of 0.25.

prange marks the loop to parallelise. We put it on the inner loop deliberately — the outer loop over detectors cannot be parallelised, because every iteration accumulates into the same out.

def _fma_loop_par_src(out, weights, *arrays):
    for i in range(len(weights)):
        w = weights[i]
        a = arrays[i]
        for j in prange(out.shape[0]):
            out[j] += w * a[j]


_fma_loop_par = jit(nopython=True, nogil=True, parallel=True)(_fma_loop_par_src)

chk[:] = 0
_fma_loop_par(chk, weights, *arrays)
print("matches:", np.allclose(chk, ref))
print("threads available:", numba.config.NUMBA_NUM_THREADS)
matches: True
threads available: 32
t_loop_par = %timeit -o _fma_loop_par(out, weights, *arrays)
25 ms ± 107 μs per loop (mean ± std. dev. of 7 runs, 10 loops each)

All the cores in the machine, to lose to the serial loop. That is not a fluke of thread-pool overhead; sweep the thread count and the shape of the thing is unmistakable.

def best_of(fn, *args, reps=5):
    """Fastest of `reps` calls, in seconds — `%timeit` cannot benchmark a loop variable."""
    fn(*args)  # warm up: compile, and fault in any fresh pages
    best = float("inf")
    for _ in range(reps):
        t0 = time.perf_counter()
        fn(*args)
        best = min(best, time.perf_counter() - t0)
    return best


for nt in [1, 2, 4, 8, 16, 32]:
    numba.set_num_threads(nt)
    t = best_of(_fma_loop_par, out, weights, *arrays)
    print(f"{nt:3d} thread{'s' if nt > 1 else ' '}: {t * 1e3:7.2f} ms   {READ / t / 1e9:5.1f} GB/s")

numba.set_num_threads(numba.config.NUMBA_NUM_THREADS)
  1 thread :   17.35 ms    51.5 GB/s
  2 threads:   16.57 ms    53.9 GB/s
  4 threads:   16.78 ms    53.2 GB/s
  8 threads:   16.78 ms    53.2 GB/s
 16 threads:   23.03 ms    38.8 GB/s
 32 threads:   24.46 ms    36.5 GB/s

The curve peaks at two to four threads and then falls over. A couple of cores are enough to saturate this machine’s memory controllers; every core after that adds contention — for bandwidth, for the shared last-level cache that out was living in, for DRAM page locality — and gives back nothing, because there was no arithmetic left to overlap in the first place. By the time all 32 threads are running, the kernel is slower than one.

So parallel=True on this kernel is a mechanism for spending most of a CPU to run slower. It is not that parallelism is wrong here; it is that the kernel’s ceiling is bandwidth, the ceiling is reached almost immediately, and past that the only variable left to move is contention. Worth being clear that this is a property of the kernel, not of Numba: my C++ replacement ships the same parallel for and hits the same wall, and the ten-core i9 I benchmarked on in 2021 had plenty of cores to hit it with. I did not test that either.

The general lesson is the one you can read straight off the arithmetic intensity we printed at the top. parallel=True multiplies your compute; if compute is not the constraint, it multiplies nothing.

The comment I left in the C++

The C++ file ends with an alternative version, commented out, and a note to nobody in particular:

// the following produces correct results, but slower.
// void toast::inplace_weighted_sum(...) {
//     # pragma omp parallel for simd
//     for (int j = 0; j < n_out; ++j) {
//         for (int i = 0; i < n_weights; ++i) {
//             out[j] += weights[i] * arrays[i][j];
//         }
//     }
// }

That is the loop nest interchanged: sample on the outside, detector on the inside. It ought to be the better order. It reads each element of every array exactly once, same as before, but out[j] is now touched once per sample instead of \(n\) times — it never even has to be a memory location, just an accumulator in a register. Strictly less traffic, on paper.

In 2021 I established that it was slower and stopped there. We can find out why without a C++ toolchain at all, because Numba will compile the same transposition:

def _fma_j_src(out, weights, *arrays):
    for j in range(out.shape[0]):
        acc = out[j]
        for i in range(len(weights)):
            acc += weights[i] * arrays[i][j]
        out[j] = acc


t0 = time.perf_counter()
_fma_j = jit(nopython=True, nogil=True)(_fma_j_src)
chk[:] = 0
_fma_j(chk, weights, *arrays)
compile_j = time.perf_counter() - t0
print("matches:", np.allclose(chk, ref))
matches: True
t_j = %timeit -o _fma_j(out, weights, *arrays)
210 ms ± 5.41 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

Nearly an order of magnitude slower than the loop we had, for a version that moves less data. The LLVM IR says what happened:

def llvm_ir(fn):
    return list(fn.inspect_llvm().values())[0]


for name, fn in [("_fma_loop", _fma_loop), ("_fma_j", _fma_j)]:
    ir = llvm_ir(fn)
    print(f"{name:12s} <8 x double> ops: {ir.count('<8 x double>'):3d}")
_fma_loop    <8 x double> ops:  23
_fma_j       <8 x double> ops:   0

Zero. The interchange destroyed the vectorisation, and it had to: SIMD needs eight consecutive samples in one register, and j — the only index that walks contiguous memory — is now the outer loop. The inner loop walks i, which strides 13 MB per step. There is nothing adjacent to pack.

And that stride is the second problem. Every inner iteration touches a different array, so the loop has 68 concurrent streams running through it. Hardware prefetchers track on the order of a dozen; TLBs hold a comparable number of pages for a working set this size. At 68 streams both give up, and the loop reverts to paying full memory latency, one scalar element at a time.

Five years ago I found this by rebuilding the extension, running the benchmark, seeing a worse number, and writing “correct results, but slower” — an empirical fact with no explanation attached. Here it took three lines of Python to reproduce and one line of IR to explain.

Which is the practical lesson I would most like to have had at the time: Numba is an excellent instrument for reasoning about a kernel you intend to write in C++. The optimisation questions — loop order, vectorisation, streams, traffic — are properties of the loop nest and the machine, not of the language. Numba lets you ask them in seconds, in a notebook, with the IR right there; in C++ every one of those questions costs a rebuild.

The bill for asking

Except that “in seconds” needs an asterisk, and this is where JIT stops being free. Ask for the same interchanged loop to be parallelised — prange on the outer loop, one word changed:

def _fma_j_par_src(out, weights, *arrays):
    for j in prange(out.shape[0]):
        acc = out[j]
        for i in range(len(weights)):
            acc += weights[i] * arrays[i][j]
        out[j] = acc


t0 = time.perf_counter()
_fma_j_par = jit(nopython=True, nogil=True, parallel=True)(_fma_j_par_src)
chk[:] = 0
_fma_j_par(chk, weights, *arrays)
compile_j_par = time.perf_counter() - t0

print("matches:", np.allclose(chk, ref))
print(f"_fma_j     compiled in {compile_j:7.2f} s")
print(f"_fma_j_par compiled in {compile_j_par:7.2f} s")
matches: True
_fma_j     compiled in    0.40 s
_fma_j_par compiled in   51.51 s

Two orders of magnitude, for one word. The parallel-accelerator pass has to reason about a loop body that indexes a 68-element tuple by a runtime index, and the only way it can is to unroll: what was a four-line loop becomes a basic block with hundreds of live values in it, and every LLVM pass downstream pays for that superlinearly. Nothing about the result is unusual — the compiled function runs at the same bandwidth as everything else — the cost is entirely in getting there.

A minute of compilation is an annoyance in a notebook. It is something else on the machines this code runs on, and the call I shipped shows I knew it:

# cache is False to avoid IO on HPC.
_fma = jit(_fma, nopython=True, nogil=True, parallel=True, cache=False)

cache=False because thousands of MPI ranks starting at once and each writing a compilation cache is worse than not caching at all. Which means: every rank, every job, recompiles from scratch before doing any science. Multiply that by the rank count and JIT warm-up becomes a line item in the allocation request. An ahead-of-time compiled extension has this cost exactly once — at build time, on someone else’s machine — and that, rather than any instruction the compiler emits, is the strongest argument in my commit message. It is in the first line, and I filed it under “eliminates a dependency” rather than under performance.

The alignment hypothesis

Which leaves the third bullet: “performance gain likely due to guaranteed memory alignment in C++ code.” I hedged it with “likely” and never followed it up. It deserved better, because it is testable — the C++ side stores its data in toast::AlignedVector, which allocates on a 64-byte boundary, and NumPy makes no such promise.

We cannot benchmark against the C++ here; we have not built it, and a cross-toolchain comparison would confound compiler, flags, and runtime all at once. But we do not need to. Alignment is a property of addresses, and we can vary an address inside Numba while holding literally everything else fixed. Start by looking at what NumPy actually hands us:

print("array data addresses, modulo 64 :", sorted({a.ctypes.data % 64 for a in arrays}))
print("array data addresses, modulo 4096:", sorted({a.ctypes.data % 4096 for a in arrays}))
array data addresses, modulo 64 : [16]
array data addresses, modulo 4096: [16]

Every single one sits 16 bytes past a page boundary — the allocator maps a fresh page and spends the first 16 bytes on its own bookkeeping. So NumPy arrays this size are reliably 16-byte aligned and reliably not 64-byte aligned. A 64-byte AVX-512 load from such an address straddles two cache lines. Every time.

Numba, for its part, cannot help. Look at what it tells LLVM about these pointers:

ir = llvm_ir(_fma_loop)
for line in ir.splitlines():
    s = line.strip()
    if "<8 x double>" in s and ("load" in s or "store" in s):
        print(s.split(", !")[0])
        break
print()
print("alignment annotations in the whole module:")
for a in ["8", "16", "32", "64"]:
    print(f"  align {a:>2}: {ir.count('align ' + a):4d}")
%wide.load = load <8 x double>, ptr %sunkaddr118, align 8

alignment annotations in the whole module:
  align  8:  489
  align 16:    0
  align 32:    0
  align 64:    0

align 8 — and not one annotation stronger anywhere in the module. Eight bytes is just sizeof(double): it is what LLVM can deduce from the element type alone, which is to say Numba told it nothing. Numba’s array model carries shape, strides and contiguity, and no alignment field at all, so there is nothing to propagate. LLVM emits unaligned vector moves because unaligned is all it can prove.

Now the experiment. Allocate the arrays with a little slack, and take views starting at different byte offsets from a 64-byte boundary. Same pages, same buffers, same function, same 893 MB — only the address changes.

del arrays  # 893 MB; make room before allocating the padded copies

PAD = 16  # elements of slack, so every offset below still yields m samples
bases = [np.full(m + PAD, 1.0) for _ in range(n)]
obase = np.zeros(m + PAD)


def at_offset(buf, off):
    """View of length `m` starting `off` elements past the next 64-byte boundary."""
    start = (-buf.ctypes.data) % 64 // 8 + off
    return buf[start : start + m]


sweep = {}
for _ in range(3):  # interleave the offsets, so drift cannot masquerade as signal
    for off in [0, 1, 2, 4, 8]:
        views = [at_offset(b, off) for b in bases]
        o = at_offset(obase, off)
        t = best_of(_fma_loop, o, weights, *views)
        sweep[off] = min(sweep.get(off, t), t)

for off, t in sorted(sweep.items()):
    mark = {0: "  aligned", 8: "  aligned", 2: "  ← what NumPy gives you"}.get(off, "")
    print(
        f"start {off * 8:2d} B in  (address % 64 = {off * 8 % 64:2d}): "
        f"{t * 1e3:6.2f} ms   {READ / t / 1e9:5.1f} GB/s{mark}"
    )

penalty = max(sweep.values()) / min(sweep.values())
print(f"\nmisalignment costs {100 * (penalty - 1):.0f}% of the runtime")
start  0 B in  (address % 64 =  0):  16.90 ms    52.8 GB/s  aligned
start  8 B in  (address % 64 =  8):  17.31 ms    51.6 GB/s
start 16 B in  (address % 64 = 16):  17.90 ms    49.9 GB/s  ← what NumPy gives you
start 32 B in  (address % 64 = 32):  17.29 ms    51.6 GB/s
start 64 B in  (address % 64 =  0):  16.76 ms    53.3 GB/s  aligned

misalignment costs 7% of the runtime

The signal is clean and it has period 64. Offsets 0 and 8 elements are both 64-byte aligned and both fast; every offset in between is a fifth slower; and the penalty does not grow with the offset — it switches on the moment the address stops being a multiple of 64 and stays flat. That is the signature of cache-line splitting and nothing else: a misaligned 64-byte vector load touches two lines instead of one, and on a kernel that does nothing but load, that is the whole cost.

So the guess was right, and five years late, here is a number for it: on this machine, alignment alone is worth about a fifth of the runtime — comfortably enough to account for a large part of the ~30% I reported, arrived at without changing a compiler, a language, or a line of the loop.

Which is exactly the problem with having used it as an argument. That fifth is not something C++ has and Python lacks. It is something toast::AlignedVector has and np.empty lacks — and the five-line at_offset above closes the gap from inside Python. Numba compiled the identical function for every row of that table. Alignment was never a reason to change language; it was a reason to change allocator, and Python has one of those too.

The scoreboard

Times carried down from the %timeit -o cells above rather than retyped, so the table cannot drift out of step with the measurements that produced it.

rows = [
    ("_fma", "NumPy, interpreted", t_np, "—"),
    ("_fma_jit", "same source, @jit", t_jit, 68),
    ("_fma_par", "same source, parallel=True", t_par, 68),
    ("_fma_loop", "explicit loop", t_loop, 0),
    ("_fma_loop_par", "explicit loop, parallel=True", t_loop_par, 0),
    ("_fma_j", "loop interchanged", t_j, 0),
]

print(f"{'':16s}{'':30s}{'ms':>8s}{'vs np':>8s}{'GB/s':>8s}{'temps':>7s}")
for name, note, t, temps in rows:
    print(
        f"{name:16s}{note:30s}{t.best * 1e3:8.2f}{t_np.best / t.best:7.2f}×"
        f"{READ / t.best / 1e9:8.1f}{str(temps):>7s}"
    )
print(f"\n{'_fma_loop':16s}{'explicit loop, 64B-aligned':30s}{sweep[0] * 1e3:8.2f}"
      f"{t_np.best / sweep[0]:7.2f}×{READ / sweep[0] / 1e9:8.1f}{0:>7d}")
print("\ntemps = full-size arrays allocated per call (NRT, less the 70-argument baseline)")
print("GB/s  = 893 MB of input ÷ best time; ignores the read–modify–write on `out`,")
print("        which is 13 MB and stays in last-level cache")
                                                    ms   vs np    GB/s  temps
_fma            NumPy, interpreted               30.43   1.00×    29.3      —
_fma_jit        same source, @jit                30.58   1.00×    29.2     68
_fma_par        same source, parallel=True       24.35   1.25×    36.7     68
_fma_loop       explicit loop                    17.29   1.76×    51.7      0
_fma_loop_par   explicit loop, parallel=True     24.76   1.23×    36.1      0
_fma_j          loop interchanged               199.90   0.15×     4.5      0

_fma_loop       explicit loop, 64B-aligned       16.90   1.80×    52.8      0

temps = full-size arrays allocated per call (NRT, less the 70-argument baseline)
GB/s  = 893 MB of input ÷ best time; ignores the read–modify–write on `out`,
        which is 13 MB and stays in last-level cache

Read the column of ratios and the interesting thing is how narrow it is. Interpreted NumPy and the best thing we built are less than a factor of two apart, and one row — the interchanged loop — is far worse than doing nothing at all. There is no headroom here, because there is no arithmetic here. Every row runs against the same wall at a different distance from it, and the distance is measured entirely in wasted traffic: a temporary, a cache-line split, a lost vectorisation.

What the C++ actually cost

The kernel I replaced all of this with is six lines:

void toast::inplace_weighted_sum(int n_out, int n_weights, double * out,
                                 double const * weights, double ** arrays) {
    for (int i = 0; i < n_weights; ++i) {
        double const weight = weights[i];
        double * const array = arrays[i];
        # pragma omp parallel for simd
        for (int j = 0; j < n_out; ++j) {
            out[j] += weight * array[j];
        }
    }
}

Line for line, that is _fma_loop. Same loop order, same scalar weight hoisted out of the inner loop, same in-place accumulation. Two languages, one algorithm — which is the point: the thinking transferred completely, and the previous sections are what the thinking consisted of. Note also what is not different: the Numba version I deleted was two lines, and the C++ that beat it is six. The extra four lines are the loop I could have written in Python and didn’t.

The diff is +230 −36 across 14 files. The six lines above are not what happened to the repository. What happened is:

toast_math_fma.cpp — the kernel +30
math_fma.hpp — the declaration +15
_libtoast_math_fma.cpp — the pybind11 binding, init_math_fma +95
toast_test_fma.cpp + toast_test.hpp — a unit test the Python version did not need +69
two CMakeLists.txt, _libtoast.cpp, _libtoast.hpp — registration and build wiring ~+13
docs/ — because a C extension is not self-documenting +8
crosstalk.py, utils.py — the actual call sites +3 −36

The binding alone — init_math_fma, 95 lines — is longer than everything else combined. It is not doing anything clever: it requests a buffer from each argument, checks that the format is double, checks that every array has the same length as out, raises through TOAST’s logger if not, builds a double **, and frees it afterwards. All of it is machinery to re-establish, by hand and at runtime, the facts that @jit derived from the arguments for free. Numba’s type dispatcher is that file.

And the diff is only the visible half. The invisible half is that the project now needs a C++17 compiler with OpenMP in every environment that builds it, a CMake configuration that finds it on Linux and macOS, a wheel per platform per Python version, and a debugging story where a segfault is a segfault. Contributors who could have fixed a Python function need a toolchain to fix a C++ one.

So was it the right call?

I still think yes — but on the evidence above, not for any of the reasons I wrote down, and the reasons I wrote down were the performance ones.

The ~30% I measured was three separable things, and none of them was “C++ is faster than Numba”. It was a temporary the array notation forced and the loop removes — recovered in Python, at no cost, in nine lines. It was 64-byte alignment — recovered in Python, at no cost, in five more. What is left once you account for those is small enough that a fair comparison would need controls this notebook does not have. And the parallel for simd I was proud of turns out to be worth less than nothing past two threads, in either language.

In other words I benchmarked my C++ against a Numba version I had not optimised, and attributed the difference to the language. That is the single most common way this comparison goes wrong, and knowing it is common is not much of a defence — I had already written the loops notebook’s lesson into the C++ file without noticing it applied on the other side of the binding too.

What survives is the first line of the commit message, the one that does not mention speed: “this not only eliminates Numba as an optional dependency”. TOAST already had libtoast, a CMake build, pybind11 bindings, and a C++ test suite. Adding a seventh kernel to an existing extension module costs a binding file; it does not cost a toolchain, because the toolchain is already paid for. Against that, Numba was an optional dependency guarded by try: from numba import jit / except ImportError, with a fallback path that silently used more memory — and, as we now know, a compiled path that used just as much. Plus a cache=False recompile on every rank of every job. Deleting a conditional dependency and its fallback, and moving a compile from every job start to one build, is worth real money on an HPC allocation.

That is a decision about the shape of the project, not about the shape of the loop. And it inverts cleanly: a project without libtoast — without the CMake, the wheels, the C++ CI — would be paying all 230 lines and the whole toolchain to buy back a factor we just showed is mostly reachable from Python. For them the same evidence points the other way.

Which is the moral, and it is not about Numba at all. The optimisation work is the same in both languages — this notebook redid mine in Python in an afternoon, and explained a finding I had left as a bare assertion in a C++ comment. What differs is the cost of carrying the result. Do the thinking where it is cheap to think; choose the language for what it costs your project to keep. And benchmark your own best effort, not your first draft — especially when you are about to conclude something about the language.