Speaker
Description
A build is bitwise reproducible when compiling the same source with the same configuration and toolchain yields byte-for-byte identical output — an identical vmlinux and bzImage, verifiable by a simple sha256sum. For a normal kernel this is a hygiene property; for a Trusted Execution Environment it is foundational. A platform that measures the code it boots and reports a cryptographic hash is only meaningful if the expected value can be independently reproduced i.e. if an auditor or WhatsApp itself, or an external reviewer can take the published source and configuration, rebuild the kernel, and arrive at exactly the same hash the hardware attests to. Without bitwise reproducibility there is an unverifiable gap between "the source we published" and "the binary that is running," and the attestation degrades from a proof into a promise. Reproducibility closes that gap and lets a TEE's trust chain be checked end-to-end by anyone. The central difficulty is that a stock kernel build is not deterministic by default, it embeds build timestamps, the building user and host, git-derived version strings, an archive of kernel headers, and ephemeral module-signing keys all of which vary run to run and perturb the final hash even when no source changed. The engineering problem is therefore to systematically identify and eliminate every source of non-determinism until two independent builds collapse to a single hash.
The strategy we used to obtain a proof of concept was divide and conquer, subsystem by subsystem. The final vmlinux/bzImage hash is just a deterministic function of all the compiled object files linked into it clearly if every .o in every subsystem compiles identically, the whole-kernel hash is necessarily identical. That observation turns an intractable "the whole image differs" problem into a localizable one. Rather than chase the top-level hash blindly, the approach hashes every object file in the tree after each of two builds and then diffs the two manifests. The resulting diff lists exactly which objects and subsystems still differ between builds. Each divergent object is a concrete lead pointing at a specific source of non-determinism; fix that source, rebuild, re-diff, and the list shrinks. Reproducibility is reached when the diff is empty. This breaks the problem down into a tractable, iterative hunt that pinpoints where in the build the divergence originates instead of guessing at the image as a whole.
The work proceeded as many rebuild-and-compare cycles against a copy of the previous build (t1/, build1/), each cycle narrowing the set of differing objects. Recurring offenders and their fixes
included:
- kheaders.o (CONFIG_IKHEADERS), embeds a freshly-archived, timestamped snapshot of the kernel headers that never matches between builds; disabled for the POC.
- Version strings — CONFIG_LOCALVERSION_AUTO appends a git-derived suffix; disabled (LOCALVERSION_AUTO off, empty LOCALVERSION/BUILD_SALT) and the generated version files under init/ and arch/ were pinned.
- Module signing (CONFIG_MODULE_SIG) — signs modules with a per-build ephemeral key, guaranteeing divergence; disabled (captured as a standalone remove_module_signing change).
- Debug/introspection artifacts — CONFIG_GDB_SCRIPTS disabled to drop non-deterministic generated content.
- Build environment — pinned via KBUILD_BUILD_TIMESTAMP, KBUILD_BUILD_USER=builder, KBUILD_BUILD_HOST=buildhost, and SOURCE_DATE_EPOCH, so timestamps and identity strings are fixed rather than sampled from the build machine. The build environment here was a standard Meta devserver, however, to deploy this we would build and deploy an image for a build environment.
The scope of the proof of concept was to establish a clean baseline, the POC was built against Linus's mainline branch (the vanilla upstream tree, origin/linus-upstream) rather than the internal Meta proprietary TEE kernels, isolating the reproducibility problem from the additional out-of-tree patches carried by the production kernel. With the fixes above, two independent builds produced an empty object-level diff every one of the ~4,800 tracked objects matched and identical sha256sum/md5sum values for both vmlinux and arch/x86/boot/bzImage. This demonstrates that a bitwise-reproducible kernel is achievable for the target and validates the subsystem-by-subsystem manifest-diffing method as the path to extend reproducibility from vanilla upstream to the full WhatsApp TEE production kernel.