New software deployment model (from which Nix has been the first example).
{stdenv,lib,fetchFromGitHub}:stdenv.mkDerivation rec{version="1.3.7";pname="htpdate";src= fetchFromGitHub {owner="twekkel";repo= pname;rev="v${version}";sha256="sha256-X7r95Uc4oGB0eVum5D7pC4tebZIyyz73g6Q/D0cjuFM=";};makeFlags=["prefix=$(out)"];meta=with lib;{description="Utility to fetch time and set the system clock over HTTP";platforms= platforms.linux;license= licenses.gpl2Plus;maintainers=with maintainers;[ julienmalka ];};}
Evaluation->Build pipeline
Figure 1: Eval-build pipeline
Functional package managers for SSC security
Functional package managers also have interesting properties for software supply chain security (which are of interest for us):
Builds from source;
Sandboxed compilation.
Functional package managers for SSC security
Installed packages create a static graph structure (a Merkle tree) that can be analysed in order to find known vulnerability in the dependencies.
Figure 2: Example of a package dependency graph
The binary distribution problem
It is not always reasonable to compile all the software a user wants to install on their own machine: creates the necessity of binary caches ;
But binary caches make us lose some of the interesting security properties of functional package managers.
Reproducible builds
A build is reproducible if given the same source code, build environment and build instructions, any party can recreate bit-by-bit identical copies of all specified artifacts.
Why is build reproducibility important?
Reproducible builds can be useful to improve the security of the binary distribution problem;
Instead of having to trust a single third party to provide them the correct binary, they can divide the trust between multiple parties.
Figure 3: Leveraging reproducible-builds to increase trust in distributed artifacts.
Research questions
How reproducible is software in the functional package management model?
Is Nix evaluation reproducible? Can we reproduce build environments of Nix packages?
Do functional package management enable bitwise build reproducibility?
Reproducibility of build environments
Is Nix evaluation reproducible? Can we reproduce build environments of Nix packages?
“Reproducibility of Build Environments through Space and Time”, ICSE 2024 (New Ideas and Emerging Results track), J. Malka, S. Zacchiroli, T. Zimmermann.
Reproducibility of build environments
We say that two build environments are identical if they contain the exact same set of executables, up to their specific versions.
Reproducibility in Space
Figure 4: Reproducibility of build environments in Space
Reproducibility in Time
Figure 5: Reproducibility of build environments in Time
Research questions
RQ1: Is space and time reproducibility of build environments achievable with Nix ?
RQ2: Does it allow rebuilding of past software versions ?
Experimental protocol
Sample 200 revisions of the Nix software repository, picked from 2017 to 2023;
For each sampled revision, perform the evaluation of each package and compare with the historical truth (historical CI results);
For the oldest revision of our samples, perform the build of each package and compare with the historical truth.
Results
RQ1:Reproducibility of build environments
We were able to reproduce the build environment of 99.99% of the packages we tested;
Discrepancies we found were due to the (unfortunate) use of some of Nix’s impure builtins.
Results
RQ1:Reproducibility of build environments
We were able to reproduce the build environment of 99.99% of the packages we tested;
Discrepancies we found were due to the (unfortunate) use of some of Nix’s impure builtins.
RQ2:Rebuilding past software versions
We were able to build successfully 14233 out of the 14242 (99.94%) packages that were built successfully by CI in 2017;
Discrepancies we found were due to leakages of the Nix build sandbox, that we wish to investigate further.
Research questions
How reproducible is software in the functional package management model?
Is Nix evaluation reproducible? Can we reproduce build environments of Nix packages?
Do functional package management enable bitwise build reproducibility?
Build reproducibility
Do functional package management enable bitwise build reproducibility?
“Does Functional Package Management Enable Reproducible Builds at Scale? Yes.”, MSR 2025, J. Malka, S. Zacchiroli, T. Zimmermann.
Motivations
Reproducible builds are good for software supply chain security, however…
Achieving build reproducibility is difficult: there exist a lot of non- determinisms in compilers that have effects on produced artifacts;
Uncertainty among reproducibility experts on the feasibility to achieve build reproducibility at scale (even though good players like Debian already achieve impressive performance) 1
Research questions
RQ1: What is the evolution of bitwise reproducible packages in nixpkgs between 2017 and 2023?
RQ2: What are the unreproducible packages?
RQ3: Why are packages unreproducible?
RQ4: How are unreproducibilities fixed? (Not covered in this talk)
Research methodology
Figure 6: Pipeline summarizing our research methodology.
A few figures
Figure 7: Evolution of the size of the nine most popular software ecosystems in nixpkgs.
709 816 packages built;
14 296 total build hours;
548 390 tracked by name and corresponding to 59 103 unique packages associated to a specific software ecossytem .
RQ1: Evolution of bitwise reproducible packages
Figure 8: Proportion of reproducible, rebuildable and non-rebuildable packages over time.
RQ1: Evolution of bitwise reproducible packages
Figure 9: Absolute numbers of reproducible, rebuildable and non-rebuildable packages over time.
RQ1: Evolution of bitwise reproducible packages
Figure 10: Reproducibility regression around June 2020.
RQ2: What are the unreproducible packages?
Figure 11: Proportion of reproducible packages belonging to the three most popular ecosystems and the base namespace of nixpkgs.
RQ3: Why are packages unreproducible?
Figure 12: Example of a diffoscope.
RQ3: Why are packages unreproducible?
Figure 13: Evolution of the number of packages that are matched by each of our heuristics, over time.
Conclusion
Bitwise reproducibility in nixpkgs as of 2023: 91%;
This justifies investing resources/conducting research on distributed cache solutions relying on build reproducibility.
Thank you for your attention!
My socials:
luj@chaos.social
julien.malka@telecom-paris.fr
RQ1: Evolution of bitwise reproducible packages
Figure 14: Sankey graph of the average flow of packages between two revisions, excluding the revision from June 2020, considered as an outlier.
