Skip to content

Pipeline for Nix - #2209

Open
chinyeungli wants to merge 73 commits into
mainfrom
1938_pipeline_for_nix
Open

Pipeline for Nix#2209
chinyeungli wants to merge 73 commits into
mainfrom
1938_pipeline_for_nix

Conversation

@chinyeungli

Copy link
Copy Markdown
Contributor

Issues

Changes

This PR creates a dedicated pipeline for Nix packages.

Requirements

  • This pipeline only works on packages that found in the official nixpkgs.
  • It takes a single Nix PURL as an input (i.e. pkg:nix/nixpkgs/...)
  • A version or a 'commit' qualifier is required in the input PURL as this is needed to fetch the correct source and binary for deployment to development mapping scan.
  • It also requires the input PURL to have a "system" qualifier to resolve system-specific binaries.
  • Docker is required. The code use docker to build sources by applying patches/configs that's defined in the upstream .nix files.
  • This pipeline will try to fetch the "debug" output if available (Debug dwarf symbols are needed for the deployment to development mapping scan). If a "debug" build cannot be found, it'll fetch the "out" output (if no "output" qualifier is defined).

Summary

  • The code will get metadata from https://search.devbox.sh and find and fetch the binary package from https://cache.nixos.org/
  • If a Nix package cannot be found in https://cache.nixos.org/ and a commit hash is provided, the code will try to fetch and build the source from https://github.com/NixOS/nixpkgs and patch the sources using the exact configuration defined in the upstream .nix files.
  • The pipeline will check if there is any license mismatch between the codebase and the declared license from the detected packages

Notes for deployment to development mapping

As Nix is not tied to a single programming language ecosystem, this pipeline executes all currently supported D2D steps across multiple languages.

Handling Missing Debug Symbols:

For pkg:nix/nixpkgs/SDL_mixer@1.2.12?system=x86_64-linux , as seen in https://search.devbox.sh/v2/pkg?name=SDL_mixer , there is no debug build found under the "outputs" section. Therefore, even the ELF binary is not stripped:
Screenshot 2026-08-18 140703

there is no dwarf symbols collected and no development to deployment mapping can be performed

Screenshot 2026-08-18 140719

Template File Mapping (.in files):

For pkg:nix/nixpkgs/openssl@3.6.0?out=debug&system=x86_64-linux, following are some of the dwarf symbols collected:

include/openssl/asn1.h
include/openssl/asn1t.h

However, these exact header files do not exist in the source codebase; they are generated from GNU Autoconf template files. To complete the mapping, the code has been updated to map to the corresponding .in template files:
Screenshot 2026-08-18 135225

Source-Only Scans:
For pkg:nix/nixpkgs/haskellPackages.aasam@0.2.0.0?system=x86_64-linux&commit=a3ae4cdd64f675cf2580affecdee01c401b43638 , this package does not exist on https://search.devbox.sh/, so there is no binary can be fetched, but the code can still fetch and build the sources based on the provided commit hash and run the basic scan:
Screenshot 2026-08-18 141322

Notes

Because the pipeline relies on a Linux-based Docker container (nixos/nix) to build sources, providing a PURL for a non-Linux system (e.g., darwin) triggers a system barrier warning.
When this happens, the pipeline safely falls back to evaluating the source using the container's native Linux environment. The extracted source tree will contain Linux-specific patches rather than macOS patches. The impact on the d2d mapping is expected to be minimal, though a small number of files may remain unmapped due to missing OS-specific structural patches.

Checklist

  • I have read the contributing guidelines
  • I have linked an existing issue above
  • I have added unit tests covering the new code
  • I have reviewed and understood every line of this PR

chinyeungli and others added 30 commits March 13, 2026 13:19
 - Get the input and extract content into the from/ codebase.
 - Build the source and place the built files into the to/ codebase.
 - Run scans.
 - Identify sources in from/ that are used in the build.

Signed-off-by: Chin Yeung Li <tli@nexb.com>
…ses detected in the codebase #1767

 * Introduce new "LICENSE_ISSUE" tag
 * License deduplication/simplification is not working well; work in progress

Signed-off-by: Chin Yeung Li <tli@nexb.com>
This commit is for testing purpose and is definitely not ready.

Signed-off-by: Chin Yeung Li <tli@nexb.com>
Signed-off-by: tdruez <tdruez@aboutcode.org>
Signed-off-by: tdruez <tdruez@aboutcode.org>
Signed-off-by: Chin Yeung Li <tli@nexb.com>
Signed-off-by: Chin Yeung Li <tli@nexb.com>
 * Use Docker for building instead of Cargo
 * Accept only one PURL as input
 * Use the .rlib found in .d files for D2D mapping
 * Remove unnecessary code

Signed-off-by: Chin Yeung Li <tli@nexb.com>
 * Add comparison logic
 * Add tests
 * Update extra_data fields
 * Better code organization
 * etc..

Signed-off-by: Chin Yeung Li <tli@nexb.com>
Signed-off-by: tdruez <tdruez@aboutcode.org>
Signed-off-by: tdruez <tdruez@aboutcode.org>
Signed-off-by: Rishabh Rohil <rishabhrohil024@gmail.com>
Signed-off-by: Rishabh Rohil <rishabhrohil024@gmail.com>
Signed-off-by: tdruez <tdruez@aboutcode.org>
Signed-off-by: tdruez <tdruez@aboutcode.org>
Signed-off-by: tdruez <tdruez@aboutcode.org>
Signed-off-by: OmAnand857 <allansmith2561@gmail.com>
Signed-off-by: tdruez <tdruez@aboutcode.org>
Signed-off-by: Aayush Kumar <code@aayushk.dev>
Signed-off-by: tdruez <tdruez@aboutcode.org>
Signed-off-by: tdruez <tdruez@aboutcode.org>
tdruez and others added 26 commits August 5, 2026 18:48
Signed-off-by: tdruez <tdruez@aboutcode.org>
Signed-off-by: tdruez <tdruez@aboutcode.org>
Signed-off-by: tdruez <tdruez@aboutcode.org>
Signed-off-by: tdruez <tdruez@aboutcode.org>
Signed-off-by: tdruez <tdruez@aboutcode.org>
Signed-off-by: tdruez <tdruez@aboutcode.org>
Signed-off-by: tdruez <tdruez@aboutcode.org>
Signed-off-by: tdruez <tdruez@aboutcode.org>
Signed-off-by: tdruez <tdruez@aboutcode.org>
Signed-off-by: tdruez <tdruez@aboutcode.org>
Signed-off-by: Mrityunjay Raj <mr.raj.earth@gmail.com>
Signed-off-by: Prajakta Kamble <prajuu2812@gmail.com>
Signed-off-by: dikshaa2909 <dikshadeware@gmail.com>
… docstring (#2184)

Signed-off-by: Prajakta Kamble <prajuu2812@gmail.com>
Signed-off-by: tdruez <tdruez@aboutcode.org>
Signed-off-by: tdruez <tdruez@aboutcode.org>
Signed-off-by: Chin Yeung Li <tli@nexb.com>
 - Merge branch '1767_dedicalted_pipeline_for_rust' into 1938_pipeline_for_nix without merging rsut.py and scan_rust_package.py
 - Update d2d matching to include <name>.h/.c to <name>.h.in/.c.in

Signed-off-by: Chin Yeung Li <tli@nexb.com>
…#1938

 - Remove the "rust" code/refernece that's delivered from 1767

Signed-off-by: Chin Yeung Li <tli@nexb.com>
Signed-off-by: Chin Yeung Li <tli@nexb.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

nixpkgs-clarity: Design and implement Scancode pipeline for single nixpkg

9 participants