Create pipeline for symbol reachability

[ziadhany]

Issues

• cravex2-reachability: Create pipeline for symbol reachability #1756

Changes

Add Pipeline(s) that can retrieve vulnerable / fix symbols, collect local symbols (pur2sym) and match them
Pipeline Graph:

flowchart TD
    %% Main Entry Point
    Start([collect_and_store_symbol_reachability_results]) --> GetResources[Filter candidate_resources]
    GetResources --> GroupPatches[Group fixed_in_patches by vcs_url]
    GroupPatches --> RepoLoop{For each repository}

    %% Repository Processing
    RepoLoop -->|Process Repo| GitContext[GitRepositoryContext]
    GitContext -->|Clone & Enter| PatchLoop{For each patch}
    
    %% Patch Processing
    PatchLoop -->|Next Patch| GitCheckout[Checkout commit_hash]
    GitCheckout --> GenReport[generate_reachability_report]
    
    %% Patch Analysis Stage
    GenReport --> PatchAnalyzer[PatchAnalyzer.collect_patch_symbols]
    PatchAnalyzer --> AnalyzeDiff[Get Diff & Changed Files]
    AnalyzeDiff --> PatchAnalyzeLoop[Analyze vulnerable & fixed texts]
    PatchAnalyzeLoop --> DetectLang[detect_language_with_scancode]
    DetectLang --> ParsePatchAST[LanguageQuery.parse_code_to_ast]
    ParsePatchAST --> ExtractPatchSymbols[SymbolExtractor: Extract Changed Symbols]
    ExtractPatchSymbols --> DiffSymbols[diff_changed_symbols]
    DiffSymbols --> ResourceLoop{For each candidate_resource}

    %% Resource Matching Stage
    ResourceLoop -->|Next Resource| LangCheck{Language matches patch?}
    LangCheck -->|No| ResourceLoop
    LangCheck -->|Yes| ResAnalyzer[ResourceAnalyzer.build_index]
    
    ResAnalyzer --> ParseResAST[Parse Resource to AST]
    ParseResAST --> IndexResSymbols[Extract definitions, imports, calls]
    IndexResSymbols --> Matcher[ResourcePatchMatcher.match]
    
    Matcher --> MatchVuln[Match Vulnerable Symbols]
    Matcher --> MatchFixed[Match Fixed Symbols]
    
    MatchVuln & MatchFixed --> CheckEvidence{Evidence found?}
    CheckEvidence -->|No| ResourceLoop
    
    %% Reporting Stage
    CheckEvidence -->|Yes| Classify[classify_reachability]
    Classify --> StatusReach[REACHABLE / POTENTIALLY / NOT_REACHABLE]
    StatusReach --> UpdateData[Update resource.extra_data]
    UpdateData --> ResourceLoop

    %% Loop Backs and Exit
    ResourceLoop -->|All resources checked| PatchLoop
    PatchLoop -->|All patches checked| RepoLoop
    RepoLoop -->|All repos checked| End([End Process])

    %% Styling
    classDef functionNode fill:#e1f5fe,stroke:#03a9f4,stroke-width:2px;
    classDef logicNode fill:#fff3e0,stroke:#ff9800,stroke-width:2px;
    classDef loopNode fill:#f3e5f5,stroke:#9c27b0,stroke-width:2px;
    
    class Start,GenReport,PatchAnalyzer,ResAnalyzer,Matcher,Classify functionNode;
    class RepoLoop,PatchLoop,ResourceLoop loopNode;
    class LangCheck,CheckEvidence logicNode;

Loading

Checklist

• I have read the contributing guidelines
• I have linked an existing issue above
• I have added unit tests covering the new code
• I have reviewed and understood every line of this PR

[ziadhany]

The pipeline requires the unidiff dependency to parse diffs/patch text

[ziadhany]

This depends on:

• Migrate VulnerableCode integration to API v3 #2148

We have an API support for patching VulnerableCode, see:

• fedcode-next: Support UI and API for patches and diffs vulnerablecode#2152

[ziadhany]

I'm also not sure why Run unit tests on macOS / run-unit-tests (3.13) is failing.

[TG1999]

@ziadhany try to fix the failing test please

[TG1999]

You should only go for reachability if collect_symbols_reachability has called it, not for every VCIO call

[ziadhany]

We're currently relying on the find_vulnerabilities pipeline to retrieve vulnerabilities. One possible solution is to introduce an optional field reachability in the find_vulnerabilities pipeline.

[TG1999]

@ziadhany this mostly looks good, let's have a session soon and do a demo and understand the limitations of the current approach

[tdruez]

• The new code does not follow current codebase conventions: remote all the type hint.
• Make sure all functions have a proper docstrings and are properly covered by a unit test
• Lack of a global test for the new pipeline

[keshav-space]

@ziadhany thanks. We should not rebuild index over and over again for the same resource each time we match against a new patch. I suggest that instead of having one big monolithic step, we break the pipeline into minimum 4 logical steps.

• Step 1: get vcs_url and patch for all vulnerabilities.
• Step 2: for each patch, collect vulnerable symbols.
• Step 3: iterate over resource, parse and collect resource symbols, and match them against vulnerable symbols.
• Step 4: prepare and store consolidated reachability for each vulnerability.

The final reachability report should be keyed by vulnerability. For each vulnerability I should get reachability (Yes, No, or Unknown) along with tool-specific details.
Also please remove type hints as we don't yet use/support type hints in SCIO

[ziadhany]

test package:
‎pkg:pypi/b2sdk‎

• https://github.com/pypa/advisory-database/blob/b96aefbfb2aad245a47883ed929639f9a3080ed4/vulns/b2sdk/PYSEC-2022-33.yaml

pipeline input:

• https://files.pythonhosted.org/packages/51/05/8e3a694095ff7c24d2496413133784aab81a4854db92d528c8e5d546d287/b2sdk-1.14.0.tar.gz

pipeline output:

• reachability-2026-08-23-23-03-11.json