Overview
The SCANOSS Engine is a command-line tool. It compares files and directories against the SCANOSS Knowledgebase (KB) to find open-source component matches. It prints results toSTDOUT as JSON, with licence, copyright, and component identification data.
The engine runs on the server. It receives WFP (Winnowing FingerPrint) files from
clients through the SCANOSS API’s ../scan-direct path. Each WFP file uses this
format:
TARGET can be a single file, a .wfp fingerprint file, or a directory.
Scanning procedure overview
The engine gives each scanned file one of three match types:- None. The engine found nothing in the KB that matches the scanned file.
- Snippet. Part of the scanned file matches part of a known open-source file.
- Full. The whole scanned file matches an existing file in the KB.
- Determine the type of match (full file or snippet).
- For snippet matches only: identify the most meaningful open-source file (MD5) among multiple candidates containing the matched snippet.
- Link the matched file with component information, such as URL, purl, and release date. There may be more than one candidate. By default, the engine selects the best match.
- Query component-level information, such as licence and vulnerabilities.
- Produce the JSON report.
File matching logic
The engine tries to match each scanned file against the KB in this order:- URL match. If the file exactly matches a known package archive at a registered
URL, the identification type (
id) is"url". - File match. The engine checks the “file table” for the file’s MD5 (
FILE_MD5). If the KB has it, the identification type (id) is"file". If not, snippet analysis begins. - Snippet match. If neither of the above applies, the engine compares snippets
using the CRC32 fingerprints from the WFP. The identification type (
id) is"snippet". - Binary match. The engine identifies binary files through binary fingerprinting.
The identification type (
id) is"binary". - No match. If none of the above apply, the identification type (
id) is"none".
Snippet matching in detail
When the engine finds no full file match, it runs snippet analysis. Snippet analysis takes several steps and trades some match accuracy for processing speed.How snippet matching works
Each WFP fragment is a CRC32 value that fingerprints a piece of code. In the KB, each fragment links to a series of file MD5s. When two WFP fragments share an MD5, that MD5 gets a “hit”. The more hits an MD5 has, the larger the snippet the scanned file shares with the matched open-source file. Each CRC32 can link to many MD5s, so the number of comparisons can grow large. To keep processing time down, the engine handles CRC32s linked to fewer MD5s first. These are rarer code fragments, so they say more about where the code came from.Snippet scanning procedure
-
Pre-processing map. The engine queries the KB for the MD5s linked to each CRC32
and builds a map in this format:
Each row records a CRC32, its line number, the number of linked MD5s (
LIST_SIZE), and the MD5s themselves. - Sort by list size. The engine sorts the map by list size, smallest first, so it handles the rarer fingerprints first.
- Build the matchmap. The engine adds each MD5 to a matchmap. If the MD5 is already in the map, it gets a hit. The matchmap tracks line ranges and holds at most 10,000 MD5s. Processing stops when the engine has handled every CRC32 or the matchmap is full.
- Select the largest snippets. The engine selects the MD5s with the most hits, which are the largest shared snippets. It may select one or more. The next steps run for each selected MD5.
- Resolve to components. Each selected MD5 may appear in several repositories, such as clones, forks, or dependents. The engine resolves these to a list of candidate components. See Component ranking logic below.
- Refine line ranges. The engine removes small or spurious line ranges as false positives and merges overlapping ranges.
Line range generation
Snippet matches report approximate line ranges, because of how the winnowing algorithm works. Each matched CRC32 carries two positions:- The line number is the line in the scanned file where the fingerprint starts.
- The OSS line number is the line in the matching open-source file where the fingerprint starts.
- If a line range already exists within a configurable tolerance gap, the engine extends that range to include the new fingerprint.
- If no nearby range exists, the engine creates a new range.
OSS from value is the starting line in the matching open-source file.
Component ranking logic
When a file appears in several components or versions in the KB, the engine picks the best match with these rules:- Context comes first. The engine can receive context, such as an SBOM (Software Bill
of Materials) passed with
--sbom. The engine always selects any component that matches this context and puts it at the top of the candidate list, whatever its release date. - Without a context match, the engine picks the oldest component in the KB, the first one released.
- If two components share a release date, the engine uses the project-level release date to break the tie.
- The scanning client can pass a component hint, the name of the most recently detected component. The engine favours files that belong to a component matching this hint.
JSON report
After it selects the best match and its component, the engine queries more component-level metadata, such as licences and known vulnerabilities. It writes everything as a JSON result toSTDOUT.
By default, the output includes only the best match. To see the full candidate list,
the top M components for each of the N matched files, pass the -F256 flag. By
default, M is 3. The engine sets N at scan time.