Install Knowledge Base

The kb-download.sh script provides a unified interface for retrieving the full Knowledge Base (KB), a KB update, the test KB, a SQLite KB snapshot or an HFH KB snapshot directly from the SCANOSS SFTP server. It connects to the server, displays the available versions and allows you to select the desired package for download. Before proceeding, it verifies that sufficient disk space is available. For KB updates, the downloaded package includes a separate import utility (ldb-import.sh) which is used to import the update into your local LDB instance.

KB Download Prerequisites

lftp (recommended): Enables faster, parallel, and resumable downloads. You can install it using apt install lftp. If lftp is not available, the script can fall back to the standard sftp client. When running with the -y flag, the fallback to sftp is handeld automatically.
sshpass (required only for sftp fallback): Needed to provide the password non-interactively when using sftp. Install it with apt install sshpass. This dependency is not required when using lftp.
SFTP credentials: Supplied by the SCANOSS sales team, including host, port, username and password.

Running the Script

Make sure the script has execution permissions:

chmod +x install/kb-download.sh

Run the script without arguments for a fully interactive session:

./kb-download.sh

Command-line Options

All connection details, paths, and the download mode can be passed as arguments to skip the matching interactive prompt:

./kb-download.sh -m update -h sftp.test.xyz -P 12345 -u USER21 -p mypassword

Connection / mode

Flag	Description
`-m`	Download mode: `full` (full KB), `update` (KB update), `test` (test KB), `sqlite` (SQLite KB snapshot), or `hfh` (HFH KB snapshot)
`-h`	SFTP host
`-P`	SFTP port
`-u`	SFTP username
`-p`	SFTP password
`-t`	lftp parallel download threads (default: 25)
`-d`	Force download tool: `lftp` or `sftp`
`-w`	Connection timeout in seconds (default: 15). Applied to the pre-flight connection check and every sftp/lftp connection, so a wrong host/port fails fast instead of hanging.
`-C`	Enable SSH transport compression. Off by default. Useful on slow links; on fast links it typically slows the transfer down (zlib CPU exceeds the bandwidth saving for dense payloads like hashes and indexes).

Path / version overrides

Flag	Description
`-V`	KB version (full/update/sqlite/hfh mode); default: latest from `LATEST.txt` on the server
`-o`	Destination path. Test mode: test-KB destination. Full mode: `oss` folder destination. (default: `/var/lib/ldb/oss`)
`-r`	Full mode only: destination for non-oss items (default: `/tmp/scanoss_kb_full_<version>`)
`-D`	Update / sqlite / hfh mode: download base directory. Final path is `<D>/<version>` for update/sqlite and `<D>/<version>.zip` for hfh. Defaults: `/tmp/scanoss_kb_update` for update, current directory for sqlite and hfh

Non-interactive flags

Flag	Description
`-y`	Don’t prompt; use defaults for unspecified values and auto-confirm the download. Required values (`-m`, `-h`, `-P`, `-u`, `-p`) must be supplied — the script exits with an error if any are missing.
`-f`	Force download even if free disk space is below the required size. Only meaningful with `-y`; without `-y` you are prompted to confirm.
`-?`	Show help

Any options not provided on the command line will be prompted interactively, unless -y is set.

Non-interactive Examples

# Defaults (latest version, /var/lib/ldb/oss + /tmp/scanoss_kb_full_<version>):
./kb-download.sh -y -m full -h sftp.test.xyz -P 12345 -u USER21 -p mypassword

# Pin a version, custom paths, force on low disk space:
./kb-download.sh -y -f -m full -V 26.03 \
    -o /data/oss -r /data/kb-rest \
    -h sftp.test.xyz -P 12345 -u USER21 -p mypassword

# Update mode with a custom download base directory:
./kb-download.sh -y -m update -D /opt/scanoss/updates \
    -h sftp.test.xyz -P 12345 -u USER21 -p mypassword

# SQLite KB into the current directory (final path: ./<version>):
./kb-download.sh -y -m sqlite \
    -h sftp.test.xyz -P 12345 -u USER21 -p mypassword

# SQLite KB into a specific base directory:
./kb-download.sh -y -m sqlite -D /opt/scanoss/sqlite \
    -h sftp.test.xyz -P 12345 -u USER21 -p mypassword

# HFH KB (single .zip; lands at ./<version>.zip):
./kb-download.sh -y -m hfh -h sftp.test.xyz -P 12345 -u USER21 -p mypassword

# Test KB:
./kb-download.sh -y -m test -h sftp.test.xyz -P 12345 -u USER21 -p mypassword

How It Works

Connection: the script prompts for (or reads from arguments) the SFTP host, port, username, and password
Mode selection: you choose between downloading the full KB, a KB update, the test KB, or a SQLite KB snapshot, or an HFH snapshot
Version discovery (full/update/sqlite/hfh only): it connects to the SFTP server and lists all available versions of the chosen type, marking which one is the latest. Test KB has no versions — there is only one current test KB on the server.
Selection (full/update/sqlite/hfh only): you choose which version to download (defaults to the latest)
Disk space check: the script fetches the metadata and verifies there is enough free disk space before downloading. If not enough is available, you are warned and asked to confirm before continuing. (For HFH the size comes from ls -l on the remote .zip rather than metadata.json.)
Download:
- For full KB: the download is split between two destinations. The oss folder (the main LDB data) goes to its own destination (default: /var/lib/ldb/oss), and all remaining files and folders go to a separate directory (default: /tmp/scanoss_kb_full_<version>). Both destinations can be changed when prompted.
- For updates: the whole update folder is downloaded to a single directory (default: /tmp/scanoss_kb_update/<version>).
- For test KB: the test KB’s oss folder is downloaded to a single destination (default: /var/lib/ldb/oss). This will populate your LDB with the test data for verification.
- For SQLite KB: the whole version folder is downloaded to <base>/<version> (default base: the current working directory). The folder contains one or more .sqlite files plus a metadata.json. The script downloads every file you have access to in that folder — different users may see different sets of .sqlite files depending on their SFTP permissions.
- For HFH KB: a single <version>.zip archive is downloaded to <base>/<version>.zip (default base: the current working directory). With lftp the file is fetched in parallel chunks (pget -n <threads>); with sftp it’s a single-stream get with the live progress poller. The script does not extract the archive — leave that to your downstream tooling.
Import (updates only): after downloading an update, run the ldb-import.sh script included in the downloaded folder

Example Sessions

Update

SCANOSS Knowledge Base Download
================================
Using lftp for downloads (parallel, resumable).

What do you want to download?
  1) Full KB
  2) KB update
  3) Test KB
  4) SQLite KB
  5) HFH KB

Select [1-5]: 2

SFTP host: sftp.test.xyz
SFTP port: 12345
SFTP username: USER21
SFTP password:

Fetching available update versions...

Available update versions:
-------------------
  1) 26.01
  2) 26.02
  3) 26.03  (latest)
-------------------

Select a update version to download [1-3] (default: 3):

Selected update: 26.03

Download directory [/tmp/scanoss_kb_update]:

Fetching update metadata...
Update size: 5.2G
Free space:  120G (on /dev/sda1)
Disk space OK.

lftp parallel threads [25]:

Download update 26.03 to /tmp/scanoss_kb_update? [Y/n]
Downloading with lftp (25 parallel threads, resumable)...
Update downloaded to /tmp/scanoss_kb_update/26.03

Finished downloading update.
Please run the ldb-import.sh script provided inside the update folder to import into the KB:
  /tmp/scanoss_kb_update/26.03/ldb-import.sh

Full KB

./kb-download.sh -m full -h sftp.test.xyz -P 12345 -u USER21

SCANOSS Knowledge Base Download
================================
Using lftp for downloads (parallel, resumable).

SFTP password:

Fetching available full KB versions...

Available full KB versions:
-------------------
  1) 26.01
  2) 26.02
  3) 26.03  (latest)
-------------------

Select a full KB version to download [1-3] (default: 3):

Selected full KB: 26.03

Destination for 'oss' folder [/var/lib/ldb/oss]:
Destination for remaining files/folders [/tmp/scanoss_kb_full_26.03]:

Fetching full KB metadata...
Full KB size: 210G
Free space:  500G (on /dev/sda1)
Disk space OK.

lftp parallel threads [25]:

Download full KB 26.03 to /var/lib/ldb/oss + /tmp/scanoss_kb_full_26.03? [Y/n]
Downloading kb/full/26.03/oss with lftp (25 parallel threads, resumable)...
Downloading kb/full/26.03/md5sum with lftp (25 parallel threads, resumable)...
Downloading kb/full/26.03/26.03_files.md5sum with lftp (25 parallel threads, resumable)...

Full KB downloaded:
  oss folder:  /var/lib/ldb/oss
  other files: /tmp/scanoss_kb_full_26.03

Finished downloading full KB.

Test KB

./kb-download.sh -m test -h sftp.test.xyz -P 12345 -u USER21

SCANOSS Knowledge Base Download
================================
Using lftp for downloads (parallel, resumable).

SFTP password:

Destination for test KB [/var/lib/ldb/oss]:

Fetching test KB metadata...
Test KB size: 180M
Free space:   500G (on /dev/sda1)
Disk space OK.

lftp parallel threads [25]:

Download test KB to /var/lib/ldb/oss? [Y/n]
Downloading kb/test/oss with lftp (25 parallel threads, resumable)...
Test KB downloaded to /var/lib/ldb/oss

Finished downloading test KB.

SQLite KB

./kb-download.sh -m sqlite -h sftp.test.xyz -P 12345 -u USER21

SCANOSS Knowledge Base Download
================================
Using lftp for downloads (parallel, resumable).

SFTP password:

Fetching available sqlite KB versions...

Available sqlite KB versions:
-------------------
  1) 26.04  (latest)
-------------------

Select a sqlite KB version to download [1-1] (default: 1):

Selected sqlite KB: 26.04

Download directory [.]:

Fetching sqlite KB metadata...
Sqlite KB size: 1.2G
Free space:  500G (on /dev/sda1)
Disk space OK.

lftp parallel threads [25]:

Download sqlite KB 26.04 to ./26.04? [Y/n]
Downloading kb/sqlite/26.04 with lftp (25 parallel threads, resumable)...

Sqlite KB downloaded to ./26.04

Finished downloading sqlite KB.

The downloaded folder contains all .sqlite files the SFTP user has access to (plus metadata.json). Different users may see different sets of files depending on their permissions.

HFH KB

./kb-download.sh -m hfh -h sftp.test.xyz -P 12345 -u USER21

SCANOSS Knowledge Base Download
================================
Using lftp for downloads (parallel, resumable).

SFTP password:

Fetching available hfh KB versions...

Available hfh KB versions:
-------------------
  1) 26.06  (latest)
-------------------

Select a hfh KB version to download [1-1] (default: 1):

Selected hfh KB: 26.06

Download directory [.]:

Fetching hfh KB file size...
Hfh KB size: 35G
Free space:  1.7T (on /dev/sda1)
Disk space OK.

lftp parallel threads [25]:

Download hfh KB 26.06 to ./26.06.zip? [Y/n]
Downloading kb/hfh/26.06.zip with lftp (25 parallel chunks, resumable)...

Hfh KB downloaded to ./26.06.zip

Finished downloading hfh KB.

HFH ships as a single <version>.zip archive (no metadata.json); the disk-space check reads the file size from the remote ls -l output. The script does not extract the archive — that’s left to downstream tooling.

Verifying a KB Update

After importing an update, verify the installation by scanning the test WFP files included in the update directory:

scanoss-py scan /tmp/scanoss_kb_update/26.03/26.03_test_file.wfp
scanoss-py scan /tmp/scanoss_kb_update/26.03/26.03_test_snippet.wfp

If both scans return appropriate matches, the update was imported successfully.

If the import is interrupted, it may cause corruption in the existing LDB.

Next Steps

Once the Knowledge Base installation is complete, proceed to Verify Installation.

Documentation

Links

KB Download Prerequisites

Running the Script

Command-line Options

Non-interactive Examples

How It Works

Example Sessions

Update

Full KB

Test KB

SQLite KB

HFH KB

Verifying a KB Update

Next Steps

​KB Download Prerequisites

​Running the Script

​Command-line Options

​Non-interactive Examples

​How It Works

​Example Sessions

​Update

​Full KB

​Test KB

​SQLite KB

​HFH KB

​Verifying a KB Update

​Next Steps

KB Download Prerequisites

Running the Script

Command-line Options

Non-interactive Examples

How It Works

Example Sessions

Update

Full KB

Test KB

SQLite KB

HFH KB

Verifying a KB Update

Next Steps