Install Knowledge Base

The kb-download.sh script downloads packages from the SCANOSS SFTP server. It can fetch the full Knowledge Base (KB), a KB update, the test KB, a SQLite KB snapshot, or an HFH KB snapshot. It connects to the server, lists the available versions, and lets you pick a package. Before it downloads, it checks that you have enough free disk space. A KB update package includes a separate import utility, ldb-import.sh. Use it to import the update into your local LDB instance.

KB download prerequisites

lftp is recommended. It gives faster, parallel, resumable downloads. Install it with apt install lftp. If lftp isn’t available, the script can fall back to the standard sftp client. With the -y flag, the script falls back to sftp without asking.
sshpass is required only for the sftp fallback. It passes the password to sftp non-interactively. Install it with apt install sshpass. You don’t need it when you use lftp.
SFTP credentials come from the SCANOSS sales team: host, port, username, and password.

Running the script

Make sure the script has execution permissions:

chmod +x install/kb-download.sh

Run the script without arguments for an interactive session:

./kb-download.sh

Command-line options

You can pass the connection details, paths, and download mode as arguments. The script then skips the prompt for each value you pass:

./kb-download.sh -m update -h sftp.test.xyz -P 12345 -u USER21 -p mypassword

Connection and mode

Flag	Description
`-m`	Download mode: `full` (full KB), `update` (KB update), `test` (test KB), `sqlite` (SQLite KB snapshot), or `hfh` (HFH KB snapshot)
`-h`	SFTP host
`-P`	SFTP port
`-u`	SFTP username
`-p`	SFTP password
`-t`	lftp parallel download threads (default: 25)
`-d`	Force download tool: `lftp` or `sftp`
`-w`	Connection timeout in seconds (default: 15). Applies to the pre-flight connection check and every sftp/lftp connection, so a wrong host or port fails fast instead of hanging.
`-C`	Enable SSH transport compression. Off by default. Useful on slow links. On fast links it usually slows the transfer, because zlib costs more CPU than it saves in bandwidth for dense payloads like hashes and indexes.

Path and version overrides

Flag	Description
`-V`	KB version (full/update/sqlite/hfh mode); default: latest from `LATEST.txt` on the server
`-o`	Destination path. Test mode: test-KB destination. Full mode: `oss` folder destination. (default: `/var/lib/ldb/oss`)
`-r`	Full mode only: destination for non-oss items (default: `/tmp/scanoss_kb_full_<version>`)
`-D`	Update / sqlite / hfh mode: download base directory. Final path is `<D>/<version>` for update/sqlite and `<D>/<version>.zip` for hfh. Defaults: `/tmp/scanoss_kb_update` for update, current directory for sqlite and hfh

Non-interactive flags

Flag	Description
`-y`	Don’t prompt. Use defaults for unspecified values and confirm the download without asking. You must supply the required values (`-m`, `-h`, `-P`, `-u`, `-p`). The script exits with an error if any are missing.
`-f`	Force the download even if free disk space is below the required size. Only has an effect with `-y`. Without `-y`, the script asks you to confirm.
`-?`	Show help

Unless you set -y, the script prompts for any option you didn’t pass on the command line.

Non-interactive examples

# Defaults (latest version, /var/lib/ldb/oss + /tmp/scanoss_kb_full_<version>):
./kb-download.sh -y -m full -h sftp.test.xyz -P 12345 -u USER21 -p mypassword

# Pin a version, custom paths, force on low disk space:
./kb-download.sh -y -f -m full -V 26.03 \
    -o /data/oss -r /data/kb-rest \
    -h sftp.test.xyz -P 12345 -u USER21 -p mypassword

# Update mode with a custom download base directory:
./kb-download.sh -y -m update -D /opt/scanoss/updates \
    -h sftp.test.xyz -P 12345 -u USER21 -p mypassword

# SQLite KB into the current directory (final path: ./<version>):
./kb-download.sh -y -m sqlite \
    -h sftp.test.xyz -P 12345 -u USER21 -p mypassword

# SQLite KB into a specific base directory:
./kb-download.sh -y -m sqlite -D /opt/scanoss/sqlite \
    -h sftp.test.xyz -P 12345 -u USER21 -p mypassword

# HFH KB (single .zip; lands at ./<version>.zip):
./kb-download.sh -y -m hfh -h sftp.test.xyz -P 12345 -u USER21 -p mypassword

# Test KB:
./kb-download.sh -y -m test -h sftp.test.xyz -P 12345 -u USER21 -p mypassword

How it works

The script prompts for the SFTP host, port, username, and password, or reads them from arguments.
You choose what to download: the full KB, a KB update, the test KB, a SQLite KB snapshot, or an HFH snapshot.
For full, update, sqlite, and hfh modes, the script connects to the SFTP server, lists every available version of the chosen type, and marks the latest. The test KB has no versions. The server holds one current test KB.
For full, update, sqlite, and hfh modes, you choose a version. The default is the latest.
The script fetches the metadata and checks for enough free disk space. If there isn’t enough, it warns you and asks you to confirm before it continues. For HFH, it reads the size from ls -l on the remote .zip instead of metadata.json.
The script downloads the package:
- The full KB goes to two destinations. The oss folder, which holds the main LDB data, goes to its own destination (default: /var/lib/ldb/oss). The remaining files and folders go to a separate directory (default: /tmp/scanoss_kb_full_<version>). You can change both destinations at the prompt.
- An update downloads as one folder to a single directory (default: /tmp/scanoss_kb_update/<version>).
- The test KB’s oss folder goes to a single destination (default: /var/lib/ldb/oss). This fills your LDB with the test data for verification.
- A SQLite KB downloads as a version folder to <base>/<version> (default base: the current working directory). The folder holds one or more .sqlite files and a metadata.json. The script downloads every file you have access to in that folder. Different users may see different sets of .sqlite files, depending on their SFTP permissions.
- An HFH KB downloads as a single <version>.zip archive to <base>/<version>.zip (default base: the current working directory). With lftp, the script fetches the file in parallel chunks (pget -n <threads>). With sftp, it uses a single-stream get with the live progress poller. The script doesn’t extract the archive. Leave that to your downstream tooling.
For updates only, run the ldb-import.sh script in the downloaded folder after the download finishes.

Example sessions

Update

SCANOSS Knowledge Base Download
================================
Using lftp for downloads (parallel, resumable).

What do you want to download?
  1) Full KB
  2) KB update
  3) Test KB
  4) SQLite KB
  5) HFH KB

Select [1-5]: 2

SFTP host: sftp.test.xyz
SFTP port: 12345
SFTP username: USER21
SFTP password:

Fetching available update versions...

Available update versions:
-------------------
  1) 26.01
  2) 26.02
  3) 26.03  (latest)
-------------------

Select a update version to download [1-3] (default: 3):

Selected update: 26.03

Download directory [/tmp/scanoss_kb_update]:

Fetching update metadata...
Update size: 5.2G
Free space:  120G (on /dev/sda1)
Disk space OK.

lftp parallel threads [25]:

Download update 26.03 to /tmp/scanoss_kb_update? [Y/n]
Downloading with lftp (25 parallel threads, resumable)...
Update downloaded to /tmp/scanoss_kb_update/26.03

Finished downloading update.
Please run the ldb-import.sh script provided inside the update folder to import into the KB:
  /tmp/scanoss_kb_update/26.03/ldb-import.sh

Full KB

./kb-download.sh -m full -h sftp.test.xyz -P 12345 -u USER21

SCANOSS Knowledge Base Download
================================
Using lftp for downloads (parallel, resumable).

SFTP password:

Fetching available full KB versions...

Available full KB versions:
-------------------
  1) 26.01
  2) 26.02
  3) 26.03  (latest)
-------------------

Select a full KB version to download [1-3] (default: 3):

Selected full KB: 26.03

Destination for 'oss' folder [/var/lib/ldb/oss]:
Destination for remaining files/folders [/tmp/scanoss_kb_full_26.03]:

Fetching full KB metadata...
Full KB size: 210G
Free space:  500G (on /dev/sda1)
Disk space OK.

lftp parallel threads [25]:

Download full KB 26.03 to /var/lib/ldb/oss + /tmp/scanoss_kb_full_26.03? [Y/n]
Downloading kb/full/26.03/oss with lftp (25 parallel threads, resumable)...
Downloading kb/full/26.03/md5sum with lftp (25 parallel threads, resumable)...
Downloading kb/full/26.03/26.03_files.md5sum with lftp (25 parallel threads, resumable)...

Full KB downloaded:
  oss folder:  /var/lib/ldb/oss
  other files: /tmp/scanoss_kb_full_26.03

Finished downloading full KB.

Test KB

./kb-download.sh -m test -h sftp.test.xyz -P 12345 -u USER21

SCANOSS Knowledge Base Download
================================
Using lftp for downloads (parallel, resumable).

SFTP password:

Destination for test KB [/var/lib/ldb/oss]:

Fetching test KB metadata...
Test KB size: 180M
Free space:   500G (on /dev/sda1)
Disk space OK.

lftp parallel threads [25]:

Download test KB to /var/lib/ldb/oss? [Y/n]
Downloading kb/test/oss with lftp (25 parallel threads, resumable)...
Test KB downloaded to /var/lib/ldb/oss

Finished downloading test KB.

SQLite KB

./kb-download.sh -m sqlite -h sftp.test.xyz -P 12345 -u USER21

SCANOSS Knowledge Base Download
================================
Using lftp for downloads (parallel, resumable).

SFTP password:

Fetching available sqlite KB versions...

Available sqlite KB versions:
-------------------
  1) 26.04  (latest)
-------------------

Select a sqlite KB version to download [1-1] (default: 1):

Selected sqlite KB: 26.04

Download directory [.]:

Fetching sqlite KB metadata...
Sqlite KB size: 1.2G
Free space:  500G (on /dev/sda1)
Disk space OK.

lftp parallel threads [25]:

Download sqlite KB 26.04 to ./26.04? [Y/n]
Downloading kb/sqlite/26.04 with lftp (25 parallel threads, resumable)...

Sqlite KB downloaded to ./26.04

Finished downloading sqlite KB.

The downloaded folder holds every .sqlite file the SFTP user has access to, and metadata.json. Different users may see different sets of files, depending on their permissions.

HFH KB

./kb-download.sh -m hfh -h sftp.test.xyz -P 12345 -u USER21

SCANOSS Knowledge Base Download
================================
Using lftp for downloads (parallel, resumable).

SFTP password:

Fetching available hfh KB versions...

Available hfh KB versions:
-------------------
  1) 26.06  (latest)
-------------------

Select a hfh KB version to download [1-1] (default: 1):

Selected hfh KB: 26.06

Download directory [.]:

Fetching hfh KB file size...
Hfh KB size: 35G
Free space:  1.7T (on /dev/sda1)
Disk space OK.

lftp parallel threads [25]:

Download hfh KB 26.06 to ./26.06.zip? [Y/n]
Downloading kb/hfh/26.06.zip with lftp (25 parallel chunks, resumable)...

Hfh KB downloaded to ./26.06.zip

Finished downloading hfh KB.

HFH ships as a single <version>.zip archive with no metadata.json. The disk-space check reads the file size from the remote ls -l output. The script doesn’t extract the archive. Leave that to your downstream tooling.

Verifying a KB update

After you import an update, check it by scanning the test WFP files in the update directory:

scanoss-py scan /tmp/scanoss_kb_update/26.03/26.03_test_file.wfp
scanoss-py scan /tmp/scanoss_kb_update/26.03/26.03_test_snippet.wfp

If both scans return the expected matches, the import worked.

An interrupted import can corrupt the existing LDB.

Next steps

When the Knowledge Base is installed, continue to Verify installation.

Get started

Recipes

CLI reference

API reference

Self-hosting & knowledge base

KB download prerequisites

Running the script

Command-line options

Non-interactive examples

How it works

Example sessions

Update

Full KB

Test KB

SQLite KB

HFH KB

Verifying a KB update

Next steps

​KB download prerequisites

​Running the script

​Command-line options

​Non-interactive examples

​How it works

​Example sessions

​Update

​Full KB

​Test KB

​SQLite KB

​HFH KB

​Verifying a KB update

​Next steps

KB download prerequisites

Running the script

Command-line options

Non-interactive examples

How it works

Example sessions

Update

Full KB

Test KB

SQLite KB

HFH KB

Verifying a KB update

Next steps