KB Update Guide

Download the KB with `kb-download.sh`

kb-download.sh is the supported way to download the KB and its updates. It connects to the SCANOSS SFTP server, lists the available versions, lets you pick one, checks you have enough free disk space, and downloads it. It can fetch the full KB, a KB update, the test KB, a SQLite KB snapshot, or an HFH KB snapshot.

Prerequisites

lftp (recommended): provides parallel, resumable downloads. Install with apt install lftp. If lftp is not installed, the script offers to fall back to the standard sftp client.
sshpass (only needed for the sftp fallback): install with apt install sshpass. Not required when using lftp.
SFTP credentials: host, port, username, and password, provided by the SCANOSS team by email.

Get the script

The script lives in the SCANOSS on-premise repository. Clone it (or pull the latest changes if you already have it) and make the scripts executable:

git clone https://github.com/scanoss/on-premise.git
cd on-premise/install/
chmod +x *.sh

The script is also documented in the on-premise install README.

Run it interactively

Run the script with no arguments for a fully guided session. It prompts for your connection details, the download mode, and the version:

./kb-download.sh

You will be asked what to download, then shown the available versions with the latest marked, for example:

SCANOSS Knowledge Base Download
================================
Using lftp for downloads (parallel, resumable).

What do you want to download?
  1) Full KB
  2) KB update
  3) Test KB
  4) SQLite KB
  5) HFH KB

Select [1-5]: 2

SFTP host: sftp.test.xyz
SFTP port: 12345
SFTP username: USER21
SFTP password:

Fetching available update versions...

Available update versions:
-------------------
  1) 26.01
  2) 26.02
  3) 26.03  (latest)
-------------------

Select a update version to download [1-3] (default: 3):

Selected update: 26.03

Download directory [/tmp/scanoss_kb_update]:

Fetching update metadata...
Update size: 5.2G
Free space:  120G (on /dev/sda1)
Disk space OK.

lftp parallel threads [25]:

Download update 26.03 to /tmp/scanoss_kb_update? [Y/n]
Downloading with lftp (25 parallel threads, resumable)...
Update downloaded to /tmp/scanoss_kb_update/26.03

Finished downloading update.
Please run the ldb-import.sh script provided inside the update folder to import into the KB:
  /tmp/scanoss_kb_update/26.03/ldb-import.sh

For long-running downloads (such as the full KB), we recommend running the script inside a tmux session so the transfer survives a dropped connection. Start one with tmux new-session -s scanoss, run the script, then detach with Ctrl+B d and reattach later with tmux attach -t scanoss.

Run it non-interactively

Any prompt can be skipped by passing the matching flag. Add -y to accept defaults and run without prompts (the connection flags -m, -h, -P, -u, -p are then required):

# Defaults (latest version, /var/lib/ldb/oss + /tmp/scanoss_kb_full_<version>):
./kb-download.sh -y -m full -h sftp.test.xyz -P 12345 -u USER21 -p mypassword

# Pin a version, custom paths, force on low disk space:
./kb-download.sh -y -f -m full -V 26.03 \
    -o /data/oss -r /data/kb-rest \
    -h sftp.test.xyz -P 12345 -u USER21 -p mypassword

# Update mode with a custom download base directory:
./kb-download.sh -y -m update -D /opt/scanoss/updates \
    -h sftp.test.xyz -P 12345 -u USER21 -p mypassword

# SQLite KB into the current directory (final path: ./<version>):
./kb-download.sh -y -m sqlite \
    -h sftp.test.xyz -P 12345 -u USER21 -p mypassword

# SQLite KB into a specific base directory:
./kb-download.sh -y -m sqlite -D /opt/scanoss/sqlite \
    -h sftp.test.xyz -P 12345 -u USER21 -p mypassword

# HFH KB (single .zip; lands at ./<version>.zip):
./kb-download.sh -y -m hfh \
    -h sftp.test.xyz -P 12345 -u USER21 -p mypassword

# Test KB:
./kb-download.sh -y -m test -h sftp.test.xyz -P 12345 -u USER21 -p mypassword

Commonly used options:

Flag	Description
`-m`	Download mode: `full`, `update`, `test`, `sqlite`, or `hfh`
`-h`	SFTP host
`-P`	SFTP port
`-u`	SFTP username
`-p`	SFTP password
`-V`	KB version (default: the latest reported by the server)
`-D`	Download directory for `update` / `sqlite` / `hfh` modes
`-o`	Destination for the `oss` folder (full/test modes; default `/var/lib/ldb/oss`)
`-y`	Non-interactive: use defaults and auto-confirm
`-?`	Show the full list of options

Run ./kb-download.sh -? for the complete set of flags (thread count, connection timeout, compression, disk-space override, and path overrides). The full option reference and more example sessions are in the install README.

After downloading an update

A KB update download includes an ldb-import.sh helper. After the download finishes, run that script from inside the downloaded folder to import the update into your local database (LDB):

/tmp/scanoss_kb_update/26.03/ldb-import.sh

The script prints the exact path to ldb-import.sh when it finishes.

Do not interrupt the import. An interrupted import can corrupt the existing LDB.

Verify the update

Each update ships with test WFP files. After importing, confirm the update worked by scanning them:

scanoss-py scan /tmp/scanoss_kb_update/26.03/26.03_test_file.wfp
scanoss-py scan /tmp/scanoss_kb_update/26.03/26.03_test_snippet.wfp

If both scans return matches, the update imported successfully.

Documentation

Links

Download the KB with `kb-download.sh`

Prerequisites

Get the script

Run it interactively

Run it non-interactively

After downloading an update

Verify the update

​Download the KB with kb-download.sh

​Prerequisites

​Get the script

​Run it interactively

​Run it non-interactively

​After downloading an update

​Verify the update

Download the KB with `kb-download.sh`

Prerequisites

Get the script

Run it interactively

Run it non-interactively

After downloading an update

Verify the update