KB Update Guide

Download the KB with `kb-download.sh`

kb-download.sh is the supported way to download the KB and its updates. It connects to the SCANOSS SFTP server, lists the available versions, lets you pick one, checks you have enough free disk space, and downloads it. It can fetch the full KB, a KB update, the test KB, a SQLite KB snapshot, or an HFH KB snapshot.

Update Frequency and Size

New KB updates are published on a monthly cadence, versioned YY.MM, for example 26.01, 26.02, 26.03. Running kb-download.sh in update mode against the same LDB about once a month is enough to stay current, without ever re-downloading the full KB. Each monthly update is a small, incremental download rather than another full snapshot, it only carries the delta since the previous version. The script reports its exact size before it downloads anything, for example Update size: 5.2G in the sample session below. That figure varies from one month to the next depending on how much the upstream knowledgebase changed, so treat the printed size as the number to plan bandwidth and download time around, not a fixed constant. For the full KB’s own footprint on disk, see Hardware Requirements.

Prerequisites

lftp (recommended): provides parallel, resumable downloads. Install with apt install lftp. If lftp is not installed, the script offers to fall back to the standard sftp client.
sshpass (only needed for the sftp fallback): install with apt install sshpass. Not required when using lftp.
SFTP credentials: host, port, username, and password, provided by the SCANOSS team by email.

Get the script

The script lives in the SCANOSS on-premise repository. Clone it (or pull the latest changes if you already have it) and make the scripts executable:

git clone https://github.com/scanoss/on-premise.git
cd on-premise/install/
chmod +x *.sh

The script is also documented in the on-premise install README.

Run it interactively

Run the script with no arguments for a fully guided session. It prompts for your connection details, the download mode, and the version:

./kb-download.sh

You will be asked what to download, then shown the available versions with the latest marked, for example:

SCANOSS Knowledge Base Download
================================
Using lftp for downloads (parallel, resumable).

What do you want to download?
  1) Full KB
  2) KB update
  3) Test KB
  4) SQLite KB
  5) HFH KB

Select [1-5]: 2

SFTP host: sftp.test.xyz
SFTP port: 12345
SFTP username: USER21
SFTP password:

Fetching available update versions...

Available update versions:
-------------------
  1) 26.01
  2) 26.02
  3) 26.03  (latest)
-------------------

Select a update version to download [1-3] (default: 3):

Selected update: 26.03

Download directory [/tmp/scanoss_kb_update]:

Fetching update metadata...
Update size: 5.2G
Free space:  120G (on /dev/sda1)
Disk space OK.

lftp parallel threads [25]:

Download update 26.03 to /tmp/scanoss_kb_update? [Y/n]
Downloading with lftp (25 parallel threads, resumable)...
Update downloaded to /tmp/scanoss_kb_update/26.03

Finished downloading update.
Please run the ldb-import.sh script provided inside the update folder to import into the KB:
  /tmp/scanoss_kb_update/26.03/ldb-import.sh

Notice the version list: 26.01, 26.02, 26.03, one entry per month, with the newest always marked (latest). The Update size: 5.2G line just above the download prompt is that month’s incremental download size, checked against your free disk space automatically. Both are worth reading before confirming, especially on a metered or slow connection.

For long-running downloads (such as the full KB), we recommend running the script inside a tmux session so the transfer survives a dropped connection. Start one with tmux new-session -s scanoss, run the script, then detach with Ctrl+B d and reattach later with tmux attach -t scanoss.

Run it non-interactively

Any prompt can be skipped by passing the matching flag. Add -y to accept defaults and run without prompts (the connection flags -m, -h, -P, -u, -p are then required):

# Defaults (latest version, /var/lib/ldb/oss + /tmp/scanoss_kb_full_<version>):
./kb-download.sh -y -m full -h sftp.test.xyz -P 12345 -u USER21 -p mypassword

# Pin a version, custom paths, force on low disk space:
./kb-download.sh -y -f -m full -V 26.03 \
    -o /data/oss -r /data/kb-rest \
    -h sftp.test.xyz -P 12345 -u USER21 -p mypassword

# Update mode with a custom download base directory:
./kb-download.sh -y -m update -D /opt/scanoss/updates \
    -h sftp.test.xyz -P 12345 -u USER21 -p mypassword

# SQLite KB into the current directory (final path: ./<version>):
./kb-download.sh -y -m sqlite \
    -h sftp.test.xyz -P 12345 -u USER21 -p mypassword

# SQLite KB into a specific base directory:
./kb-download.sh -y -m sqlite -D /opt/scanoss/sqlite \
    -h sftp.test.xyz -P 12345 -u USER21 -p mypassword

# HFH KB (single .zip; lands at ./<version>.zip):
./kb-download.sh -y -m hfh \
    -h sftp.test.xyz -P 12345 -u USER21 -p mypassword

# Test KB:
./kb-download.sh -y -m test -h sftp.test.xyz -P 12345 -u USER21 -p mypassword

Commonly used options:

Flag	Description
`-m`	Download mode: `full`, `update`, `test`, `sqlite`, or `hfh`
`-h`	SFTP host
`-P`	SFTP port
`-u`	SFTP username
`-p`	SFTP password
`-V`	KB version (default: the latest reported by the server)
`-D`	Download directory for `update` / `sqlite` / `hfh` modes
`-o`	Destination for the `oss` folder (full/test modes; default `/var/lib/ldb/oss`)
`-y`	Non-interactive: use defaults and auto-confirm
`-?`	Show the full list of options

Run ./kb-download.sh -? for the complete set of flags (thread count, connection timeout, compression, disk-space override, and path overrides). The full option reference and more example sessions are in the install README.

After downloading an update

A KB update download includes an ldb-import.sh helper. After the download finishes, run that script from inside the downloaded folder to import the update into your local database (LDB):

/tmp/scanoss_kb_update/26.03/ldb-import.sh

The script prints the exact path to ldb-import.sh when it finishes.

Do not interrupt the import. An interrupted import can corrupt the existing LDB.

Verify the update

Each update ships with test WFP files. After importing, confirm the update worked by scanning them:

scanoss-py scan /tmp/scanoss_kb_update/26.03/26.03_test_file.wfp
scanoss-py scan /tmp/scanoss_kb_update/26.03/26.03_test_snippet.wfp

If both scans return matches, the update imported successfully.

A collate error E078 means records were dropped. Since LDB 4.3.0 / 5.0.0-crc64, a collate that has to discard records because a single key holds more than its configured cap now fails loudly with error E078 instead of silently reporting success. If you see E078 in an import or collate log, re-run against a smaller MAX_RECORD for the affected table, or contact SCANOSS support.

Upgrade LDB before importing with collate. LDB 4.1.12 and 5.0.2-crc64 fix collate bugs that could silently drop records while still reporting the full record count. LDB 5.0.2-crc64 no longer discards a record of an adjacent key whose data is identical (only tables with variable-length records were affected), and LDB 4.1.12 no longer loses a record group whose first record has an all-zero subkey. Tables collated with an earlier version may be missing records. Collating again doesn’t recover them: re-import those tables with the fixed version.

Documentation

Links

Download the KB with `kb-download.sh`

Update Frequency and Size

Prerequisites

Get the script

Run it interactively

Run it non-interactively

After downloading an update

Verify the update

​Download the KB with kb-download.sh

​Update Frequency and Size

​Prerequisites

​Get the script

​Run it interactively

​Run it non-interactively

​After downloading an update

​Verify the update

Download the KB with `kb-download.sh`

Update Frequency and Size

Prerequisites

Get the script

Run it interactively

Run it non-interactively

After downloading an update

Verify the update