KB update guide

Download the KB with `kb-download.sh`

kb-download.sh is the supported way to download the KB and its updates. It connects to the SCANOSS SFTP server, lists the available versions, and lets you pick one. It then checks that you have enough free disk space and downloads the package. It can fetch the full KB, a KB update, the test KB, a SQLite KB snapshot, or an HFH KB snapshot.

Update frequency and size

SCANOSS publishes a KB update every month, versioned YY.MM, for example 26.01, 26.02, 26.03. To stay current, run kb-download.sh in update mode against the same LDB about once a month. You never need to download the full KB again. Each monthly update is a small incremental download, not another full snapshot. It holds only the changes since the previous version. The script reports the exact size before it downloads anything, for example Update size: 5.2G in the sample session below. The size changes from month to month, depending on how much the upstream knowledgebase changed. Plan bandwidth and download time around the printed size, not a fixed number. For the full KB’s footprint on disk, see Hardware requirements.

Prerequisites

lftp is recommended. It gives parallel, resumable downloads. Install it with apt install lftp. If lftp isn’t installed, the script offers to fall back to the standard sftp client.
sshpass is needed only for the sftp fallback. Install it with apt install sshpass. You don’t need it with lftp.
You need SFTP credentials: host, port, username, and password. The SCANOSS team sends them by email.

Get the script

The script is in the SCANOSS on-premise repository. Clone the repository, or pull the latest changes if you already have it, and make the scripts executable:

git clone https://github.com/scanoss/on-premise.git
cd on-premise/install/
chmod +x *.sh

The script is also documented in the on-premise install README.

Run it interactively

Run the script with no arguments for a guided session. It prompts for your connection details, the download mode, and the version:

./kb-download.sh

The script asks what to download, then lists the available versions and marks the latest. For example:

SCANOSS Knowledge Base Download
================================
Using lftp for downloads (parallel, resumable).

What do you want to download?
  1) Full KB
  2) KB update
  3) Test KB
  4) SQLite KB
  5) HFH KB

Select [1-5]: 2

SFTP host: sftp.test.xyz
SFTP port: 12345
SFTP username: USER21
SFTP password:

Fetching available update versions...

Available update versions:
-------------------
  1) 26.01
  2) 26.02
  3) 26.03  (latest)
-------------------

Select a update version to download [1-3] (default: 3):

Selected update: 26.03

Download directory [/tmp/scanoss_kb_update]:

Fetching update metadata...
Update size: 5.2G
Free space:  120G (on /dev/sda1)
Disk space OK.

lftp parallel threads [25]:

Download update 26.03 to /tmp/scanoss_kb_update? [Y/n]
Downloading with lftp (25 parallel threads, resumable)...
Update downloaded to /tmp/scanoss_kb_update/26.03

Finished downloading update.
Please run the ldb-import.sh script provided inside the update folder to import into the KB:
  /tmp/scanoss_kb_update/26.03/ldb-import.sh

The version list has one entry per month, 26.01, 26.02, 26.03, and marks the newest (latest). The Update size: 5.2G line above the download prompt is that month’s incremental download size. The script checks it against your free disk space. Read both before you confirm, especially on a metered or slow connection.

For long downloads such as the full KB, run the script inside a tmux session so the transfer survives a dropped connection. Start one with tmux new-session -s scanoss, run the script, then detach with Ctrl+B d and reattach later with tmux attach -t scanoss.

Run it non-interactively

To skip a prompt, pass the matching flag. Add -y to accept the defaults and run with no prompts. With -y, the connection flags -m, -h, -P, -u, and -p are required:

# Defaults (latest version, /var/lib/ldb/oss + /tmp/scanoss_kb_full_<version>):
./kb-download.sh -y -m full -h sftp.test.xyz -P 12345 -u USER21 -p mypassword

# Pin a version, custom paths, force on low disk space:
./kb-download.sh -y -f -m full -V 26.03 \
    -o /data/oss -r /data/kb-rest \
    -h sftp.test.xyz -P 12345 -u USER21 -p mypassword

# Update mode with a custom download base directory:
./kb-download.sh -y -m update -D /opt/scanoss/updates \
    -h sftp.test.xyz -P 12345 -u USER21 -p mypassword

# SQLite KB into the current directory (final path: ./<version>):
./kb-download.sh -y -m sqlite \
    -h sftp.test.xyz -P 12345 -u USER21 -p mypassword

# SQLite KB into a specific base directory:
./kb-download.sh -y -m sqlite -D /opt/scanoss/sqlite \
    -h sftp.test.xyz -P 12345 -u USER21 -p mypassword

# HFH KB (single .zip; lands at ./<version>.zip):
./kb-download.sh -y -m hfh \
    -h sftp.test.xyz -P 12345 -u USER21 -p mypassword

# Test KB:
./kb-download.sh -y -m test -h sftp.test.xyz -P 12345 -u USER21 -p mypassword

Common options:

Flag	Description
`-m`	Download mode: `full`, `update`, `test`, `sqlite`, or `hfh`
`-h`	SFTP host
`-P`	SFTP port
`-u`	SFTP username
`-p`	SFTP password
`-V`	KB version (default: the latest reported by the server)
`-D`	Download directory for `update` / `sqlite` / `hfh` modes
`-o`	Destination for the `oss` folder (full/test modes; default `/var/lib/ldb/oss`)
`-y`	Non-interactive: use defaults and auto-confirm
`-?`	Show the full list of options

Run ./kb-download.sh -? for every flag, including thread count, connection timeout, compression, the disk-space override, and path overrides. The full option reference and more example sessions are in the install README.

After downloading an update

A KB update download includes an ldb-import.sh helper. After the download finishes, run it from inside the downloaded folder. It imports the update into your local database (LDB):

/tmp/scanoss_kb_update/26.03/ldb-import.sh

The script prints the exact path to ldb-import.sh when it finishes.

Do not interrupt the import. An interrupted import can corrupt the existing LDB.

Verify the update

Each update ships with test WFP files. After the import, scan them to confirm the update worked:

scanoss-py scan /tmp/scanoss_kb_update/26.03/26.03_test_file.wfp
scanoss-py scan /tmp/scanoss_kb_update/26.03/26.03_test_snippet.wfp

If both scans return matches, the import worked.

A collate error E078 means records were dropped. Since LDB 4.3.0 and 5.0.0-crc64, a collate fails with error E078 when it has to discard records because a single key holds more than its configured cap. Earlier versions dropped those records and still reported success. If you see E078 in an import or collate log, run the collate again with a smaller MAX_RECORD for the affected table, or contact SCANOSS support.

Upgrade LDB before importing with collate. LDB 4.1.12 and 5.0.2-crc64 fix collate bugs that could drop records while still reporting the full record count. Before 5.0.2-crc64, LDB could discard a record of an adjacent key whose data was identical. This affected only tables with variable-length records. Before 4.1.12, LDB could lose a record group whose first record had an all-zero subkey. Tables collated with an earlier version may be missing records. Collating again doesn’t recover them. Re-import those tables with the fixed version.

Get started

Recipes

CLI reference

API reference

Self-hosting & knowledge base

Download the KB with `kb-download.sh`

Update frequency and size

Prerequisites

Get the script

Run it interactively

Run it non-interactively

After downloading an update

Verify the update

​Download the KB with kb-download.sh

​Update frequency and size

​Prerequisites

​Get the script

​Run it interactively

​Run it non-interactively

​After downloading an update

​Verify the update

Download the KB with `kb-download.sh`

Update frequency and size

Prerequisites

Get the script

Run it interactively

Run it non-interactively

After downloading an update

Verify the update