Make the installation easier, with a script that builds rocksdb
scripts/build-rocksdb.sh builds rocksdb once, with the shared and the
static library, into a prefix of its own. It builds the version of the
rocksdb package of the distribution where there is one, so that the
static library matches what collocatordb is compiled against, and a
pinned one where there is none (Rocky Linux, RHEL). Older rocksdb
versions are made to compile with newer compilers, e.g. Fedora's 10.2.1
with gcc 16.
-DSTATIC_ROCKSDB=ON links rocksdb and collocatordb statically, which
makes counting collocations about 22% faster, without naming the two
libraries by their paths.
The installation section of the README is rewritten as four steps that
cover Fedora, Rocky Linux 9 and 10, RHEL and MacOS.
Change-Id: Idbb588ad517c626fc387973b676126480edb6c2e
diff --git a/CMakeLists.txt b/CMakeLists.txt
index f391940..8abe345 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -1,6 +1,7 @@
cmake_minimum_required(VERSION 3.9)
project(dereko2vec VERSION 0.9.1 DESCRIPTION "Fork from wang2vec with extensions for re-training and count based models")
option(STATIC_DEREKO2VEC "Build dereko2vec as a static binary" OFF)
+option(STATIC_ROCKSDB "Link rocksdb and collocatordb statically, everything else shared" OFF)
if (STATIC_DEREKO2VEC AND APPLE)
message(FATAL_ERROR "STATIC_DEREKO2VEC does not work on MacOS, which ships "
"no static system libraries. Build without it.")
@@ -31,12 +32,14 @@
include_directories(/usr/local/include /usr/local/kl/include /opt/homebrew/include)
link_directories(/usr/local/kl/lib /usr/local/lib /usr/local/lib64 /opt/homebrew/lib)
-# A static dereko2vec needs everything as a static library. An ordinary build
-# takes the shared ones, and it has to take them explicitly: a leftover static
-# library below /usr/local would otherwise win over the shared library of the
-# distribution, because cmake weighs the search path higher than the suffix.
-# Set -DCOLLOCATORDB=<path> or -DROCKSDB=<path> to pick specific ones.
-if (STATIC_DEREKO2VEC)
+# A static dereko2vec needs everything as a static library. With STATIC_ROCKSDB
+# only rocksdb and collocatordb are static and the compression libraries stay
+# shared, which is the fast indexing build the README describes. An ordinary
+# build takes the shared ones, and it has to take them explicitly: a leftover
+# static library below /usr/local would otherwise win over the shared library
+# of the distribution, because cmake weighs the search path higher than the
+# suffix. Set -DCOLLOCATORDB=<path> or -DROCKSDB=<path> to pick specific ones.
+if (STATIC_DEREKO2VEC OR STATIC_ROCKSDB)
find_library(COLLOCATORDB libcollocatordb_static.a)
find_library(ROCKSDB librocksdb.a)
else ()
@@ -50,16 +53,17 @@
if (NOT COLLOCATORDB)
message(FATAL_ERROR "No collocatordb found. Please install it from "
"https://korap.ids-mannheim.de/gerrit/plugins/gitiles/ids-kl/collocatordb"
- " - a static dereko2vec needs libcollocatordb_static.a, otherwise "
+ " - a static build needs libcollocatordb_static.a, otherwise "
"libcollocatordb.so is enough.")
endif ()
if (NOT ROCKSDB)
message(FATAL_ERROR "No rocksdb found. Install the rocksdb development "
"package of your distribution, e.g. librocksdb-dev or "
- "rocksdb-devel. Where there is none, or where it is older than the "
- "one collocatordb was built against, build rocksdb and say where "
- "it is: -DCMAKE_PREFIX_PATH=<prefix it was installed to>. It has "
- "to be the same rocksdb that collocatordb was built against.")
+ "rocksdb-devel. Where there is none, or where a static one is "
+ "needed, scripts/build-rocksdb.sh builds one into a prefix of its "
+ "own; then say where it is: -DCMAKE_PREFIX_PATH=<prefix>. It has "
+ "to be the same rocksdb that collocatordb was built against, "
+ "which the script takes care of on its own.")
endif ()
message(STATUS "Using collocatordb ${COLLOCATORDB}")
message(STATUS "Using RocksDB ${ROCKSDB}")
diff --git a/README.md b/README.md
index ff91da1..bf06a9f 100644
--- a/README.md
+++ b/README.md
@@ -4,21 +4,73 @@
## Installation
-### Dependencies
+### 1. Prerequisites
-* cmake
-* rocksdb. `librocksdb-dev` on Debian and Ubuntu and `rocksdb-devel` on Fedora
- do. Rocky Linux and RHEL package a rocksdb that is too old, there it has to
- be built, which the
- [collocatordb README](https://korap.ids-mannheim.de/gerrit/plugins/gitiles/ids-kl/collocatordb)
- describes
-* [libcollocatordb](https://korap.ids-mannheim.de/gerrit/plugins/gitiles/ids-kl/collocatordb) >= v1.6.0
+* on Fedora:
-It has to be the same rocksdb that collocatordb was built against. dereko2vec
-itself does not include any rocksdb header, but it links what collocatordb
-refers to, and symbols of one version do not exist in another.
+ ```bash
+ sudo dnf install cmake gcc-c++ git rocksdb-devel snappy-devel zlib-devel bzip2-devel lz4-devel libzstd-devel
+ ```
-### Build and install
+* on Rocky Linux 9 and 10 and RHEL: the same without `rocksdb-devel`, rocksdb
+ is built in the next step:
+
+ ```bash
+ sudo dnf install cmake gcc-c++ git snappy-devel zlib-devel bzip2-devel lz4-devel libzstd-devel
+ ```
+
+* on MacOS:
+
+ ```bash
+ brew install cmake rocksdb snappy zlib bzip2 lz4 zstd
+ ```
+
+### 2. RocksDB, where the static one is wanted or there is no package
+
+Linking rocksdb statically makes counting collocations about 22% faster -
+on an indexing run of two weeks that is about three days. Debian, Ubuntu and
+MacOS ship `librocksdb.a` in their rocksdb package, Fedora, Rocky Linux and
+RHEL do not. One script builds rocksdb once, with the shared and the static
+library, into a prefix of its own:
+
+```bash
+scripts/build-rocksdb.sh
+```
+
+It builds the version of the rocksdb package of the distribution where there
+is one, so that the static library matches the headers collocatordb is
+compiled against, and a pinned recent one where there is none (Rocky Linux,
+RHEL). The version and the prefix can be changed, see
+`scripts/build-rocksdb.sh --help`.
+
+On Rocky Linux and RHEL, which have no rocksdb package, this step is not
+optional, and programs that use the shared library need the prefix in the
+search path of the loader, which the script prints at its end.
+
+### 3. CollocatorDB
+
+[libcollocatordb](https://korap.ids-mannheim.de/gerrit/plugins/gitiles/ids-kl/collocatordb) >= v1.6.0:
+
+```bash
+git clone "https://korap.ids-mannheim.de/gerrit/ids-kl/collocatordb"
+cd collocatordb
+cmake -S . -B build -DCMAKE_INSTALL_PREFIX=/usr/local
+cmake --build build -j $(nproc)
+sudo cmake --install build && sudo ldconfig
+ctest --test-dir build --extra-verbose
+cd ..
+```
+
+Where rocksdb was built in step 2 and it is not the one of the distribution -
+on Rocky Linux and RHEL always, elsewhere when `-v` named another version -
+collocatordb has to be built against it, and naming its prefix covers the
+headers and both libraries at once:
+
+```bash
+cmake -S . -B build -DCMAKE_PREFIX_PATH=$HOME/rocksdb -DCMAKE_INSTALL_PREFIX=/usr/local
+```
+
+### 4. Build and install dereko2vec
```bash
cmake -S . -B build
@@ -30,34 +82,30 @@
This installs `dereko2vec` and `vecs2mmap`. The build directory has to be
`build` inside the sources, the test looks for the binary relative to it.
-Where rocksdb or collocatordb were installed into a prefix of their own, name
-it, which covers the libraries and the header at once. Several prefixes are
-separated by semicolons:
+For the fast build with rocksdb and collocatordb linked statically, name the
+prefix of the rocksdb from step 2. The compression libraries stay shared:
+
+```bash
+cmake -S . -B build -DSTATIC_ROCKSDB=ON -DCMAKE_PREFIX_PATH=$HOME/rocksdb
+```
+
+On Debian, Ubuntu and MacOS, whose rocksdb package brings the static library,
+`-DSTATIC_ROCKSDB=ON` without a prefix does. And where rocksdb was installed
+into a prefix of its own but is to be linked shared, name it without the
+option:
```bash
cmake -S . -B build -DCMAKE_PREFIX_PATH=$HOME/rocksdb
```
-### Faster indexing
+It has to be the same rocksdb that collocatordb was built against. dereko2vec
+itself does not include any rocksdb header, but it links what collocatordb
+refers to, and symbols of one version do not exist in another. Where the
+script of step 2 was used for both, this is already taken care of.
-Linking rocksdb and collocatordb statically makes counting collocations about
-22% faster, measured with 2 million increments against rocksdb 7.8.3. On an
-indexing run of two weeks that is about three days. Name the two static
-libraries, the compression libraries stay shared:
-
-```bash
-cmake -S . -B build \
- -DCOLLOCATORDB=/usr/local/lib64/libcollocatordb_static.a \
- -DROCKSDB=$HOME/rocksdb/lib64/librocksdb.a
-```
-
-`lib64` on Fedora, Rocky Linux and RHEL, `lib` on Debian and Ubuntu. Debian and
-Ubuntu ship `librocksdb.a` in `librocksdb-dev`, elsewhere it comes out of a
-rocksdb built by hand, see the collocatordb README.
-
-This does not need static versions of zlib, snappy, lz4 and zstd, which Rocky
-Linux, RHEL and Fedora do not ship. Only `-DSTATIC_DEREKO2VEC=ON`, which builds
-a completely static binary, needs those, and is rarely worth the trouble.
+A completely static binary, `-DSTATIC_DEREKO2VEC=ON`, needs static versions of
+zlib, snappy, lz4 and zstd on top, which Rocky Linux, RHEL and Fedora do not
+ship, and is rarely worth the trouble.
### If dereko2vec does not start
diff --git a/scripts/build-rocksdb.sh b/scripts/build-rocksdb.sh
new file mode 100755
index 0000000..36e4565
--- /dev/null
+++ b/scripts/build-rocksdb.sh
@@ -0,0 +1,147 @@
+#!/usr/bin/env bash
+#
+# Builds rocksdb once, with the shared and the static library, and installs it
+# into a prefix of its own (default: $HOME/rocksdb). Works on Fedora, Rocky
+# Linux, RHEL, Debian, Ubuntu and MacOS.
+#
+# The version to build is, in this order:
+# 1. the one given with -v
+# 2. the one of the rocksdb package of the distribution, so that the static
+# library matches the headers and the shared library collocatordb was
+# built against
+# 3. a pinned one that is known to compile with the compilers of Rocky Linux
+# 9 (gcc 11) up to Fedora (gcc 16) and MacOS (clang)
+#
+# Examples:
+# scripts/build-rocksdb.sh # build the version of the distribution
+# scripts/build-rocksdb.sh -v 10.2.1 # build a specific version
+# scripts/build-rocksdb.sh -p /opt/rocksdb # install somewhere else
+#
+set -euo pipefail
+
+PINNED_VERSION=10.2.1
+PREFIX=$HOME/rocksdb
+VERSION=
+SOURCE_DIR=
+STATIC_ONLY=0
+
+usage() {
+ sed -n '2,17p' "$0"
+ exit "${1:-0}"
+}
+
+while [ $# -gt 0 ]; do
+ case $1 in
+ -p|--prefix) PREFIX=$2; shift 2 ;;
+ -v|--version) VERSION=$2; shift 2 ;;
+ -s|--source) SOURCE_DIR=$2; shift 2 ;;
+ --static-only) STATIC_ONLY=1; shift ;;
+ -h|--help) usage 0 ;;
+ *) echo "Unknown option: $1" >&2; usage 1 ;;
+ esac
+done
+
+SOURCE_DIR=${SOURCE_DIR:-$PREFIX-src}
+
+# --- the version -------------------------------------------------------------
+if [ -z "$VERSION" ]; then
+ if command -v rpm > /dev/null && rpm -q rocksdb > /dev/null 2>&1; then
+ VERSION=$(rpm -q --qf '%{VERSION}' rocksdb)
+ elif command -v dpkg-query > /dev/null && dpkg-query -W librocksdb-dev > /dev/null 2>&1; then
+ VERSION=$(dpkg-query -W -f '${Version}' librocksdb-dev | cut -d- -f1)
+ elif command -v brew > /dev/null && brew list --versions rocksdb > /dev/null 2>&1; then
+ VERSION=$(brew list --versions rocksdb | awk '{print $2}' | cut -d' ' -f1)
+ else
+ VERSION=$PINNED_VERSION
+ echo "No rocksdb package found, building the pinned version $VERSION."
+ echo "Give another one with -v, see https://github.com/facebook/rocksdb/tags"
+ fi
+fi
+echo "Building rocksdb $VERSION"
+
+# --- the prerequisites -------------------------------------------------------
+missing=
+for tool in git cmake make; do
+ command -v $tool > /dev/null || missing="$missing $tool"
+done
+if ! command -v c++ > /dev/null && ! command -v g++ > /dev/null && ! command -v clang++ > /dev/null; then
+ missing="$missing c++"
+fi
+# The compression libraries rocksdb is built with below. Where they are
+# missing the build still succeeds, just without them, so this is a hint.
+compression_hint=
+if command -v pkg-config > /dev/null; then
+ for lib in snappy zlib bzip2 liblz4 libzstd; do
+ pkg-config --exists $lib 2> /dev/null || compression_hint="$compression_hint $lib"
+ done
+fi
+if [ -n "$missing" ]; then
+ echo "Missing build tools:$missing" >&2
+ echo >&2
+ echo "Install them and the compression libraries:" >&2
+ echo " Fedora, Rocky, RHEL: sudo dnf install cmake gcc-c++ git snappy-devel zlib-devel bzip2-devel lz4-devel libzstd-devel" >&2
+ echo " Debian, Ubuntu: sudo apt-get install cmake g++ git libsnappy-dev zlib1g-dev libbz2-dev liblz4-dev libzstd-dev" >&2
+ echo " MacOS: brew install cmake snappy zlib bzip2 lz4 zstd" >&2
+ exit 1
+fi
+if [ -n "$compression_hint" ]; then
+ echo "Warning: pkg-config does not find:$compression_hint" >&2
+ echo "rocksdb will be built without them. The devel packages named above provide them." >&2
+fi
+
+# --- the sources -------------------------------------------------------------
+if [ -d "$SOURCE_DIR/.git" ]; then
+ echo "Reusing the checkout in $SOURCE_DIR"
+ git -C "$SOURCE_DIR" fetch --depth 1 origin "refs/tags/v$VERSION"
+ git -C "$SOURCE_DIR" checkout -q FETCH_HEAD
+else
+ git clone https://github.com/facebook/rocksdb.git --branch "v$VERSION" --depth 1 "$SOURCE_DIR"
+fi
+
+# --- the build ---------------------------------------------------------------
+# lib64 where the distribution has one (Fedora, Rocky, RHEL, openSUSE), lib
+# elsewhere. Both are searched for a prefix by cmake, this is for the user.
+if [ -d /usr/lib64 ] && [ ! -e /etc/debian_version ]; then
+ LIBDIR=lib64
+else
+ LIBDIR=lib
+fi
+
+# WITH_GFLAGS and WITH_LIBURING are off so that the static library does not
+# drag them into everything that links it. FAIL_ON_WARNINGS off so that a
+# newer compiler does not turn rocksdb's own warnings into errors. And
+# -include cstdint because older rocksdb versions declare uint64_t and friends
+# without including it, which newer compilers no longer tolerate - Fedora 44
+# packages 10.2.1 and builds it with gcc 16, for instance. Where it is already
+# included, the flag changes nothing.
+cmake -S "$SOURCE_DIR" -B "$SOURCE_DIR/build" \
+ -DCMAKE_BUILD_TYPE=Release \
+ -DCMAKE_CXX_FLAGS="-include cstdint" \
+ -DFAIL_ON_WARNINGS=OFF \
+ -DWITH_TESTS=OFF -DWITH_ALL_TESTS=OFF \
+ -DWITH_BENCHMARK_TOOLS=OFF -DWITH_TOOLS=OFF -DWITH_CORE_TOOLS=OFF \
+ -DWITH_GFLAGS=OFF -DWITH_LIBURING=OFF \
+ -DWITH_SNAPPY=ON -DWITH_LZ4=ON -DWITH_ZLIB=ON -DWITH_ZSTD=ON -DWITH_BZ2=ON \
+ -DROCKSDB_BUILD_SHARED=$([ $STATIC_ONLY = 1 ] && echo OFF || echo ON) \
+ -DCMAKE_INSTALL_PREFIX="$PREFIX" \
+ -DCMAKE_INSTALL_LIBDIR="$LIBDIR"
+cmake --build "$SOURCE_DIR/build" -j "$(getconf _NPROCESSORS_ONLN 2> /dev/null || sysctl -n hw.ncpu)"
+cmake --install "$SOURCE_DIR/build"
+
+# --- what to do with it ------------------------------------------------------
+echo
+echo "Installed rocksdb $VERSION:"
+echo " headers: $PREFIX/include"
+echo " static: $PREFIX/$LIBDIR/librocksdb.a"
+[ $STATIC_ONLY = 1 ] || echo " shared: $PREFIX/$LIBDIR/$( [ "$(uname)" = Darwin ] && echo librocksdb.dylib || echo librocksdb.so )"
+echo
+echo "Build collocatordb against it with:"
+echo " cmake -S . -B build -DCMAKE_PREFIX_PATH=$PREFIX -DCMAKE_INSTALL_PREFIX=/usr/local"
+echo
+echo "Build dereko2vec against it with:"
+echo " cmake -S . -B build -DSTATIC_ROCKSDB=ON -DCMAKE_PREFIX_PATH=$PREFIX"
+if [ $STATIC_ONLY = 0 ] && [ "$(uname)" = Linux ]; then
+ echo
+ echo "Programs that use the shared library find it after:"
+ echo " echo $PREFIX/$LIBDIR | sudo tee /etc/ld.so.conf.d/rocksdb.conf && sudo ldconfig"
+fi