Stac Report: kdb+ 4.0 With Intel Persistent Memory Pool Under Stac-M3

STAC (Securities Technology Analysis Center LLC ) recently performed STAC-M3 benchmarks on a stack involving kdb+ 4.0 and a server cluster supporting a pool of Intel Optane DC Persistent Memory.

The results are available (registration required).

STAC-M3 is the set of standard enterprise tick-analytics benchmarks for database software/hardware stacks that manage large timeseries of market data (‘tick data’). In this project, the firm ran both the baseline benchmark suite (code named Antuco) and the optional scaling suite (code named Kanaga).

The stack included Kx Systems, Inc.‘s kdb+ 4.0 running on a cluster of ten 2-socket Super Micro Computer, Inc. SYS-6019U-TN4RT servers. Each server had 2 Xeon Platinum 8280L CPUs @2.70GHz, 192GB of DRAM, and 6TB of Optane DC Persistent Memory. The database storage was distributed over the cluster’s 60TB pool of persistent memory. The kdb+ processes ran in Docker v1.13 containers orchestrated by Red Hat OpenShift v3.11. The SUT was embedded in an implementation of the Intel Data Management Platform, a solution targeted at data scientists and data engineers within large organizations. Additional configuration information is available in the report linked above (and micro-detailed configuration information is also available at the same link).

Intel Corp. chose to highlight that this solution:

Outperformed all publicly disclosed results in five of 24 Kanaga mean-response time benchmarks:
- 50-user 12-day VWAB benchmark year 1 (STAC-M3.β1.50T.YR1VWAB-12D-HO.TIME).
- 10-user market snapshot: benchmark years 2 through 4 (STAC-M3.β1.10T.YR[n]-MKTSNAP.TIME), including:
  - 23.9x speedup in benchmark year 2 (STAC-M3.β1.10T.YR2-MKTSNAP.TIME)
  - 20.9x speedup in benchmark year 4 (STAC-M3.β1.10T.YR4-MKTSNAP.TIME)

Outperformed all publicly disclosed results in 3 of 17 Antuco mean-response time benchmarks, including:
- 1.2x speedup in 100-user interval stats (STAC-M3.β1.100T.STATS-UI.TIME)
- 2.7x speedup in 100-user 12-day VWAB – no overlap (STAC-M3.β1.100T.VWAB-12D.NO.TIME)
- 4.3x speedup in 10-user market snapshot (STAC-M3.β1.10T.MKTSNAP.TIME)

Detailed results are in the report. Premium subscribers have access to the code used in this project as well as the configuration details.