Securities Technology Analysis Center Report: kdb+ 3.6 With DDN Exascaler and AI400X Flash Appliances Under Stack-M3

STAC (Securities Technology Analysis Center LLC ) recently performed STAC-M3 Benchmarks on a stack involving kdb+, the DDN EXAScaler parallel file system, and 2 DDN AI400X all-flash storage appliances.

DDN AI400X all-flash storage appliance

The results are available (registration required).

STAC-M3 is the set of standard enterprise tick-analytics benchmarks for database software/hardware stacks that manage large time series of market data (‘tick data’). In this project, we ran both the baseline benchmark suite (code named Antuco) and the optional scaling suite (code named Kanaga).

The stack under test was Kx’s kdb+ 3.6 database system distributed across 15 Intel S2600BPB servers accessing 2 DDN AI400X all-flash appliances via the EXAScaler 5.1.1 parallel filesystem. DDN (DataDirect Networks, Inc.) chose to highlight that this solution:

• Outperformed all publicly disclosed results in all year-high bid throughput benchmarks (STAC-M3.?1.1T.*.BPS)

• vs. a kdb+ solution based on a parallel file system with 14 database servers and 18 storage servers (SUT ID KDB200401):

  • • was faster in 4 of 17 Antuco mean response time benchmarks, including:

      • • 9.4x speedup in NBBO (STAC-M3.β1.1T.NBBO.TIME)

        • 46% faster in 10-user market snapshot (STAC-M3.β1.10T.MKTSNAP.TIME)

        • 15% faster in 10-user aggregate stats (STAC-M3.β1.10T.STATS-AGG.TIME)

        • 3% faster in 100-user interval stats (STAC-M3.β1.100T.STATS-UI.TIME)

    • was within 0.2% of the mean response time for 10-user volume curve (STAC-M3.β1.10T.VOLCURV.TIME )

• vs. a kdb+ solution with 4 database servers and networked flash storage (SUT ID KDB190430):

  • • was faster in 14 of 17 Antuco mean response time benchmarks, including:

      • • 12x speedup in 10-user market snapshot (STAC-M3.β1.10T.MKTSNAP.TIME)

        • 6x speedup or greater in all 4 user counts doing intervalized stats (STAC-M3.β1.*.STATS-UI.TIME)

        • 5x speedup in 10-user aggregate stats (STAC-M3.β1.10T.STATS-AGG.TIME)

   

• • • was faster in 20 of 24 Kanaga mean response time benchmarks, including:

      • • 19x speedup or greater in all four 10-user market snapshots (STAC-M3.β1.10T.YR[n]-MKTSNAP.TIME)

        • 3x speedup or greater in year-high bid for all four years (STAC-M3.β1.1T.[n]YRHIBID.TIME)

Details are in the STAC report (registration required). Premium subscribers have access to the code used in this project and the micro-detailed configuration information for the solution.

Read also:
R&D: Stack Usage Analysis for Efficient Wear Leveling in Non-Volatile Main Memory Systems
Approach allows to save up to 85% of write overhead for wear-leveling in experiments.
October 2, 2020 | Press Release
Stac Report: kdb+ 4.0 With Intel Persistent Memory Pool Under Stac-M3
Solution sets records across scale and baseline benchmarks.
September 11, 2020 | Press Release