Cost Effective In-Memory Analytics at Scale: Using DuckDB as Application-Layer Offload

Serving high concurrency, low-latency analytical queries typically points toward a data warehouse such as Athena, Redshift or Snowflake. These solutions work well but whilst they are “interactive” speed they aren’t near real-time. Also, their costs scale steeply, they introduce operational complexity and requires additional skills. This is often untenable for small teams or cost-conscious organisations. This session explores an alternative pattern: embedding the analytics query engine and data in the same process as the application to deliver low latency, high concurrency, autoscaling, resilient and low-cost analytical queries with a few lines of familiar code and commodity hardware.

This session is for both application developers and data engineers. Whilst it will use AWS it is equally valid on other clouds or on-premises.

In this session we will discuss the pattern, show the code, show it in action and share some measurements. We will address some limitations and sign-post solutions.

Using mature data science tools in high performance realtime applications Establishing in-memory analytics as an application-layer offload pattern Identify alternative patterns to traditional high-end data warehouse systems How "boring" technology on commodity hardware goes an incredibly long way and why this is perfect for small teams and resource-constrained organisations