3.1 Gzip (GNU zip)

Gzip uses the DEFLATE algorithm, which balances speed and compression ratio. It is widely supported across different platforms and programming languages, making it effective for both text and binary data. Its primary strength lies in providing a good compromise between compression speed and ratio, making it a popular choice for general-purpose compression. However, Gzip is not the fastest in terms of compression or decompression speeds, and its compression ratios are moderate compared to newer algorithms. It is ideal for general-purpose compression where compatibility and ease of use are important, such as web servers for compressing HTTP responses and archiving files where space saving is needed but not critical.

3.2 Snappy

Snappy is optimized for speed, making it ideal for applications requiring rapid read/write operations. It excels in scenarios where quick compression and decompression are more critical than achieving the highest compression ratio. Snappy’s primary strength is its high speed, both in compression and decompression, which minimizes latency in data processing. However, this comes at the cost of lower compression ratios compared to other algorithms. Snappy is best suited for real-time data processing, log compression, and systems where performance is a higher priority than storage efficiency.

3.3 Zstandard (Zstd)

Zstandard, or Zstd, offers high compression ratios and fast decompression, making it excellent for scenarios that need high compression without significant speed loss. Zstd is highly versatile, providing various compression levels to balance speed and compression ratio according to the application’s needs. Its strengths include superior compression ratios and efficient decompression speeds, often outperforming older algorithms like Gzip. One downside is that higher compression levels can be resource-intensive in terms of CPU and memory usage. Zstd is ideal for large-scale data storage, backup solutions, and systems where both speed and storage efficiency are critical.

3.4 Brotli

Brotli achieves high compression ratios, especially for text data, making it perfect for web content. Developed by Google, Brotli provides excellent compression efficiency for web fonts and other web assets, significantly reducing file sizes and improving load times. Its primary strength lies in its ability to achieve superior compression ratios for text data, which is crucial for optimizing web performance. However, Brotli can be slower than other algorithms in both compression and decompression, particularly at higher compression levels. Brotli is best used for web content delivery, where minimizing bandwidth usage and improving load times are paramount.

3.5 LZ4

LZ4 prioritizes speed over compression ratio, making it ideal for real-time data processing. It is designed to provide extremely fast compression and decompression, which is critical in performance-sensitive environments. LZ4’s strengths include its lightning-fast speeds, which make it suitable for scenarios where data needs to be processed in real-time. However, its compression ratios are lower than those of more advanced algorithms like Zstd or Brotli. LZ4 is particularly useful for in-memory compression, real-time logging, and applications where speed is more important than achieving the highest compression ratio.

3.6 Uncompressed

Storing data in its original format without compression ensures the fastest possible read and write speeds, as there is no overhead from compression or decompression processes. This approach is best when speed is crucial, and storage space is not a concern. The primary advantage of uncompressed data is the elimination of latency related to compression algorithms, providing immediate access to data. However, the downside is the significantly larger storage requirement. Uncompressed storage is ideal for applications where performance is paramount and storage costs are negligible, such as temporary data storage and real-time data analytics.

By understanding these compression techniques’ specific advantages and limitations, you can make informed decisions to optimize data storage and access for various applications.

5.1 Device specifications

13th‑Gen Intel Core i7‑13620H 32 GB RAM - Windows 64‑bit

5.2 Dataset generation

We benchmark row counts 10^6, and three structures (numeric, character, mixed). The helper below fabricates a mixed‑type frame:

knitr::opts_chunk$set(echo = TRUE, cache = TRUE)
source("utils.R")

generate_sample_data <- function(n, type = "mixed"){
  set.seed(123)
  if(type == "numeric"){
    data.frame(ID = seq_len(n), Value = rnorm(n))
  } else if(type == "character"){
    data.frame(ID = seq_len(n),
               Description = replicate(n, paste0(sample(letters,20,TRUE), 
                                                 collapse="")))
  } else {
    data.frame(
      ID          = seq_len(n),
      Value       = rnorm(n),
      Category    = sample(letters[1:5], n, TRUE),
      Description = replicate(n, paste0(sample(letters,20,TRUE), collapse=""))
    )
  }
}

sample_size <- 1e6
dt_list <- list(
  numeric   = as.data.table(generate_sample_data(sample_size, "numeric")),
  character = as.data.table(generate_sample_data(sample_size, "character")),
  mixed     = as.data.table(generate_sample_data(sample_size, "mixed"))
)Copy

7.1 Summarising

pretty_method <- c(
  fst_uncompressed      = "fst (uncompressed)",
  qs_fast               = "qs (preset = 'fast')",
  parquet_snappy        = "Parquet + Snappy",
  parquet_zstd          = "Parquet + Zstd",
  parquet_uncompressed  = "Parquet (uncompressed)",
  parquet_gzip          = "Parquet + Gzip",
  dt_none               = "CSV via data.table (none)",
  dt_gzip               = "CSV via data.table (gzip)"
)

write_s <- write_summary_all %>%
  dplyr::mutate(method = method_from_expr(expr)) %>%
  select(structure, method,
         write_median_ms = median,
         write_mean_ms   = mean,
         write_sd_ms     = sd)

read_s <- read_summary_all %>%
  dplyr::mutate(method = method_from_expr(expr)) %>%
  select(structure, method,
         read_median_ms = median,
         read_mean_ms   = mean,
         read_sd_ms     = sd)

cons <- write_s %>%
  inner_join(read_s, by = c("structure","method")) %>%
  left_join(size_summary_all %>%
              select(structure, method, size_MB, relative_size),
            by = c("structure","method")) %>%
  dplyr::mutate(Method = recode(method, !!!pretty_method)) %>%
  arrange(structure, write_median_ms)

unique(cons$structure) %>%
  walk(~ print_per_structure(cons, .x))

write_wide <- cons %>%
  select(structure, Method, `Write (ms)` = write_median_ms) %>%
  pivot_wider(names_from = structure, values_from = `Write (ms)`) %>%
  arrange(`mixed`)  # or order by 'numeric' etc.

knitr::kable(write_wide, digits = 1,
             caption = "Median write time (ms) by storage method and data structure")Copy

# Wide table: read medians
read_wide <- cons %>%
  select(structure, Method, `Read (ms)` = read_median_ms) %>%
  pivot_wider(names_from = structure, values_from = `Read (ms)`) %>%
  arrange(`mixed`)
knitr::kable(read_wide, digits = 1,
             caption = "Median read time (ms) by storage method and data structure")Copy

# Wide table: sizes
size_wide <- cons %>%
  select(structure, Method, `Size (MB)` = size_MB) %>%
  pivot_wider(names_from = structure, values_from = `Size (MB)`)
knitr::kable(size_wide, digits = 2,
             caption = "File size (MB) by method and structure")Copy

method_from_expr <- function(expr) sub("^write_|^read_", "", expr)

# Prepare write and read with ratios
write_s <- write_summary_all %>%
  mutate(method = method_from_expr(expr)) %>%
  select(structure, method, write_median_ms = median) %>%
  normalise_ratio("write_median_ms")

read_s <- read_summary_all %>%
  mutate(method = method_from_expr(expr)) %>%
  select(structure, method, read_median_ms = median) %>%
  normalise_ratio("read_median_ms")

# Prepare sizes with ratio to smallest
size_s <- size_summary_all %>%
  normalise_ratio("size_MB") %>%
  select(structure, method, size_MB, size_MB_ratio)

# Join all
consolidated_ratio <- write_s %>%
  inner_join(read_s,  by = c("structure", "method")) %>%
  left_join(size_s,   by = c("structure", "method")) %>%
  mutate(Method = recode(method, !!!pretty_method)) %>%
  arrange(structure, write_median_ms)

knitr::kable(
  consolidated_ratio %>%
    select(structure, Method,
           `Write (ms)` = write_median_ms,
           `Write (×)`  = write_median_ms_ratio,
           `Read (ms)`  = read_median_ms,
           `Read (×)`   = read_median_ms_ratio,
           `Size (MB)`  = size_MB,
           `Size (×)`   = size_MB_ratio),
  digits = 2,
  caption = "Median write/read times and file size with ratios (from fastest/smallest) by method and structure"
)Copy

7.2 Throughput plot

# Plot labels
plot_df <- bind_rows(
  dplyr::mutate(write_summary_all, phase = "write"),
  dplyr::mutate(read_summary_all,  phase = "read")
) %>%
  dplyr::mutate(method = sub("^(write|read)_", "", expr),
         Method = dplyr::recode(method, !!!pretty_method))

unique(plot_df$structure) %>%
  walk(~ print(make_structure_plot(.x, log_scale = FALSE)))Copy

8.1 Interpretation

• Write path
  • Uncompressed fst is one of the fastest writer across all structures.
    
  • qs (preset = "fast") is consistently the next best, offering strong throughput with modest size reduction.
    
  • data.table is equivalent to fst in mixed scenario.
  • Parquet with Gzip is the slowest writer by a large margin in all cases.
• Read path
  • For numeric data, uncompressed fst is the fastest reader.
  • For character and mixed data, Parquet (uncompressed) yields the lowest read latency, with Snappy and Zstd close behind; these formats benefit from reduced on-disk size that offsets decompression overhead.
    
  • CSV + Gzip is consistently the slowest reader, particularly for mixed data.
• File size
  • Zstd and Gzip deliver the smallest files across structures.
    
  • Snappy and qs strike a practical balance, with much smaller than uncompressed, and materially better write/read performance than Gzip.
    
  • CSV is largest on disk; CSV + Gzip is smaller but slow to read.

8.2 Practical perspective

• High-frequency writes should prefer uncompressed fst (numeric-heavy) or qs (mixed/character) to maximise throughput.
  
• Read-optimised pipelines should prefer Parquet (uncompressed) for minimum latency, or Parquet + Snappy/Zstd when storage reduction matters with minimal read-time penalty.
  
• Archival or size-constrained storage should favour Parquet + Zstd (or Gzip when maximum compression is required and slower I/O is acceptable).