Production Configuration Guide
IMPORTANT
This guide is essential reading for deploying Blaze in production environments.
Recommended System Sizes
The following tables lists the minimum recommended system resources based on patient count, assuming approximately 1,000 resources per patient. Each configuration should be dedicated to a single Blaze instance. Avoid running other memory-intensive processes on the same system.
| # Patients | Cores | RAM | SSD | Heap Mem | Block Cache | DB Scale Factor | CQL Cache |
|---|---|---|---|---|---|---|---|
| 10 k | 2 | 8 GiB | 100 GB | 2 GiB | 2 GiB | 1 | 128 MiB |
| < 50 k | 4 | 16 GiB | 250 GB | 4 GiB | 4 GiB | 1 | 128 MiB |
| < 100 k | 4 | 32 GiB | 500 GB | 8 GiB | 8 GiB | 2 | 512 MiB |
| 100 k | 8 | 64 GiB | 1 TB | 16 GiB | 16 GiB | 2 | 512 MiB |
| 1 M | 16 | 128 GiB | 2 TB | 32 GiB | 32 GiB | 4 | 1 GiB |
| > 1 M | 32 | 256 GiB | 4 TB | 64 GiB | 64 GiB | 4 | 1 GiB |
Memory Allocation Strategy
In general, available RAM should be distributed as follows:
- 25% → JVM heap memory
- 25% → RocksDB block cache
- 50% → OS page cache (for RocksDB database file access)
IMPORTANT
Leave half of the available system memory "free" for the OS page cache
See System Memory
DB Scale Factor
Sizes of DB in-memory buffers (memtables), SST files, and internal data structures are scaled by the DB_SCALE_FACTOR environment variable. This affects key RocksDB parameters including write buffer size, target file size, and maximum bytes per level.
The default value of 1 is appropriate for small databases with less than 50k patients or 50M resources in total. For larger databases, you should increase this factor.
Impact of Higher Scale Factors:
- Memory: Increases memory usage outside the JVM heap
- Disk: Creates larger individual SST files
- File Count: Results in fewer total files in large databases
When to Increase: Consider increasing the scale factor when you encounter:
- Database size exceeding 50k patients or 50M resources
- File descriptor (
ulimit) problems due to too many SST files - Compaction performance issues with many small files
Configuration
The list of all environment variables can be found in the Environment Variables Section under Deployment. The variables important here are:
| Name | Use for | Default | Examples |
|---|---|---|---|
JAVA_TOOL_OPTIONS | Heap Mem | — | -Xmx2g, -Xmx4g, -Xmx8g, -Xmx16g, -Xmx32g or -Xmx64g |
DB_BLOCK_CACHE_SIZE | Block Cache | 128 | 2048, 4096, 8192, 16384, 32768 or 65536 (in megabyte) |
DB_SCALE_FACTOR | DB Buffers/File Sizes | 1 | 1, 2, 4, 8 or 16 |
CQL_EXPR_CACHE_SIZE | CQL Expression Cache | — | 128, 512, 1024 (in megabyte) |
DB_RESOURCE_STORE_KV_THREADS | Read-/Writing Resources | 4 | 16, 32, 64, 128 |
DB_RESOURCE_INDEXER_THREADS | Indexing Resources | 4 | 16, 32, 64, 128 |
Performance Benchmarks
Review performance metrics for properly configured systems:
Architecture Considerations
CPU Cores
Blaze performance scales directly with CPU core count (vertical scaling). Transaction and CQL query processing is parallelized even for single-client workloads, meaning more cores translate to better performance. Testing has confirmed efficient utilization of systems with up to 64 cores.
Memory
Blaze uses three main memory regions, namely the JVM heap memory, the RocksDB block cache, and the OS page cache.
- The JVM heap memory is used for general operation and specifically for the resource cache which caches resources in their ready to use object form and the resource handle cache which caches the resource handles of the newest versions of resources. By default, they take up to 25% and 10% of the heap.
- The RocksDB block cache is separate from the JVM heap memory because it's implemented as an off-heap memory region. The block cache stores resources and index segments in uncompressed blob form. After the resource cache, it functions as a second-level cache which stores resources in a more compact form and is not subject to JVM garbage collection.
- The OS page cache functions as a third level of caching because it contains all recently accessed database files. The database files are compressed and so store data in an even more compact form.
Heap and Block Cache Size
The split between JVM heap and RocksDB block cache depends on your workload. Start by monitoring JVM heap usage and garbage collection through Blaze's Prometheus metrics endpoint (available on port 8081 by default). As a starting point, divide the allocated RAM equally: 50% for the JVM heap and 50% for the RocksDB block cache.
OS Page Cache
We recommend reserving half of your system memory for the OS page cache. For instance, on a 16 GB system, allocate only 8 GB total between the JVM heap and RocksDB block cache, leaving the other 8 GB for the OS.
TIP
This "cache" memory might appear unused in typical monitoring tools, which might lead administrators to think it can be reclaimed or reallocated. However, it should not be reduced or allocated to other applications.
The operating system’s page cache can significantly improve performance, particularly on systems with limited disk I/O throughput. If your cache can hold the most actively accessed database files, you'll see significant gains. When all database files fit in memory, the system effectively operates as an in-memory database.
To see how much of the database files the page cache currently holds, run vmtouch on the host for the index, the transaction and the resource database directories, adjusting the paths to your data volume:
sudo vmtouch /path/to/data/index
sudo vmtouch /path/to/data/transaction
sudo vmtouch /path/to/data/resourceWithout options, vmtouch only reports and doesn't change the page cache. The line Resident Pages shows the cached part of all files in the directory, like 21G/675G 3.11%. If accessing many resources pushes the index out of the page cache, see RESOURCE_DB_DIRECT_READS.
Storage Requirements
TIP
Local NVMe SSD storage is recommended.
Blaze's architecture is optimized for the low-latency characteristics of local solid-state storage. RocksDB, the underlying storage engine, exploits the fast random access patterns that SSDs provide. Since local NVMe latency is typically lower than network round-trip time, network-attached storage offers no architectural advantage.
Legacy Storage: If only spinning disks are available, maximize the OS page cache allocation to compensate. However, expect higher latency, particularly for interactive operations.
Transactions
Blaze maintains indexes for FHIR search parameters and CQL evaluation. The indexing process can be executed in parallel if transactions contain more than one resource. Transactions containing only one resource and direct REST interactions like create/update/delete don't benefit from parallel indexing.
Depending on your system, indexing can be I/O or CPU-bound. Performance tests show that at least 64 CPU cores can be utilized for resource indexing.
Resource Store Write Concurrency
IMPORTANT
Set DB_RESOURCE_STORE_KV_THREADS to the maximum number of transactions you expect to be in flight at the same time.
DB_RESOURCE_STORE_KV_THREADS sizes the thread pool the resource store uses to read and write resources. Every transaction hands all its resources to that pool as a single unit, so the pool size caps how many transactions can write their resources concurrently. For a server answering HTTP requests, the number to size it by is the number of clients writing to it at the same time. The default of 4 is meant for small deployments.
A pool smaller than that concurrency costs throughput twice over. RocksDB merges the writers waiting at any moment into a group commit: their write-ahead log entries are written and synced together, so the cost of the fsync is shared by the whole group. A small pool keeps the groups small and so pays for the fsync far more often than necessary. On top of that, every client that finds no free thread has to wait for a full write of another client before its own even starts, which shows up as a step in the processing time as soon as the clients outnumber the threads.
This matters most on SSDs with slow syncs, where the fsync dominates the processing time of a write. In the transaction load test, A5N46 — a consumer NVMe SSD that acknowledges only 178 syncs per second — reaches over 4000 transactions/s at 128 concurrent clients with the pool sized to match, nearly twenty-three transactions per fsync, at a median processing time that stays flat at 29 ms over the whole sweep. Disks that sync from a write cache are far less sensitive, because their fsync is nearly free to begin with.
Sizing the pool generously is cheap. Its threads are created on demand and are blocked on disk I/O rather than on the CPU while they work, so a pool that is only occasionally filled costs next to nothing.
Resource Indexer Concurrency
IMPORTANT
Set DB_RESOURCE_INDEXER_THREADS to the number of cores of your system.
DB_RESOURCE_INDEXER_THREADS sizes the thread pool that indexes resources. Every resource of a transaction is indexed as one task on that pool, across transaction boundaries, so the pool size caps how many resources are indexed at the same time, no matter how many transactions they belong to.
Indexing a resource is mostly CPU work, so the default of 4 threads uses at most four cores for it, no matter how many cores the system has. That is fine for small deployments, but leaves most of a larger system idle while transactions queue in front of the pool. In the transaction load test, a system with 128 cores stays at about 8,000 transactions/s from 8 concurrent clients on with the default of 4 threads, and reaches about 19,600 transactions/s at 128 clients with 128 threads. Raising the pool further to 512 threads makes no measurable difference.