KQL databases are a powerful way to analyze large amounts of data in real-time, using the Kusto Query Language (KQL). However, managing the storage costs and fees for KQL databases can be challenging, especially when dealing with different types of databases and varying data usage patterns. In this post, we will explore the following topics:
- The different types of KQL databases and when are they needed
- The factors that affect the storage costs and fees for KQL databases
- The best practices to maintain costs and budget for KQL databases under control
Types of KQL Databases
A KQL database is a logical container for data that is stored in OneLake, a unified data lake that supports multiple analytics workloads in Microsoft Fabric. A KQL database can be configured to use one of the three types of storage tiers: hot, warm, or cold. Each storage tier has different characteristics in terms of performance, availability, and cost. The following table summarizes the main differences between the storage tiers1:
|
Storage tier |
Performance |
Availability |
Cost |
|---|---|---|---|
|
Hot |
High |
High |
High |
|
Warm |
Medium |
Medium |
Medium |
|
Cold |
Low |
Low |
Low |
The storage tier of a KQL database determines how the data is stored in OneLake. Data in the hot tier is stored in both OneLake Cache Storage and OneLake Standard Storage. OneLake Cache Storage is a premium storage that provides fast query response times, while OneLake Standard Storage is a persistent storage that ensures data durability. Data in the warm tier is stored only in OneLake Standard Storage, but with a higher replication factor than the cold tier. Data in the cold tier is stored only in OneLake Standard Storage, but with a lower replication factor than the warm tier.
The storage tier of a KQL database also affects the compute resources that are allocated to the database. A KQL database uses an autoscale mechanism to adjust the number of virtual cores (v-cores) that are used by the database, based on the data usage pattern. The autoscale mechanism ensures cost and performance optimization for the database. However, the storage tier of the database sets the minimum and maximum number of v-cores that can be used by the database. The following table shows the default v-core limits for each storage tier2:
|
Storage tier |
Minimum v-cores |
Maximum v-cores |
|---|---|---|
|
Hot |
4 |
128 |
|
Warm |
2 |
64 |
|
Cold |
1 |
32 |
The choice of the storage tier for a KQL database depends on the data usage scenario and the business requirements. Generally, the hot tier is suitable for data that is frequently accessed and requires high performance and availability. The warm tier is suitable for data that is occasionally accessed and requires moderate performance and availability. The cold tier is suitable for data that is rarely accessed and requires low performance and availability.
For example, a KQL database that stores real-time sensor data for monitoring and alerting purposes may use the hot tier, while a KQL database that stores historical data for archival and compliance purposes may use the cold tier.
Factors Affecting Storage Costs and Fees for KQL Databases
The storage costs and fees for KQL databases are determined by several factors, such as the amount of data stored, the storage tier, the data retention policy, the data compression ratio, the data ingestion rate, and the data query rate. The following sections explain how each factor affects the storage costs and fees for KQL databases.
Amount of Data
The amount of data stored in a KQL database is the primary factor that affects the storage costs and fees for the database. The more data is stored, the higher the storage costs and fees. The storage costs and fees for a KQL database are calculated based on the amount of data stored in OneLake Cache Storage and OneLake Standard Storage, which are billed separately from the Fabric capacity units. The following table shows the pay-as-you-go rates for OneLake Cache Storage and OneLake Standard Storage3:
|
Storage type |
Rate |
|---|---|
|
OneLake Cache Storage |
$0.15 per GB per month |
|
OneLake Standard Storage |
$0.02 per GB per month |
Storage Tier
The storage tier of a KQL database affects the storage costs and fees for the database in two ways. First, the storage tier determines the amount of data stored in OneLake Cache Storage and OneLake Standard Storage. As mentioned earlier, data in the hot tier is stored in both OneLake Cache Storage and OneLake Standard Storage, while data in the warm and cold tiers is stored only in OneLake Standard Storage. Therefore, the hot tier has higher storage costs and fees than the warm and cold tiers.
Second, the storage tier determines the replication factor of the data stored in OneLake Standard Storage. The replication factor is the number of copies of the data that are stored across different regions for data durability and availability. The warm tier has a higher replication factor than the cold tier, which means that the warm tier has higher storage costs and fees than the cold tier. The following table shows the default replication factors for each storage tier4:
|
Storage tier |
Replication factor |
|---|---|
|
Hot |
3 |
|
Warm |
2 |
|
Cold |
1 |
Data Retention Policy
The data retention policy of a KQL database affects the storage costs and fees for the database by controlling how long the data is stored in the database. The data retention policy can be set at the database level or the table level, and it can be specified in terms of days or size. The data retention policy deletes the data that is older than the specified period or exceeds the specified size, which reduces the storage costs and fees for the database. However, the data retention policy also affects the data availability and usability for the database, so it should be carefully chosen based on the business needs and compliance requirements.
Data Compression Ratio
The data compression ratio of a KQL database affects the storage costs and fees for the database by reducing the amount of data stored in the database. The data compression ratio is the ratio between the original size of the data and the compressed size of the data. The data compression ratio depends on the data type, the data format, and the compression algorithm used by the database. The data compression ratio can vary from 1:1 (no compression) to 10:1 (high compression) or more. The higher the data compression ratio, the lower the storage costs and fees for the database. However, the data compression ratio also affects the data ingestion and query performance for the database, as it requires more CPU and memory resources to compress and decompress the data.
Data Ingestion Rate
The data ingestion rate of a KQL database affects the storage costs and fees for the database by increasing the amount of data stored in the database. The data ingestion rate is the rate at which the data is ingested into the database, either from streaming sources or batch sources. The data ingestion rate can vary from a few KB per second to a few GB per second or more. The higher the data ingestion rate, the higher the storage costs and fees for the database. However, the data ingestion rate also affects the data freshness and timeliness for the database, as it enables the database to capture and analyze the data in real time.
Data Querey Rate
The data query rate of a KQL database affects the storage costs and fees for the database by consuming the compute resources that are allocated to the database. The data query rate is the rate at which the data is queried from the database, either by interactive users or automated applications. The data query rate can vary from a few queries per hour to a few queries per second or more. The higher the data query rate, the higher the compute costs and fees for the database. However, the data query rate also affects the data value and insight for the database, as it enables the database to provide answers and solutions to the data users.
Best Practices to KQL Database Maintain Costs
The storage costs and fees for KQL databases can be optimized by following some best practices, such as:
- Choosing the right storage tier for the data usage scenario and the business requirements
- Setting the appropriate data retention policy for the data availability and usability needs
- Using the materialize () function to cache the results of frequently used queries and reduce the data processing load
- Using the summarize operator to aggregate and group the data and reduce the data size
- Using the project operator to select only the relevant columns and reduce the data size
- Using the has operator instead of the contains operator to search for full tokens and reduce the data scanning load
- Using the == operator instead of the =~ operator to perform case-sensitive comparisons and reduce the data scanning load
- Using the limit operator to limit the number of rows returned by the query and reduce the data transfer load
- Monitoring the KustoUpTime metric to track the compute usage of the database and adjust the v-core limits if needed
- Monitoring the OneLake Read and Write metrics to track the data transactions of the database and optimize the data ingestion and query patterns
- Monitoring the OneLake Cache Storage and OneLake Standard Storage metrics to track the data storage of the database and optimize the data compression and deletion policies
By following these best practices, you can optimize the storage costs and fees for KQL databases and get the most out of your data analytics in Microsoft Fabric.
The post KQL Databases: How to Optimize Storage Costs and Fees appeared first on Dynamics Communities.