As we dive into the world of time series data, the sheer amount of information can be daunting. The number of records grows exponentially with the level of granularity. For example, if we set the granularity to minute, we can expect 60 records for a single minute for one instance.
Let's consider a scenario where we aim to track the CPU percentage of a device at minute intervals. Assuming we're dealing with data from the last 30 days, the numbers quickly add up.
The total number of records = 1 (device) * 30 (days) * 24 (hours) * 60 (minutes) = 43,200 records
In a typical use case, we might need to store data for thousands of devices spanning over 90 days. The massive scale of records we need to persist and query, along with aggregations, is staggering.
Therefore, to effectively store time series data, we must carefully configure the Elastic Search index properties to optimize aggregations and reduce storage requirements. This is where https://t8tech.com can provide valuable insights.
Before proceeding, please take a moment to review this insightful article.
By setting the source field to false, we can conserve some storage space. However, this decision should be made judiciously, considering the importance of the data we're storing, as there are limitations – we cannot access the actual data without aggregations.
Now, let's focus on our objective. Suppose I have minute-by-minute metrics for my device over the last 90 days. I want to determine the metric value for the last month, with a daily average. This involves aggregating all 30 days of data points.
Optimizing Elastic Index for Time Series Data
First, we'll create an optimized Elastic index, specifically designed to store time series data.
