Frozen tier queries: Search S3 without rehydration

Elastic's frozen tier cuts storage costs by up to 90 percent compared to hot or warm tiers while maintaining searchability. This architecture flips long-term retention on its head: compute decouples from storage, letting you query massive datasets directly from object stores like AWS S3 without local rehydration. Traditional archival methods force a choice between price and accessibility. The frozen tier refuses that trade-off.

You get direct querying of data sitting in low-cost classes like S3 Glacier Deep Archive via searchable snapshots. No more waiting hours for retrieval. Index Lifecycle Management (ILM) handles the heavy lifting, automating the shuffle from hot to warm to cold before hitting the frozen state. It's a complete lifecycle, not a patchwork of manual exports.

While standard archives nap for hours during retrieval, this method leverages S3 Glacier Instant Retrieval for millisecond access. The cold tier still clings to local primary data; the frozen tier drops it entirely. The result? IT teams keep unlimited lookback for compliance and trend analysis without burning budget on high-performance local disks that sit idle 99% of the time.

The Role of the Frozen Tier in Cost-Effective Data Retention

Frozen Tier Definition: Decoupled Compute and Searchable Snapshots

On March 23, 2021, Elastic 7.12 dropped the frozen tier, and it changed the game by completely decoupling compute from storage. You query data sitting in AWS S3, Microsoft Azure Storage, or Google Cloud Storage directly. No local rehydration. Unlike the cold tier, which keeps a local primary copy, the frozen tier stores indices exclusively as searchable snapshots in remote repositories. A tiny local cache holds recently queried segments to speed up repeat access, but the bulk of the data lives remotely. Organizations see up to 90% storage savings over hot tiers by moving infrequently accessed data to these backends.

Migrate to the frozen tier when your security team needs unlimited historical lookback without manual bottlenecks. Security operations centers drown in data; keeping years of logs on local hardware for forensics is financially suicidal. Searchable snapshots let you query those massive datasets while paying pennies. It's perfect for deep compliance audits in medical or financial sectors where records must sit for years yet remain instantly accessible. Kibana dashboards keep working against frozen indices without the complex rehydration workflows legacy archives demand.

APM teams gain the same advantage, drilling into years of trend data to spot slow-burn degradation. You stop trading retention duration for search capability. Engineers correlate current incidents with historical baselines instantly. While AWS S3 Glacier Deep Archive targets data accessed once or twice a year, the frozen tier is built exactly for that profile within the Elastic stack.

Don't ignore the latency tax. Querying remote data is slower than local disk.rabata.io recommends setting Index Lifecycle Management policies to shift aged data automatically based on age or size. This keeps costs down while preserving the ability to run complex aggregations across multi-year datasets.

Cold Tier vs Frozen Tier: Local Primary Data Versus Object Store Snapshots

The cold tier keeps local primary data. The frozen tier deletes it. This isn't a minor tweak; it fundamentally alters how clusters handle redundant copies and density. Cold tier users can store twice the data on the same hardware by removing local redundancy, but primary segments still eat node disks. Frozen architecture relies exclusively on searchable snapshots stored remotely, needing only a fractional cache for active queries.

Latency versus density. That's the choice. Data destined for frozen storage fits long-term archive patterns: rare access, but it must be immediate. AWS S3 Glacier Deep Archive targets data accessed once or twice yearly, matching the frozen index profile perfectly. But remote storage means network dependency. If the object store hiccups, frozen queries stall. Cold tier searches keep running if local data persists.

Here's the rub: maximizing density shrinks local failure domains but spikes reliance on external SLAs. Rabata.io suggests this move for datasets blowing past standard retention windows where query volume crashes. You stop planning for disk space and start planning for network throughput.

Architecture of Direct S3 Search via Searchable Snapshots

Mechanics: Frozen Tier Architecture: Local Cache and Object Store Decoupling

The frozen tier kills the local copy to achieve radical density. Indices live entirely in an object store like S3 via searchable snapshots; compute nodes fetch only the segments they need. Look at the scaling: a warm node manages 10 TB, but a frozen node handles 100 TB. That's a 1:1500 RAM-to-storage ratio, a conservative estimate that slashes hardware footprints.

Cloud storage tiers scale the backend to petabytes without dragging down hotter layers. Local RAM stays static even as data volume explodes. But don't expect local-disk speeds. Initial queries hitting cold object storage introduce latency variance. Network throughput becomes your bottleneck, not disk IOPS. Bandwidth provisioning matters now.rabata.io recommends this topology for AI/ML training data where full dataset locality is unnecessary.

Asynchronous Search Execution and Block-Max WAND Optimization

Elasticsearch uses asynchronous search to render Kibana dashboards in the background, masking object store latency. This stops user-facing timeouts during complex queries on massive datasets. Text searches use block-max WAND, an optimization skipping entire document blocks when their max possible score can't beat current top results. Searchable snapshots pull only necessary index subsets, minimizing network transfer.

Fix frozen tier performance issues by leaning on these selective retrieval methods, not full rehydration. Tools like Storage Class Analysis identify cold data candidates before you write policies. Yes, initial query latency is higher than local disk. Real-time alerting on frozen data is impractical without serious tuning.

This shifts retention strategy design:

• Queries scan concise index structures, not raw blobs.
• Background execution lets heavy aggregation jobs finish without blocking the UI.
• Network egress costs drop because the engine fetches minimal segments.

Rabata.io recommends setting asynchronous thresholds based on concurrent user loads to prevent starvation. Interactive debugging might feel sluggish if the underlying S3 bucket lacks throughput.

Storage Density Scaling: 100 TB Frozen Nodes Versus 10 TB Warm Nodes

A single frozen tier node supports 100 TB of data. A standard warm node using identical 64 GB RAM configurations tops out at 10 TB. That's a tenfold density jump because the architecture eliminates local data copies, relying on searchable snapshots in remote repositories. The local cache holds a fraction of the total dataset, letting operators manage massive volumes with minimal hardware.

This density shift breaks traditional failure domain math. A single node outage now impacts ten times more data than in warm tiers. Millisecond retrieval is possible with instant access tiers, but local disk I/O bottlenecks just move to the network interface. Storage Class Analysis tools spot candidates for this tier by detecting access patterns that tolerate slightly higher latency.rabata.io recommends validating network bandwidth before scaling frozen nodes to avoid saturating links during broad queries.

Configuring Index Lifecycle Management for Frozen Migration

ILM Policy Mechanics for Frozen Tier Migration

Defining ILM policy phase actions triggers the shift from local warm nodes to remote object storage. The configuration removes the local copy, moving primary data to S3-compatible repositories while keeping indices searchable. Once the frozen action executes, the cluster depends on a local cache holding only a fraction of the dataset to accelerate recent queries. Clusters using this architecture manage notably larger historical data volumes per node. Such density lets organizations cut storage costs substantially compared to the standard cold tier. Tools like AWS S3 Intelligent-Tiering complement this by shifting data between access tiers based on frequency, though Elasticsearch ILM explicitly controls the index state transition. The following configuration demonstrates the specific JSON structure required to automate this lifecycle shift:

Maximizing storage density via the frozen tier inherently increases latency for random access patterns compared to local disk. It's designed for occasional searches, not high-speed ops. The frozen tier uses optimizations like asynchronous search and block-max WAND to mitigate slower speeds. Searchable snapshots use Lucene expertise to pull down only the index subsets needed to answer a query.

Validating S3-Compatible Repositories with Elastic Test Kit

Elastic officially supports AWS S3, Azure Cloud Storage, Google Cloud Storage, and the provider, yet many enterprises try alternative S3-compatible backends. The platform provides a repository test kit available as an easily consumable API to address compatibility verification. Administrators should execute this validation workflow prior to production deployment:

1. Deploy the test kit container within the cluster network boundary.
2. Target the specific bucket intended for long-term retention.
3. Analyze the output for latency spikes or multipart upload failures.

Passing these tests confirms basic connectivity but does not guarantee performance parity with native endpoints. Successful validation does not equate to official support if issues arise later. Engineering teams must reproduce any anomalies on supported devices to engage vendor assistance effectively. Searchable snapshots rely heavily on consistent object retrieval speeds. Real-time search demands low-latency access patterns that some clones struggle to maintain consistently, unlike batch-oriented lifecycle policies found in standard cloud storage. The frozen tier eliminates the local copy altogether, yet the underlying storage must handle concurrent read requests without throttling. Strategic validation prevents cost optimization efforts from compromising operational reliability during critical investigations.

Support Limitations for Unvalidated Object Stores

Passing the repository validation test kit does not grant official support status for arbitrary S3-compatible backends. Elastic explicitly limits supported devices to AWS S3, Azure Cloud Storage, Google Cloud Storage, and the provider, leaving other validated stores in a conditional state. If a failure occurs on an unvalidated provider, the issue must be reproducible on one of these supported platforms before engineers will address the defect. This constraint creates a specific operational risk for teams using cost-optimized storage class tiering on niche object stores. The validation API confirms basic compatibility, but it does not verify edge-case behavior under heavy frozen tier load. Troubleshooting becomes speculative when searchable snapshots fail to initialize correctly without this redundancy. Validation proves connectivity, not reliability. Teams must accept that unique features of their chosen object store may remain unsupported if they cannot demonstrate the same failure on approved hardware. This policy ensures stability but demands that architects plan for potential migration if their preferred low-cost storage exhibits undiscovered incompatibilities.

Strategic Value of Decoupled Compute and Storage Architectures

Application: Decoupled Compute Mechanics in Frozen Tier Architecture

Elasticsearch reads index segments directly from remote object storage when the frozen tier removes local copies entirely. Data stays immediately searchable in Kibana without manual rehydration steps or restoration delays. A small local cache accelerates repeat queries against identical data subsets. Cost models distinguish "hot" application data from "cold" or "frozen" archives, where the latter accepts specific access constraints for notably lower storage rates. One node managing warm configurations addresses a much larger volume in the frozen tier, slashing the required compute footprint. This density introduces latency because initial queries on cold data paths lag behind local disk access speeds. Operators balance retention depth against query responsiveness by tuning Index Lifecycle Governance policies with care. The decoupled model turns storage from a fixed hardware constraint into a scalable utility for long-term analytics.

Real-World Scenarios: Security Lookback and APM Trend Analysis

Observability, security, and enterprise search generate IT data at an exponential rate that pressures storage budgets immediately. Unlimited lookback for security investigations becomes feasible without the prohibitive costs of high-performance hardware maintenance. Drilling into years of APM data aids trend identification while keeping historical records accessible. Query latency remains the primary limitation since frozen data suits occasional searches rather than real-time dashboards. Asynchronous search configurations maintain user experience despite slower response times for cold data access. Organizations retain critical historical context without sacrificing financial efficiency or requiring manual data rehydration processes.

Standard AWS S3 Glacier retrieval times range from minutes to hours, creating unavoidable latency gaps during incident response. Searchable snapshots eliminate this delay by allowing Elasticsearch to query object storage directly without manual restoration steps. Traditional workflows force operators to initiate restore requests, wait for system rehydration into temporary buckets, and remap indices before analysis begins. Security investigations stall when immediate access to historical logs is required. The frozen tier maintains data in a queryable state so analysts run complex aggregations instantly. Standard archives prioritize storage density over availability whereas searchable snapshots optimize for immediate readability.

Deep archives feature infrequent access patterns that operators must weigh against the operational risk of delayed insights. S3 Glacier Deep Archive offers minimal storage costs yet the inability to instantly validate a hypothesis proves costly during a breach. Waiting for data restoration extends the window of vulnerability during an active incident. Direct S3 search transforms long-term retention from a passive compliance exercise into an active investigative tool. Financial savings matter less than temporal efficiency when responding to active threats. The constraint is time rather than money in these critical moments.

References

• Amazon S3 - Cloud Object Storage - AWS: Amazon
• S3 Glacier Flexible Retrieval: Cheaper (~$0.0036/GB) for long-term archives