Zero-Downtime Migration: How We Moved a High-Load Search Infrastructure from Elastic Cloud to AWS OpenSearch Serverless
Key Takeaways
- Successfully migrated more than 100 million production records from Elastic Cloud to AWS OpenSearch Serverless without service interruption.
- Implemented a dual-write architecture that allowed the application to switch between search engines at any time with no risk of data loss.
- Developed a custom migration utility with automatic retries and controlled batch processing to minimize production impact.
- Solved multiple OpenSearch Serverless limitations, including the lack of Alias support and performance bottlenecks caused by uneven index distribution.
Migrating an enterprise-grade live search platform is never just about copying data from one system to another. It means keeping production fully operational while millions of records continue to change every day.
That was exactly the challenge our Customer faced when deciding to replace Elastic Cloud with AWS OpenSearch Serverless. The transition represented a fundamental shift from a node-managed, license-restricted ecosystem to an open-source, fully serverless architecture. The project required moving an extensive production dataset, adapting the application to a different search engine architecture, and ensuring that users experienced no downtime throughout the migration.
This case study explains how SaM Solutions’ engineers executed a zero-downtime migration strategy for a high-traffic customer support ticketing system and overcame several technical challenges unique to OpenSearch Serverless.
Need a trusted partner for secure, cloud-native development?
The Business Need
Our Customer provides a comprehensive ticketing platform used by businesses to manage customer support requests, knowledge bases, and internal wikis.
The platform’s search functionality was based on the Elastic Cloud search engine. The setup was as follows:
- More than 10 logical indexes for multiple supported languages, resulting in 25+ physical indexes.
- Over time, these indexes grew to approximately 100 million records in production.
- Strict multi-tenancy requirements: To prevent data leaks between different enterprise accounts, the architecture enforced complex data filtering via Elastic index aliases so users could only view records authorized for their specific organization.
To optimize operational expenses and infrastructure costs, the Customer chose to migrate to AWS OpenSearch Serverless. However, as an active B2B SaaS platform, downtime was not an option. Since new tickets and knowledge base entries were continuously created, updated and deleted, they needed a solution that would keep both search platforms synchronized until the final cutover.
Project Implementation
The core directive for SaM Solutions’ experts was clear: migrating live production data and ensuring the ability to switch between the two systems on the fly without data loss or user interruption.
Building a zero-downtime migration strategy
Executing a live migration on a system handling active users means the data target is constantly moving. The transition needed to support seamless rollback options at any stage without losing records created during the migration window.
To achieve this, SaM Solutions’ engineers used dual-writing via AWS Lambda. The Customer’s platform already had AWS Lambda functions for indexing data. However, there was a custom in-house library that supported writing only to Elastic Cloud. Our experts modified this library to execute parallel dual-writes, simultaneously piping live incoming data directly into both Elastic Cloud and AWS OpenSearch Serverless based on the configuration.
As a result, every newly created or updated document immediately became available in both systems, which means both search engines always contained the latest version of each document. Even if a record was migrated and then updated afterward, the change was instantly replicated to both systems, eliminating the risk of data inconsistencies during the migration process.
This dual-write approach allowed engineers to switch between search engines whenever necessary (or even roll back if unexpected issues occurred) without losing newly generated data.
Migrating historical data
With live changes covered, our team had to transfer the bulk historical data (100+ million production records).
To accomplish this, our experts developed a dedicated migration utility based on Hangfire, processing documents in batches of 2,500 records. Larger batches triggered OpenSearch Serverless throttling, making smaller controlled batches the optimal balance between throughput and stability.
The migration was executed during weekends to minimize production load. Our team created a separate migration job for each index, so multiple indexes could be migrated in parallel. This significantly reduced the overall migration time. Automatic retry mechanisms handled temporary failures, ensuring that transient infrastructure issues did not interrupt the migration process.
Verifying data integrity
Completing the migration was only part of the task. The team also needed to guarantee that every document had been transferred correctly.
Random production records created during the previous week were sampled and compared between Elastic Cloud and OpenSearch Serverless. In addition, the team developed a suite of integration tests that executed identical search queries against both platforms and automatically compared the returned results.
The validation was performed in two modes. The first compared results by position, verifying that both search engines returned documents in exactly the same order. For example, if Elastic Cloud returned records James, Jannet, and Jordan, while OpenSearch Serverless returned Jannet, Jordan, and James, the test was considered unsuccessful, even though the same documents were present.
The second mode ignored result order and compared only the returned data sets. In this case, the test passed as long as both systems returned the same documents, regardless of their ranking.
Together, these checks allowed the team to separate genuine data discrepancies from expected differences in search ranking and confirm that the migrated data was complete and consistent before the final cutover.
To further reduce migration risks, two dedicated testing environments were deployed. QA engineers could execute identical search scenarios against both search engines and compare results side by side.
Overcoming Technical Challenges
Although OpenSearch Serverless offers significant operational advantages, its architecture differs considerably from Elastic Cloud. Several implementation challenges required custom engineering solutions.
Re-engineering tenant isolation without Elastic alias support
In Elastic Cloud, tenant isolation was managed using index aliases with embedded filters, automatically blocking multi-tenant accounts from accessing each other’s sensitive records.
AWS OpenSearch Serverless does not support this feature. Leaving this unaddressed risked exposing cross-tenant data.
Our developers redesigned the application’s search query generation layer to build filtering conditions dynamically for every request. This way, they emulated the security behavior of traditional aliases without degrading search performance.
Resolving performance bottlenecks and OCU allocation
Another challenge emerged after the initial migration. One production index contained approximately 60 million documents, while the remaining indexes shared only 20–40 million records. This uneven distribution created performance bottlenecks despite additional OpenSearch Compute Units (OCUs) allocated by AWS. At the same time, the request volume generated by that largest index was not high enough to trigger AWS automatic scaling.
Working together with AWS engineers, our team evaluated several performance tuning approaches.
Ultimately, the team decided to reorganize the search architecture by moving the largest index into a separate OpenSearch collection. This provided greater flexibility for resource allocation and scaling.
The change required another controlled migration, followed by another round of validation and manual OCU tuning. After these adjustments, the search performance reached the required production level.
Adapting to search engine differences
The migration also revealed behavioral differences between Elastic Cloud and OpenSearch Serverless. One example involved exceptionally long search queries.
Unlike Elastic Cloud, OpenSearch Serverless enforces stricter query size limits, meaning certain complex requests could exceed supported thresholds. Addressing these limitations required additional query optimization, and further improvements are planned as the platform continues to evolve.
Results
The migration was completed successfully with no production downtime and no data loss.
Throughout the project:
- More than 100 million production records were migrated safely.
- Production indexing continued uninterrupted thanks to the dual-write synchronization mechanism.
- Historical data was transferred without affecting system availability.
- Extensive validation confirmed data consistency before the final cutover.
- Search functionality was fully adapted to AWS OpenSearch Serverless despite architectural differences.
The Customer was totally satisfied with the achieved results.
Summing Up
Large-scale search engine migrations are rarely straightforward, especially when production systems continue processing new data around the clock.
This project demonstrates that careful planning, incremental synchronization, and thorough validation can eliminate the risks traditionally associated with live infrastructure migrations. SaM Solutions’ team successfully moved a mission-critical search platform from Elastic Search to AWS OpenSearch Serverless without interrupting business operations. The transition was effectively invisible to end users. The platform remained fully operational during the entire migration process.
For organizations planning a similar transition, the biggest lesson is clear: successful migration means designing an architecture that allows both old and new systems to coexist safely until the very last switch.
Andrey Kopanev, Senior .NET Developer, AI Enthusiast
