Fast Data is an emerging industry term for information that is arriving at high volume and incredible rates, faster than traditional databases can manage. Three years ago, as part of our AWS Fast Data journey we introduced Amazon ElastiCache for Redis, a fully managed in-memory data store that operates at sub-millisecond latency. Since then we’ve introduced Amazon Kinesis for real-time streaming data, AWS Lambda for serverless processing, Apache Spark analytics on EMR, and Amazon QuickSight for high performance Business Intelligence.

While caching continues to be a dominant use of ElastiCache for Redis, we see customers increasingly use it as an in-memory NoSQL database. Developers love the blazing fast performance and in-memory capabilities provided by Redis, making it among the most popular NoSQL key-value stores. However, until now ElastiCache for Redis customers could only run single-shard Redis. This limited the workload size and write throughput to that of a single VM, or required application level sharding. Today, as a next step in our Fast Data journey, we have extended the ElastiCache for Redis service to support “Redis Cluster,” the sharding capability of Redis. Customers can now scale a single deployment to include up to 15 shards, making each Redis-compatible data store up to 3.5 terabytes in size, that operate on microsecond time scales. We also do this at very high rates: up to 4.5 million writes per second and 20 million reads per second. Each shard can include up to five read replicas to ensure high availability so that both planned and unforeseen outages of the infrastructure do not cause application outages.

Building upon Redis

There are some great examples and use cases for Redis, which you can see at companies like Hudl, which offers mobile and desktop video analytics solutions to sports teams and athletes. Hudl is using ElastiCache for Redis to provide millions of coaches and sports analysts with near real-time data feeds that they need to help drive their teams to victory. Another example is Trimble, a global leader in location services who is using ElastiCache for Redis as their primary database for workforce location, helping customers like DirecTV get the right technician to the right location as quickly and inexpensively as possible, enabling both reduced costs and increased satisfaction for their own subscribers.

Increasingly, ElastiCache for Redis has become a mission critical in-memory database for our customers whose availability, durability, performance and scale matter to their business. We have therefore been enhancing the Redis engine running on ElastiCache for the last few years using our own expertise in making enterprise infrastructure scalable and reliable. Amazon’s enhancements address many day-to-day challenges with running Redis. By utilizing techniques such as granular memory management, dynamic I/O throttling and fine grained replica synchronization, ElastiCache for Redis delivers a more robust Redis experience. It enables customers to run their Redis nodes at higher memory utilization without risking swap usage during events such as snapshotting and replica synchronization. It also offers improved synchronization of replicas under load. In addition, ElastiCache for Redis provides smoother Redis failovers by combining our Multi-AZ automated failover with streamlined synchronization of read replicas. Replicas now recover faster as they no longer need to flush their data to do a full resynchronization with the primary. All these capabilities are available to customers at no additional charge, and maintain open-source Redis compatibility.

With this launch, we augmented the client-based failover logic of Redis 3.2 with ElastiCache for Redis Multi-AZ. If a customer is running a self-managed Redis environment on EC2 instead of using ElastiCache for Redis and the primary node fails, the cluster relies on a majority of primaries to determine and execute a failover. If such a majority doesn’t exist, the cluster will go into failed state, rejecting any further reads and writes. This could lead to a major availability impact on the application, requiring human intervention to manually salvage the cluster. This does not happen with ElastiCache for Redis. ElastiCache for Redis Multi-AZ capability is built to handle any failover case for Redis Cluster with robustness and efficiency. The combination of ElastiCache for Redis with the intelligent Redis 3 clients leads to maximum performance and availability of your Redis environment. The client keeps a map of Redis nodes, which is updated in case of failover. This allows for faster failover times while minimizing latency. Alternative solutions frequently use proxy layers to achieve failover and sharding, which slow down your application by requiring requests to do double the network hops.

Redis and Fast Data

Fast data can have a transformational impact on data-driven development that is underlying a lot of the interesting things happening in the Cloud today. This enables “here-and-now” real-time processing and dashboards as well as predictions that enable smart applications. As data sizes grow and expectations move from analytics on a daily basis to analytics on a real-time basis, the need to process data quickly increases. With the latest enhancement we have made to ElastiCache for Redis, we are excited to help these customers with a more robust, high performance, highly scalable in-memory database solution.

Many of our customers share my excitement:

Interactive Intelligence, Inc. is a software company providing unified business communications solutions for call centers, including real-time reporting and analytics. “We have been eagerly awaiting ElastiCache for Redis support for Redis Cluster, and are excited to take advantage of it for easy to setup redundancy, fast failure recovery, and ultra-high scalability” said Anthony Roach, Chief Architect. “We are heavy users of ElastiCache for Redis for both caching and fast data structure storage due to its ease of management and reliability, and the addition of Redis Cluster makes it even more compelling.”

Team Internet AG is an ad tech company with a focus on domain monetization and real-time bidding. “In the last years we have moved quite a significant workload over to ElastiCache for Redis for caching and ephemeral data,” said Markus Ostertag, Head of Development. “Now with the support for Redis Cluster, we’re very happy to be able to scale out much more easily, get higher performance and better reliability for our whole Redis infrastructure.”

This is a great time to be watching the rapid development of AWS Cloud capabilities for Fast Data management and I urge you to take a few minutes to take a look at the new ElastiCache for Redis and see how you might be able to use it for your own projects.