Skip to main content

Redis Cluster: Fact Sheet (Not Just Issues)

Redis and the Redis clustering works very differently from the other data stores and data store clusters. The differences are not always as obvious and may come up as realizations down the line while using Redis, like what happened in our case. We are using a Redis cluster, with which, fortunately, we have not faced many issues so far. But that does not mean we will not and we shall need to be prepared.

Recently we were working on getting a Redis cluster up and working with docker compose and was enlightened to some of the differences which later led to disillusionment for me. Thought that there should be a ‘document of facts‘ on Redis and Redis cluster which people/myself can refer to. So I decided to create one, enjoy:

  1. Redis is great as a single server.
  2. In a Redis cluster, all your masters behave as if they are simultaneously active (not sure if they all are masters at the same time technically, but they behave as such).
  3. Every master in a cluster knows every other master/node in the cluster.
  4. There is no single master looking over the orchestration job.
  5. The masters, during clustering (sharding) agree upon the division of load: who shall have which hash slots.
  6. Each master speaks only for itself. If you ask for a key, and if the hash-slot for the same happens to be on the master you asked, it will return a value. Otherwise it returns a ‘redirection’ to the master that has the slot for this key.
  7. It is then the client’s job to resend the request to this new master based on the redirection.
  8. Clients try to sync up with master for which hash-slots lie with which master in order to speed up the retrieval.
  9. Every master knows other master by IP and IP only. It is not possible to use a hostname.
  10. The knowledge about the other nodes in the cluster is stored in a file called: nodes.conf. Although the extension is gives an impression of user modifiable configuration file, it is not a file for humans to modify.
  11. Every master must know other master by actual public ip, it is not possible to use a loopback (like 127.0.0.1). If you do that it ends up in a max-redirection error. How it works is, when a client asks for a key and server responds with a redirection, the ‘smart’ client is expected to follow this redirection and get the value from this other node. Now the ‘dumb’ server responds with only ip it knows other node by, that is your loopback on the Redis server. But this ‘smart’ client (Jedis) is not smart enough to understand that the loopback is actually of the node and apparently starts looking for a Redis node on its own host! Whatever.. Just avoid doing that.
  12. When two nodes meet to form a cluster, one of them has to forgo its data. Either one must be empty.
  13. Replicas are not within master or any other nodes for that matter. Unlike what we know about clustering in Elastic Search or Kafka like services, replicas in Redis are independent nodes. So if you want a replication factor of 2 and have 3 masters, you effectively need 3 * 2 + 3 = 9 nodes in the cluster.
  14. If a master drops off, it is not possible to bring it back into cluster with data. Implication of point 12.
  15. If you need to perform any updates to any of the nodes/servers, take point 12 and 14 into consideration. Take out the master, upgrade, flush and reconnect as a slave, that is how it works.
  16. Converting a single server to cluster is not supported officially. There is one blog of a smart person showing a workaround for such a migration. Inverse of this, cluster to single server shall be equally painful.
  17. Redis / Redis Clustering is not officially supported on a Windows machine. There are unofficial ways to achieve something of the sort, the MSOpenTech’s Redis implementation, which now also supports clusters.
  18. The Java client, Jedis, has two different classes, one for connecting to a single standalone (JedisClient) and other for connecting to a cluster (JedisClusterClient). So if you decide to use the cluster in production, you cannot choose to use a single server during development. Implication is un-necessary load on your laptops. It can be managed by using environment aware wiring. We worked around by creating a jar, with a class that on post-construct just replaces the cluster-client reference of our internal cache utility class with a single server jedis-client. Just placing this jar on classpath during development solves it for us.
  19. Running Redis cluster in docker has its own pain points, on that later. (A different fact sheet for docker soon.)
  20. Extending point 11, if you have two network interfaces on the nodes, and have two isolated networks for two services that use this Redis cluster, how will that work out? Such is a setup is expected in a docker compose, where we isolate the service into different networks. Will need to see how Redis behaves in such a setup.

Although it was not the intention, while reading what I wrote I realized that the points above do look like a rant. In spite of these Redis is a solid, fast cache store and I love it for that. These are merely a few nuisances and related implications which we learnt about and experienced in our use of Redis cluster. Please use them only as points to ponder on when designing your application. Also, these nuisances are based on the state of Redis and Redis cluster at the time of writing which will change in time to come.

Comments

Popular posts from this blog

Using Docker and a Private Registry with VPN On Windows

Wasn’t that a very specific title? Docker has a very good documentation and reading that alone is enough for most of the straightforward tasks we might want to do. But as always some practical tasks are not straightforward, hence this blog. What we are going to see here today is how to setup docker toolbox on a Windows machine, make it work even when VPN is connected, make it talk to a private, insecure docker registry (that is why VPN) and configure it so it can run docker compose and see how we can set this config as a one-time activity. That’s quite a mouthful, but yes this is what we are going to do. All ready? Let us begin then.

Install Docker ToolboxGo and download the docker toolbox and install it. That should create a shortcut called “Docker Quickstart Terminal”. Run it. That should show you an error about virtualization.

Enable VirtualizationRestart your machine, enter the BIOS settings and enable virtualization. It may be under advanced settings. On this Laptop, it is under th…

Yet another packager for node

Yet another packager for node There are so many packaging systems for node already, or maybe not as many, so here I am presenting another way to package your applications into an self extracting executable that has no dependencies. Ah well, a few dependencies, like the processor architecture, and Linux operating system may be, but that is all. What is it? It is a modified shell script originally used to create self-extracting and installing applications for Linux platforms. What it does is, it creates a tarball which includes your code, the modules it depends on, the specific node binary it uses, and appends it to a script with the command to execute your code. It is essentially a binary merge of the files, the shell script and the tar.This is not something new, people have used such a system in the past to deliver applications for Linux, every time you see an obscenely large ‘.sh’ file (for being that, a shell file) that can install or execute an application without requiring any ot…