Redis as SQL Cache, take one

I was worried about the performance of our Database Servers. Our access patterns are mostly read-only, so why not cache the data we need in an intermediate server? Redis appears to be a good solution.

Too many readers, few writers

From a data life cycle point of view, my current domain has the following characteristics:

1. It evolves by big chunks and the number of users allowed to make changes on it is very limited.
2. There are hundreds of concurrent users on read-mode.
3. It's not mission critical for the consumers of the data to be perfectly up to date. They can wait some minutes.
4. My budget is close to nil.

I didn't want to route the readers of the data to the main DBs. That'd create the perfect bottleneck. And I've been looking into caching all that information, in memory, for a couple weeks.

Theres quite a lot of solutions out there. Microsoft has a couple: Velocity or AppFabric Cache. But in the Linux world there are way more options. But at the begining I was lazy and silly and I wanted a full Windows stack.

First approach: memcached

Memcached is one of the veteran solutions in this endeavor. It's incredibly stable and Facebook (among many other) has been mantaining it for quite a long time. Here you have a chat by the man himself.

It's pretty rare to have scaling problems that compare to FB's. So I decided to take a look. There are at least 2 major versions of this solution that are precompiled for Win32. They work. But everybody agrees that the performance it's not the same.

VirtualBox + Debian 9

Once I admitted that I should host my services on Linux I went for one of the virtualization solutions I know, VirtualBox, and I noticeed with glee that it was possible to install Debian. That was my first Linux distro. Feels a bit like returning home.

Then it was a matter of apt-getting make, gcc, vim, terminator, etc.

And I went to fetch Memcached sources. But, on my way there, I thought that since I had a "full fledged" Linux, why not checking around a little. And then Redis happened. On paper, Redis' features are a super set of Memcached. So I decided to give it a go. With a Linux in place, it was painless 4 steps to build from sources.

Then you end with something like this:

The base sources includes the tooling of the DB. Which is super nice.

C# + Redis: a lot of "Stacks"

Since I wanted a fast start on all this Redis biz. I checked in PluralSight for a fast start. That, in hindsight, was a bit of a mistake. Redis has a great amount of material in youtube, they even have a conference.

My first approach, was to write something in C# to feed a RedisDB. Following the advice from the PS Course I opted for ServiceStack.Redis and it works very well. Except for one detail. My budget for all this is exactly zero dolars, and servicestack is clear regarding its pricing Needless to say I reached the starter limits in exactly one hour. All that was, clearly, my bad. I should've read the services better.

Thankfully there is a good list of other C# Clients and I decided to take a look into StackExchange.Redis. Yup I know the names are super confusing. But that's what happened. Combine that with some fever and you have a glorious headache just waiting for you.

The code itself it's reasonably clear, in a somewhat "unittest" format:

{% codeblock lang:Cpp %} const string redisConnectionString = "YourServerIP:6379,allowAdmin=true"; ConnectionMultiplexer cm = ConnectionMultiplexer.Connect( redisConnectionString ); IDatabase db = cm.GetDatabase(); Assert.IsNotNull( db ); string value = "abdcdfge"; db.StringSet( "myKey", value ); string recovered = db.StringGet( "myKey" ); Assert.IsTrue( value.Equals( recovered ) ); {% endcodeblock %}

With this library in place, projecting my data in a Redis-Friendly format is just a matter of wiggly Linq enough.

Consuming the cache from C++

Unfortunately the vast majority of the consumers of my domain work over C++ stacks. So there was the problem of finding a library that could communicate with the database with the minimum number of dependencies. I believe I have a good candidate here: cpp_redis

Painless to compile and try. But it's still too soon to have a full formed opinion about it. I might post somehting more down the line.

Some lessons learned

1. First and foremost you should check your tweets three times before clicking "send". I wrote "StackExchange" when I wanted to say "ServiceStack" and all hell broke loose.
2. Redis is part of the NoSQL family of DBs. No schema enforced. That gives you a lot of opportunities. But it puts a lot of pressure on the main keys.
3. This DB supports several data types as primitives: sets, lists, hash, ... the natural candidate to persist objects seems to be Hash, but I need to dig deeper on all this.
4. The subscribe commands are incredibly powerful. Just for those alone, Redis is worth your time.
5. This DB supports Lua on server side. And who doesn't love Lua, right?