When my manager Alex first started at Terracotta, he blogged a simple Hello Terracotta example to demonstrate POJO clustering in action. When I first started last month, he walked me through this same example and went into quite a bit more depth, decompiling some code and showing me a glimpse of how client POJO’s are instrumented by Terracotta and thus plug into our clustering framework. Here’s what we did.

First, if you haven’t yet, please go and run through Alex’s Hello Terracotta example. I’ll be starting right where he left off.

Now, as Alex pointed out to me, I need to make a slight correction to the example POJO code he is using. Here is the code:

The correction is the commented out bit. The compiler complains about that code, since the root field is final. And anyway, it is not necessary - if an L1 node starts up and Terracotta sees that the clustered root already exists, Terracotta will ignore the assignment in the source code (Item 2 above) and instead set that field to the pre-existing root. That is why, if you run this sample program twice in a row without bouncing the Terracotta server, you will see that the counter continues to increment from where it left off the first time. Terracotta clustered objects persist.

To continue, today when I run this example I am adding the special super-secret -Dtc.classloader.writeToDisk=true flag to the vm flags when I start up the dso process. My full command looks like this:

dso-java.sh -Dtc.config=config/tc-config.xml -Dtc.classloader.writeToDisk=true -classpath bin test.HelloTerracotta

This additional flag causes Terracotta to dump the modified classes under ~/adapted/. Under that folder you should find a test/ directory containing the instrumented HelloTerracotta.class file.

At this point you can examine the raw bytecode to see what we’ve added - Terracotta uses ASM to instrument bytecode on the fly in order to cluster arbitrary client code.

Or, you can decompile the file like I did using Jad. This likely won’t be able to decompile everything all the way, but it’s close enough that you can see what’s going on.

When I decompile HelloTerracotta using Jad, after cleaning up the go() method by hand, this is what I see:

First thing to notice, the instrumented version of the class now implements two interfaces, Manageable and TransparentAccess. These are both in the dso-l1-api project if you download the Terracotta source code. The api is our internal api for dealing with all clustered objects in the L1 nodes.

Next, notice that all direct reads of the root field have been replaced with calls to a new dynamically-generated Set __tc_getroot() method, and the one spot in the constructor where we were setting the root field directly is now replaced by a call to a new __tc_setroot(Set) method. And you can see that those getter/setter methods are doing the work of ensuring any pre-existing root is set on this object, or else creating the new clustered root.

You can see that there are a lot of calls into ManagerUtil - this is a dynamically clustered object’s hook into the Terracotta runtime. ManagerUtil can be thought of as a facade of static methods, encapsulating a more interesting system which I won’t go into here.

The go() method didn’t decompile entirely correctly, so I’ve had to tweak it by hand into what you see here. What I find most interesting is that now, in addition to the synchronized block, there are calls (in a try-finally block) to ManagerUtil to acquire/release a clustered lock. Remember - since the root field is actually a clustered object, modifying it requires obtaining a cluster-wide exclusive write lock. It’s also interesting to note that a chunk of that tc-config.xml file, specifically the lock information including the method expression, is being passed to the monitorEnterWithContextInfo method.

In conclusion, this example illustrates the pattern for any POJO being clustered by Terracotta. Terracotta dynamically instruments a class to adhere to the dso api, an api for handling all of the distributed objects and locks within Terracotta. Of course that barely scratches the surface of all that Terracotta does, but at least you can see that conceptually what Terracotta is doing behind the scenes to cluster your code is really not all that complicated.

For further reading, the Terracotta website offers some very straightforward articles about how Terracotta works, and how Terracotta scales. Additionally, there are quite a few other Terracotta employees who blog, and their blogs are listed here.

This week at Terracotta we accomplished something that two weeks ago I thought was both impossible and dangerous - we instrumented java.lang.String to be compressible.

What, you ask, the heck is going on? String doesn’t implement any interface called JavaLangString. It doesn’t have a __tc_decompress() method. String is final and immutable! It has to be for thread-safety! Are you mad?

I offered these same objections to my boss two weeks ago, but the fact is that something like the above code will be in an upcoming Terracotta release. What makes this all possible is that we instrument Java bytecode on the fly.

Terracotta already does some large String compression when clustering Strings across cluster nodes, but we wanted to improve on that. What if, when we decoded a compressed String data from across the cluster and constructed a String instance on a remote node, we actually kept the String instance compressed until it absolutely needed to decompress to be read?

Through the voodoo black magic of bytecode instrumentation, we have accomplished this and made it completely transparent to the application’s use of String. Here’s a basic outline of what happens:

• data about clustered compressed String is sent over the wire (we call it “hydration”) and decoded at one of the cluster nodes. The following data is encoded:
  • actual compressed String data, in byte[] array form
  • uncompressed String length (int)
  • String hashcode (int)
• when decoded, the compressed byte[] array is encoded into a char[] array - basically two bytes can fit into a single char.
• a String instance is constructed with that char[] array, and also the uncompressed length and original hashcode

So, if the resulting String were actually read, it would be gibberish (if it were displayable at all). (By “read” I mean, if it’s internal character content needed to be accessed.) That’s why we need to instrument the String class - if and when the String needs to be read, we have instrumented it so that it knows how to decompress itself. We have basically added a new private boolean field (indicating if compressed or not) a new constructor and some additional methods.

There’s a lot going on here so l’ll point a few things out. First, we intercept any field-level access of the private internal char[] value and we route it through this __tc_getvalue() getter method. This is how we transparently decompress the contents no matter how the String instance is accessed.

Secondly, there are important concurrency issues in play here. String is thread-safe because it is immutable, or was until we got our grubby mits on it. We need for String to remain both thread-safe and also highly concurrent, so we wanted to avoid synchronized blocks. Our solution?

We have instrumented value to be volatile, as well as our new $__tc_compressed field. Now if you once more examine the __tc_getvalue() method above, you’ll see that there is a benign race condition. It’s possible two threads could both decompress and set the value field. That’s fine, since the decompression is deterministic and outputs the same result each time. Because the fields are volatile, once they are set those changes should be visible to all other threads. What should not ever happen is, no thread should attempt to decompress the characters once they are already decompressed - the call to StringCompressionUtil inspects the data to see if it has already been decompressed, and if it has it returns null.

The above methods don’t exist anywhere in source code per se. Rather, we use ASM to add the bytecode for these new methods dynamically to the String class as it is loaded at runtime. So what we actually have somewhere is code that looks like this:

This creates the bytecode to implement the __tc_decompress() method.

When does an instrumented String instance need to decompress itself? Basically, when it’s internal char[] array needs to be accessed. It does not need to decompress when length() method is called, because we have set the uncompressed length when we constructed the String. It does not need to decompress when hashCode() is called, since we have preset the hash code, so that means a compressed String can sit around in a Map all day long without needing to decompress. We even have plans to muck with the equals() method - if two Strings are unequal due to hash code we can exit early and avoid decompressing them to compare character content.

Let’s look once more at the code way at the top:

Because we have instrumented String, at runtime instances of String do implement the interface JavaLangString. So, within Terracotta we can cast an instance of String to JavaLangString, which is an interface containing the __tc_decompress() method. In this way we avoid using reflection. One interesting thing to note is that we first have to upcast to Object to trick the compiler - at compile time the compiler knows (or thinks it knows) that String does not implement this interface, so it errors.

In conclusion, a bunch of us were talking after Alex’s Terracotta talk last night, and someone made the point that, with any good technology, your users will inevitably take advantage of it in a way that you never dreamed of. And that, I think, is what Terracotta is doing.

Somehow my first month at Terracotta has already flown by. Here’s what I’ve been up to.

Mostly I’ve been testing. We have a very cool distributed testing framework called “Droid”, and using it we’ve written a number of test scripts (in Groovy) to simulate some real-world use cases and see where we can make improvements. Droid, in a nutshell, allows us to easily and automatically run tests on multiple nodes, and provides means of synchronizing those nodes. In fact, Droid achieves this clustered synchronization using Terracotta, which just goes to show you that we’re not afraid to eat our own dog food.

In the process of working on this I’ve gotten to dive fairly deep into both Groovy and the java.util.concurrent classes, both of which I’ve enjoyed immensely. My humble contribution to Droid was to add some Groovy utilities enabling us to set up multiple test “phases”, which are simply points at which the nodes all synchronize by waiting on a CyclicBarrier. Our test scripts can now register closures to run during a phase, which cleans up the scripts themselves somewhat. Droid already had the ability to “kill” a test node, which was implemented rather ingeniously by using a cluster-wide (via Terracotta) Object to wait on and notify all - basically a daemon thread in each node would wait on this object, and when notified would check to see if it had been killed and should do a System.exit(-1). Building upon this, in my Groovy utilities I added methods to easily set up a kill phase, in which all testing would be suspended while a node was killed off - this leverages the Runnable CyclicBarrier action, which is run before any threads are released from the barrier.

Also as part of all of this, I’ve become reacquainted with Unix. Terracotta has an impressive array of “perf” machines dedicated entirely to continuous performance testing. Some are tuned to act as client L1 nodes, other more powerful multicore machines are tuned to act as the L2 servers. These machines can be reserved, scheduled and used all remotely. So my third week was largely spent actually running some of these distributed tests on multiple perf machines, which involved ssh and sudo.

This week, based in part on some findings from running these tests, I have finally begun (along with Alex) diving into actual production code to see what improvements can be made. In particular, we are looking into the case where cache values are comprised of very large strings, for example large XML documents, and we are investigating how we can improve our string compression code, or whether we can avoid unnecessarily faulting String values into an L1 node.

I love working at Terracotta. I love it! I could go on and on about other cool stuff that’s happening, but I won’t. Suffice to say, the technologies we use and the problems we are tackling are fascinating, to say the least. All of my colleagues are both very sharp and very down-to-earth - no power trips going on that I can tell. Working at home is wonderful - it’s so nice to just take a 15 minute break and walk my son to preschool, or have lunch with my family. Or, in theory, work in my boxers, not that I’ve done that, yet. And working with Alex again rocks.