After two weeks of debugging, on Friday I added five magic lines of code to make the last three automated Spring tests pass. With this change we can say we support Spring up to 2.0.8 in our upcoming 2.6.2 release. We still have targeted Spring 2.5.x support for an upcoming summer release.

Two weeks of debugging, five lines of code, one bug fix. This has to be improved on.

Just for my own amusement, I’m going to try to list and describe all of the things that accounted for all the time spent on this.

By midweek last week, I had finally gotten into a groove with the Eclipse debugger, stepping through one of the failing Spring container tests. By “container” test, we mean parts of the test were actually running in a web app container, Tomcat in this case. Even so, this was not a very good, tight feedback loop. The basic pattern was for me to start the test, then attach to each of the two processes running in tomcat with the debugger. (Luckily, our code was already set up to allow for debugging, although it took me awhile to find and enable the magic property.) Then I just had to step through code in the debugger, not really sure what I was looking for. I had to alternate doing this, first with Spring 2.0.1 in the classpath, since our test was passing for that version, then with Spring 2.0.8 in the classpath, and look for what was going wrong. (We have a way of easily varying the version of a library such as Spring. However, I spent basically all of Tuesday helping Hung debug a problem in our build process concerning those variants.) Each time I alternated Spring variants, I also had to tweak the source lookup in my Eclipse remote debug configuration, to pull up source from the right Spring version.

The test uses our DSO functionality, it’s basically a semi-complete running instance of the DSO client. At one point I found I needed to hit breakpoints that were occuring during DSO bootstrapping, which meant I had to set yet another magic property (in ClassProcessorHelper) to enable debugging which is normally not available prior to TC instrumentation. I had to dig through my IM chat transcripts to find the name and location of that property, which my boss had mentioned once weeks ago, and then enable it, and find through painful trial and error that it didn’t work unless I did a full clean recompile (but it worked like a charm after that).

The code I was stepping through was instrumented using AspectWerkz, although for the most part the breakpoints seemed to work fine. But the amount of code was just vast. All I can say is, I spent probably twelve to fourteen hours of straight debugging on Thursday and Friday, just hitting breakpoints, comparing state, following hunches and wild goose chases and red herrings. In the end I found the Spring class whose source had changed between 2.0.1 and 2.0.5, and again in 2.0.8. (It was org.springframework.aop.config.ScopedProxyBeanDefinitionDecorator, deep in the guts of Spring’s aop framework.)

So in the final tally, I spent unexpected amounts of time

• helping debug a build problem which prevented us from using the “variants” feature, to vary the Spring library
• setting up Eclipse projects for different versions of Spring, so I could browse 2.0.1 and 2.0.8 source code side by side
• tweaking the Spring source lookup for the Eclipse remote debugging configuration, to alternate between 2.0.1 and 2.0.8
• figuring out how to enable debugging of our container tests
• figuring out how to enable debugging during DSO client startup
• trying (in vain) to write a more lightweight unit test to tackle the problem with
• debugging for many, many hours once it was all working

And that’s not even counting actually learning AspectWerkz and writing a new pointcut and advice, which I would have had to do anyway, but which itself involved some painful trial and error.

And at the end of all of this, I sort of feel like I’ve just patched some big honking beast that I don’t fully understand. I feel like our code as it stands now (the AspectWerkz pointcuts and advices for clustering Spring) are still very tightly coupled to Spring source code in a very fragile way, and it’s only a matter of time before a new minor Spring version comes along and breaks it again. All week I was thinking, there has got to be a better way.

Our code for instrumenting and clustering Spring is really cool and mind-blowing, don’t get me wrong. I never would have thought of it in a million years. But it is a classic case of code that is tightly coupled, not modular at all. There is an impressive amount of test coverage through automated integration and system tests, which we need. But the Spring test suite takes about an hour to run, and that’s not even every test. There is a lot of AspectWerkz advices and pointcuts being used to cluster Spring, and none of them can be run and tested in isolation. You just have to start up the whole shebang and debug.

I consider myself a TDD and refactoring proponent. All week long, the pain of trying to debug this was screaming out to me that this part of the code needed refactoring and unit tests. Honestly, I’m not sure I could have done any worthwhile refactoring and still figured out the problem in the same amount of time. That’s the classic TDD/refactoring chicken and egg problem - when you’re in a time crunch, the prospect of trying to clean up some code in order to make things easier at some undermined future point seems insurmountable, and meanwhile you always tend to mentally downplay the amount of time it will take to just debug and fix the code as is. That’s the fear that comes with TDD.

In an interesting twist, I was talking to my boss Steve, and he asked me why I wasn’t refactoring. My boss wants me to refactor! He didn’t order me to or anything, but he sounds very much in favor of it. He said he almost always regrets not refactoring as he goes.

I think keeping the code clean and testable is so important to the long term health of a large, constantly evolving code base. I mean, we can’t just keep having software developers spend two weeks debugging one bug.

There’s a lot more I want to think about along these lines, but I’ve written enough for now.

This weekly summary actually encompasses the last three weeks. Sigh.

Lots of activity throughout dev is centered around the Terracotta 2.6 and 2.6.1 releases, as well as the upcoming 2.6.2 release.

Primarily I’ve been working on updating Terracotta’s Spring support to 2.5.x. Currently we only support up to 2.0.5. I had thought I had gotten it working up through Spring 2.0.8, but late last week we fixed a bug in our build process which then revealed three failing automated TC Spring tests which were previously (incorrectly) passing. So Spring 2.0.8 is not quite there…but close. Meanwhile, my compadre Nitin had made some changes that got TC working with Spring 2.5, but those changes are not backwards compatible to Spring 2.0.x, so I’m investigating whether they can be merged together somehow. Since we are dependent on the Spring source code in order to instrument their code (by using Aspectwerkz), we are subject to the whims of whatever source code changes occur between even minor releases (such as differences between Spring 2.0.5 and 2.0.8).

The other thing of note that I got accomplished was to respond to this post on our forums about a deadlock occurring in Terracotta L1. The poster had nicely laid it all out for us, with a stack trace excerpt clearly showing the deadlock. My teammates and I reviewed the pertinent class, and I cleaned up a number of synchronization bugs or missing synchronization. The deadlock itself was cleaned up by moving to a CopyOnWriteArrayList for a collection, which previously was being locked while iterating through it (read-only) and doing expensive stuff. The fix will be in 2.6.2 release.

I was without internet connection at my house a couple weeks ago for a few days. I had to do bloody battle with Charter to get that fixed. Ultimately a technician came and found that the line to my house had been put on a splitter at some undetermined point in the past, and so my signal strength was no longer strong enough. Meanwhile, luckily, I was able to go to my parents’ house and get some work done there. Have I mentioned that I love my MacBook Pro, and wireless internet?

My previous post was about whether Guice should be used in unit testing, and Crazy Bob himself commented:

I prefer the simpler unit test. I think the real moral of this story is that unit tests alone are never enough.
Also, you did get the error right away when you started your app. That’s like an automatic test that you needn’t replicate manually (comparable to a compiler check).

I started to reply and my comment quickly grew long enough that I decided to just write another post. So, should Guice be used in unit tests? I hate to be wishy washy but I think the answer is it depends. As happens so often in software development, there are conflicting forces at work. In this case I think two conflicting forces are ease of unit testing versus fast feedback loop.

One thing I do completely agree with Bob on is that unit tests alone are never enough. Alex Miller wrote a great post about that called Weaving the Test Fabric.

Fast feedback loop

Different parts of the test fabric perform differently and are intended to be run at different frequencies. It’s common to only run all of the automated tests once during a nightly build, especially if any of the system tests are long-running and/or resource-intensive enough. It’s common to have a medium-weight suite of tests which a developer is expected to run once prior to committing the code, but which may still take quite a few minutes to run. And I prefer to have a suite of as many automated tests as possible which run fast. If you have a fast running suite of automated tests, then you have a fast feedback loop. I’m talking under twenty seconds, the sort of suite you can run in between every single change, as you work. Such tests most likely make heavy use of mocks and have no dependencies on external resources like db’s, file systems or network connections. Such tests are most likely unit tests.

With Guice, as with any code, the question is: how long can you stand to wait to find out if you implemented, or broke, something?

I would not be able to stand having to start the whole application up every time I was debugging my Guice usage. Bob Lee reminded me that you can still have automated tests that test your usage of Guice, they could be component or system level automated tests, not necessarily unit tests. They could even be included in the fast running suite.

Ease of unit testing

It is so important to keep unit tests, and unit testing, simple. For the sake of all current and future developers working on the code, the barriers to unit testing need to be as few as possible - basically just know JUnit. It’s hard enough as it is to get people to write unit tests. So, having thought about it, now I’m not so sure it’s worth having Guice in unit tests if it burdens my peers. I know how overwhelmed I feel when I work on a project and find I have to learn additional in-house unit test conventions and special TestCase subclasses that I’m expected to start with. It’s an awfully attractive idea to keep unit testing confined to simple JUnit tests, and leave fancier stuff for more complicated integration-style tests.

This is all predicated on the idea that Guice is not mainstream yet, as JUnit is, and would still be a burden to have to learn just to unit test with. If Guice ever becomes more widely used and understood, then I might revert back to my earlier conclusion. After all, Guice is the new new - why not embrace it in unit test code if it is being embraced in production? I like the API very much and think it could easily become part of the unit testing vernacular.

Conclusion

The two forces I mentioned above do not have to be diametrically opposed. Thinking about it now, I could envision component-level or subsystem-level automated tests that do nothing except test the Guice dependency injection for that component. Such a test could still be fast-running if mocks are used as appropriate - Guice itself is performant enough thanks to it’s pure Java implementation. And such tests would naturally force the developer to place the proper Guice annotations in the proper place. Unit tests could remain simple JUnit tests.

Last week was shortened by jury duty on Monday. Fortunately, I was never selected from the pool, and on Tuesday I was back to work.

There are (still) a number of monkey failures (such as this one) that I need to get working on.

However, I discovered I could procrastinate tackling those by checking the forums. I decided to try to answer this post (which has since been addressed by a couple of my teammates). Almost two days later, I conceded that it’s really really hard to try to cluster the underlying javax.swing.text.AbstractDocument of a JTextField. I still haven’t got it. (See gkeim’s response for a clever workaround.)

I finished up the week by working on droid. One of my peers was having trouble running a test in which he wanted one of the spawned workers to have a different tc-config than all the others. There are some weird subtleties in passing vm arguments through the agent which are intended for the worker, but as it turned out, I believe the functionality is already there and didn’t require any changes on my part, just an explanation of how to do it.

At my Guice presentation recently, I used some small code exercises to demonstrate the basics of Guice. Here is my simplest example, involving a Service, a Client needing a Service injected, a test case, and an Application.

First, the Service:

Next, the Client, and it’s test (JUnit 4):

And finally, an Application that ties it all together using Guice:

During the talk, Jeff Grigg had a question followed by an interesting comment. For his question, he asked me to remove the @Inject attribute from Client and re-run the test. I did, thinking I was about to demonstrate a typical helpful Guice Exception. But the test passed. I was momentarily lost, until I remembered that the unit test did not use Guice at all. I ran the Application just as a sanity check, and it was indeed broken.

Jeff was on it like a hobo on a ham sandwich - I had broken the application but the test still passed. My test did not justify the use of the @Inject attribute.

He was absolutely right. I realize now that I had been following the example from the Guice Developer Day Slides, which had taken a non-Guice example and migrated it to Guice, leaving the JUnit test unmodified so that, like my test above, it continued to manually pass a mock to the client code and did not use Guice. This, I realize now, is fine for a tutorial but it not truly Test Driven Development, where every line of code has some test to justify it’s existence.

Naturally, Jeff raised the question - should Guice then be used in unit tests?

I think the answer is yes, why not? After all, Guice suggests it be thought of as the new new, and that using Guice is no more work than plain old factories.

If Guice were used in unit tests, then the tests would naturally require that the code have the necessary @Inject attributes, or whatever other attributes are necessary for Guice. The only difference would be that different Modules would have to be created for the tests, which contained bindings which differed slightly from the production Module(s).

I tried implementing a new client, strictly adhering to TDD and using Guice in my unit tests. Here is what I came up with.

There are two things I notice. One is that my mission is accomplished - my Client’s @Inject attribute is now truly unit tested - the test will fail without it. As a bonus, Client is slightly simplified in that it no longer needs a constructor which accepts a Service. The other thing I notice is that the test is somewhat more complicated, but really it’s only the addition of a Module and Injector.

My feeling is that it’s worth the extra effort to use Guice in unit tests, if you’re going to embrace Guice in your production code anyway.

In a future post, I’d like to investigate whether there is a better way to integrate Guice with JUnit, alleviating the need to write boilerplate unit testing Modules over and over again. Just tonight I noticed both GuiceBerry and AtUnit which potentially address this and look somewhat promising.

I gave my Guice presentation tonight. The best part of the talk for me is that a lot of people chimed in with comments and questions, which is just the way I like it. Thanks to everyone for attending and for generating some good discussion. My talk was comprised mostly of code samples, the presentation materials will soon be available on the JUG knowledge base.

It was great to see some old compadres (Greg, Rasesh, Jeff C and Dennis) as well as introduce myself to some familiar faces in the St. Louis Java community (Michael, Mark, Jeff G, Weiqi, Kyle).

The differences between Guice and Spring were definitely a frequent line of questions and comments. Some open questions and comments that came up related to this are:

• Is it ever preferable to have your application’s configuration truly outside of the source code (a la Spring xml)? Is there value in reconfiguring the app without rebuilding, or is that just a pipe dream? Is external configuration desirable so that non-developers can reconfigure a system, or is that fraught with peril?
• What would be a reason to integrate Guice into an existing code base which uses Spring, especially if you are already leveraging Spring 2.5’s annotation-based dependency injection? I conceded there may be no reason. Kyle Cordes pointed out that in a pristine, pie-in-the-sky world this may be true, but so often in the real world there are large clunky heterogeneous revenue-generating apps and components which need integrating, and you may be in a situation where you have no choice but to integrate Guice with Spring. In that department, Guice scores points for having integration with Spring built in.
• Is there a Guice parallel to Spring’s named instances? That is, in Spring you can configure beans which are named multiple instances of a single concrete type, and said beans can be injected. Jeff C suggested that a Guice Provider might supply that behavior. Kyle pointed out that, by Guice’s very nature you may not need to declaratively configure named instances because Guice entirely operates as Java, and you should be able to just programmatically get or create whatever instances you want. It’s definitely a different mindset than Spring.

Weiqi urged some of us to read the Guice source code, particularly to be impressed by some clever use of generics. I plan to read a little of it right now.

This just in - Weiqi has already written a writeup of my talk on his own blog. Thanks Weiqi, I’m flattered!

This Thursday night I’m giving a presentation on Guice at the St. Louis Java Users Group. Come on out and see!

My talk is shaping up to be mostly code and very few slides. In the first half I will be demonstrating the basics of Guice, what it is and what it can do, via the usual simple contrived code samples that you would expect. In the second half I hope to address how you might introduce Guice to an existing codebase. Specifically, I want to look at Guice’s integration with Spring, and also look at some basic strategies for refactoring out smelly code like Singletons or tightly coupled spaghetti code.

Boy, I hope it turns out as good as the above paragraph suggests!

If you’ve never been to the JUG before I encourage you to come check it out. It’s very informal and relaxed, it’s a great place to network and plug in to the St. Louis Java community. Plus there’s food and refreshments, and almost always cool swag being given away.

One last thing - I would be remiss if I didn’t mention that my company, Terracotta, has support for clustering both Spring and Guice at the JVM level. (There - now I can expense this blog.)

Last week I had the opportunity to use the Extract Method refactoring. I’m generally a proponent of this refactoring - I think it’s a worthy goal to have more numerous, shorter methods, if that means that any given method is more quickly understood at a glance.

The code I had, after I had made some further complicating modifications, looked something like this.

I didn’t like the nearly-duplicate logic duplicated in the if-else block. I felt there was an extract method in there somewhere. However, it took me three times before I came up with a solution I liked, which is basically this:

I won’t bore you with the first two tries, but I’ll just say that for some reason the use of the third defaultTo arg didn’t occur to me right away, but was just what I needed to make the method calls nice and clean.

After I looked at the finished product, I was reminded that another benefit of Extract Method is that differing levels of abstraction are separated. In this case, the determination of whether WebLogic or Tomcat is in use, and what delimiter and default to use to determine the web app name, is a sort of mid-level policy. The nuts and bolts of parsing that app name out of a String instance is the lowest level of abstraction and, in my opinion, is better off not inlined into the policy code. In a similar vein, we would not want to see the implementation of String.substring() inlined into every single invocation of that method.

I don’t think my finished product is perfect. I think maybe there’s a better method name that could be used, which communicates the intent better, because it’s not perfectly obvious at a glance why the delimiter and default need to be passed to the method and what happens. But then again that’s part of the point, we don’t want to know the details. Meanwhile, on the plus side, we see that the same method is invoked in both cases, and only the arguments differ, so it’s obvious that there’s a pattern there which wasn’t necessarily obvious before.

When my manager Alex first started at Terracotta, he blogged a simple Hello Terracotta example to demonstrate POJO clustering in action. When I first started last month, he walked me through this same example and went into quite a bit more depth, decompiling some code and showing me a glimpse of how client POJO’s are instrumented by Terracotta and thus plug into our clustering framework. Here’s what we did.

First, if you haven’t yet, please go and run through Alex’s Hello Terracotta example. I’ll be starting right where he left off.

Now, as Alex pointed out to me, I need to make a slight correction to the example POJO code he is using. Here is the code:

The correction is the commented out bit. The compiler complains about that code, since the root field is final. And anyway, it is not necessary - if an L1 node starts up and Terracotta sees that the clustered root already exists, Terracotta will ignore the assignment in the source code (Item 2 above) and instead set that field to the pre-existing root. That is why, if you run this sample program twice in a row without bouncing the Terracotta server, you will see that the counter continues to increment from where it left off the first time. Terracotta clustered objects persist.

To continue, today when I run this example I am adding the special super-secret -Dtc.classloader.writeToDisk=true flag to the vm flags when I start up the dso process. My full command looks like this:

dso-java.sh -Dtc.config=config/tc-config.xml -Dtc.classloader.writeToDisk=true -classpath bin test.HelloTerracotta

This additional flag causes Terracotta to dump the modified classes under ~/adapted/. Under that folder you should find a test/ directory containing the instrumented HelloTerracotta.class file.

At this point you can examine the raw bytecode to see what we’ve added - Terracotta uses ASM to instrument bytecode on the fly in order to cluster arbitrary client code.

Or, you can decompile the file like I did using Jad. This likely won’t be able to decompile everything all the way, but it’s close enough that you can see what’s going on.

When I decompile HelloTerracotta using Jad, after cleaning up the go() method by hand, this is what I see:

First thing to notice, the instrumented version of the class now implements two interfaces, Manageable and TransparentAccess. These are both in the dso-l1-api project if you download the Terracotta source code. The api is our internal api for dealing with all clustered objects in the L1 nodes.

Next, notice that all direct reads of the root field have been replaced with calls to a new dynamically-generated Set __tc_getroot() method, and the one spot in the constructor where we were setting the root field directly is now replaced by a call to a new __tc_setroot(Set) method. And you can see that those getter/setter methods are doing the work of ensuring any pre-existing root is set on this object, or else creating the new clustered root.

You can see that there are a lot of calls into ManagerUtil - this is a dynamically clustered object’s hook into the Terracotta runtime. ManagerUtil can be thought of as a facade of static methods, encapsulating a more interesting system which I won’t go into here.

The go() method didn’t decompile entirely correctly, so I’ve had to tweak it by hand into what you see here. What I find most interesting is that now, in addition to the synchronized block, there are calls (in a try-finally block) to ManagerUtil to acquire/release a clustered lock. Remember - since the root field is actually a clustered object, modifying it requires obtaining a cluster-wide exclusive write lock. It’s also interesting to note that a chunk of that tc-config.xml file, specifically the lock information including the method expression, is being passed to the monitorEnterWithContextInfo method.

In conclusion, this example illustrates the pattern for any POJO being clustered by Terracotta. Terracotta dynamically instruments a class to adhere to the dso api, an api for handling all of the distributed objects and locks within Terracotta. Of course that barely scratches the surface of all that Terracotta does, but at least you can see that conceptually what Terracotta is doing behind the scenes to cluster your code is really not all that complicated.

For further reading, the Terracotta website offers some very straightforward articles about how Terracotta works, and how Terracotta scales. Additionally, there are quite a few other Terracotta employees who blog, and their blogs are listed here.

Here’s what I’ve been working on this week at Terracotta.

The Terracotta 2.6 stable 0 release was released this last week. Personally, my contribution was to help implement the dynamic String compression feature. This week, emphasis shifted from achieving feature-freeze to testing.

We wanted to add some tests around serializing Terracotta-instrumented objects between instrumented and uninstrumented code, particularly our new compressed Strings. We needed to make sure that we hadn’t broken serialization with our bytecode manipulation - particularly for String, which is a special case (although this wasn’t apparent to me until I read the source code for java.io.ObjectInputStream#readString).

I mentioned that we have a system test framework, built on JUnit, which is able to start up and run multiple L1 nodes in a test fixture, complete with instrumented classes. My first attempt was to try to test serialization in-process. That is, for each Class I was interested in testing serialization for, I wanted to load two different Class instances (from two different ClassLoaders) and serialize a sample object back and forth using both Classes. The first ClassLoader would be the normal Terracotta ClassLoader, which loads the instrumented version of a Class (either from the Terracotta bootjar or using dynamic instrumentation - in this case I was just trying to test the bootjar classes). I wrote a second, custom (nondelegating) ClassLoader which would load a non-instrumented Class for the same classname - it did this by loading from the unmodified bootclasspath (without Terracotta bootjar prepended to it). This was tricky, since (as my teammate Tim showed me) ClassLoader is hardcoded to not allow loading of any class beginning with “java.”, except from the bootstrap loader only. However, galvanized with the spirit of reckless abandon that I’ve come to embrace since working here, I quickly circumvented this with reflection:

Don’t try this at home. Anyway, this actually worked - I was able to get a test running which loaded different Class objects for each Class I was interested in, including the JDK classes.

However, I was thwarted in my attempts to test Java Serialization in process. I haven’t taken the time to track down the exact cause, but no matter what I tried I either got java.lang.LinkageErrors or some other problem I can’t remember. I’m sure it’s something to do with the fact that my hacked ClassLoader doesn’t play well with serialization, somewhere deep in the bowels of ObjectInputStream or ObjectOutputStream.

As a compromise, I honed in on compressed Strings specifically. At Alex’s suggestion, I manually serialized a sample large String to a file, using a regular non-instrumented String. Then I wrote a test which took the same String, compressed it, and then Serialized it, and I compared the two resulting byte[] arrays. Luckily, the test passed.

I also added some more unit tests to our StringCompressionUtil class, which is the class that actually does the compression of the String for us. It has two responsibilities: (1) actually compressing the byte[] array which we get from the original String, and (2) encoding that compressed byte[] array as a char[] array which can be stuck back into the same String instance. In the course of adding more tests I did a little refactoring and was pleased to find that even with such low-level code I was able to eliminate some duplication and extract some methods which made it a little more clear what our algorithm was intended to do.

At the end of the week I was just beginning to look into some monkey failures. The monkey machines are our continuous integration machines - read all about them here at Hung’s blog. Basically we do continuous integration on a whole bunch of different OS’s using different JDK’s and versions, all the time. This particular failure was using the IBM JDK on a linux machine.