My Office This Week
I’m working remotely this week in Destin, FL. It’s rough, but that’s life working for Terracotta.
Archive for July 2008
Weekly Summary
I struggled last week. It was just one of those weeks where nothing seemed to be easy. The week started off nicely with me adding some more functionality to our distributed cache testing framework, that’s a peripheral and relatively new code base that I’m pretty familiar and comfortable with by now. But after that things started queuing up and I couldn’t close anything out.
First, I was asked to take a look at a customer issue involving Serialization of Terracotta-instrumented ConcurrentHashMap instances. The instrumenting we do on that class breaks Serialization of CHM instances between non-Terracotta processes and Terracotta processes. I’d like to blog about this separately, and actually I think someone smarter than me could do a PhD dissertation on how Terracotta instruments CHM (there’s a lot of it) and why it’s necessary. So in two days I was really only able to evaluate the bug, get it reproduced in a test case, and decide on an approach, but was not able to actually make a fix. That was disappointing.
Next, I was asked to see about removing the JSR 107 dependency from our ehcache TIM. By this time it was later in the week and I was packing for our trip to Florida, and I only finally found what I think is the problem while en route to Florida without an internet connection.
Thirdly, a teammate and India and I have been trying to do some followup testing on the initial cache testing I last blogged about, but it has been going slower than we’d like due to various roadblocks too boring to mention. Just a little while ago I committed some more functionality to our distributed testing framework which will allow us to vary the number of Segments used in CHM instances when we test, and that should’ve been done last week.
This week I’m going to make a concerted effort to eliminate potential distractions, and to better manage all the various feeds that bombard me. IM has to be on pretty much all the time, so my coworkers can reach me, but otherwise this week I’m going to turn of Twitter, RSS feeds and all e-mail for long stretches at a time and get…things…done! I’m also going to go to bed early (soon) and try to get up very early, take a walk to wake myself up, and get to work early. I’d like to do that all week and see if it helps.
My family and I are in Destin, FL for two weeks. I’m going to work this week and take next week as vacation. I’m excited to be down here and have a change of scenery, but at the same time I’m surrounded by many more potential distractions.
Weekly Summary - Clustered Performance Testing
I haven’t been writing here as much as I’d like to, and truthfully there is a ton of stuff I want to write about! But it’s hard to make the time. Especially when, given extra time, I’d rather just keep working on what I’m doing 
Over the last couple weeks I’ve been running a set of clustered performance tests using our homemade distributed testing framework, nicknamed “Droid”. I went into some detail about this in my last . The testing I did was to measure the cluster-wide throughput (transactions per second) given one, two, four and eight nodes in a cluster (not counting the Terracotta server as a node). We repeated these test using both ConcurrentHashMap and Ehcache as our distributed cache, and we repeated all of the tests with a new (2.6.2) and older (2.5.4) version of Terracotta.
I had a one-on-one with my boss Steve, also one of the co-founders and originally an engineer himself (now head of engineering). We had an interesting discussion about the testing results, and concurrent testing in general. He reminded me to always be aware of unexpected bottlenecks when testing, and always make sure you’re measuring what you think you’re measuring. For example, we designed the test so that none of the machines would be memory or CPU bound - but did I verify that that was in fact the case? Not really. I just set the jvm memory high enough and hoped for the best. We were really trying to get a feel for how Terracotta distributed lock contention would bog down linear scalability as more L1 nodes were added, so Steve’s point was that we don’t want other unexpected resource constraints to mar the measurements.
Late last week, continuing into this week, I started taking a first swipe at collecting cluster-wide statistics in Droid. We already have single-node statistics, but it would save us (primarily Alex) some time if the framework did the number crunching for us. Of course this means we have to use Terracotta and create another distributed object for doing such collecting and processing.
I also spent some time with a new engineer in India, Himadri, trying to get him started on Droid. His development machine runs Windows and I have a MBP, so there’s been some pain there. In particular, we ran into what turned out to be a known (but not by me) issue in our build process that occurs only in Windows.
In other news, so far I absolutely love working out of my home, but on Thursday two weeks ago I experienced the downside. My internet connection went out. Grrr. So I packed up and drove to my parents’ house, but it was out there, too. Stupid of me - my parents and I both have Charter cable, and it turned out that Charter had an area-wide outage that day. At the time I was very irritated - it’s so easy to hate Charter. I finally decided to go to McAlister’s Deli, which has free wifi. I started my working day at 11 o’clock that day. But it ended up being a great experience: the wifi worked fine, and McAlister’s has sweet tea, which to me is like crack cocaine. My boss Alex and I even ended up meeting there last week for a working lunch. Incidentally, Alex has DSL at his house, so chances are good that we won’t both have an outage at the same time.
This Friday I’m leaving with my family for Florida for two weeks. I’m going to work down there the first week.
