Bulkheads
Bulkheads are used in ships to create seperate watertight compartments which serve to limit the effect of a failure – ideally preventing the ship from sinking. The bold vertical lines in Samuel Halpern’s diagram illustrate them:
If water breaks through the hull in one compartment, the bulkheads prevent it from flowing into other compartments, limiting the scope of the failure.
This same concept is useful in the architecture of large systems for the same reason – limiting the scope of failure.
If we look at a very simple system, say something that easily partitions by user, like a wish list of some kind. We can put bulkheads in between sets of app servers talking to distinct databases. In this system a given app server only talks to the database in its partition.
Given this setup, if a single app server goes berserk and starts lashing out with a TCP hatchet at everything it talks to, no matter how angry it gets it only takes out a vertical slice of the system, the rest goes about business happily.
If we take a slightly fancier system (ie, slightly more realistic) we can see we develop (mostly) identical vertical slices:
On a ship we’d call the groups compartments, but we’ll call them clusters because each vertical bunch of stuff forms a logical unit which can be thought of as one thing (say, a cluster!). In this setup, if one of the caches started blackholing requests the damage done (hopefully just latency increasing a small bump to a reasonable timeout) would stop at the bulkheads around the cluster. Yea!
If we look closely at the slightly fancier system, we note that a cluster consists of:
- 3 App Servers
- 2 Caches
- 2 Log Servers
- 4 Somethings
- 1 Database
Typically, you can use clusters as units by which to add capacity, and the exact contents of the cluster will be determined by finding the limiting element (usually the one which needs to maintain lots of state), on the most constrained axis of scale embodied in the cluster, and sizing out the rest of the elements based on their capacity relative to the limiting element. Add to this enough capacity to handle spikes, provide acceptable redundancy, and voila, you have a cluster. In theory.
In practice, some things simply do not work well with hard boundaries like this. In this example, note that the load balancers are not part of a cluster, but span clusters – they need to as they are responsible for determining which cluster can handle a given request!
It gets worse, notice that we have two log servers per cluster. Given a reasonable number of clusters, say 25, that amounts to 50 log servers. A single log server (in this case) is capable of servicing about 1000 app servers, but logs are really important so we need to run them redundantly, hence two per cluster. Given 25 clusters and three app servers per cluster, a single log server has plenty of capacity, yet we have 50 for fault isolation in this setup. The accountants are not happy.
Another variant on the inefficieny problem are the Somethings. Somethings utilization is very bursty. Under average conditions one is enough for each cluster, but they occasionally (once a week or so) burst to four times that activity, so we have four, in each cluster. Now, we know the usage pattern is such that bursts don’t overlap, so we’d really like to have shared burst capacity rather than per-cluster, but that will then breach the bulkhead.
This leads to system classification based upon the scope of failure directly causable by a given component (how is that for absract sounding?). If we revise our system to make the accountants and engineers happier (centralized things are generally easier to build) it might look like this:
The system is now much more efficient, but we have increased the risk of a cascading failure taking down more, or all, of the system if any of the load balancers or log servers are effected, or if two something’s burst at the same time (I am sure you have heard someone say, “but that can never happen!” before, right?).
We’ll call services which span clusters Class A services, and services are fully contained within a cluster Class B services. Working with Class A services is harder than Class B because you need to be extra special super really really careful that the Class A service cannot take down your Class B service.
A famous example, going back to the naval usage of bulkhead, of a Class A service would be the passenger decks (the E deck in particular) on the Titanic.
The Titanic was built to suffer failure in a number of its clusters^W compartments, but in fact it suffered failure in too many, and the cluster spanning passenger decks allowed the water to cascade across bulkheads, leading to tragedy.
Treasure Trove: jmxutils
Gianugo and I used to do a talk at JavaOne called “The Long Tail Treasure Trove.” The goal of the talk was to introduce at least thirty or so small, open source, useful libraries which the majority of attendees had never heard of. They were great talks. We haven’t done one, in a while, so at Henri’s prompting (very belatedly), I’m just going to start blogging them!
So, we’ll start with a great one – Martin’s JMX exporting library, jmxutils. Writing JMX beans tends to be agonizing, so unless someone is holding a blowtorch to you toes, you avoid it. Well, it doesn’t need to be, examine this:
public class Something
{
private volatile String color;
@Managed(description="Favorite Color")
public String getColor() {
return color;
}
@Managed
public void setColor(String color) {
this.color = color;
}
@Managed
public void pickOne(String first, String second) {
this.color = second;
}
}
MBeanExporter exporter
= new MBeanExporter(ManagementFactory.getPlatformMBeanServer());
Something s = new Something();
exporter.export("org.skife.example:name=Something", s);
This little tidbit exports a JMX Bean, building a model mbean by inspecting the annotations. This particular one exports a mutable property (color) and an operation (pickOne). Nicely, it uses Paranamer (subject for another post) to even get the parameter names on the operation right!
Now, to get it into maven central…
Emacs 23.1, for the Designers
Embedding Clojure
There is some information out there on embedding Clojure in Java, but it isn’t the easiest to find, and the examples don’t tend to come with explanations, so… here is yet another!
Let’s take a silly example and say we want to embed clojure as a validation language on something, so that it looks something like this
public class Thing
{
private int num = 0;
@Validate("(> num 0)")
public void setNum(@Name("num") Integer num) {
this.num = num;
}
@Validate("(< first second)")
public void setInOrder(@Name("first") Integer first,
@Name("second") Integer second) {
this.num = first + second;
}
}
We want the validation function, expressed in the @Validate annotation to be invoked on every call to the method, binding the appropriate parameters to their @Name, etc. That is, for the second one, we want ensure that first is less than second, and so forth. We want it to be really fast – the validation will be called on every invocation of the validated method, so we need it to be really fast.
While fairly contrived, and rather absurd, it makes a nice example :-)
What we’d like to do is hold a reference to an otherwise anonymous clojure function (we don’t want to pollute the global namespace) and invoke it on every method call with some kind of method interceptor.
We can create the Clojure function reference with something like:
public IFn define(String func) throws Exception {
String formish = String.format("(fn [val] (true? %s))", func);
return (IFn) clojure.lang.Compiler.load(new StringReader(formish);
}
/* ... */
IFn fn = define("(> val 0)");
assertTrue((Boolean) fn.invoke(7));
The clojure compiler (inconeniently in Java 6) is named Compiler and provides a handy load(String) function which will read and evaluate a String, returning whatever it evaluates to. In this case we return a function which wraps our validation function in a test for true-ishness. In this example, our passed in value has a hard coded name, val, which is unfortunate, but can be worked around.
We can invoke this function directly via one of its’ invoke methods – it has a ton of overloads for different argument counts.
This approach will generate a Java class (well, a .class anyway) implementing our function.
To wrap behavior of a class, rather than an interface, and in a performant way, we’ll break out the ever-scary-but-awesome CGLIB and create a runtime extension of the class being validated. CGLIB is fast, but you pay for that with some gnarly low-level-feeling hackey. Not as low as ASM, though :-)
Our object factory looks like
public <T> T build(Class<T> type) throws Exception {
Enhancer e = new Enhancer();
e.setSuperclass(type);
List<Callback> callbacks = new ArrayList<Callback>();
callbacks.add(NoOp.INSTANCE);
final Map<String, Integer> callback_mapping = new HashMap<String, Integer>();
int count = 0;
for (Method method : type.getDeclaredMethods()) {
if (method.isAnnotationPresent(Validate.class)) {
callbacks.add(new Handler(method));
callback_mapping.put(method.getName(), ++count);
}
else {
callback_mapping.put(method.getName(), 0);
}
}
e.setCallbacks(callbacks.toArray(new Callback[callbacks.size()]));
e.setCallbackFilter(new CallbackFilter()
{
public int accept(Method method) {
Integer i = callback_mapping.get(method.getName());
if (i == null) {
return 0;
}
else {
return i;
}
}
});
return (T) e.create();
}
Which is pretty gnarly. Basically, for methods without the @Validate annotation, it provides a NOOP, passing through to the parent class, for methods with the annotation, it delegates to a special Callback. We do some calback filter hackery to allow it to avoid dynamically dispatching at runtime (like a reflection proxy would), allowing CGLIB to generate a method body that invokes out handler directly. None of this is really what we are interested in, the Handler has the good stuff, let’s see it.
private static class Handler implements MethodInterceptor
{
private final IFn fn;
private int[] boundParamOffsets;
private final Method method;
private Handler(Method m) throws Exception {
final Validate v = m.getAnnotation(Validate.class);
Annotation[][] panno = m.getParameterAnnotations();
ArrayList<String> names = new ArrayList<String>();
List<Integer> counts = new ArrayList<Integer>();
for (int i = 0; i < panno.length; i++) {
Annotation[] param_annotations = panno[i];
for (Annotation a : param_annotations) {
if (a instanceof Name) {
names.add(((Name) a).value());
counts.add(i);
}
}
}
final StringBuilder args = new StringBuilder();
for (String name : names) {
if (name != null) {
args.append(name).append(" ");
}
}
String form = format("(fn [%s] (false? %s))",
args.toString(),
v.value());
fn = (IFn) load(new StringReader(form));
this.boundParamOffsets = new int[counts.size()];
Class[] arglets = new Class[this.boundParamOffsets.length];
for (int i = 0; i < this.boundParamOffsets.length; i++) {
arglets[i] = Object.class;
this.boundParamOffsets[i] = counts.get(i);
}
method = fn.getClass().getDeclaredMethod("invoke", arglets);
}
public Object intercept(Object o,
Method method,
Object[] objects,
MethodProxy proxy) throws Throwable {
Object[] args = new Object[boundParamOffsets.length];
for (int i = 0; i < boundParamOffsets.length; i++) {
args[i] = objects[boundParamOffsets[i]];
}
final Boolean bad = (Boolean) this.method.invoke(fn, args);
if (bad) {
throw new IllegalArgumentException("Failed validation!");
}
return proxy.invokeSuper(o, objects);
}
}
YIKES! Okay, this is where it gets ugly, though it is a darned nice example of why macros are handy… if your language supports em. Java doesn’t, so it’s ugly.
We’ll step through it, though. The constructor wants to build three things, the Clojure function (fn), an array of offsets into the parameter list for the named parameters (boundParamOffsets), and finally grab the Java Method for the correct invoke on the Clojure function so we can invoke it via reflection (okay, we could optimize one step further and create concrete invoker classes which do it without reflection, but it’s getting late) imaginatively named method.
The first chunk of the constructor finds the relevant annotations and builds up a list of their names, in the order they appear, as well as the offsets, in that same order. Luckily, we are going to define a wrapper function which binds them, so we can control the order.
final Validate v = m.getAnnotation(Validate.class);
Annotation[][] panno = m.getParameterAnnotations();
ArrayList<String> names = new ArrayList<String>();
List<Integer> counts = new ArrayList<Integer>();
for (int i = 0; i < panno.length; i++) {
Annotation[] param_annotations = panno[i];
for (Annotation a : param_annotations) {
if (a instanceof Name) {
names.add(((Name) a).value());
counts.add(i);
}
}
}
final StringBuilder args = new StringBuilder();
for (String name : names) {
if (name != null) {
args.append(name).append(" ");
}
}
At the end, we build up a list of the names in the form they’ll be embedded into our function template
String form = format("(fn [%s] (false? %s))",
args.toString(),
v.value());
fn = (IFn) load(new StringReader(form));
This looks a lot like the earlier example of function definition, the main difference being that we are templating in the names of the arguments now, so that they match what is expected.
The last bit of the constructor,
this.boundParamOffsets = new int[counts.size()];
Class[] arglets = new Class[this.boundParamOffsets.length];
for (int i = 0; i < this.boundParamOffsets.length; i++) {
arglets[i] = Object.class;
this.boundParamOffsets[i] = counts.get(i);
}
method = fn.getClass().getDeclaredMethod("invoke", arglets);
just stores off the bound parameter offsets and grabs a handle on the Method, straightforward… compared to the rest, anyway.
The rest of the fun stuff happens on method invocation:
public Object intercept(Object o,
Method method,
Object[] objects,
MethodProxy proxy) throws Throwable {
Object[] args = new Object[boundParamOffsets.length];
for (int i = 0; i < boundParamOffsets.length; i++) {
args[i] = objects[boundParamOffsets[i]];
}
final Boolean bad = (Boolean) this.method.invoke(fn, args);
if (bad) {
throw new IllegalArgumentException("Failed validation!");
}
return proxy.invokeSuper(o, objects);
}
In this case, we build up the array of arguments to pass to the clojure function, invoke it, and if it returns true, we raise an exception. Otherwise, we pass the invocation on to the parent class.
Whoo! It’s a fair number of hoops to jump through, but the performance requirement is to blame for a lot of them. The code can probably be simplified a lot, but… it works, and is damned fast. In testing it, the Clojure IFn invocation was nearly indistinguishable from a pure-Java Callable invocation in the informal microbenchmarks I ran. Not really surprising considering it is generating a class to implement the function… Adding a type hint on the arguments actually lead to the Clojure one being frequently faster… somehow, I am not sure how. Need to pull out javap to see what is being generated, but I digress :-)
Some key takeaways from this exercise, for me, were a mind shift in how I think about Clojure and Java interop. I am used to working with languages which embed and have their own runtime, like Lua. Clojure doesn’t get embedded, it just coexists. It doesn’t have its’ own runtime – you invoke functions and manipulate state (not that the final form we see here does that) directly via APIs which look like Java reflection or references, respectively. I really like it.
To wrap up, here is the test case for the whole thing:
package org.skife.example;
import static org.testng.Assert.assertEquals;
import org.testng.annotations.BeforeMethod;
import org.testng.annotations.Test;
public class TestValidatemajig
{
private Thing t;
@BeforeMethod
public void setUp() throws Exception {
t = new Validatemajig().build(Thing.class);
}
@Test(expectedExceptions = IllegalArgumentException.class)
public void testValidationFailure() throws Exception {
t.setNum(-1);
}
@Test
public void testValidationSuccess() throws Exception {
t.setNum(1);
assertEquals(1, t.num);
}
@Test(expectedExceptions = IllegalArgumentException.class)
public void testMultipleParamFailure() throws Exception {
t.setInOrder(2, 1);
}
@Test
public void testMultipleParamSuccess() throws Exception {
t.setInOrder(1, 2);
assertEquals(3, t.num);
}
public static class Thing
{
private int num = 0;
@Validate("(> num 0)")
public void setNum(@Name("num") Integer num) {
this.num = num;
}
@Validate("(< first second)")
public void setInOrder(@Name("first") Integer first,
@Name("second") Integer second) {
this.num = first + second;
}
}
}
Hopefully more Clojure fun to come, though after wrestling with doing Clojure from Java, I may just switch to Clojure and do some Java from that angle.
Setting up TokyoCabinet and Ruby
I ran into a couple weirdnesses setting up tokyocabinet and the Ruby API, so am adding this to my external memory. Hopefully it will help anyone else bumping into the same issue.
Assuming you install tokyocabinet at a non-standard location, such as /Users/brianm/.opt/tokyocabinet-1.4.27 and then want to build the ruby bindings for it via a gem, the trick is to add the bin/ directory for the tokyocabinet install dir to your $PATH (in my case, that is just export PATH=/users/brianm/.opt/tokyocabinet-1.4.27/bin:$PATH). The ruby API’s extconf.rb shells out to tc’s tcucodec to find paths to libraries, etc. Alternately you could modify the extconf.rb, which is very short and sweet, but I hate doing that for aesthetic reasons.
To build the gem, you need to build via extconf but not install. After the build, use the normal gem tokyocabinet.gemspec command to build a gem. Install the gem (in my case, via rip) and glod’s your uncle.
Now to figure out if anyone has done a convenience API wrapper around the table database in TC…