tr ouwens

by the way: things I want to say

On profiling

Several times now, I’ve been in a situation where we would have some performance problem, and my team mates would “know” immediately what the problem was and set to work. “No, no,” I would say, “we must profile first. It could be something else than we think!” And then they’d ask me for an example, and I couldn’t come up with one, and they wouldn’t believe me, or they would believe me but think using a profiler is too much work, and they would go ahead with their original idea, and after some work, they would find out that they gained some improvement but not nearly enough. And they would be disappointed and then I would teach them to use the profiler, and profile for 15 minutes, and find something silly to fix, and get a 75% speed increase.

I’m writing this down so next time I can’t come up with an example, I can refer back to this post.

Yesterday, I was doing my yearly foobal touch-up. One of the things I wanted to fix was a performance problem where, as the year progressed and new scores were added to the data set, the program would get increasingly slow, up to the point where I’d have to remove old data from the data set if I ever was going to get an answer from the damn thing.

Here’s how the program works:

  1. Scrape the latest match results from some website,
  2. Append them to a huge xml file with all the data,
  3. Read the xml file and convert all the elements into data objects,
  4. Inject the data objects into the rule engine,
  5. Let the rule engine do its thang,
  6. Print the answer to the screen.

There were two things that I suspected might cause the slowness. The first was reading the xml and converting it into data objects: there’s a lot of data and xml might not be the most efficient way to store that. It’s a lot of string parsing, and a csv file might serve just as well. The other one was the rule engine: I’m not an expert rule engine developer, and I know at least one of the rules I wrote is very awkwardly implemented. Given the amount of data it has to process, and the fact that I don’t even know if it performs linearly, quadratically, or even exponentially, I figured it might also be the cause of the slowness.

But which of these two suspicions was the real culprit? I had no idea, so I busted out the profiler. (By the way, did you know Oracle ships for free with the Java JDK? It’s called VisualVM and it’s actually very good. Go try it out!)

After about 10 minutes, I found out that Foobal was spending most of its time not in the rule engine. It was also spending most of its time not in the xml parser. No, it was spending most of its time in the org.joda.time.LocalDate.toDate() method. What!?

Turns out I use that method only once in my application, in the data object that goes between the xml and the rule engine. Here it is:

case class Outcome(
    homeTeam: String,
    outTeam: String,
    homeScore: Int,
    outScore: Int,
    date: LocalDate) {
  
  def millis: Long = date.toDate.getTime

  // other methods elided for brevity
}

I added the millis method because I like to use Joda-Time, but the rule engine doesn’t. Using Longs makes date comparisons a lot easier to do for the rule engine. But why does the program spend so much time there?

If you’re familiar with Scala, you will see that millis is a def, meaning that the body of the method is evaluated each time it’s called. Turns out that it gets called a lot. Like, really a lot. So I changed it into a val, making it a property which gets evaluated only once, when the object is created.

Here’s a graph of the speed-up that I got out of that.

Speed-up

Oh, by the way: the scale of the vertical axis is logarithmic. OMG.

So the moral of the story: ten minutes of profiling and 2 seconds of changing 3 bytes, saved me a lot of time rewriting the xml to csv or messing with a rule engine that I don’t understand fully.

Another moral of the story is that I, too, am not immune to the premature optimization bug, either. If I’d had only one supsicion, instead of two, I might never have fired up VisualVM to find out the cause was actually a third thing that I would never have come up with otherwise.

Let me finish up with Donald Knuth’s quote about premature optimization:

“Programmers waste enormous amounts of time thinking about, or worrying about, the speed of noncritical parts of their programs, and these attempts at efficiency actually have a strong negative impact when debugging and maintenance are considered. We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil. Yet we should not pass up our opportunities in that critical 3%.”

Comparing ints and shorts in C#

My colleague Ralph and I recently discovered an interesting bit of C# equality behaviour. Consider the following piece of code:

[Test]
public void Equals()
{
    int i = 42;
    short s = 42;

    Assert.IsTrue(i == s);
    Assert.IsTrue(s == i);
    Assert.IsTrue(i.Equals(s));
    Assert.IsTrue(s.Equals(i)); // fails
}

One would expect the last assert to pass, just like the others. The fact that it doesn’t, tells us two things:

  • The behaviour of == and Equals is different on C#’s primitive integral types. At least, when comparing objects of (slightly) different types.
  • Equals is not symmetric when comparing ints with shorts.

Both are surprising to me. While I don’t like pointing out bugs in things like the .NET Framework (it’s like farting: he who points it out, is usually at fault himself), these do seem to qualify; especially the a-symmetry in Equals, which violates the contract.

There’s probably some arcane piece of .NET legacy at work here, so of course, I sent a question to Eric Lippert. I hope I’ll get a response; I’ll post an update if and when I do.

In the mean time: can you, dear readers, offer an explanation?

A better way of typing

I’m happy! I’m happy because I just found out how to get a Compose key on my shiny MacBook. I want you to be happy too, so I’ll show you how you can get one, too. I’ll even show you how to do it on Windows and Ubuntu. But first, I will explain why this makes me so happy. (If you don’t care about the why, just click here and go straight to the how. But you’ll miss a perfectly good geek-out.)

Why?

I’m a language geek. I like to spell words correctly. Words such as naïveté. Actually, I’m also Dutch, so I have to deal with words like coëfficiënt and of course and (which, really, are two very distinct words). I speak French (or should I say: français) and I live in a country where we pay with . And if that’s not enough, I like to spell people’s names correctly. I’m a software engineer; we’re a pretty international bunch. I’ve worked with people called René, Radovanović, and even Enikő. Yes, that’s an o with a double accent aigu. It’s Hungarian. Try finding that key on your keyboard.

I want it to be easy to type these special characters. I also want it to not get in the way.

Windows has this nifty trick. If you use the US International keyboard layout, you can press a " followed by an e and you automatically get an ë. Sounds nice in theory, but for me, it gets in the way. I’m a programmer; I need to be able to type things like String vowel = "e";. And when I do, I don’t want that to show up like String vowel = ë";. That’s just annoying. On a day to day basis, I have to type "e much more often than I have to type ë.

Of course, if I type a space after I type ", I get my precious ", and I can then type an e and it won’t turn into an ë anymore. I know people, programmers like me even, who have this extra key stroke ingrained in their muscle memory. If they work on a computer where this option is disabled, they type things like String vowel = " e";. Notice the extra space? Now I have nothing against these people, but this is just plain stupid. It’s like hitting your face every five minutes hoping to catch a fly.

OS X has something slightly smarter. To get a special character, you press a special key combination. For example, to get an é, you press ⌥E, followed by e. If you want an ë, you press ⌥U, followed by e. The problem with those combinations is that they’re pretty arbitrary: e for ´, u for ¨, and i for ˆ. In fact, I can never remember what key corresponds to what symbol. It’s not easy.

Also, both methods only support a very limited set of special symbols. Enikő is out of luck; she has to hunt through the Character Map program to spell her name on a non-Hungarian keyboard. If you think that’s too exotic, then you should realise that isn’t directly supported on many systems, either.

I’m not even going to mention Alt codes.

Surprisingly, Linux, for all its usability-issues, has an easy and elegant solution to this mess: the Compose key. You pick a key you don’t use often (I like the Menu key: nobody uses that button anyway) which becomes the Compose key. If you want to enter a special symbol, you hit this button, followed by the two (or more) symbols that you want to ‘compose’. For example, Compose followed by " followed by e becomes ë. Compose followed by = followed by o becomes ő, and Compose followed by = followed by e becomes . Easy!

And it goes even further than that. Combine - and >, and you get an arrow: , T and M become , < and 3 become , and, I was surprised to find out while researching this article, Compose-C-C-C-P becomes . Seriously.

I like this. It doesn’t get in the way at all. I’m free to type String vowel = "e"; without any ë’s showing up. Also, it’s super easy. In fact, it’s so easy that sometimes when I’m bored, I amuse myself by trying out various combinations of keys to see what comes out. How would you type Æ, ¿ or ©?

By now, I have probably convinced you that you want to have a Compose key, too. So how do you get one? It’s easy.

How?

  • OS X: Install the US custom keyboard layout: download and installation instructions are here.
    Note that MacBooks don’t have a Menu key. This tool uses the § key instead, which is fine, because who uses it anyway? And even if you do, it’s a Compose-s-o away.
  • Windows: Install this nifty little program.
  • Ubuntu: Go to System Settings → Keyboard Layout → Options → Compose key position.
  • Any other Linux? Then you probably already know how to do this.

Enjoy a better way of typing! ☺

Foobal: A New Hope

This weekend, a new soccer season started! It’s way past time I told you all how I, or rather, how Foobal did last season.

As you may remember, Foobal started out pretty well. After half a season, it was still doing quite OK. And then… not so much. Unfortunately. Foobal and I finished fourth; a position shared with three other people. In other words: slightly above the middle.

But I shan’t be deterred! This year is going to be better. Actually, I know this for a SOLID FACT, because when I was dusting off the code and preparing the program for the new season, I discovered a bug. A bug that had been there the whole time. A bug that caused Foobal to ignore over 30% of all the data it could have used. A bug that caused foobal to ignore all matches that were played before the 10th of each month.

Oh dear.

How I haven’t seen this sooner, I will never know. It’s not like I didn’t notice many matches were missing from the data file… But here we are.

It turns out that the website I’m scraping my data from, does something unexpected for single-digit dates: it pads them with an extra space. So, instead of mar 2 or mar 02, it actually prints mar  2 in the HTML. Note the extra space between mar and 2.

The code that parses this date, uses a regex that splits on spaces. So in the case of mar 12, it splits into mar and 12, and then passes the 2nd part to the number parsing routine, which is fine. However, in the case of mar 2, it splits into mar,   and 2. And when it passes the 2nd part, which happens to be the empty string, into the number parsing routine, it throws an exception… which promptly gets eaten, never to be seen again. If ever you need an argument for a fail-fast architecture, this is it.

To make matters even more amusing, the entire fix consisted of one. single. character. (OK, I updated some unit tests, too. But still.)

So! This year will be a lot better, I’m pretty sure. Even though Foobal’s first prediction of the season was already incorrect. At least it’s going to be based on much more accurate data.