I got pulled into yet another demo, involving building yet another prototype. I managed to get something functional in under a week, in about 2000 lines of Java. The next two weeks involved rewriting lots of the code either due to my misunderstanding of what it was supposed to do, or changes in what it should do after having something available to play with. Since I was the only one working on the code, I felt free to rewrite big chunks of it. The demo ended up being about 3000 lines of Java and seems to have gone successfully.

Two things, which I've said before, apply to this experience.

• Being able to throw away the first few iterations is valuable. The design of later iterations can incorporate issues that weren't forseen in the earlier iterations.


• Static typing makes replacing large chunks of code manageable. The compiler caught numerous little oversights, whereas using a dynamically typed language would have been much more painful.

So, for prototyping and rapid iteration, I strongly prefer using statically typed languages.

For an established code base that has many people working on it, rewriting large chunks of code is less of an option.

• It's harder to understand all of the issues in a large codebase.


• Management conservatism discourages potentially disruptive changes.

After setting up my computer to take pictures of my television screen when finishing a Rock Band song, I can go back and see if I got any new high scores afterwards. However, the leaderboard interface is limited by the controllers and navigating hundreds of songs is clumsy, so I came up with a way to make the computer do most of that work. I can get my scores from rockbandscores.com in csv format, and use OCR (optical character recognition) software to get the name of the song out of the picture.

The scores and song names show up in two different ways. In the middle of a set, the song name appears at the bottom left of the screen. At the end of the set, the song name appears at the top of the screen, in uppercase, and in a different font. There are also lots of non-textual graphics on the screen. However, the OCR software doesn't have to be anywhere near perfect, since all it needs to do is to enable distinguishing between a limited set of a few hundred song titles. So I got ocrad, which is free, lightweight, and fast, though not very sophisticated or flexible.

I send the upper part of each image and the lower left of each image through OCR after processing the piece of the image into monochrome, selecting the whitest and brightest pixels. Originally, I selected for the brightest pixels, but I found that also selecting for the whitest pixels worked much better. There are still lots of non-text clutter left, which the OCR software interprets. So after gathering data on what results the OCR software got from various pictures, I made and refined a heuristic for matching the OCR results with the song titles.

Here are some initial results after getting it to work pretty well. First, a song in the middle of a setlist:


Processing the upper part and the lower left part causes the song title to stand out in the lower left.


The OCR software does a reasonable job on the lower left:

M1dn19htR1d9r. . .
\ a ' ;' '' .. , 1
_-__.

The OCR text is good enough that it matches one song in the scores.csv file, the correct one, and I did not get a new high score:

[Midnight Rider, The Allman Brothers Band, GUITAR:75218 6.18, DRUMS:127100 5.37]

After finishing the set:


Processing the upper part and lower left part of the image causes the song name and score to show up cleanly in the upper part:


The OCR software returns:

FRRNH11N'5 7DWER
225,225 _K1,v,v_

and the A in the picture does look like an R, and the K does look like an H, and all the other misread characters do look like the characters returned by the OCR software. The score was also correctly recognized, but that usually doesn't happen.

Matching the text with the scores.csv file, I got a new high score:

[Franklin's Tower, Grateful Dead, DRUMS:223275 5.98, GUITAR:155807 6.67]

One thing I like to do is play Rock Band. However, I often find that I don't remember what songs I just played in a playlist, and sometimes I want to check and see if I got a new high score. My first thought was to sniff the packets sent to the online leaderboards and pull the songs and the scores out of that data. Unfortunately, the packets are encrypted, so I gave up on that idea. But I had a new idea. I could point my computer's camera at the screen and record a video.

Making a video of the gameplay would eat up disk space and it would be a pain to go through a long video to see all the songs. But, going back to my first idea of sniffing packets, I could have some scheme of taking pictures and saving the ones around the time when packets were sent to the leaderboards. After finding out how to take pictures using the quicktime API and trying it out, I wrote some code that took a picture every 5 seconds, and interleaved the pictures with tcpdump output. I found that 32 byte packets were being sent every few seconds to Xbox Live, and 80 byte packets were being sent every once in a while, but larger packets were always sent after finishing a song. So then I wrote it to take a few pictures when seeing the larger packets. It worked really well. It takes some extraneous pictures due to other packets. Fortunately, I don't have lots of Xbox friends that would cause lots of notification packets.

Now I can play a dozen or so songs, and then go back and see how I did, instead of worrying about not remembering what I just played after playing only 3 or 4 songs.

import java.io.BufferedReader;
import java.io.File;
import java.io.FileOutputStream;
import java.io.InputStreamReader;
import java.io.PrintStream;
import java.util.Date;
import java.util.concurrent.ArrayBlockingQueue;
import java.util.concurrent.TimeUnit;
import quicktime.QTSession;
import quicktime.io.QTFile;
import quicktime.qd.Pict;
import quicktime.qd.QDConstants;
import quicktime.qd.QDGraphics;
import quicktime.qd.QDRect;
import quicktime.std.StdQTConstants4;
import quicktime.std.image.GraphicsExporter;
import quicktime.std.sg.SequenceGrabber;

public class Scorer {
    private static ArrayBlockingQueue<Long> queue = new ArrayBlockingQueue<Long>(20);
    private static boolean running = true;

    private static void capturePhoto(String filename) throws Exception {
        try {
            QTSession.open();
            QDRect qdRect = new QDRect(640, 480);
            QDGraphics qdGraphics = new QDGraphics(qdRect);
            SequenceGrabber sequenceGrabber = new SequenceGrabber();
            sequenceGrabber.setGWorld(qdGraphics, null);
            GraphicsExporter graphicsExporter = new GraphicsExporter(StdQTConstants4.kQTFileTypePNG);
            graphicsExporter.setInputPicture(Pict.fromSequenceGrabber(sequenceGrabber, qdRect, 0, 0));
            graphicsExporter.setOutputFile(new QTFile(filename));
            graphicsExporter.doExport();
        } finally {
            QTSession.close();
        }
    }

    private static void recorder(String dir) {
        try {
            PrintStream out = new PrintStream(new FileOutputStream(dir + "index.html"));
            out.println("<html><body>");
            int i = 0;
            long t = 0L;
            while (running) {
                queue.take();
                if (System.currentTimeMillis() - t < 30000L)
                    continue;
                out.println("<br>Start:" + new Date() + "<br>");
                for (int j = 0; j < 3; j++) {
                    capturePhoto(dir + i + ".png");
                    out.println("<img src=\"" + i + ".png\">");
                    i++;
                    TimeUnit.SECONDS.sleep(1);
                }
                out.println("<br>End:" + new Date() + "<br>");
                queue.clear();
                t = System.currentTimeMillis();
            }
            out.println("</body></html>");
            out.flush();
            out.close();
        } catch (Exception e) {
            e.printStackTrace();
        }
    }

    private static void sniffer() {
        try {
            Process process = new ProcessBuilder("/usr/bin/sudo", "/usr/sbin/tcpdump", "-n", "-l", "port", "3074").start();
            BufferedReader in = new BufferedReader(new InputStreamReader(process.getInputStream()));
            while (running) {
                String s = in.readLine();
                if (s == null)
                    break;
                if (s.endsWith("length 32") || s.endsWith("length 80"))
                    continue;
                queue.offer(System.currentTimeMillis());
            }
            process.destroy();
        } catch (Exception e) {
            e.printStackTrace();
        }
    }

    public static void main(String[] args) throws Exception {
        new File("/tmp/score").mkdir();
        new Thread() { public void run() { recorder("/tmp/score/"); } }.start();
        new Thread() { public void run() { sniffer(); } }.start();
        System.in.read();
        running = false;
    }
}

I played around a little with Clojure's Java interoperability at work. Since I made making a new module in the main product that I work on a matter of putting some classes and some xml in a jar file and putting that jar file and any jars it depends on in a directory, it shouldn't be a problem.

The Clojure code was all straightforward, using gen-class. How to actually get the class files generated was all in the documentation, but having the correct classpath was a big snag for me, especially the need to have ./classes in the classpath, which kept causing mysterious NoClassDefFoundErrors. After getting that figured out, I stuck it into ant:

    <target name="compile">
        <mkdir dir="${build.dir}"/>
        <java classname="clojure.main">
            <arg value="-e"/>
            <arg value="(set! *compile-path* &#34;${build.dir}&#34;) (compile 'name.of.test.ModuleClass)"/>
            <classpath>
                <fileset dir="${lib.dir}">
                    <include name="*.jar"/>
                </fileset>
                <fileset dir="${clojure.home}">
                    <include name="*.jar"/>
                </fileset>
         <pathelement location="${src.dir}"/>
         <pathelement location="${build.dir}"/>
            </classpath>
        </java>
    </target>

Then, after getting the jar files built, I dropped the jar file and clojure.jar into the directory and got a giant stack trace, leading to

Caused by: java.io.FileNotFoundException: Could not locate clojure/core__init.class or clojure/core.clj on classpath: 
        at clojure.lang.RT.load(RT.java:402)
        at clojure.lang.RT.load(RT.java:371)
        at clojure.lang.RT.doInit(RT.java:406)
        at clojure.lang.RT.(RT.java:292)
        ... 45 more

I hadn't the classloader set properly for the initial loading of the modules. For all the other modules, there wasn't much done in the class initializers, so this problem didn't come up. I had the classloader set for all other operations, though, so it was merely a matter of doing the same thing at initialization, and it worked.

I also noted that I handled the classloader by hand-creating a proxy class and implementing each method to wrap the classloader switch,

    final ClassLoader handlerClassLoader = handler.getClass().getClassLoader();
    return new Handler() {
        public Object getObject(Object parameter) {
            ClassLoader cl = Thread.currentThread().getContentClassLoader();
            Thread.currentThread().setContextClassLoader(handlerClassLoader);
            try {
                return handler.getObject(parameter);
            } finally {
                Thread.currentThread().setContextClassLoader(cl);
            }
        }
    };

which I could improve by using java.lang.reflect.Proxy, which I could use to wrap all the calls with one method. The code was about 100 lines shorter, and would no longer have to be changed if the Handler interface changed. I'll check in these changes, but not the test module in Clojure.