Tuesday, December 16, 2008

OS X and Java SE 6

It appears that you can run Java SE 6 on a Mac, but only if you're Intel 64-bit. No PowerPC, no 32-bit Intel.

But from my terminal on a 64-bit Intel machine with all updates:

$ type java
java is /usr/bin/java

$ java -version
java version "1.5.0_16"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_16-b06-284)
Java HotSpot(TM) Client VM (build 1.5.0_16-133, mixed mode, sharing)

In order to get Java SE 6 running on your Mac, open the Java Preferences (at /Applications/Utilities/Java/Java Preferences) and drag Java SE 6 to the top of the list for Java applet versions and/or Java application versions. Now I get:

$ java -version
java version "1.6.0_07"
Java(TM) SE Runtime Environment (build 1.6.0_07-b06-153)
Java HotSpot(TM) 64-Bit Server VM (build 1.6.0_07-b06-57, mixed mode)

Note that you may not be very happy with the result... Given the history of Java support on OS X, perhaps I should try SoyLatte.

Friday, December 12, 2008

git is slow; too many lstat operations


I'm working on a new project and using git for the first time; I really like the distributed source control model, it makes a lot of sense. I come from using p4 for quite a long time, so git feels a lot like CVS/SubVersion, but obviously a lot more powerful than CVS. Also, I want to make sure that I do not say that git is hard to understand and use.

The way I work with my projects is by submitting not only all my source code for my project, but also all tools that are used to generate my project. (Obviously I draw the line somewhere, I don't submit my Linux distribution that I use on the build machine.) That way I can guarantee reproducibility and no one on the team has to install their own tools before being productive.

A short time into my new project I already have more than 500,000 files in my git repository. Now every time I use any git command, it just sits there for many seconds... even when I open a file using Emacs, I have a severe delay because git is invoked to check its status. Using dtrace I've determined that it is doing an lstat on every single file in the repository, to make sure that it's not out of date.

Therefore beware: git is not good for large repositories. And when I say large, I don't mean "sorta large" like 20,000 files. I mean more on the order of 500,000 or 2.5 million files (the current size of our p4 repository). This is because git is just like CVS: it automatically determines for you which files you have edited/deleted/added, and which you have not, but it does this by doing an lstat on every single file in your tree. p4 does not make this assumption, and requires you to tell it what files you have edited/deleted/added, therefore it never does an lstat of the files in your tree (unless you invoke some special commands to ask it to look).

It seems there are attempts to optimize the lstat operations, but it is part of the design of git, and would likely be unnatural to avoid it or suppress it.

I found the git benchmarks, where it describes how efficient it is for the size of its repositories, which is great. However, it does not mention the number of files in a repository. And I couldn't find the current size of the linux kernel repository, but I found a mention that the pull of an entire tree was 5,000 files, which included all of git's metadata (if I understood it correctly).

In order to work around git's slowness with large numbers of files, I decided to split my git repository into two halves; the half with the tools, and the other half with my project source. I found a tool called git split that seems to do the job (seehttp://people.freedesktop.org/~jamey/git-split), but it didn't work on my git 1.6.0 repository. It got stuck because it couldn't read my .git/info/grafts. So I gave up there and just deleted all the tools out of my tree, and added them into a separate git repository. That made things much better.

Update: I found this blog article that also claims that git does not scale: Why Perforce is more scalable than Git

Thursday, December 11, 2008

Hibernate JPA annotations work only on get method, not set


I have a project where I'm using Hibernate with JPA annotations. I got stuck where the annotations to one of my attributes wouldn't stick, but I finally figured it out:


You have to annotate either the member (in which case all attributes must be annotated via member), or the get method for your attribute. You can't annotate the set method.

Sunday, November 23, 2008

Beware Windows exit codes

I have a Java process that is forked from a Windows service; the service process monitors the child Java process to make sure it's OK.

However, I was seeing in my logs that sometimes I would get the exit code 143. My code does not generate this exit code; further, there are no .dmp files or Java crash reports or exceptions logged or any other indication in my code that the process exited. The explanation is simple: my service and all other processes were receiving CTRL_SHUTDOWN_EVENT. My service process ignores this, but Java does not (see IBM's excellent document on Java Signal Handling).

The result is that my Java process would exit, my monitor would not know why, and restart it, at which time it would again get a shutdown event, etc, until the machine actually completed its shutdown.

Needless to say my service process now watches for those events...

Wednesday, November 12, 2008

AddVectoredExceptionHandler prevents Purify from working


In all our Windows executables we AddVectoredExceptionHandler so that we can get .dmp files when things crash. However, I recently discovered that doing so preventsPurify from working correctly on Windows. This should probably be expected, and I'm probably not chaining to the next handler correctly, but it's something to watch out for.

Subtopic #1: In all other operating systems, if you want a core dump file, it's very straight-forward; you ulimit -c unlimited, and you get core files. In Windows, you have to write your own Dump file from code that you write. You could depend on Dr. Watson, or Windows Error Reporting, but if you want your own dump file in a place where you or your customers can find it, you have to write it yourself.

Subtopic #2: Fifteen years ago, Purify was the most important development tool you could have. It's still the most important development tool you could have, except that it hasn't really changed in 15 years. I have several friends that are ex-Pure, and they are also dismayed at how it's just not keeping up. For instance:
  • It wasn't until 2004 (or was it 2006?) that you could really use Purify on Linux. Wow.
  • We still can't Purify JNI C code in a Java JVM on Linux. The JVM does magic things that Purify can't handle. Oh well.
  • Purify for Windows doesn't even work on x64.

Tuesday, November 11, 2008

WDK 6001.18001 NDISPROT 5x cancel code bug


I spent some time determining that the sample code for ndisprot that is included in the 6001.18001 WDK has a bug in the IRP cancel code. What's worse is that the bug only exists in the directory labeled 5x, not the directory labeled 60.

For some (at least 2) of the samples in the latest WDK's, they made a clean break from the NDIS 5.x and earlier sample code when they wrote the samples for NDIS 6.0. In the NDIS 6.0 sample they wrote the cancel code according to the pseudo code in MSDN. Unfortunately, they didn't back-port those changes/fixes to the sample code in the 5x directory (sample code intended for NDIS 5.x)

My full description/conversation is here:

http://www.osronline.com/showthread.cfm?link=142677

Thursday, August 14, 2008

Java SSL connection calls InetAddress.getLocalHost


That sounds innocuous, right?
When you call InetAddress.getLocalhost(), a reverse DNS lookup for your hostname is done. In the worst case, you’ve specified a DNS server that isn’t reachable, and so you have to wait for the DNS timeout, which can be quite long, like 30 seconds or 2 minutes. The reason the crypto code in JCE is doing this is for a random seed generator. Seems you could find something else more random than your hostname…
Below I’ve replicated the sample code that I created for this fix, in case it’s of any use to anyone:
I’ve found what I believe is a workaround to this problem, that seems to work against Java6. It works by setting the system property impl.prefix, and using implementations derived from the following classes:
java.net.PlainDatagramSocketImpl
java.net.Inet4AddressImpl
java.net.Inet6AddressImpl
The override implementations of Inet4AddressImpl and Inet6AddressImpl are designed to make sure that InetAddress.getLocalHost() returns an answer without causing any network access. That means that SSL connections, when constructing their random seed that includes the local hostname, will not hang when DNS cannot be reached.
The reason PlainDatagramSocketImpl is overridden is because the system propertyimpl.prefix is also used to construct it; if impl.prefix is not specified, then a prefix of “Plain” is assumed, and thus PlainDatagramSocketImpl is loaded. Therefore we must provide an implementation that with our own matching prefix.
The main class, DefeatGetLocalHost sets the system property impl.prefix to “DefeatGetLocalHost”. This will cause the following classes to be loaded when they are needed:
java.net.DefeatGetLocalHostDatagramSocketImpl
java.net.DefeatGetLocalHostInet4AddressImpl
java.net.DefeatGetLocalHostInet6AddressImpl
The reason that these derived classes are set in the same package, java.net, is because constructors and methods are package protected; therefore placing them in the same package provides the highest level of compatibility.
Also, in order to get our derived classes in package java.net to load in the Java runtime, we have to append the boot classpath. This is done with: -Xbootclasspath/a: after which we specify the directory with our class files.
In the next comment are the source files that I wrote to demonstrate. Compile it and execute DefeatGetLocalHost using -Xbootclasspath/a: to include the overridden classes.
java/net/DefeatGetLocalHostDatagramSocketImpl.java:
package java.net;

class DefeatGetLocalHostDatagramSocketImpl extends PlainDatagramSocketImpl {
}
java/net/DefeatGetLocalHostInet4AddressImpl.java:
package java.net;
import java.io.IOException;

class DefeatGetLocalHostInet4AddressImpl extends Inet4AddressImpl {

    public String getLocalHostName() {
        System.out.println("Using implementation " +
                           this.getClass().getName() + ".getLocalHostName");
        return "localhost";
    }

    public InetAddress[] lookupAllHostAddr(String hostname)
        throws UnknownHostException {

        System.out.println("Using implementation " +
                           this.getClass().getName() + ".lookupAllHostAddr");

        if (hostname.equals("localhost")) {
            return new InetAddress[] {
                InetAddress.getByAddress(new byte[] {
                        (byte)127, (byte)0, (byte)0, (byte)1
                    })
            };
        }

        return super.lookupAllHostAddr(hostname);
    }
}
java/net/DefeatGetLocalHostInet6AddressImpl.java:
package java.net;
import java.io.IOException;

class DefeatGetLocalHostInet6AddressImpl extends Inet6AddressImpl {

    public String getLocalHostName() {
        System.out.println("Using implementation " +
                           this.getClass().getName() + ".getLocalHostName");
        return "localhost";
    }

    public InetAddress[] lookupAllHostAddr(String hostname)
        throws UnknownHostException {

        System.out.println("Using implementation " +
                           this.getClass().getName() + ".lookupAllHostAddr");

        if (hostname.equals("localhost")) {
            return new InetAddress[] {
                InetAddress.getByAddress(new byte[] {
                        (byte)127, (byte)0, (byte)0, (byte)1
                    })
            };
        }

        return super.lookupAllHostAddr(hostname);
    }
}
DefeatGetLocalHost.java:
public class DefeatGetLocalHost {

    public static void main(String[] args) {
        try {
            safeMain(args);
        } catch(Throwable e) {
            e.printStackTrace();
        }
    }

    private static void safeMain(String[] args)
        throws java.net.UnknownHostException, java.net.SocketException {

        System.setProperty("impl.prefix", "DefeatGetLocalHost");

        System.out.println("Getting localhost:");
        System.out.println(java.net.InetAddress.getLocalHost().getHostAddress());

        System.out.println("Creating DatagramSocket:");
        java.net.DatagramSocket dg = new java.net.DatagramSocket();
        dg.close();

        System.out.println("Success");
    }
}