Java uses /dev/random; may block forever creating SSL connections

The software that we're developing creates SSL connections when it starts up, and it does so at S13 (has to be after network, but before other services start). The result is that on an NFS booted Linux machine, it sits there forever, and never completes the connection.

Clue #1: if you move the mouse or type on the machine's keyboard, eventually the connection will complete.

Of course the reason for it hanging is that Java is using /dev/random to generate the keys for the SSL connection. And /dev/random gets all of its entropy from the physical environment, and refuses to return random values until it gets some input from the outside world.

We don't see this on a machine that boots from disk; I assume that /dev/random gets entropy from the interaction with the drive, via interrupts and so forth. For some reason the network activity doesn't yield the same entropy data, or at least not enough.

I found this article that discusses the usefulness of /dev/random given its current design.

In order to work around this, we decided to use /dev/urandom. We could do this by a link in the file system, but a much superior solution is to set the following system property in Java:

-Djava.security.egd=file:/dev/urandom

Now all you have to worry about is attacks against your SSL connection from those who know that you are using the pseudo-random number generator...

Linux and Java and Threads and setuid

Today I learned something very interesting. I learned that you can't setuid on a process in Linux; not when you have multiple threads. Please see #8 in this list.

What they refer to as interesting times probably includes the following:

• When calling setuid, only the caller thread will actually get its uid changed. All other existing threads in the "process" retain their original uid.
• I believe any sane person should recognize this as meaning Linux is broken when using threads and setuid.
• This is a security hole, because root threads still exist in the process. If the non-root threads are hijacked by an attacker, they can stack stomp on the root threads and execute arbitrary code as root.
• Because synchronization depends on the ability to deliver signals, and delivering signals depends on priviledges, it's easy to see how synchronization between a thread running as root and another running as non-root can wedge the process.
• Even if I did call setuid in the first bytecode instruction in a Java process, it's too late; Java has already forked threads to do things like garbage collection, and those threads present the security hole described above, and the synchronization problem described above.
• I'm sure there's a long list of other reasons why this is bad, but I can't think of them now, and the above is sufficient.

In our project we have a Java process that uses forked processes written in C; the purpose of these forked processes is to run as root, or at least elevated privileges, while the Java process runs as some sort of nobody user. Unfortunately this doesn't work very well at all on Linux because we cannot downgrade the uid of the Java process after it starts.

This also means that if we want to listen on a port under 1024, we'll have to do that some other way; there's no way we could get the Java process to bind to that port as root and then downgrade to a nobody uid.

Also the processes I refer to have to be forked before the JVM starts. This means that we have to rendezvous with them in some manner that either means some sort of JNI code to hook up the file descriptors in the pipes, or use some other form of IPC.

Analysis of why creating socket in Java takes 3 minutes

Here's what we think is happening, thanks to Evan's suggestion to use gdb and Carol's assistance in recreating the loopback's IP address and route:

The first attempt by Java to open a socket is preceeded with an initialization of its socket code.
The socket initialization code calls java.net.PlainDatagramSocketImpl.leave, as is indicated in this stack trace from gdb:

#0  0xb75ebc32 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
#1  0xb75e3bf8 in connect () from /lib/tls/libpthread.so.0
#2  0xaa4b7c24 in Java_java_net_PlainDatagramSocketImpl_leave ()
   from /opt/scalent/jre/lib/i386/libnet.so
#3  0xaa4b8029 in Java_java_net_PlainSocketImpl_initProto ()
   from /opt/scalent/jre/lib/i386/libnet.so
#4  0xb2fa6bf2 in ?? ()
#5  0xb2fa0ddb in ?? ()
#6  0xb2f9e104 in ?? ()
#7  0xb721bb44 in JavaCalls::call_helper ()
   from /opt/scalent/jre/lib/i386/client/libjvm.so
#8  0xb72cfa6d in os::os_exception_wrapper ()
   from /opt/scalent/jre/lib/i386/client/libjvm.so
#9  0xb721bd96 in JavaCalls::call ()
   from /opt/scalent/jre/lib/i386/client/libjvm.so
#10 0xb7200f6f in instanceKlass::call_class_initializer_impl ()
   from /opt/scalent/jre/lib/i386/client/libjvm.so
#11 0xb720569c in instanceKlass::call_class_initializer ()
   from /opt/scalent/jre/lib/i386/client/libjvm.so
#12 0xb72001cb in instanceKlass::initialize_impl ()
   from /opt/scalent/jre/lib/i386/client/libjvm.so
#13 0xb72059af in instanceKlass::initialize ()
   from /opt/scalent/jre/lib/i386/client/libjvm.so
#14 0xb720d6d4 in InterpreterRuntime::_new ()
   from /opt/scalent/jre/lib/i386/client/libjvm.so
#15 0xb2fad510 in ?? ()
#16 0xb2fa0ddb in ?? ()
#17 0xb2fa0ddb in ?? ()
#18 0xb2fa0ddb in ?? ()
#19 0xb2fa0ddb in ?? ()
#20 0xb2fa0ddb in ?? ()
#21 0xb2fa0d04 in ?? ()
#22 0xb2fa0ddb in ?? ()
#23 0xb2fa10e1 in ?? ()
#24 0xb2f9e104 in ?? ()
#25 0xb721bb44 in JavaCalls::call_helper ()
   from /opt/scalent/jre/lib/i386/client/libjvm.so
#26 0xb72cfa6d in os::os_exception_wrapper ()
---Type  to continue, or q  to quit---
   from /opt/scalent/jre/lib/i386/client/libjvm.so
#27 0xb721bd96 in JavaCalls::call ()
   from /opt/scalent/jre/lib/i386/client/libjvm.so
#28 0xb721b666 in JavaCalls::call_virtual ()
   from /opt/scalent/jre/lib/i386/client/libjvm.so
#29 0xb721c1df in JavaCalls::call_virtual ()
   from /opt/scalent/jre/lib/i386/client/libjvm.so
#30 0xb7274f25 in thread_entry ()
   from /opt/scalent/jre/lib/i386/client/libjvm.so
#31 0xb7319caa in JavaThread::thread_main_inner ()
   from /opt/scalent/jre/lib/i386/client/libjvm.so
#32 0xb7315674 in JavaThread::run ()
   from /opt/scalent/jre/lib/i386/client/libjvm.so
#33 0xb72d1083 in _start () from /opt/scalent/jre/lib/i386/client/libjvm.so
#34 0xb75dedac in start_thread () from /lib/tls/libpthread.so.0

• The call in question seems to be to make the machine leave the multicast group. See here.
• Leaving the multicast group must involve connecting to loopback on a random port (the port it chooses changes and is always above 32768), and then shoving some random bytes through. That's my theory. Update: This is almost certainlyrtnetlink.
• It just sits there trying to communicate with itself, and times out after ~3.5 minutes.
• It appears that it has its problem because the loopback device is not configured with an IP address and is not in the route table.
• After we issued the following commands, everything works just fine and the 3.5 minute delay turns into 18ms delay:
  • ip addr add 127.0.0.1/8 dev lo
  • /sbin/route add -net 127.0.0.0/8 dev lo
• We found a most interesting thread on the Java forum that seems to mirror our problem. But the guy apparently never figured it out. Maybe I should post a solutionthere.
• We went through the following other possible problems:
• I always thought it was some kind of nfs file locking problem, but that's not the case at all.
• We redirected the logging output to a local disk on the machine. That didn't help at all.
• We saw the process was blocked reading /dev/random; /dev/random must use loopback to generate random numbers. To solve this we used/dev/urandom which is not as random, but removed the block. But the connection delay persisted. This might explain why it took 50 minutes to send the first message on the SSL connection. Once we removed the block on/dev/random the 50 minute message send delay seemed to go away.
• We wrote some code to try to connect without SSL, but I'm not convinced that ever worked. It's still in the code and can be activated with a configuration setting, and I tested that configuration setting in my client.
• We then thought it was a delay caused by doing a reverse DNS lookup on the peer's IP address--likely so it could do certificate validation/throw nice exceptions. We saw in gdb that the stack trace was deep in some Java code that was trying to do some kind of DNS operation. /etc/resolv.conf was empty, so we added our name server to it and rebooted the machine. That didn't help, but the stack trace changed.
• Then the stack trace was stuck inJava_java_net_PlainDatagramSocketImpl_leave; I thought that might have still been some DNS hosage, so I changed /etc/hosts to include the addresses of the peers. That didn't help and didn't change the stack trace.
• Finally we typed /sbin/ifconfig. That showed us that lo did not have an IP address.
• Carol told us the correct magic commands to type.

Creating a socket in Java takes 3 minutes

Sometimes we would see on some of our Linux boxes a 3 minute delay between an attempt to open a socket and a successful connection. This did not make sense... but I eventually determined that this was caused by the loopback device not having an address, or not having a route in the route table.

You can probably fix this by typing the following:

/sbin/ip addr add 127.0.0.1/8 dev lo
/sbin/route add -net 127.0.0.0/8 dev lo

Also, we had these other symptoms:

• /dev/random would block for about 3 minutes, probably because it depends on loopback to get its results.
• Java trying to do a reverse DNS lookup would block for about 3 minutes, probably because it was trying to get results from 0.0.0.0, because /etc/resolv.conf was empty, and 0.0.0.0 was being interpreted as 127.0.0.1... Update: see this postabout a related fix...

Here is a thread where I replied with this information; the thread also suggests other solutions, perhaps to the same or similar problems.

Now if I could just figure out how that Linux box got into that situation...

libstdc++ and compatibility

I've been studying the C/C++ build and I think I learned some things:

• glibc is inextricably linked with the Linux operating system. You can't run with a new glibc.
• LD_LIBRARY_PATH can affect libc, but cannot affect ld-linux.so.2 (ld-2.3.2.so). It seems you can get around this with chroot, but then you have other problems.
• glibc 2.3.2 has the symbol GLIBC_PRIVATE which is in ld-2.3.2.so, but not inld-2.2.x.
• libstdc++ 3.2 (it comes with g++ 3.2) requires glibc 2.3. Redhat 7 ships with 2.2 or earlier. See previous point. You cannot take a libstdc++ from Redhat 9 and run it on Redhat 7 unless you upgrade glibc and just about everything else in the OS, at which point it's not really Redhat 7.
• libstdc++ is more than STL. It's the C++ runtime and STL. Therefore STLport can never replace libstdc++.
• I can chroot with Redhat 7 (actually Mandrake 8) and get my Redhat 9 compiled binary and libstdc++ 3.2 shared object. However, once I do that I can't do things like read /proc or modify /etc which is something we need to do.
• Starting with g++ 3.2, libstdc++ is attempting to be forward/backwards compatible in its ABI where possible. At this point compatibility was completely broken.
• Redhat 9 ships with compat-libstdc++ which contains the C++ runtime libraries for gcc 2.96 as used in Redhat 7.3. This means C++ stuff compiled on Redhat 7 will work on Redhat 9, but only when this package is installed.
• glibc works very well forward/backwards compatibility-wise, with the GLIBC_2.0,GLIBC_2.1, etc. symbols. If you build a binary that is C only, it's probably going to run anywhere, as long as it's glibc v2 or better, preferably glibc v2.1.
• It is impossible to statically link libstdc++ into an executable when exceptions are thrown/caught. This is because symbols such as _Unwind_DeleteExceptionexist in libgcc.so but do not exist in libgcc.a.

perl and relocating its installation

While setting up our development system and source control, I'm taking the philosophy that all tools are to be checked into source control, not installed on individual machines; in that way a developer's tools are never out of date. Unfortunately some tools don't like this approach, they like to hard-code or "relocate" their position during installation.

One of those is perl.

This link explains a bit how ActivateState relocates perl on install.

What happens is that the @INC path must be embedded in the perl executable on
Unix platforms, or so they claim. When install.sh is run, it calls reloc_perl,
which uses an ActiveState perl module Relocate which then uses this trick to
replace things like
/tmp/.TheInstallScriptWasNotRunTheInstallScriptWasNotRunTheInstallScriptWasNotRun-perl/lib/5.8.0
with the appropriate path. Unfortunately, when I first tried this, the path just happens to be my home directory where I downloaded it.

By the way, there is only 0x80 (128) bytes of space to put the path in, so there is a limit to what location it can be relocated into.

So, the procedure I used to get an ActivePerl that works on anyone's machine no matter where their source directory is mapped to their file system:

• Installed ActiveState Perl normally, into a place such as your home directory: in my case this was /home/jared.oberhaus/p4/tools/linux/ActivePerl-5.8.3.809
• Found all instance of text and binary files under the installation directory that contain /home/jared.oberhaus and replaced them with the original files from the install tar. The original files still have encoded strings such as/tmp/.TheInstallScriptWasNotRunTheInstallScriptWasNotRunTheInstallScriptWasNotRun-perl/lib/5.8.0 inside them.
• Submitted these files to source control as-is.
• Modified ActiveState's install.sh by adding to it (not removing the original install procedures). First it links the magic /tmp path to the file location where the source control version is mapped. This is controlled by detecting where the install script exists and processing that. When reloc_perl executes it will copy everything into /home/user/p4/tools/linux/perl-5.8.3 and at the same time replace the magic /tmp string with the correct location.

Preventing System.exit()

You can prevent System.exit() by setting the appropriate thing in the SecurityManager. Try something like this in your JUnit test:

public void setUp() {
    System.setSecurityManager(new CatchSystemExit());
}
public void tearDown() {
    System.setSecurityManager(null);
}
private static class CatchSystemExit extends SecurityManager {
    /** @see SecurityManager */
    public void checkExit(int status) {
        m_exitCode = status;
        throw new SecurityException("System.exit() attempt caught");
    }
    /** @see SecurityManager */
    public void checkPermission(Permission perm, Object context) {
    }
    /** @see SecurityManager */
    public void checkPermission(Permission perm) {
    }
}