Zoneminder: Got unexpected memory map file size

I'm trying to capture video from a device that's 640x480 24bit color, but I'm getting this in /var/log/syslog:

ERR [Got unexpected memory map file size 9217192, expected 36865192

Google-ing for this doesn't turn up anything except the Zoneminder source code; but it turns out that this is caused by too small a setting in /proc/sys/kernel/shmmax. This is explained the Zoneminder FAQ.

After updating my /proc/sys/kernel/shmmax to the recommended 128MB, it works just fine...

EasyCap DC60 and Ubuntu 11.10

I got an EasyCap DC60 from Amazon to see if it would work with my Ubuntu box to capture video. It looked promising at first... The output from dmesg looks like this:

[ 3391.497363] easycap::0adjust_standard: selected standard: PAL_BGHIN
[ 3391.808248] easycap::0adjust_format: sought:    640x480,UYVY(0x59565955),1=field,0x00=std mask
[ 3391.808256] easycap::0adjust_format: sought:    V4L2_FIELD_NONE
[ 3391.808264] easycap::0adjust_format: actioning: 640x480 PAL_BGHIN_AT_640x480_FMT_UYVY-n
[ 3391.832121] easycap::0adjust_brightness: adjusting brightness to  0x7F
[ 3391.856121] easycap::0adjust_contrast: adjusting contrast to  0x3F
[ 3391.880121] easycap::0adjust_saturation: adjusting saturation to  0x2F
[ 3391.904122] easycap::0adjust_hue: adjusting hue to  0x00

and it shows up as a video device just fine:

$ ls -altr /dev/video0 

crw-rw----+ 1 root video 81, 0 2011-11-05 22:58 /dev/video0

But when actually using it, it shows an image at first, and then shows grey bars. Note that the above message shows PAL, but I've tried NTSC, and other encodings with the same result. I found a thread about this on SourceForge.net, but the SourceForge project appears to be empty now. Google Cache currently contains the thread. The interesting pieces of text are:

---

The vertical grey bars usually mean that the front-end video chip cannot lock onto the analogue input signal.  This can happen if, for example, the video signal is really NTSC while the mplayer command line specifies PAL.  But in these circumstances, I would not expect to see any video output at all (apart from the grey bars).  The puzzling thing is that you do get some valid video, which then disappears.

---

One possibility is that the SAA7113H chip is raising a hardware "lost signal" flag when in reality the signal remains entirely viable (some aspects of this chip are known to be flaky).  To investigate this, it would be useful if you could reinstall the driver after changing line 31 of the script ./install.sh to
BARS=0
and run ./test4PAL.sh again.  This is a straight go/no-go test:  either it solves the problem or it is doesn't.  We don't need a verbose kern.log this time, so you can set DEBUG=0 in the installation script.

Since the SourceForge project seems to have disappeared, I'm not sure where install.sh is, or how to find it. Maybe the whole thing's moved to kernel.org. I suppose I could start digging around in the kernel source and see if I can figure out what that does...

Update: OK, this was a lot easier than I thought:

• Google for "easycap site:kernel.org"
• Find the README.
• Remove all users of easycap kernel mod (shut down apps/services)
• sudo rmmod easycap
• sudo modprobe easycap 'bars=0'
• Restart and try again...

Update 2: The next thing you should probably do is:

• sudo vi /etc/modprobe.d/easycap.conf
• options easycap bars=0
• :wq
• sudo reboot

Zoneminder Memory.pm bug in 1.24.4

If you have Zoneminder installed and you're seeing this in /var/log/syslog:

Nov  6 14:36:11 machine zmwatch[2741]: ERR [Shared data size conflict in shared_data for monitor monitor, expected 328, got 316]

It's probably because of a bug that's been patched. On my distribution (Ubuntu 11.10) the file Memory.pm is at

/usr/share/perl5/ZoneMinder/Memory.pm

and the patch in question was applied at line 130.

Ubuntu 8.10 and Static IP

I just installed Ubuntu 8.10 Desktop and had a very interesting time trying to configure a static IP address. There are plenty of discussions on the forums about how this doesn't just work with the standard Network Manager. And you can't just edit/etc/network/interfaces because that is ignored when you have Network Manager installed.

To make it work, I followed this procedure:

First, remove the Network Manager packages:

sudo apt-get remove network-manager
sudo apt-get remove gnome-netstatus-applet

Now you'll have to manually set an IP so that you can connect to the Internet (modify this to be appropriate to your setup):

sudo ifconfig eth0 10.x.x.y netmask 255.255.255.0
sudo ip route add default via 10.x.x.1
sudo vi /etc/resolv.conf

Set nameserver 10.x.x.z appropriately. Next install the old gnome network admin tool:

sudo apt-get install gnome-network-admin

Finally use the old GUI to set networking configuration:

network-admin

This will store the network configuration in /etc/network/interfaces where it belongs. And it seems to work when you reboot. I'll keep it this way until Network Manager is fixed.

Compiling xemacs 21.5.28 on Ubuntu 7.10

I'm actually now a GNU Emacs user, not an XEmacs user, but in case this is helpful to anyone I've recorded how I figured out how to the get the XEmacs source to compile on Ubuntu 7.10.

The reason I'm no longer an XEmacs user is because it's just not keeping up with GNU Emacs, but more importantly, there is an extremely annoying bug...

Update: It appears that the "Xlib: unexpected async reply" that I refer to above is fixed in Ubuntu 8.04. Too bad I'm now using GNU Emacs...

Platform specific IP routing behavior

The code I'm working on manipulates routing tables on three different platforms: Linux, Windows and Solaris. Each of them has a different behavior for different scenarios. Here I attempt to document those differences.

First some definitions:

• Interface A device which can directly reach a subnet via ARP or other protocols. An example is eth0.
• Direct route A route which indicates which Interface to use to reach a directly connected subnet.
• Gateway route A route which indicates a gateway to use to reach a subnet which is connected via a router.
• Default route A special case of a Gateway route in which the destination subnet is all possible addresses.
• Host route A special case of any of either a Direct route or a Gateway route in which the destination is a single machine.
• Multicast route A special case of a Direct route in which the destination subnet is the multicast address space, 224.0.0.0/8 or a subset thereof.

Note that example route entries in this table is based on the format emitted from Linux's /sbin/ip/route.

		Linux 2.6		Windows 2003		Solaris 10
The interface chosen to access a gateway via a route is determined by traversing the route table, not hardcoded into the route entry. For instance, default via 192.168.0.1 does not need the specification dev eth0, because that is determined by finding the direct route 192.168.0.0/24 dev eth0.		no		no		yes1
A Gateway route which is also a Host route can be added where the destination is an address that exists in a subnet of another Direct route. For instance, the route 192.168.0.20/32 via 192.168.0.1 dev eth0 exists while the direct route192.168.0.0/24 dev eth0 also exists.		yes2		yes2		yes2
A Direct route which is also a Host route can be added where the destination is an address that exists in a subnet of another Direct route. For instance, the route 192.168.0.20/32 dev eth0 exists while the direct route 192.168.0.0/24 dev eth0 also exists.		yes		yes		yes
Direct routes can be deleted.		yes		yes3		yes
Route priority can be programmatically controlled.		yes		yes		no4
When an interface is administratively taken down, do the associated Direct route entries disappear?		yes		yes		yes
When an interface is administratively taken down, and associated Direct route entries disappear, do they return when the interface is brought up again?		yes		yes		yes
When an interface is administratively taken down, do the associated Gateway route entries disappear?		yes		yes		yes
When an interface is administratively taken down, and associated Gateway route entries disappear, do they return when the interface is brought up again?		no		yes		no
When an interface is administratively down, is it an error to add a route that references that interface?		yes		yes		yes
When an interface is unplugged, do the associated route entries disappear?		no		yes		no
When an interface is unplugged, and associated route entries disappear, do they return when the interface is plugged in again?		N/A		yes		N/A
When an interface is unplugged, is it an error to add a route that references that interface?		no		yes5		no
When an interface is unplugged, and associated route entries do not disappear, will an alternate route be chosen because the interface is unplugged?		no		N/A		no
If two interfaces are connected to the same subnet, will ARPrespond on either interface for an address on one of the interfaces?		yes		no		no
Can routes be modified for all attributes including priority? An answer of no means they must be destroyed and recreated to modify attributes.		no		yes		no
Does the operating system create Multicast routes by default?		no		yes		no
Can multicast routes be deleted?		yes		yes3		yes
If Multicast routes do not exist, do multicast packets exit the machine? To where?		yes6		no		no
Can two routes be created with the same destination and priority, but a different interface?		no		yes		yes7
Can a Gateway route be specified with an interface, where that interface does not have a Direct route for the gateway in the Gateway route?		no8		yes9		no8
When you remove a Direct route which is required by a Gateway route to reach the gateway, does the Gateway route disappear?		no		no		no
If you specify a Gateway route with a gateway that is not reachable via a Direct route, is this allowed?		no		yes10		no

¹ This configuration is possible if the route entry is set for this behavior. The route entry can also be configured for a specific interface.

² When a ping is performed on the destination address, it is sent to the gateway, not via the direct route; this is what should be expected by following the rules in reading a route table. However, the gateway in my test was a Linux 2.6 machine, and it rejected the ping with an ICMP of "unreachable." This means such a configuration is possible, but useless.

³ One cannot directly delete a default route or other "protected" routes, but there is a way to fool Windows into deleting it. I found this fascinating discussion

⁴ The metric attribute cannot be set for a route in Solaris.

⁵ The error that's returned is ERROR_INVALID_PARAMETER. That doesn't differentiate this condition from other problems.

⁶ It appears to choose the first available interface.

⁷ This question is partly irrelevant in Solaris; the priority or metric cannot be set for a route. However, you can create two routes with the same destination but different interfaces.

⁸ It works even if the Direct route is on another interface, but it must exist.

⁹ You can set the route, but it doesn't do anything.

¹⁰ This strange behavior is apparently allowed; the source address that's used is the address on the interface that is preferred for the Default gateway.

Problems with JVM crashing

I suddenly seem to have all kinds of problems with the JVM crashing when I'm creating it in our monitor code. The way things work is that I have an executable that linksjava.so instead of using the shipped java exectuable. I call this the "driver." Here's what I've found:

The driver will often (but not most of the time) crash, only when -Xdebug is given, with the following stack trace:

gdb build/debug.linux.x86.rhel3/bin/scdriver_debug core.28224
GNU gdb 6.0
Copyright 2003 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i686-pc-linux-gnu"...
Core was generated by `/home/jared.oberhaus/jared.oberhaus-linux3-all/shared/1.2/build/debug.linux.x86'.
Program terminated with signal 11, Segmentation fault.
Reading symbols from /home/jared.oberhaus/jared.oberhaus-linux3-all/tools/linux/j2sdk1.4.2_06/jre/lib/i386/libjava.so...done.
Loaded symbols for /home/jared.oberhaus/jared.oberhaus-linux3-all/tools/linux/j2sdk1.4.2_06/jre/lib/i386/libjava.so
Reading symbols from /lib/tls/libpthread.so.0...done.
Loaded symbols for /lib/tls/libpthread.so.0
Reading symbols from /lib/tls/libc.so.6...done.
Loaded symbols for /lib/tls/libc.so.6
Reading symbols from /home/jared.oberhaus/jared.oberhaus-linux3-all/tools/linux/j2sdk1.4.2_06/jre/lib/i386/server/libjvm.so...done.
Loaded symbols for /home/jared.oberhaus/jared.oberhaus-linux3-all/tools/linux/j2sdk1.4.2_06/jre/lib/i386/server/libjvm.so
Reading symbols from /home/jared.oberhaus/jared.oberhaus-linux3-all/tools/linux/j2sdk1.4.2_06/jre/lib/i386/libverify.so...done.
Loaded symbols for /home/jared.oberhaus/jared.oberhaus-linux3-all/tools/linux/j2sdk1.4.2_06/jre/lib/i386/libverify.so
Reading symbols from /lib/libnsl.so.1...done.
Loaded symbols for /lib/libnsl.so.1
Reading symbols from /lib/libdl.so.2...done.
Loaded symbols for /lib/libdl.so.2
Reading symbols from /lib/ld-linux.so.2...done.
Loaded symbols for /lib/ld-linux.so.2
Reading symbols from /lib/tls/libm.so.6...done.
Loaded symbols for /lib/tls/libm.so.6
Reading symbols from /home/jared.oberhaus/jared.oberhaus-linux3-all/tools/linux/j2sdk1.4.2_06/jre/lib/i386/native_threads/libhpi.so...done.
Loaded symbols for /home/jared.oberhaus/jared.oberhaus-linux3-all/tools/linux/j2sdk1.4.2_06/jre/lib/i386/native_threads/libhpi.so
Reading symbols from /lib/libnss_files.so.2...done.
Loaded symbols for /lib/libnss_files.so.2
Reading symbols from /lib/libnss_ldap.so.2...done.
Loaded symbols for /lib/libnss_ldap.so.2
Reading symbols from /lib/libresolv.so.2...done.
Loaded symbols for /lib/libresolv.so.2
Reading symbols from /lib/libnss_dns.so.2...done.
Loaded symbols for /lib/libnss_dns.so.2
Reading symbols from /usr/lib/sasl/libanonymous.so...done.
Loaded symbols for /usr/lib/sasl/libanonymous.so
Reading symbols from /usr/lib/sasl/libcrammd5.so...done.
Loaded symbols for /usr/lib/sasl/libcrammd5.so
Reading symbols from /usr/lib/sasl/libdigestmd5.so...done.
Loaded symbols for /usr/lib/sasl/libdigestmd5.so
Reading symbols from /usr/kerberos/lib/libdes425.so.3...done.
Loaded symbols for /usr/kerberos/lib/libdes425.so.3
Reading symbols from /usr/kerberos/lib/libkrb5.so.3...done.
Loaded symbols for /usr/kerberos/lib/libkrb5.so.3
Reading symbols from /usr/kerberos/lib/libcom_err.so.3...done.
Loaded symbols for /usr/kerberos/lib/libcom_err.so.3
Reading symbols from /usr/kerberos/lib/libk5crypto.so.3...done.
Loaded symbols for /usr/kerberos/lib/libk5crypto.so.3
Reading symbols from /usr/lib/sasl/libgssapiv2.so...done.
Loaded symbols for /usr/lib/sasl/libgssapiv2.so
Reading symbols from /usr/kerberos/lib/libgssapi_krb5.so.2...done.
Loaded symbols for /usr/kerberos/lib/libgssapi_krb5.so.2
Reading symbols from /usr/lib/sasl/liblogin.so...done.
Loaded symbols for /usr/lib/sasl/liblogin.so
Reading symbols from /lib/libcrypt.so.1...done.
Loaded symbols for /lib/libcrypt.so.1
Reading symbols from /lib/libpam.so.0...done.
Loaded symbols for /lib/libpam.so.0
Reading symbols from /usr/lib/sasl/libplain.so...done.
Loaded symbols for /usr/lib/sasl/libplain.so
Reading symbols from /home/jared.oberhaus/jared.oberhaus-linux3-all/tools/linux/j2sdk1.4.2_06/jre/lib/i386/libzip.so...done.
Loaded symbols for /home/jared.oberhaus/jared.oberhaus-linux3-all/tools/linux/j2sdk1.4.2_06/jre/lib/i386/libzip.so
Reading symbols from /home/jared.oberhaus/jared.oberhaus-linux3-all/tools/linux/j2sdk1.4.2_06/jre/lib/i386/libjdwp.so...done.
Loaded symbols for /home/jared.oberhaus/jared.oberhaus-linux3-all/tools/linux/j2sdk1.4.2_06/jre/lib/i386/libjdwp.so
#0  0x0066e6c1 in pthread_mutex_init () from /lib/tls/libpthread.so.0
(gdb) where
#0  0x0066e6c1 in pthread_mutex_init () from /lib/tls/libpthread.so.0
#1  0x01070e3c in ObjectMonitor::ObjectMonitor ()
   from /home/jared.oberhaus/jared.oberhaus-linux3-all/tools/linux/j2sdk1.4.2_06/jre/lib/i386/server/libjvm.so
#2  0x01000517 in CreateRawMonitor ()
   from /home/jared.oberhaus/jared.oberhaus-linux3-all/tools/linux/j2sdk1.4.2_06/jre/lib/i386/server/libjvm.so
#3  0x0039a872 in JVM_OnLoad ()
   from /home/jared.oberhaus/jared.oberhaus-linux3-all/tools/linux/j2sdk1.4.2_06/jre/lib/i386/libjdwp.so
#4  0x00ff8a2e in JvmdiInternal::post_event ()
   from /home/jared.oberhaus/jared.oberhaus-linux3-all/tools/linux/j2sdk1.4.2_06/jre/lib/i386/server/libjvm.so
#5  0x01002a0e in jvmdi::post_vm_initialized_event ()
   from /home/jared.oberhaus/jared.oberhaus-linux3-all/tools/linux/j2sdk1.4.2_06/jre/lib/i386/server/libjvm.so
#6  0x010f109c in Threads::create_vm ()
   from /home/jared.oberhaus/jared.oberhaus-linux3-all/tools/linux/j2sdk1.4.2_06/jre/lib/i386/server/libjvm.so
#7  0x00fb4388 in JNI_CreateJavaVM ()
   from /home/jared.oberhaus/jared.oberhaus-linux3-all/tools/linux/j2sdk1.4.2_06/jre/lib/i386/server/libjvm.so
#8  0x08048e6b in exec_java (java_library_path=0x0, 
    jre_home=0xbfffcdda "/home/jared.oberhaus/jared.oberhaus-linux3-all/tools/linux/j2sdk1.4.2_06/jre", 
    java_class=0xbfffce27 "com/scalent/shared/tools/test/MonitorTest3", 
    classpath=0x0) at driver.c:300
#9  0x0804895d in main (argc=5, argv=0xbfffb054) at driver.c:81

• I thought it was something I did because in the stack trace I can see thatclasspath and java_library_path, parameters to exec_java are null, and sometimes contain other bad values. Examining this with the debugger I've determined that this is just the optimizer. The compiler is passing in the right values for these when they're needed, but otherwise they reflect the value of the register $esi which can vary.
• I tried to use Purify on this, but there is something seriously broken with Purify on the machine that I'm running on right now. It seems to work better with root, but when I try it as my own user, I get a MSE on almost every malloc and pthreadoperation, whether my code does it or not. Another red/green/blue herring.
• I tried Valgrind on it to try to find something, but that didn't seem to discover anything either. Of course, Valgrind can't really execute the whole JVM, but that's not what I was looking for; I was just trying to get it to execute my non-JVM code and find some sort of memory corruption.
• I also tried the j2sdk1.4.2_06, better than our j2sdk1.4.2_03. That didn't help at all. It still crashes at least 1/3 times.
• Finally, I went into our code and turned off all the options. After 34 runs ofcom.scalent.shared.tools.test.TestMonitor it did not fail once. I believe the whole thing has something to do with the -Xdebug and related options, as I've never seen a crash in the non-debug version of the driver.
• I think I really proved that it has something devious to do with -Xdebug and friends. I commented out just the -Xdebug and -Xrunjdwp:transport=dt_socket,address=9300,server=y,suspend=noptions and ran the test com.scalent.shared.tools.test.TestMonitor160 times and it didn't fail once.
• I tried putting a 5 second delay between tests incom.scalent.shared.tools.test.TestMonitor, but that didn't help. It still failed on the third test.
• I tried again with strict=y on the -Xrunjdwp:transport line, but that didn't help.
  I also tried using the dt_shmem transport for -Xrunjdwp, but that didn't help either.

• I have resigned myself to the fact that this is a bug in the JVM, at least with the way that I'm calling it. Fortunately it only happens while we have -Xdebug turned on.

RedHat Enterprise Linux and iowait

We use RedHat Enterprise Linux 3.1 at work for a developer box, and I was seeing problems where the machine would get into 100% iowait lockups; it wouldn't completely lock, but it would get REALLY slow. The answer is here, and for me it worked instantly:

http://www.daniweb.com/techtalkforums/post38052.html#post38052

Java uses /dev/random; may block forever creating SSL connections

The software that we're developing creates SSL connections when it starts up, and it does so at S13 (has to be after network, but before other services start). The result is that on an NFS booted Linux machine, it sits there forever, and never completes the connection.

Clue #1: if you move the mouse or type on the machine's keyboard, eventually the connection will complete.

Of course the reason for it hanging is that Java is using /dev/random to generate the keys for the SSL connection. And /dev/random gets all of its entropy from the physical environment, and refuses to return random values until it gets some input from the outside world.

We don't see this on a machine that boots from disk; I assume that /dev/random gets entropy from the interaction with the drive, via interrupts and so forth. For some reason the network activity doesn't yield the same entropy data, or at least not enough.

I found this article that discusses the usefulness of /dev/random given its current design.

In order to work around this, we decided to use /dev/urandom. We could do this by a link in the file system, but a much superior solution is to set the following system property in Java:

-Djava.security.egd=file:/dev/urandom

Now all you have to worry about is attacks against your SSL connection from those who know that you are using the pseudo-random number generator...

Linux and Java and Threads and setuid

Today I learned something very interesting. I learned that you can't setuid on a process in Linux; not when you have multiple threads. Please see #8 in this list.

What they refer to as interesting times probably includes the following:

• When calling setuid, only the caller thread will actually get its uid changed. All other existing threads in the "process" retain their original uid.
• I believe any sane person should recognize this as meaning Linux is broken when using threads and setuid.
• This is a security hole, because root threads still exist in the process. If the non-root threads are hijacked by an attacker, they can stack stomp on the root threads and execute arbitrary code as root.
• Because synchronization depends on the ability to deliver signals, and delivering signals depends on priviledges, it's easy to see how synchronization between a thread running as root and another running as non-root can wedge the process.
• Even if I did call setuid in the first bytecode instruction in a Java process, it's too late; Java has already forked threads to do things like garbage collection, and those threads present the security hole described above, and the synchronization problem described above.
• I'm sure there's a long list of other reasons why this is bad, but I can't think of them now, and the above is sufficient.

In our project we have a Java process that uses forked processes written in C; the purpose of these forked processes is to run as root, or at least elevated privileges, while the Java process runs as some sort of nobody user. Unfortunately this doesn't work very well at all on Linux because we cannot downgrade the uid of the Java process after it starts.

This also means that if we want to listen on a port under 1024, we'll have to do that some other way; there's no way we could get the Java process to bind to that port as root and then downgrade to a nobody uid.

Also the processes I refer to have to be forked before the JVM starts. This means that we have to rendezvous with them in some manner that either means some sort of JNI code to hook up the file descriptors in the pipes, or use some other form of IPC.

Analysis of why creating socket in Java takes 3 minutes

Here's what we think is happening, thanks to Evan's suggestion to use gdb and Carol's assistance in recreating the loopback's IP address and route:

The first attempt by Java to open a socket is preceeded with an initialization of its socket code.
The socket initialization code calls java.net.PlainDatagramSocketImpl.leave, as is indicated in this stack trace from gdb:

#0  0xb75ebc32 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
#1  0xb75e3bf8 in connect () from /lib/tls/libpthread.so.0
#2  0xaa4b7c24 in Java_java_net_PlainDatagramSocketImpl_leave ()
   from /opt/scalent/jre/lib/i386/libnet.so
#3  0xaa4b8029 in Java_java_net_PlainSocketImpl_initProto ()
   from /opt/scalent/jre/lib/i386/libnet.so
#4  0xb2fa6bf2 in ?? ()
#5  0xb2fa0ddb in ?? ()
#6  0xb2f9e104 in ?? ()
#7  0xb721bb44 in JavaCalls::call_helper ()
   from /opt/scalent/jre/lib/i386/client/libjvm.so
#8  0xb72cfa6d in os::os_exception_wrapper ()
   from /opt/scalent/jre/lib/i386/client/libjvm.so
#9  0xb721bd96 in JavaCalls::call ()
   from /opt/scalent/jre/lib/i386/client/libjvm.so
#10 0xb7200f6f in instanceKlass::call_class_initializer_impl ()
   from /opt/scalent/jre/lib/i386/client/libjvm.so
#11 0xb720569c in instanceKlass::call_class_initializer ()
   from /opt/scalent/jre/lib/i386/client/libjvm.so
#12 0xb72001cb in instanceKlass::initialize_impl ()
   from /opt/scalent/jre/lib/i386/client/libjvm.so
#13 0xb72059af in instanceKlass::initialize ()
   from /opt/scalent/jre/lib/i386/client/libjvm.so
#14 0xb720d6d4 in InterpreterRuntime::_new ()
   from /opt/scalent/jre/lib/i386/client/libjvm.so
#15 0xb2fad510 in ?? ()
#16 0xb2fa0ddb in ?? ()
#17 0xb2fa0ddb in ?? ()
#18 0xb2fa0ddb in ?? ()
#19 0xb2fa0ddb in ?? ()
#20 0xb2fa0ddb in ?? ()
#21 0xb2fa0d04 in ?? ()
#22 0xb2fa0ddb in ?? ()
#23 0xb2fa10e1 in ?? ()
#24 0xb2f9e104 in ?? ()
#25 0xb721bb44 in JavaCalls::call_helper ()
   from /opt/scalent/jre/lib/i386/client/libjvm.so
#26 0xb72cfa6d in os::os_exception_wrapper ()
---Type  to continue, or q  to quit---
   from /opt/scalent/jre/lib/i386/client/libjvm.so
#27 0xb721bd96 in JavaCalls::call ()
   from /opt/scalent/jre/lib/i386/client/libjvm.so
#28 0xb721b666 in JavaCalls::call_virtual ()
   from /opt/scalent/jre/lib/i386/client/libjvm.so
#29 0xb721c1df in JavaCalls::call_virtual ()
   from /opt/scalent/jre/lib/i386/client/libjvm.so
#30 0xb7274f25 in thread_entry ()
   from /opt/scalent/jre/lib/i386/client/libjvm.so
#31 0xb7319caa in JavaThread::thread_main_inner ()
   from /opt/scalent/jre/lib/i386/client/libjvm.so
#32 0xb7315674 in JavaThread::run ()
   from /opt/scalent/jre/lib/i386/client/libjvm.so
#33 0xb72d1083 in _start () from /opt/scalent/jre/lib/i386/client/libjvm.so
#34 0xb75dedac in start_thread () from /lib/tls/libpthread.so.0

• The call in question seems to be to make the machine leave the multicast group. See here.
• Leaving the multicast group must involve connecting to loopback on a random port (the port it chooses changes and is always above 32768), and then shoving some random bytes through. That's my theory. Update: This is almost certainlyrtnetlink.
• It just sits there trying to communicate with itself, and times out after ~3.5 minutes.
• It appears that it has its problem because the loopback device is not configured with an IP address and is not in the route table.
• After we issued the following commands, everything works just fine and the 3.5 minute delay turns into 18ms delay:
  • ip addr add 127.0.0.1/8 dev lo
  • /sbin/route add -net 127.0.0.0/8 dev lo
• We found a most interesting thread on the Java forum that seems to mirror our problem. But the guy apparently never figured it out. Maybe I should post a solutionthere.
• We went through the following other possible problems:
• I always thought it was some kind of nfs file locking problem, but that's not the case at all.
• We redirected the logging output to a local disk on the machine. That didn't help at all.
• We saw the process was blocked reading /dev/random; /dev/random must use loopback to generate random numbers. To solve this we used/dev/urandom which is not as random, but removed the block. But the connection delay persisted. This might explain why it took 50 minutes to send the first message on the SSL connection. Once we removed the block on/dev/random the 50 minute message send delay seemed to go away.
• We wrote some code to try to connect without SSL, but I'm not convinced that ever worked. It's still in the code and can be activated with a configuration setting, and I tested that configuration setting in my client.
• We then thought it was a delay caused by doing a reverse DNS lookup on the peer's IP address--likely so it could do certificate validation/throw nice exceptions. We saw in gdb that the stack trace was deep in some Java code that was trying to do some kind of DNS operation. /etc/resolv.conf was empty, so we added our name server to it and rebooted the machine. That didn't help, but the stack trace changed.
• Then the stack trace was stuck inJava_java_net_PlainDatagramSocketImpl_leave; I thought that might have still been some DNS hosage, so I changed /etc/hosts to include the addresses of the peers. That didn't help and didn't change the stack trace.
• Finally we typed /sbin/ifconfig. That showed us that lo did not have an IP address.
• Carol told us the correct magic commands to type.

Creating a socket in Java takes 3 minutes

Sometimes we would see on some of our Linux boxes a 3 minute delay between an attempt to open a socket and a successful connection. This did not make sense... but I eventually determined that this was caused by the loopback device not having an address, or not having a route in the route table.

You can probably fix this by typing the following:

/sbin/ip addr add 127.0.0.1/8 dev lo
/sbin/route add -net 127.0.0.0/8 dev lo

Also, we had these other symptoms:

• /dev/random would block for about 3 minutes, probably because it depends on loopback to get its results.
• Java trying to do a reverse DNS lookup would block for about 3 minutes, probably because it was trying to get results from 0.0.0.0, because /etc/resolv.conf was empty, and 0.0.0.0 was being interpreted as 127.0.0.1... Update: see this postabout a related fix...

Here is a thread where I replied with this information; the thread also suggests other solutions, perhaps to the same or similar problems.

Now if I could just figure out how that Linux box got into that situation...

libstdc++ and compatibility

I've been studying the C/C++ build and I think I learned some things:

• glibc is inextricably linked with the Linux operating system. You can't run with a new glibc.
• LD_LIBRARY_PATH can affect libc, but cannot affect ld-linux.so.2 (ld-2.3.2.so). It seems you can get around this with chroot, but then you have other problems.
• glibc 2.3.2 has the symbol GLIBC_PRIVATE which is in ld-2.3.2.so, but not inld-2.2.x.
• libstdc++ 3.2 (it comes with g++ 3.2) requires glibc 2.3. Redhat 7 ships with 2.2 or earlier. See previous point. You cannot take a libstdc++ from Redhat 9 and run it on Redhat 7 unless you upgrade glibc and just about everything else in the OS, at which point it's not really Redhat 7.
• libstdc++ is more than STL. It's the C++ runtime and STL. Therefore STLport can never replace libstdc++.
• I can chroot with Redhat 7 (actually Mandrake 8) and get my Redhat 9 compiled binary and libstdc++ 3.2 shared object. However, once I do that I can't do things like read /proc or modify /etc which is something we need to do.
• Starting with g++ 3.2, libstdc++ is attempting to be forward/backwards compatible in its ABI where possible. At this point compatibility was completely broken.
• Redhat 9 ships with compat-libstdc++ which contains the C++ runtime libraries for gcc 2.96 as used in Redhat 7.3. This means C++ stuff compiled on Redhat 7 will work on Redhat 9, but only when this package is installed.
• glibc works very well forward/backwards compatibility-wise, with the GLIBC_2.0,GLIBC_2.1, etc. symbols. If you build a binary that is C only, it's probably going to run anywhere, as long as it's glibc v2 or better, preferably glibc v2.1.
• It is impossible to statically link libstdc++ into an executable when exceptions are thrown/caught. This is because symbols such as _Unwind_DeleteExceptionexist in libgcc.so but do not exist in libgcc.a.