chroot
I've been working on Lifthub, which started as my personal project and then a friend joined me, and maybe I spent half of my time on playing around with "chroot". I did a lot of trial and errors and learned quite a bit. Well, the environment is far from being stable yet when running Java programs, but maybe it's worth writing something about it.
Environment
I used CentOS 5.3, but I believe most of the stuff that I'll write here applies to CentOS 5.x and other Linux distros.
For simplicity, /chroot will be used as a chroot jail in this entry.
Basic knowledge
You need to have basic understanding of how share objects(libraries) work, and how you use "ldd" command. It's not difficult. Don't worry.
For example, if you type "ldd /bin/ls", you'll get something like the following:
$ ldd /bin/ls librt.so.1 => /lib64/librt.so.1 (0x00002b5cbbee0000) libacl.so.1 => /lib64/libacl.so.1 (0x00002b5cbc0e9000) libselinux.so.1 => /lib64/libselinux.so.1 (0x00002b5cbc2ef000) libc.so.6 => /lib64/libc.so.6 (0x00002b5cbc508000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00002b5cbc85f000) /lib64/ld-linux-x86-64.so.2 (0x00002b5cbbcc3000) libattr.so.1 => /lib64/libattr.so.1 (0x0000003625a00000) libdl.so.2 => /lib64/libdl.so.2 (0x00002b5cbca7b000) libsepol.so.1 => /lib64/libsepol.so.1 (0x00002b5cbcc7f000)
The result tells us that the /bin/ls depends on the libraries or shared objects listed here. In other words, you need to have those libraries in your environment to run /bin/ls
Check dependencies
$ ldd /usr/bin/java libz.so.1 => /usr/lib64/libz.so.1 (0x0000003624600000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00002ba13d841000) libjli.so => /usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre/bin/../lib/amd64/jli/libjli.so (0x00002ba13da5c000) libdl.so.2 => /lib64/libdl.so.2 (0x00002ba13dc61000) libc.so.6 => /lib64/libc.so.6 (0x00002ba13de65000) /lib64/ld-linux-x86-64.so.2 (0x00002ba13d624000)
The java command depends on a few files, but I thought they weren't enough, so checked the package dependencies.
$ rpm -qR java-1.6.0-openjdk-1.6.0.0-1.16.b17.el5 | sort /bin/sh /bin/sh /usr/sbin/alternatives /usr/sbin/alternatives config(java-1.6.0-openjdk) = 1:1.6.0.0-1.16.b17.el5 jpackage-utils >= 1.7.3-1jpp.2 ld-linux-x86-64.so.2()(64bit) ld-linux-x86-64.so.2(GLIBC_2.3)(64bit) libX11.so.6()(64bit) libXext.so.6()(64bit) libXi.so.6()(64bit) libXrender.so.1()(64bit) libXtst.so.6()(64bit) libasound.so.2()(64bit) libasound.so.2(ALSA_0.9)(64bit) libasound.so.2(ALSA_0.9.0rc4)(64bit) libawt.so()(64bit) libc.so.6()(64bit) libc.so.6(GLIBC_2.2.5)(64bit) libc.so.6(GLIBC_2.3)(64bit) libc.so.6(GLIBC_2.3.2)(64bit) libc.so.6(GLIBC_2.4)(64bit) libdl.so.2()(64bit) libdl.so.2(GLIBC_2.2.5)(64bit) libfreetype.so.6()(64bit) libgcc_s.so.1()(64bit) libgcc_s.so.1(GCC_3.0)(64bit) libgcc_s.so.1(GCC_3.3)(64bit) libgcc_s.so.1(GCC_4.2.0)(64bit) libgif.so.4()(64bit) libjava.so()(64bit) libjava.so(SUNWprivate_1.1)(64bit) libjli.so()(64bit) libjli.so(SUNWprivate_1.1)(64bit) libjpeg = 6b libjpeg.so.62()(64bit) libjvm.so()(64bit) libjvm.so(SUNWprivate_1.1)(64bit) libm.so.6()(64bit) libm.so.6(GLIBC_2.2.5)(64bit) libmawt.so()(64bit) libmawt.so(SUNWprivate_1.1)(64bit) libnet.so()(64bit) libnet.so(SUNWprivate_1.1)(64bit) libnsl.so.1()(64bit) libpng12.so.0()(64bit) libpng12.so.0(PNG12_0)(64bit) libpthread.so.0()(64bit) libpthread.so.0(GLIBC_2.2.5)(64bit) libpthread.so.0(GLIBC_2.3.2)(64bit) libpthread.so.0(GLIBC_2.3.3)(64bit) libthread_db.so.1()(64bit) libthread_db.so.1(GLIBC_2.2.5)(64bit) libverify.so()(64bit) libverify.so(SUNWprivate_1.1)(64bit) libz.so.1()(64bit) openssl rpmlib(CompressedFileNames) <= 3.0.4-1 rpmlib(PayloadFilesHavePrefix) <= 4.0-1 rpmlib(VersionedDependencies) <= 3.0.3-1 rtld(GNU_HASH) tzdata-java
Then I checked the dependencies of those libraries by using "ldd" (and small shell scripts). I won't write all the details here.
Set up the environment
Make directories
# chown root:root /chroot/
# mkdir /chroot/{proc,bin,lib,usr,etc,tmp.dev}
# mkdir /chroot/lib64
# mkdir /chroot/usr/{bin,sbin,lib,lib64}
# chmod 777 /chroot/tmp/
# chmod +t /chroot/tmp/
Copy necessary libraries
We checked the dependencies above. So, now copy all the necessary files in the appropriate places under /chroot. This is a bit time consuming, but not that difficult. If you tried to run a program and a library was missing, you would get an error.
Copy Java related files
# rpm -ql java-1.6.0-openjdk (snip) # cp -a /usr/lib/jvm /chroot/usr/lib
Copy commands
At least, you need bash. In my case, I needed only the following commands.
So, copy them and the libraries that they require. "su" is tricky, so I'll write about it later.
Create character devices
(I don't know the correct terminology.)
# mknod -m 666 /chroot/dev/null c 1 3 # mknod -m 666 /chroot/dev/zero c 1 5 # mknod -m 644 /chroot/dev/random c 1 8
Copy config files
- /etc/passwd
- /etc/group
- /etc/shadow
- /etc/hosts
- /etc/nsswitch.conf
- /etc/resolv.conf
su
Only root can "chroot", so it's natural that you want to use "su" to another user after doing "chroot". In the old days, "su" used to be a simple command, I guess. Today, it seems to involve PAM and SELinux, which I don't need in the chroot environment.
If you used the version that comes with CentOS, you would get an error like "su: john does not exist" even though it existed.
So, get an old version of "coreutils". I used coreutils 5.97.
$ tar zxvf coreutils-5.97.tar.gz $ cd coreutils-5.97 $ ./configure && make
Then copy the binary to the chroot environment.
I didn't dig into deeper, but /lib64/libnss_files* seem to be necessary, so I copied them to the jail.
Run Java
Almost there. But, you might get some more errors.
Locale
OpenJDK 64-Bit Server VM warning: Can't detect initial thread stack location - find_vma failed *** glibc detected *** java: munmap_chunk(): invalid pointer: 0x0000000041119860 *** ======= Backtrace: ========= /lib64/libc.so.6(cfree+0x166)[0x2ac342984886] /usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre/lib/amd64/libjava.so[0x2aaaaaedfa17] /usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre/lib/amd64/libjava.so(Java_java_util_TimeZone_getSystemTimeZoneID+0x72)[0x2aaaaaedf512] [0x2aaaab94010e] ======= Memory map: ========
If you see the error above, try the following.
# cp -a /usr/share/zoneinfo/Japan /chroot/etc/localtime # cp -a /usr/lib/locale/ja_JP.utf8/ /chroot/usr/lib/locale
libjli.so
Even though you have libjli.so in your jail environment, you might get the following.
$ java java: error while loading shared libraries: libjli.so: cannot open shared object file: No such file or directory
I did a quick search and found some pages(here and here).
The solution that I took was just add the path to LD_LIBRARY_PATH.
export LD_LIBRARY_PATH=/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre/lib/amd64/jli
Other errors
Here are some other errors that I encountered when I was trying to run Java.
# # A fatal error has been detected by the Java Runtime Environment: # # SIGSEGV (0xb) at pc=0x00002b557b07cc8d, pid=29992, tid=1080588608 # # JRE version: 6.0_17-b17 # Java VM: OpenJDK 64-Bit Server VM (14.0-b16 mixed mode linux-amd64 ) # Derivative: IcedTea6 1.7.5 # Distribution: Custom build (Wed Oct 13 13:04:40 EDT 2010) # Problematic frame: # V [libjvm.so+0x615c8d] # # An error report file with more information is saved as: # /tmp/hs_err_pid29992.log # # If you would like to submit a bug report, please include # instructions how to reproduce the bug and visit: # http://icedtea.classpath.org/bugzilla #
# # A fatal error has been detected by the Java Runtime Environment: # # Internal Error (synchronizer.cpp:1800), pid=11783, tid=1104775488 # Error: guarantee(obj->mark() == markOopDesc::encode(mid),"invariant") # # JRE version: 6.0_17-b17 # Java VM: OpenJDK 64-Bit Server VM (14.0-b16 mixed mode linux-amd64 ) # Derivative: IcedTea6 1.7.5 # Distribution: Custom build (Wed Oct 13 13:04:40 EDT 2010) # An error report file with more information is saved as: # /home/lifthubuser/servers/jetty-6.1.26/hs_err_pid11783.log # # If you would like to submit a bug report, please include # instructions how to reproduce the bug and visit: # http://icedtea.classpath.org/bugzilla #
Conclusion
It was a long way. Finally, java seems to work. However, there are still some unresolved issues, which you wouldn't see in a standard Linux environment. I think normal commands should be ok, but JVM seems too complicated and untested in Linux chroot environments.
I'm thinking of trying FreeBSD jails out.
