Deadlock resolution.

Whenever you hit a web server / app server hang problem because of an hosted web application, you can try the following:

1. Take a thread dump of the process. You can issue "kill -3 pid". The thread dump will be available in the web server errors log file.

2. Most of the threads will be waiting to acquire a lock on particular resource say R1. There will be one thread who had acquired the lock on R1 and probably in the following state.
Doing an indefinite wait on trying to acquire a lock on another resource R2. This lock on R2 could be with another thread that is in the wait queue trying to acquire R1 (OR) R2 will never be available because of resource starvation.

3. All threads have entered into lock.wait() and no notify call to wake up these guys.

Getting threaddump of a Java Process.

Analyzing thread dump is the best way to debug a hanging process.
There are 3 ways to get it.

1. Issue kill -9 command.
For ex: If you have started a Sun Web Server process and if it is hanging because of an web-app deployed in it, you can issue command kill -9 pid where pid is the process id of the child process. Note that there will be two processes. You need to issue this on child process id.

2. Attaching jdb. Refer to the following document for more details.
http://wiki.caucho.com/Thread_dump

3. Using JHAT tool. You need to start web server with JDK 1.6 and above to use this. This can be set in the server.xml file for the variable JAVA_HOME. JHAT is a heap analyzer tool which will give you all info related to the heap of the process in the process address space.

Dev. tip of the day.

It is common in dev that we might have lot of versions of 3rd party jars where new APIs are introduced or deprecated with changing versions of jars. If you get unresolved symbol error from any 3rd party lib, one quick way to confirm if you are using correct jar is, do
strings filename.jar | egrep "missing symbol name"

If you do not find one, then jar is not what you expected. The API has changed.

For ex: To check if the symbol cloneConnectionManager is present in ldapjdk.jar or not, I have to do
strings /usr/share/lib/ldapjdk.jar | egrep "cloneConnectionManager"

If present, I will see symbols, else API is missing in the class that you had expected. It is quite possible that the symbol is present in a different class but not the one that you had expected. Check for it also.

Notes on wait method

• the wait() method causes a thread to release the lock it is holding on an object; allowing another thread to run
• the wait() method is defined in the Object class
• wait() can only be invoked from within synchronized code
• it should always be wrapped in a try block as it throws IOExceptions
• there are actually three wait() methods
  1. wait()
  2. wait(long timeout)
  3. wait(long timeout, int nanos)
• the timeout is measured in milliseconds
• nanos is measured in nanoseconds
• wait() can only invoked by the thread that own's the lock on the object

• when wait() is called, the thread becomes disabled for scheduling and lies dormant until one of four things occur:
  1. another thread invokes the notify() method for this object and the scheduler arbitrarily chooses to run the thread
  2. another thread invokes the notifyAll() method for this object
  3. another thread interrupts this thread
  4. the specified wait() time elapses
• when one of the above occurs, the thread becomes re-available to the Thread scheduler and competes for a lock on the object
• once it regains the lock on the object, everything resumes as if no suspension had occurred
• if the thread was interrupted by another thread, an InterruptedException is thrown BUT not until after the thread regains it's lock on the object

• the wait() method throws three exceptions
  1. IllegalArgumentException - if the timeout value passed is invalid
  2. IllegalMonitorStateException - if the current thread does not own the object's lock
  3. InterruptedException - if another thread interrupts the current thread. The interrupted status of the current thread is cleared
  Courtesy: http://www.janeg.ca/scjp/threads/wait.html
  

Great article on thread synchronization

Awesome article on thread synchronization. It also gives assembly level instructions when monitor region is executed.
http://www.artima.com/insidejvm/ed2/threadsynch.html

Synching time with centralized server on unix machines

/usr/sbin/ntpdate ntp_server_ip_addr

Difference between SPI and API

SPI for plug-ins that provide services to platform
API for plug-ins that use the functionality in the platform which is possibly provided by services.

.bashrc script - feel free to use my bash script on unix flavors.

export PATH=/usr/bin:\
/usr/sbin:\
/usr/sfw/bin:\
/usr/sfw/sbin:\
$ANT_HOME/bin:\
/net/flashy.red.iplanet.com/export/d1/S11/SUNWspro/bin:\
/home/la204265/installations/SunStudio12/SUNWspro/bin:\
$JAVA_HOME/bin:\
/bin:/sbin:\
/usr/ccs/bin:\
/usr/proc/bin:\
/usr/platform/sun4u/sbin:\
/.secure/bin:\
/.secure/arch/sparc_sun_solaris2.3:\
/usr/local/bin:\
/usr/local/sbin:\
/usr/local/netscape:\
/usr/dist/exe:\
~/bin:\
/tools/ns-arch/sparc_sun_solaris2.10/bin:\
$PATH

export MANPATH=/usr/man:\
/usr/dt/man:\
/usr/openwin/share/man:\
/usr/local/man:\
/export/man:$MANPATH

export LD_LIBRARY_PATH=/usr/lib/mps:\
/usr/lib:\
/tools/ns-arch/soft/gcc-3.2.1/run/default/sparc_sun_solaris2.10/lib:\
/usr/local/lib:\
/lib:\
/usr/X/lib:\
/usr/ucblib:\
/usr/suntools/internal/SUNWspro/lib:\
/usr/local/ssl/lib:\
/usr/local/lib/gtk/themes/engines:\
/usr/local/ssl/lib:\
/net/flashy.sfbay/usr/re/tools/sparc/lib:\
$LD_LIBRARY_PATH

export CVSROOT=:pserver:la204265@jescvs.sfbay.sun.com:/m/src

export ANT_HOME=/home/la204265/work/tools/ant/apache-ant-1.7.0

export JAVA_HOME=/share/builds/components/jdk/1.5.0/SunOS
export JAVAHOME=/share/builds/components/jdk/1.5.0/SunOS
export JDK14_COMPILER_PATH=/share/builds/components/jdk/1.4.1/SunOS

export CVS_AM71_TAG="cvs co -r AM_MAIN_BRANCH DSAME"
export CVS_AM70_TAG="cvs co -r DSAME70_PATCH_BRANCH DSAME"
export CVS_AM63_TAG="cvs co -r DSAME63_RTM_BRANCH DSAME"
export CVS_AGENT22_TAG="cvs co -r WebAgents_22_RTM_Branch Agents"
export CVS_AGENT21_TAG="cvs co -r DSAMEagents21_sustaining_branch Agents"
export CVS_CHECKIN_FORMAT="cvs commit -m 'Bug: Esc: CRT: Review:' filename"
export ENV_ECLIPSE="~/tools/eclipse/eclipse -vmargs -Xmx700M"

export SUNSTUDIO_12=/home/la204265/installations/SunStudio12/SUNWspro/bin/sunstudio

export VISUAL=vi
export EDITOR=vi
export TETEXDIR=/usr/local/teTeX
export INFOPATH=$TETEXDIR/info

function get_xserver ()
{
echo $(hostname)

if [[ $(hostname) == "muir" ||
$(hostname) == "mojave" ||
$(hostname) == "lassen" ||
$(hostname) == "avatar" ||
$(hostname) == "koval" ]]; then
DISPLAY="avatar.red.iplanet.com:0.0"
fi
}

if [ -z ${DISPLAY:=""} ]; then
get_xserver
fi

export DISPLAY

#---------------
# Some settings
#---------------

ulimit -S -c 0 # Don't want any coredumps
set -o notify
set -o noclobber
set -o ignoreeof
#set -o nounset
#set -o xtrace # Useful for debuging

# Enable options:
shopt -s cdspell
shopt -s cdable_vars
shopt -s checkhash
shopt -s checkwinsize
shopt -s mailwarn
shopt -s sourcepath
shopt -s no_empty_cmd_completion # bash>=2.04 only
shopt -s cmdhist
shopt -s histappend histreedit histverify
shopt -s extglob # Necessary for programmable completion

# Disable options:
shopt -u mailwarn
unset MAILCHECK # I don't want my shell to warn me of incoming mail

export TIMEFORMAT=$'\nreal %3R\tuser %3U\tsys %3S\tpcpu %P\n'
export HISTIGNORE="&:bg:fg:ll:h"
export HOSTFILE=$HOME/.hosts # Put a list of remote hosts in ~/.hosts

#-----------------------
# Greeting, motd etc...
#-----------------------

# Define some colors first:
red='\e[0;31m'
RED='\e[1;31m'
blue='\e[0;34m'
BLUE='\e[1;34m'
cyan='\e[0;36m'
CYAN='\e[1;36m'
NC='\e[0m' # No Color
# --> Nice. Has the same effect as using "ansi.sys" in DOS.

# Looks best on a black background.....
echo -e "${CYAN}This is BASH ${RED}${BASH_VERSION%.*}\
${CYAN} - DISPLAY on ${RED}$DISPLAY${NC}\n"
date
if [ -x /usr/games/fortune ]; then
/usr/games/fortune -s # makes our day a bit more fun.... :-)
fi

function _exit() # function to run upon exit of shell
{
echo -e "${RED}Hasta la vista, baby${NC}"
}

trap _exit EXIT

#---------------
#---------------
# Shell Prompt
#---------------

if [[ "${DISPLAY#$HOST}" != ":0.0" && "${DISPLAY}" != ":0" ]]; then
HILIT=${red} # remote machine: prompt will be partly red
else
HILIT=${cyan} # local machine: prompt will be partly cyan
fi

alias cls=clear
alias rm='rm -i'
alias cp='cp -i'
alias mv='mv -i'
# -> Prevents accidentally clobbering files.
alias mkdir='mkdir -p'

alias h='history'
alias j='jobs -l'
alias r='rlogin'
alias rt='rlogin -l root'
alias rl='rlogin -l la204265'
alias which='type -all'
alias ..='cd ..'
alias path='echo -e ${PATH//:/\\n}'
alias print='/usr/bin/lp -o nobanner -d $LPDEST'
# Assumes LPDEST is defined
alias pjet='enscript -h -G -fCourier9 -d $LPDEST'
# Pretty-print using enscript
alias background='xv -root -quit -max -rmode 5'
# Put a picture in the background
alias du='du -kh'
alias df='df -kTh'

# The 'ls' family (this assumes you use the GNU ls)
alias la='ls -Al' # show hidden files
alias ls='ls -hF' # add colors for filetype recognition
alias lx='ls -lXB' # sort by extension
alias lk='ls -lSr' # sort by size
alias lc='ls -lcr' # sort by change time
alias lu='ls -lur' # sort by access time
alias lr='ls -lR' # recursive ls
alias lt='ls -ltr' # sort by date
alias lm='ls -al |more' # pipe through 'more'

alias tree='tree -Csu' # nice alternative to 'ls'

# tailoring 'less'
alias more='less'
export PAGER=less
export LESSOPEN='|/usr/bin/lesspipe.sh %s 2>&-'
# Use this if lesspipe.sh exists.
export LESS='-i -N -w -z-4 -g -e -M -X -F -R -P%t?f%f \
:stdin .?pb%pb\%:?lbLine %lb:?bbByte %bb:-...'

# spelling typos - highly personnal :-)
alias xs='cd'
alias vf='cd'
alias moer='more'
alias moew='more'
alias kk='ll'

#----------------
# a few fun ones
#----------------

function xtitle ()
{
case "$TERM" in
*term | rxvt)
echo -n -e "\033]0;$*\007" ;;
*)
;;
esac
}

# aliases...
alias ecl='/home/la204265/tools/eclipse/eclipse -vmargs -Xmx700M'
alias top='xtitle Processes on $HOST && top'
alias make='xtitle Making $(basename $PWD) ; make'
alias ncftp="xtitle ncFTP ; ncftp"

remoteon

Interesting socket bug related to data corruption.

A DESCRIPTION OF THE PROBLEM : Asynchronous closes can result in data that should be written to one channel being written to a completely unrelated stream if another stream is opened from a concurrent thread. Link: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6285901

HTTP POST API usage.

Code snippet to do HTTP POST to a web server. Reference javaworld portal.
-------------------------------------------------------------------------
   URL                 url;
   URLConnection   urlConn;
   DataOutputStream    printout;
   DataInputStream     input;
   // URL of CGI-Bin script.
   url = new URL (getCodeBase().toString() + "env.tcgi");
   // URL connection channel.
   urlConn = url.openConnection();
   // Let the run-time system (RTS) know that we want input.
   urlConn.setDoInput (true);
   // Let the RTS know that we want to do output.
   urlConn.setDoOutput (true);
   // No caching, we want the real thing.
   urlConn.setUseCaches (false);
   // Specify the content type.
   urlConn.setRequestProperty
   ("Content-Type", "application/x-www-form-urlencoded");
   // Send POST output.
   printout = new DataOutputStream (urlConn.getOutputStream ());
   String content =
   "name=" + URLEncoder.encode ("Buford Early") +
   "&email=" + URLEncoder.encode ("buford@known-space.com");
   printout.writeBytes (content);
   printout.flush ();
   printout.close ();
   // Get response data.
   input = new DataInputStream (urlConn.getInputStream ());
   String str;
   while (null != ((str = input.readLine())))
   {
   System.out.println (str);
   textArea.appendText (str + "\n");
   }
   input.close ();

Tomcat goodies on troubleshooting standalone server.

There are only really 3 things likely to go wrong during the stand-alone
Tomcat install:

(1) The most common hiccup is when another web server (or any process for that
matter) has laid claim to port 8080. This is the default HTTP port that
Tomcat attempts to bind to at startup. To change this, open the file:

$CATALINA_HOME/conf/server.xml

and search for '8080'. Change it to a port that isn't in use, and is
greater than 1024, as ports less than or equal to 1024 require superuser
access to bind under UNIX.

Restart Tomcat and you're in business. Be sure that you replace the "8080"
in the URL you're using to access Tomcat. For example, if you change the
port to 1977, you would request the URL http://localhost:1977/ in your browser.

(2) An "out of environment space" error when running the batch files in
Windows 95, 98, or ME operating systems.

Right-click on the STARTUP.BAT and SHUTDOWN.BAT files. Click on
"Properties", then on the "Memory" tab. For the "Initial environment" field,
enter in something like 4096.

After you click apply, Windows will create shortcuts which you can use
to start and stop the container.

(3) The 'localhost' machine isn't found. This could happen if you're behind a
proxy. If that's the case, make sure the proxy configuration for your
browser knows that you shouldn't be going through the proxy to access the
"localhost".

In Netscape, this is under Edit/Preferences -> Advanced/Proxies, and in
Internet Explorer, Tools -> Internet Options -> Connections -> LAN Settings.

Notes on Runnable Jars

You can easily package an application's entire set of classes and resources into a Java Archive (JAR). In fact, that is one goal of having jar files. Another is to let users easily execute the application stored in the archive. Why then are jar files second-class citizens in the Java universe—functioning only as archives—when they can be first class, right alongside native executables?

To execute a jar file, you can use the java command's -jar option. For example, say you have a runnable jar file called myjar.jar. Because the file is runnable, you can execute it like this: java -jar myjar.jar.

Alternatively, the Java Runtime Environment (JRE), when installed on an OS like Microsoft Windows, associates jar files with the JVM so you can double-click on them to run the application. These JARs must be runnable.

The question is: How do you make a JAR runnable?

The manifest file and the Main-Class entry

Inside most JARs, a file called MANIFEST.MF is stored in a directory called META-INF. Inside that file, a special entry called Main-Class tells the java -jar command which class to execute.

The problem is that you must properly add this special entry to the manifest file yourself—it must go in a certain place and must have a certain format. However, some of us don't like editing configuration files.

Let the API do it for you

Since Java 1.2, a package called java.util.jar has let you work with jar files. (Note: It builds on the java.util.zip package.) Specifically, the jar package lets you easily manipulate that special manifest file via the Manifest class.

For more details, refer to the link:

http://www.javaworld.com/javaworld/javatips/jw-javatip127.html