Monday, December 7, 2009

BEA-000388 - JVM called WLS shutdown hook. The server will force shutdown now

Issue:

The WLS shuts down with the similar errors in log files:

<Apr 24, 2009 12:10:44 PM PDT> <Notice> <WebLogicServer> <BEA-000388> <JVM called WLS shutdown hook. The server will force shutdown now>

<Apr 24, 2009 12:10:44 PM PDT> <Alert> <WebLogicServer> <BEA-000396> <Server shutdown has been requested by <WLS Kernel>>

<Apr 24, 2009 12:10:44 PM PDT> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to FORCE_SHUTTING_DOWN>


Possible Cause and Solution:

I have the following suggestion for you for the below error:

<Apr 24, 2009 12:10:44 PM PDT> <Notice> <WebLogicServer> <BEA-000388> <JVM called WLS shutdown hook. The server will force shutdown now>

1.) It seems like some component is sending the wrong signal to the JVM and this issue is occurring. JVM monitors and catches OS signals, like: CTRL +C event, Log off event, shutdown event.When JVM catches one of the stated above signals, it shutdowns all java processes.

Please try these possible solutions of this problem:

· Specify -Xrs parameter in the JAVA startup arguments and start the admin server

· -Xrs

· Note: -Xrs is a non-standard option developed by Sun Microsystems for their HotSpot JVM. BEA JRockit continues to support this option; however the BEA JRockit non-standard option -XnoHup provides the same functionality. -Xrs reduces usage of operating-system signals by the JVM. If the JVM is run as a service (for example, the servlet engine for a web server), it can receive CTRL_LOGOFF_EVENT but should not initiate shutdown since the operating system will not actually terminate the process. To avoid possible interference such as this, the -Xrs command-line option does not install a console control handler, implying that it does not watch for or process CTRL_C_EVENT, CTRL_CLOSE_EVENT, CTRL_LOGOFF_EVENT, or CTRL_SHUTDOWN_EVENT Operation.

2.) If the issue recurs again even after trying the above option then can you please apply thefollowing JAVA_OPTION:

· Depending on the JVM version, it may be possible to get a thread dump before the process exits

· HotSpot supports the command-line option -XX:+ShowMessageBoxOnError

· The corresponding JRockit option is -Djrockit.waitonerror

· While the JVM< goes down, it may prompt the user: "Do you want to debug the problem?"

· This pauses the JVM, thereby creating an opportunity to generate a thread dump (a stack trace of every thread in the JVM), attach a debugger, or perform some other debugging activity.

7 comments:

  1. Also see this thread
    http://kr.forums.oracle.com/forums/thread.jspa?threadID=832777

    Where if on a windows server the administrator logs out from an mstsc /console session the servers are shutdown. This only occurs when /console is used.

    adding -Xnohup to the startup script seems to fix the problem

    ReplyDelete
  2. Which file do I need to modify for:
    Specify -Xrs parameter in the JAVA startup arguments and start the admin server

    ReplyDelete
  3. you can specify in any of the following files:
    - setdomainenv.sh
    - startweblogic.sh
    or the installscript for windows service

    Thx,
    Unni

    ReplyDelete
  4. Is there a way to capture the console events like Ctrl+C, Logoff Event, Shutdown event in a log file for a JVM? This would help in understanding how JVM catches these OS signals and runs the shutdown hooks.

    Also, after adding the option -Xrs prevent running the shutdown hooks upon receiving the above mentioned console events. So, the same log could now be used to test how the addition of this option allows the JVM to skip running these shutdown hooks and terminate the java processes.

    ReplyDelete
  5. http://publib.boulder.ibm.com/infocenter/realtime/v1r0/index.jsp?topic=/com.ibm.rt.doc.10/user/signals.html

    this IBM document gives and overview of how jvm handles process signals

    thx,
    Unni

    ReplyDelete
  6. I am getting this error on one node of the cluster that is running on Linux. Will the -Xnohup still help on Linux?

    ReplyDelete
  7. -Xrs: Reduces use of operating-system signals by the Java virtual machine (JVM). When -Xrs is used the signal masks for SIGINT, SIGTERM, SIGHUP, and SIGQUIT are not changed by the JVM, and signal handlers for these signals are not installed.

    So , i believe it will help!!

    Let me know... how it goes

    Enjoy Administration :)
    Unni

    ReplyDelete