J2EE Start-Up Issues/solutions – Troubleshooting

This is a general troubleshooting to assist you in solving J2EE engine start-up problems.
It details the common problems and outlines the best places to look in order to pinpoint the
cause of failure.

Start-Up Procedure:

There are different methods of starting the J2EE in NetWeaver depending on the installation type:

With a J2EE-Only installation the user will use the start-up Framework to start the Engine

This is done via the SAP MMC on the Windows server.

There are slight differences in the way this works in windows and UNIX installations.

On a UNIX box, the System is started and stopped by running the ‘startsap’ and ‘stopsap’ scripts. These are located in the /usr/sap/<SID>/ SYS/exe/run directory

The above diagram describe the start-up procedure of the J2EE Engine.

The start-up Framework or the UNIX scripts launch ‘Jcontrol’ which reads profiles located in /usr/sap/<SID>/ SYS/profile for parameters required to start the Instance. ‘Jcontrol’ is in essence, the master program. In Windows, the System is started from the SAP MMC which should be present on the servers desktop. When started this way, the profiles are also read from the same location as they are with the UNIX installation. There are important profile files that should be present in this directory.

DEFAULT.PFL - This contains the SAPSYSTEMNAME, the dbname, the dbtype and the dbhost.

START_SCS<InstanceNumber>_<host> e.g. START_SCS01_us4025. This sets global variables and  starts the Messaging Service and Locking Service (Enqeue Server)

<SID>_SCS<InstanceNumber>_<host> e.g. J2E_SCS01_us4025. This contains parameters for the messaging service and enqeue service.

START_JC<InstanceNumber>_<host> e.g. START_JC00_us4025. This sets Global Variables and start the J2EE (J2EE Engine)

<SID>_JC<InstanceNumber>_<host> e.g. J2E_JC00_us4025. This profile contains information on where to find instance properties and sdm properties.
 This file also contains the location of ‘jlaunch’ which is required to start the J2EE Engine. ‘Jlaunch’ is normally located in the /usr/sap/<SID>/JC<InstanceNumber>/j2ee/os_libs directory.

Process in Unix:

On a UNIX box, if the Engine has started properly, then the processes should appear as shown below. They can be view by using the following commands:

ps –ef|grep jlaunch and ps –ef|grep sapstart etc.

On Windows, you can use TaskManager. As illustrated on the dialgram above,
the various processes are clearly visible when the J2EE is up and running.

 JCMon that can be used to analyze the condition of the J2EE after Start-Up. As seen in the screenshot above, this tool tells us the state of each of the nodes. Therefore, if one wants to know if the dispatcher starts successfully, they can use this tool.

jcmon “pf=/usr/sap/<SID>/sys/profile/<SAPSID>_<INSTANCE_ID>_<HOST>”

Example:

jcmon pf=/usr/sap/<SID>/sys/profile/<SID>_JC00_testpc.

J2EE can also be started using the SMICM transaction in the R/3. In cases where the J2EE is not starting up in this manner, there are a number of parameters that can be checked. On the R/3 System, open transaction RZ10 and choose the profile of the SAP WebAS. Check the parameter rdisp/j2ee_start is set to 1. You should also check the patches to the Start-Up Framework.


Missing J2EE Settings in R/3  refer the snote: 741289

Tools for Start-Up Analysis:

Work Directory:

On a standalone is located at: /usr/sap/<SID>/JC<Instance Number>/work
On a Add-In, it is located at /usr/sap/<SID>/DVEMBGS<Instance Numbers>/work
On a dialog, it is located at /usr/sap/<SID>/J<Instance Number>/work

The first thing to ask the customer for is the complete work directory. Just ask them to archive (Zip) it up and attach it to the message

Because files are over written at each start-up, sometimes it is best that you ask the customer to restart the J2EE once more before attaching the logs. Then you have a fresh set of logs to look at.

Dev_jcontrol

This is the trace for the Jcontrol process.
It will tell you how far the start-up process has progressed and approximately where it has failed.

Ex 1:
Dev_Jcontrol
[Thr 3316] JControlICheckProcessList: process server0 started (PID:3972)
N.B. The memory settings are included in the dev_jcontrol
Bootstrap Logs
There are a number of logs/traces for each bootstrap
􀂄
Main bootstrap for instance
􀂄
Bootstraps for each Node
Ex: 2
Dev_bootstrap
[Thr 3420] Wed Jan 24 14:41:34 2007
[Thr 3420] JLaunchIExitJava: exit hook is called (rc=0)
[Thr 3420] JLaunchCloseProgram: good bye (exitcode=0)




Dev_server & Dev_dispatcher:

These are the trace files of the JLaunch processes
Useful for checking the JVM parameters for each process Memory settings, etc.

Also useful for checking at which stage the start-up process failed if it gets this far
􀂄
[Waiting for start (1)] to [Starting (2)]
􀂄
[Starting (2)] to [Starting applications (10)]

Ex 3
Dev_server0
[Thr 4788] Wed Jan 24 14:46:41 2007
[Thr 4788] JLaunchISetState: change state from [Starting applications (10)] to [Running (3)]
Std_server & Std_dispatcher
Output of the JVM
Shows which services failed during start-up
Core Services (e.g. Security - com.sap.security.core.ume.service) are required for successful start-up

Ex 4

Std_dispatcher0.out
ServiceManager started for 16303 ms.
Framework started for 25666 ms.
SAP J2EE Engine Version 6.40 PatchLevel 100627.313 is running!
PatchLevel 100627.313 December 14, 2005 20:06 GMT
DefaultTrace.X.trc
DefaultTrace for the server is located at: /usr/sap/<SID>/<Instance>/j2ee/cluster/serverX/log
Useful for determining if the problem is actually within an application starting up
Check if the customer can load the ‘Default’ page
􀂄
http://<J2EE_Host>:<port>
If so, it is a problem with the application starting and check the defaultTrace
Also in the dev_serverX, never goes from:
􀂄
[Starting applications (10)] to [Running (3)]
If it seems to hang/get stuck between ‘Starting Applications’ and ‘Running’ it is useful to trigger thread dumps

1. If it seems to hang/get stuck between ‘Starting Applications’ and ‘Running’ it is useful to trigger thread dumps.
2. Such cases are common with XI start-Ups. From the thread dumps we can often see that there are not enough Application Threads allocated

Common Problems:

Message Server Connection

Jcontrol cannot connect to the Message Server host/port

Often the actual port number is wrong

Check what port the MS is listening on in:

􀂄

/usr/sap/<SID>/SCS<Inst>/work/dev_ms

􀂄

I listen to internal port 3900 (3900)

Compare with the port that Jcontrol tries to connect to in the Dev_jcontrol

􀂄

Instance properties

􀂄

-> ms host : usilap34

􀂄

-> ms port : 3601

If incorrect, they need to change this in the configtool Make sure that they Save any changes and Restart Memory Fragmentation in the Std_bootstrap:

Error occurred during initialization of VM. Could not reserve enough space for object heap

Fixed through Note 835704

Must start jlaunch in this Note and this may not be possible.

Solution:

Reduce the value of the MaxHeapSize property of the server node via the configtool, navigate to Instance_ID# -> Server_ID# -> Bootsrtap (256Mb)

Restart Jlaunch and Rebase

Increase XMX again if required

Note 940893

JDK Detection Issues (One not mentioned)

􀂄

JStart-upIGetJDKInfo: Cannot find Java class

􀂄

Permission issue:

Java VM Shared library issues

Java VM Initialization Issues

Java Terminated with a Non-zero Exit code