J2EE Start-Up Issues/solutions – Troubleshooting

This is a general troubleshooting to assist you in solving J2EE engine start-up problems.
It details the common problems and outlines the best places to look in order to pinpoint the
cause of failure.

Start-Up Procedure:

There are different methods of starting the J2EE in NetWeaver depending on the installation type:

With a J2EE-Only installation the user will use the start-up Framework to start the Engine

This is done via the SAP MMC on the Windows server.

There are slight differences in the way this works in windows and UNIX installations.

On a UNIX box, the System is started and stopped by running the ‘startsap’ and ‘stopsap’ scripts. These are located in the /usr/sap/<SID>/ SYS/exe/run directory

The above diagram describe the start-up procedure of the J2EE Engine.

The start-up Framework or the UNIX scripts launch ‘Jcontrol’ which reads profiles located in /usr/sap/<SID>/ SYS/profile for parameters required to start the Instance. ‘Jcontrol’ is in essence, the master program. In Windows, the System is started from the SAP MMC which should be present on the servers desktop. When started this way, the profiles are also read from the same location as they are with the UNIX installation. There are important profile files that should be present in this directory.

DEFAULT.PFL - This contains the SAPSYSTEMNAME, the dbname, the dbtype and the dbhost.

START_SCS<InstanceNumber>_<host> e.g. START_SCS01_us4025. This sets global variables and  starts the Messaging Service and Locking Service (Enqeue Server)

<SID>_SCS<InstanceNumber>_<host> e.g. J2E_SCS01_us4025. This contains parameters for the messaging service and enqeue service.

START_JC<InstanceNumber>_<host> e.g. START_JC00_us4025. This sets Global Variables and start the J2EE (J2EE Engine)

<SID>_JC<InstanceNumber>_<host> e.g. J2E_JC00_us4025. This profile contains information on where to find instance properties and sdm properties.
 This file also contains the location of ‘jlaunch’ which is required to start the J2EE Engine. ‘Jlaunch’ is normally located in the /usr/sap/<SID>/JC<InstanceNumber>/j2ee/os_libs directory.

Process in Unix:

On a UNIX box, if the Engine has started properly, then the processes should appear as shown below. They can be view by using the following commands:

ps –ef|grep jlaunch and ps –ef|grep sapstart etc.

On Windows, you can use TaskManager. As illustrated on the dialgram above,
the various processes are clearly visible when the J2EE is up and running.

 JCMon that can be used to analyze the condition of the J2EE after Start-Up. As seen in the screenshot above, this tool tells us the state of each of the nodes. Therefore, if one wants to know if the dispatcher starts successfully, they can use this tool.

jcmon “pf=/usr/sap/<SID>/sys/profile/<SAPSID>_<INSTANCE_ID>_<HOST>”

Example:

jcmon pf=/usr/sap/<SID>/sys/profile/<SID>_JC00_testpc.

J2EE can also be started using the SMICM transaction in the R/3. In cases where the J2EE is not starting up in this manner, there are a number of parameters that can be checked. On the R/3 System, open transaction RZ10 and choose the profile of the SAP WebAS. Check the parameter rdisp/j2ee_start is set to 1. You should also check the patches to the Start-Up Framework.


Missing J2EE Settings in R/3  refer the snote: 741289

Tools for Start-Up Analysis:

Work Directory:

On a standalone is located at: /usr/sap/<SID>/JC<Instance Number>/work
On a Add-In, it is located at /usr/sap/<SID>/DVEMBGS<Instance Numbers>/work
On a dialog, it is located at /usr/sap/<SID>/J<Instance Number>/work

The first thing to ask the customer for is the complete work directory. Just ask them to archive (Zip) it up and attach it to the message

Because files are over written at each start-up, sometimes it is best that you ask the customer to restart the J2EE once more before attaching the logs. Then you have a fresh set of logs to look at.

Dev_jcontrol

This is the trace for the Jcontrol process.
It will tell you how far the start-up process has progressed and approximately where it has failed.

Ex 1:
Dev_Jcontrol
[Thr 3316] JControlICheckProcessList: process server0 started (PID:3972)
N.B. The memory settings are included in the dev_jcontrol
Bootstrap Logs
There are a number of logs/traces for each bootstrap
􀂄
Main bootstrap for instance
􀂄
Bootstraps for each Node
Ex: 2
Dev_bootstrap
[Thr 3420] Wed Jan 24 14:41:34 2007
[Thr 3420] JLaunchIExitJava: exit hook is called (rc=0)
[Thr 3420] JLaunchCloseProgram: good bye (exitcode=0)




Dev_server & Dev_dispatcher:

These are the trace files of the JLaunch processes
Useful for checking the JVM parameters for each process Memory settings, etc.

Also useful for checking at which stage the start-up process failed if it gets this far
􀂄
[Waiting for start (1)] to [Starting (2)]
􀂄
[Starting (2)] to [Starting applications (10)]

Ex 3
Dev_server0
[Thr 4788] Wed Jan 24 14:46:41 2007
[Thr 4788] JLaunchISetState: change state from [Starting applications (10)] to [Running (3)]
Std_server & Std_dispatcher
Output of the JVM
Shows which services failed during start-up
Core Services (e.g. Security - com.sap.security.core.ume.service) are required for successful start-up

Ex 4

Std_dispatcher0.out
ServiceManager started for 16303 ms.
Framework started for 25666 ms.
SAP J2EE Engine Version 6.40 PatchLevel 100627.313 is running!
PatchLevel 100627.313 December 14, 2005 20:06 GMT
DefaultTrace.X.trc
DefaultTrace for the server is located at: /usr/sap/<SID>/<Instance>/j2ee/cluster/serverX/log
Useful for determining if the problem is actually within an application starting up
Check if the customer can load the ‘Default’ page
􀂄
http://<J2EE_Host>:<port>
If so, it is a problem with the application starting and check the defaultTrace
Also in the dev_serverX, never goes from:
􀂄
[Starting applications (10)] to [Running (3)]
If it seems to hang/get stuck between ‘Starting Applications’ and ‘Running’ it is useful to trigger thread dumps

1. If it seems to hang/get stuck between ‘Starting Applications’ and ‘Running’ it is useful to trigger thread dumps.
2. Such cases are common with XI start-Ups. From the thread dumps we can often see that there are not enough Application Threads allocated

Common Problems:

Message Server Connection

Jcontrol cannot connect to the Message Server host/port

Often the actual port number is wrong

Check what port the MS is listening on in:

􀂄

/usr/sap/<SID>/SCS<Inst>/work/dev_ms

􀂄

I listen to internal port 3900 (3900)

Compare with the port that Jcontrol tries to connect to in the Dev_jcontrol

􀂄

Instance properties

􀂄

-> ms host : usilap34

􀂄

-> ms port : 3601

If incorrect, they need to change this in the configtool Make sure that they Save any changes and Restart Memory Fragmentation in the Std_bootstrap:

Error occurred during initialization of VM. Could not reserve enough space for object heap

Fixed through Note 835704

Must start jlaunch in this Note and this may not be possible.

Solution:

Reduce the value of the MaxHeapSize property of the server node via the configtool, navigate to Instance_ID# -> Server_ID# -> Bootsrtap (256Mb)

Restart Jlaunch and Rebase

Increase XMX again if required

Note 940893

JDK Detection Issues (One not mentioned)

􀂄

JStart-upIGetJDKInfo: Cannot find Java class

􀂄

Permission issue:

Java VM Shared library issues

Java VM Initialization Issues

Java Terminated with a Non-zero Exit code

JCo RFC Connection

The RFC is an SAP interface protocol, which simplifies the programming of communication processes between systems. The RFCs enable you to call and execute predefined functions in a remote system, or in the same system. The RFCs manage the communication process, parameter transfer, and error handling.

In the J2EE Engine the RFC functions are implemented by the JCo RFC Provider service, which is used for processing ABAP to Java requests. A feature is provided for receiving calls from the SAP systems – this is done by registering the J2EE Engine as an RFC destination.
The JCo RFC Provider Service processes calls from the SAP systems. It dispatches the calls to a stateless session bean, which is registered in the J2EE Engine naming system. By naming convention the JNDI name used is identical to the name of the SAP function module.
Technically, the service is based on the JCo (SAP Java Connector).

The JCo RFC Provider service uses a TCP/IP connection type. Hence, the destination must be created in the Web AS with a gateway host and gateway service for the particular system. Then, on the J2EE Engine, the same destination with the same name, host,
and service is created. For more information about registering destinations on J2EE Engine

SAP Web AS with the SAP Java Connector as the interface between the J2EE and the ABAP environment:


1.  On startup the JCo RFC Provider service connects to the Web AS repository.

2.  On startup the JCo RFC Provider service registers itself at the Gateway with a defined name.
 It is possible to register it under different names and at different Gateways.

3.  The Web AS calls a function for the registered RFC destination.

Note: Make sure that the function is defined in the repository!

4.  The Gateway forwards the call to the JCo RFC Provider service.

5.  The JCo RFC Provider service looks in the JNDI for the EJB, which is registered under the function name.

Note: The name of the function must be identical with the JNDI name of the bean, that is, it is necessary to have a JNDI name. If there is no such name, the JCo RFC Provider service will not be able to find the bean.

It is advisable for the beans to be written with a LocalHome interface.

6.  The JCo RFC Provider service calls the processFunction(JCO.Function) method of the EJB found.

  7.  The results of that call (the modified JCO.Function) are passed to the Gateway.

  8.  The Gateway passes the results back to the Web AS.

Jco RFC connection getting automatically stopped

You can resolve this issue by setting ABAP Parameter gw/acl_mode = 0
refer Note 1480644



Now it running state

Another helpful snotes

/nwa -> configuration->jco rfc provider and check the settings as per need. Also do check note 1658310 .

Change UME data source ABAP/JAVA to Database (Default) in EP

.

If any issue after change Portal — User Management — Identity Management — Data Source —- Database only to ABAP System.

After restarting the instance the value of Data Source shows only ABAP system & for somehow if you want to revert to Database only, it’s not possible. There is no any other options rather than ABAP System.

As per SAP note 718383.

dataSourceConfiguration_abap.xml

No change is possible.

This configuration supports all usages (especially SAP Exchange Infrastructure and SAP Enterprise Portal) by making ABAP users and ABAP roles available as users and groups in the UME, and supports the creation of new groups in the UME (which are then stored in the local database) as well.

sol: You can change the database only(default) option by editing UME properties in Config Tool.

Steps to follow:

·         Config Tool — Global server configuration – services – com.sap.security.core.ume.service

·         ume.persistence.data_source_configuration (Make this value Restore to Default)

·         Restart the instance.

Now when you check again in Portal— User Management — Identity Management — Data Source

It will show the Database only(Default) value.

Add Java Server Node in Configtool

1.Make sure /usr/sap/SID has atlest 10 GB space

2.Take the backup of instance.properties file

3.Launch Configtool

4.Set default server processes as 2 at the ‘template’ level

 5. Save changes

6.Define the following parameters in all the INSTANCE profiles (CI and Apps)

j2ee/HTTP/request_strategy = STATELESS
j2ee/HTTP/request_STATELESS_0=0,7,2,0,1,0
j2ee/HTTP/request_STATELESS_1=0,3,3,0.9,0.01,1
j2ee/HTTP/request_STATELESS_2=0,2,6,1,1,0.01
j2ee/HTTP/request_STATELESS_3=0,5,2,0,0.01,1
j2ee/HTTP/request_STATELESS_4=1,1,0,0,2,0.5
j2ee/HTTP/request_STATELESS_5=1,3,1,0,1,0.01
j2ee/HTTP/request_STATELESS_6=1,6,1,0,20,0.5
j2ee/HTTP/request_STATELESS_7=1,4,3,0,0.0001,0.0001

7.Restart SAP – On the CI and Apps

 
Post checks

1.Start JCMON tool on the CI and APP’s and confirm new server node is created

2. Login to the System Info URL and verify it shows the additional node for each host (CI/App)

ICM timeout parameters

icm/conn_timeout:

This parameter determines the maximum length of time for creating a
connection to an external partner (in milliseconds).
If this length of time is exceeded, the attempts to create a
are terminated and an error message returned to the caller.

icm/conn_timeout is used to set the timeout when setting up the connection. If the timeout is exceeded, an error page is generally displayed with the information "404 Resource not found" "Partner not reached".
Therefore, exceeding the icm/conn_timeout does not lead to a "500 Connection timed out" error message


icm/keep_alive_timeout:

If no communication takes place over an existing connection for more
than a certain length of time, the connection is closed by ICM. The
standard value for the timeout is set with this parameter, but can
overridden for each service (see below).

A value of -1 deactivates the timeout. The connection is then never
closed by the ICM.
Caution:
Note that the user context in the work process is not closed after
this period of time. That timeout is defined using the profile
parameter %%rdisp/plugin_auto_logout%%.


Dependent parameter:

you can override the standard value using the parameter
icm/server_port_<xxx>


icm/server_port_<xxx>

You can use this parameter to specify the service/port and the
keepalive timeout to be used for a protocol. You can optionally
specify a service name or the port number.
You can also set the following options for the parameter:
- Timeout Options: There are two timeouts that can be configured:
  The network timeout TIMEOUT and the processing timeout
  PROCTIMEOUT. The period is specified with the parameter value
  <timeout> (in seconds). The value -1 deactivates the timeout.
  If the option is not set, the value is defined by the parameter
  %%icm/keep_alive_timeout%%.
  (Note that the user context in the work process is not closed
  after this period of time. This timeout is defined by the profile
  parameter %%rdisp/plugin_auto_logout%%.)

  The processing timeout PROCTIMEOUT defines how long the ICM waits
  for the response from the backend (AS ABAP or AS Java) before it
  returns an error (ETIMEOUT) to the client.
- Use an external binding program: To bind ports < 1024 under UNIX,
  extend this parameter using the (optional) specification
  EXTBIND=1. The icmbnd then binds the port and transfers it to the
  ICM.
- Do not bind port to all host names: You can use the optional
  parameter HOST=<host name or IP address> to specify that the port
  is not to be bound to all host names (default), but only to those
  specified.


- X.509 certificate: You can use the optional parameter VCLIENT to
  specify whether the client must produce an X.509 certificate if
  SSL is used.
  There are three levels of verification in total (0-2):
   0: No certificate is required, and the server does not request a
      certificate.
   1: The server requests that the client transfers a certificate.
      If the client does not send a certificate, authentication is
      performed using a different method (such as Basic)
      (default setting).
   2: A client must transfer a valid certificate to the server.
      Otherwise, access is denied.
This server-specific value overrides the value set with the
parameter %%icm/HTTPS/verify_client%%.


The processing timeout is set for individual services using the following parameter:
icm/server_port_ < n > = ...,PROCTIMEOUT=< s >
The keepalive timeout can be set globally for all services of the ICM using the parameter icm/keepalive_timeout.
You can also set it for individual services by using the following parameter:
icm/server_port_ < n > = ...,TIMEOUT=< s >
If you set this parameter, it overwrites parameter icm/keepalive_timeout for the specified service.
Even if the ICM serves as client for a request, the TIMEOUT and PROCTIMEOUT parameters of icm/server_port_ < n > control the timeouts for the relevant protocol, provided that you have set TIMEOUT or PROCTIMEOUT. However, it is possible for a client to explicitly set the timeout in IF_HTTP_CLIENT, method SEND.

If you have not set the PROCTIMEOUT parameter, TIMEOUT (or icm/keepalive_timeout) defines both the processing timeout and the keepalive timeout. In addition, as of release 6.40, the keepalive timeout is limited to a maximum of 60 seconds.

In a system where the default timeout settings of 60 seconds for the keepalive timeout and the processing timeout are insufficient because of long-running applications, we recommend that you set the TIMEOUT and PROCTIMEOUT parameters for the relevant services so that you can configure them independent of each other. In addition, we recommend that you do not set the TIMEOUT value higher than necessary, for example, to the usual default value of 60 seconds.
We recommend, for example, the following settings:
icm/server_port_0 = PROT=HTTP,PORT=1080,TIMEOUT=60,PROCTIMEOUT=600
to allow a maximum processing time of 10 minutes.