Search This Blog

SBL-NET-01034: The SISNAPI connection was closed by the peer


Applies to:

Siebel Finance Sales - Version: 8.1 SIA [21039] and later   [Release: V8 and later ]
Information in this document applies to any platform.

Symptoms

The customer upgraded their application from 8.0.0.5 to 8.0.0.9 and experienced the following issues:

a) The OM (Object Manager) stopped responding from time to time.
b) All the logged in users were kicked out and no user could login thereafter. The server busy error message was encountered and the workaround was to restart the OM.

There were no exact sequence of steps to reproduce this issue. The error was intermittent and was happening occasionally and whenever it happened, it kicked out users and then no user could login.

The OM logs showed the following error:
SBL-NET-01034: The SISNAPI connection was closed by the peer

Cause

The customer was using Message Broadcast feature in their environment and the issue was caused because of the incorrect parameter setting. The problem was caused by the parameter "Application Enable Message Broadcast Cache" setting which was reset to False. The customer was using this setting to maintain keep alive.  Once the parameter was reset to true, the issue was fixed.

The cause of the issue is justified as the bookshelf also mentions something related to the parameter, however it does not explicitly mentions that the OM will intermittently stop responding and the users will be kicked off.

There are two methods for ways for the application to obtain the messages for display:

a) The default behavior is to have the messages read from the database each time the message bar is refreshed. This method can adversely affect performance if the message bar is set to refresh frequently.
b) Message Broadcast Caching stores messages in each application’s object manager. The messages are then broadcast through the Service Request Broker (SRBroker).

If this is not set to TRUE, it results in reads to the database.  That explains the traffic and the subsequent issues.

Solution

The following is the solution action plan for this issue:

1) Navigate to the Administration - Server Configuration screen > Enterprises view.
2) From the Enterprise Servers list, select the enterprise server that runs the application whose message broadcasting settings you want to modify.
3) Click the Component Definitions tab, then select a component whose Component Type value is Application Object Manager.
4) Select the Application Enable Message Broadcast Cache parameter in the Component Parameters subview, and type the value as TRUE.









Applies to:

Siebel System Software - Version: 7.5.3.9 [16194] and later   [Release: V7 and later ]
Oracle Solaris on SPARC (64-bit)
Product Release: V7 (Enterprise)
Version: 7.5.3.9 [16194]
Database: Oracle 9.2.0.2
Application Server OS: Sun Solaris 8
Database Server OS: Sun Solaris 8

This document was previously published as Siebel SR 38-2884485569.

Symptoms

SBL-SVR-00027we lost accesses to the Server Admin screen and srvrmgr on multiple environment (dev, test, staging, etc).

when accessing the server admin screen, we got error msg:
SBL-NET-01034: The SISNAPI connection was closed by the peer

when accessing srvrmgr, we got msg:
Connected to 0 server(s) out of a total of 1 server(s) in the enterprise

Failed to connect server s3_intf_s1: Handshake failed

The srvrmgr.llog file shows:
2021 2006-02-07 10:11:35 2006-02-07 10:12:58 -0600 00000004 001 001f 0001 09
srvrmgr 29510 1 /u00/siebel/siebsrvr/log/srvrmgr.log 7.5.3.9 [16194] ENU
GenericLog      GenericError    1       2006-02-07 10:11:35     (sasess.cpp 9(66
4) err=1801034 sys=0) SBL-NET-01034: The SISNAPI connection was closed by the pe
er
GenericLog      GenericError    1       2006-02-07 10:11:35     (sasess.cpp 9(66
4) err=902047 sys=0) SBL-ADM-02047: Could not send SISNAPI message
GenericLog      GenericError    1       2006-02-07 10:12:58     (sacmd.cpp 10(38
8) err=902049 sys=0) SBL-ADM-02049: There is no connected server targeted for th
at command
GenericLog      GenericError    1       2006-02-07 10:12:58     (sacmdl.cpp 29(6
71) err=902049 sys=0) SBL-ADM-02049: There is no connected server targeted for t
hat command
$


(to be continued)

Cause

Configuration/ Setup

Solution

Message 1

(continuing...)

The enterprise log file shows:
...
ServerLog       ProcessExit     1       2005-12-07 14:31:27     S3SuspectAssigne
d46085     SBL-OSD-02010   Process exited with error - Process exited because of
a bus error (data alignment) SIGBUS
ServerLog       ProcessExit     1       2005-12-07 14:31:27     S3RouteNational
46090     SBL-OSD-02010   Process exited with error - Process exited because of
a bus error (data alignment) SIGBUS
ServerLog       ProcessExit     1       2005-12-07 14:31:27     IBSNCSTM
46088     SBL-OSD-02010   Process exited with error - Process exited because of
a bus error (data alignment) SIGBUS
ServerLog       ProcessExit     1       2005-12-07 14:31:28     IBSNEscalate
46087     SBL-OSD-02010   Process exited with error - Process exited because of
a bus error (data alignment) SIGBUS
...


(to be continued)

Message 2

(continuing ...)

The PSTACK of the coredump shows:
eagnmnsu257(ROOT)# pstack -F core.global.sun4u.eagnmnsu257.1138992838.5252.900.siebsess.7025
core 'core.global.sun4u.eagnmnsu257.1138992838.5252.900.siebsess.7025' of 7025: siebsess 44 1 /u00/siebel/siebsrvr/admin/s3_dev_es.s3_dev_s1.shm 263 e
----------------- lwp# 1 / thread# 1 --------------------
b98c538c __1cJqeLicPath6FpkCpC_v_ (62656c2f, 62656c2f, 73696562, 73727672, 2f6d772f, 6c69623a) + 2c
----------------- lwp# 2 / thread# 2 --------------------
bfb1f090 _signotifywait (bf02c000, bfbe0b80, 0, bf6c0ef8, 2f, 7efefeff) + 8
bfb1a534 thr_errnop (0, 0, 0, 0, 0, 0) + 20
----------------- lwp# 3 --------------------------------
bf019300 private___lwp_cond_wait (bed65d98, bf02cd74, bf02c000, 0, 0, 4) + 8
bfb1cc8c _door_return (bed65cd8, bf00a380, 0, 0, 0, 0) + 68
----------------- lwp# 4 --------------------------------
bfb1cc34 _door_return (25, bf02c000, bf02d678, 3, bf02c000, 1) + 10
bf00a380 _lwp_start (bea0bd98, 0, 0, 0, 0, 0) + 18
bfb1a534 thr_errnop (0, 0, 0, 0, 0, 0) + 20
-------------------------- thread# 3 --------------------
bf00d9e0 _reap_wait (bf030988, 1e8fc, 0, bf02c000, 0, 0) + 38
bf00d738 _reaper (bf02ce08, bf032710, bf030988, bf02cde0, 1, fe400000) + 38
bf01b11c _thread_start (0, 0, 0, 0, 0, 0) + 40

(to be continued)

Message 3

Output of a truss on the srvrmgr is attached to this SR for your review.

Also, when we ran the odbcsql, we also got a core dump, the pstack is the same as the pstack for srvrmgr.

we are able to start the siebel servers and the callcenter OM, and access screens other than the server admin screen. But the OM's with SIGBUS do not work any more.

Even more confusing is that we we cold start (reboot the machines), the error does not appear. However, this is not a acceptable solution. Because we are running on Solaris machines, reboot the production won't be thinkable.

If you can assist us in find and resolve the problems before we move to production, it's great.

Thanks,

Message 4

(1/2)

For the benefit of other readers:

It was found in the truss output and in the siebsrvr_xx.log logs located in siebsrvr/log directory the error SBL-SVR-00027 error message found. The error may be due to lack of file descriptors.

Based on above, the customer was recommended to follow the instructions on the bookshelf below and adjust the kernel settings appropriately.

Siebel Server Installation Guide for UNIX > Tuning UNIX Operating Systems for Siebel Installation > Tuning Siebel eBusiness Applications for Solaris > Tuning Kernel Settings for Solaris

The kernel adjustments alleviated the problem for a while, but the SIGBUS error and core dumps start happening again.

Please, see below the findings that I have found about in the truss.out file provided, within this file there is the _LD_LIBRARY_PATH value used, as you can see below there is no reference for the /u00/oracle/product/920/v64/lib32 directory, also it is repeating some path directories some times. This is exactly what I would like to avoid in the tests that I have asked you.

The LD_LIBRARY_PATH was reviewed and it was found a reference to the Oracle 64 bit library directory. The customer was asked to remove the reference to the Oracle 64 bit library keeping only the Oracle 32 bit library as it is the only supported version of Oracle client libraries.

After removing the Oracle 64 bit library path from the LD_LIBRARY_PATH the SIGBUS error and core dumps did not appear anymore.

Message 5

(2/2)

The following is found in the Bookshelf > SIEBEL SERVER INSTALLATION GUIDE FOR UNIX VERSION 7.5.3 JULY 2003 12-FN585F > Installing the Siebel Server.

NOTE: Siebel applications support the Oracle 32-bit client, but if you have installed the Oracle 64-bit client on your Siebel Server, you need to add $ORACLE_HOME/lib32 instead of $ORACLE_HOME/lib to your LD_LIBRARY_PATH (Solaris),SHLIB_PATH (HP-UX), or LIBPATH (AIX).

Kind Regards,





Applies to:

Siebel System Software - Version 7.8.2.3 SIA [19221] and later
Oracle Solaris on SPARC (64-bit)
Product Release: V7 (Enterprise)
Version: 7.8.2.3 [19221] Com/Med
Database: Oracle 9.2.0.7
Application Server OS: Sun Solaris 9
Database Server OS: Sun Solaris 8

This document was previously published as Siebel SR 38-3360281273.


Symptoms

SBL-DAT-00803, SBL-JCA-00317, SBL-SMI-00126, SBL-NET-01034
Hi,

We are experiencing two unexpected behaviors when attempting to log into eCommunications:

1) The Login Page is not displayed. "Page Cannot Be Displayed" error after about 10 minutes.
2) The Login Page is displayed, but "Page Cannot Be Displayed" error after attempting to log in.

We are able to access the database via a Dedicated Client and can use the ServerManager commands.
We have already tried bypassing all Load Balancers and the LDAP Server unsuccessfully.
We are currently getting JCA Connection errors from Portal to Siebel.
We are seeing excessive CPU for Kernal and IOWait.

We are seeing some discrepancies with the mounted file system.
After removing the Anonymous User *.spf file from the userpref directory, we started getting the Home Page.
However, the system attempted to create the new file but it has 0 byte, and the file name convention is different, for example, <username>&<application>.spf_<hostname>_14084_13.

This seems to be related to file access rights, but we made everything 777 and still get the same error.
We know that this UNIX machine had some problems with this mount point after a power outage, but we do not know the details.
We turned up logging for eComm and SCBroker and the logs stop just prior to getting access the UserPref file.

For eCustomer, it looks like we can connect via JCA for the initial request (GetMDN), but then for the GetHierarchy we get an error and lose the JCA connection:

[SIEBEL ERROR] [2007-05-25 12:25:55.892] [SiebelConnection(1230636829)] Complex Account Hierarchy - Portal Services.GetHierarchyByUser FAILED on Connection (siebel.tcpip.none.none://<hostname>:2321/esblr/eCustomerCMEObjMgr_enu/!3.6e01.4893.46570dcc, INT_PORTAL with message <com.siebel.om.sisnapi.RequestException>
<Error><ErrorCode>8236</ErrorCode> <ErrMsg>OMRPC Request 4 on connection 18579 was abandoned after 60070 ms because it timed out. (SBL-JCA-317) </ErrMsg></Error>
</com.siebel.om.sisnapi.RequestException>

Thanks!

Cause

Configuration/ Setup

Solution

Message 1

For the benefit of other readers:

Users are getting “This page cannot be displayed – Cannot find server or DNS error” error message when trying to load the login page or after providing username and password.

After deleting the User Preferences File, this behavior was resolved, but the new user preference file was created with zero byte.
We have ruled out the possibility of this behavior being related to the situation described in Alert 1025: Siebel Server Components That Read or Write to Files on Solaris May Encounter Fopen() Behaviors.
We have checked all Operating System user limits.

After further investigation, customer found out that a lockd daemon process was not running on the server where the shared Siebel File System resides after the power outage.
All the other Siebel Servers have NFS mounts pointing to this server to access the shared File System.
Once lockd was started on that server, the behavior was no longer observed.

The following Service Requests helped troubleshooting this behavior:

    - Doc ID 504354.1: Unable to connect to Public Sector with SisnSockError
    - Doc ID 495979.1: Web client hangs when trying to login

Thank you,




Applies to:

Siebel System Software - Version: 7.8.2.2 [19219] and later   [Release: V7 and later ]
z*OBSOLETE: Microsoft Windows Server 2003
Product Release: V7 (Enterprise)
Version: 7.8.2.2 [19219]
Database: Oracle 9i
Application Server OS: Microsoft Windows 2003 Server SP1
Database Server OS: Sun Solaris 9

This document was previously published as Siebel SR 38-3035052921.

Symptoms

SBL-SVR-01015, SBL-SVR-01043, SBL-SMI-00033, SBL-NET-01034, SBL-NET-01033, SBL-ADM-01044, SBL-SCB-00015, SBL-SCB-00011Hi,

Each night a cold DB backup is taken on each environment. This is affecting a number of components on the Siebel server:

Transaction Processor
Transaction Merger
Transaction Router
Server Tables Cleanup

These components need to be restarted each morning as the backup is affecting them. I decided to use scripts to shutdown the siebel server 15 mins before backup is taken and another to restart it in the morning. I have tested these script on one environment and this works fine. On another environmnet the shutdown script appears to hang (i.e batch job does not complete). All the parameters are correct as the command would not run otherwise. I have attached the script and the log files at the time it executed (01:45). As you can see from the log file at 06:00 created after the startup script is executed, the server is still shutting down. Can you please inform me what the issue is?

Kind regards

Cause

Configuration/ Setup

Solution

Message 1

For the benefit of other readers:

The customer's script:

D:\sea78\test\siebsrvr\BIN\srvrmgr /g UKGateway /e siebtest /s SiebTest1 /u sadmin /p password /c "shutdown appserver SiebTest1"

In review of the customer's component log files, we could see that there was a problem with one or more components shutting down. As stated in SupportWeb SR 38-1147657941 [Siebel Servers stick in "Shutting down" state], the "shutdown appserver" command issued from the Server manager command-line is a synchronous process, thus the command will block until all tasks in the server have exited. There could be a problem stopping one or more components, caused either by the components themselves, or caused by the synchronous nature of the shutdown process (ie. one component is waiting on a response from another component, or the Siebel server - that may have already been shutdown).

The customer was recommended to use the Windows command, net stop. The net stop command is used as follows:

net stop <service>

For example:

net stop SiebSrvr

Which would stop the service called SiebSrvr.

Assuming you do not have any problem stopping your Siebel Server service using the Windows' Services utility, then you should not experience a problem with using the net stop command to stop the Siebel Server service. You can schedule the command using the Windows command, at.

This information resolved this for the customer.

Regards,



Applies to:

Siebel System Software - Version 7.7.1 [18306] to 8.2.2.1 SIA[23012] [Release V7 to V8]
Information in this document applies to any platform.
Error Message Area:Networking Layer - NET
Version:Siebel 7.7


Purpose

This document is intended to provide cause and corrective action information about Siebel Error Message SBL-NET-01034: The SISNAPI connection was closed by the peer.

Scope

This document is informational and intended for any user.

Details

Explanation

The connection to the server has been explicitly closed by the client. This may be due to server or client process termination. This error could also occur in any Siebel Server component such as MQ or Remote Synchronization, due to incorrect installation or a time out condition.

Corrective Action

Investigate whether this error occurs for multiple clients on different machines or for multiple server components. Determine the steps that led up to the error and determine if a long period of waiting or network interruption coincides with the error.

Depending on the cause, verify the installation requirements or verify network connectivity and re-attempt the operation.




Applies to:

Siebel Remote - Version: 7.5.3.4 [16180] to 8.1 [21039] - Release: V7 to V8
IBM AIX on POWER Systems (64-bit)
Product Release: V7 (Enterprise)
Version: 7.5.3.4 [16180]
Database: Oracle 8.1.7.4
Application Server OS: IBM AIX 5L 5.1
Database Server OS: IBM AIX 5L 5.1

This document was previously published as Siebel SR 38-1923792601.

Symptoms

SBL-GDB-00004, SBL-SMI-00024, SBL-SRB-00061, SBL-GEN-09103Hi,

I am running "GenNewDb" and immediately, I got this following error.

srvrmgr:nytids007tst> start task for component GenNewDb

start task for component GenNewDb
SBL-NET-01034: The SISNAPI connection was closed by the peer

When I checked the "GenNewDb_xxxx.log",I can see the following error message.

ContextInit     ContextInit     0       2005-04-18 23:15:50     SIEBEL_ASSERT_MODE = 0
GenericLog      GenericWarn     2       2005-04-18 23:15:50     (smisched.cpp 17(266) err=9103 sys=0) SBL-GEN-09103: Parameter value
was never set (i.e. is null)
GenericLog      GenericWarn     2       2005-04-18 23:15:50     (smisched.cpp 17(269) err=9103 sys=0) SBL-GEN-09103: Parameter value
was never set (i.e. is null)
GenericLog      GenericError    1       2005-04-18 23:15:50     (smisched.cpp 17(285) err=2100024 sys=0) SBL-SMI-00024: Unable to lo
ad gennewdb

Please look into this ASAP as we need to provide the extract to TAM for review.

Thanks & Regards

Cause

Missing siebel server files

a] \siebsrvr\lib\libgennewdb.so
b] \siebsrvr\dbtempl\sse_utf8.dbf

Solution

Message 1

1 of 2

For the benefit of other readers, the issue and resolution are as follows.

Issue Description:
----------------------
When the GenNewDB was run, wither from the UI or from server manager, an error occurred alomost immediately. In this case, customer ran from the server manager prompt as follows. Customer encountered errors as described below.

srvrmgr:nytids007tst> start task for component GenNewDb

start task for component GenNewDb
SBL-NET-01034: The SISNAPI connection was closed by the peer

When I checked the "GenNewDb_xxxx.log",I can see the following error message.

ContextInit     ContextInit     0       2005-04-18 23:15:50     SIEBEL_ASSERT_MODE = 0
GenericLog      GenericWarn     2       2005-04-18 23:15:50     (smisched.cpp 17(266) err=9103 sys=0) SBL-GEN-09103: Parameter value
was never set (i.e. is null)
GenericLog      GenericWarn     2       2005-04-18 23:15:50     (smisched.cpp 17(269) err=9103 sys=0) SBL-GEN-09103: Parameter value
was never set (i.e. is null)
GenericLog      GenericError    1       2005-04-18 23:15:50     (smisched.cpp 17(285) err=2100024 sys=0) SBL-SMI-00024: Unable to lo
ad gennewdb

Resolution:
--------------

After detailed analysis, Siebel Technical Support discovered the following.

1. Customer environment was missing the following files

a] \siebsrvr\lib\libgennewdb.so
b] \siebsrvr\dbtempl\sse_utf8.dbf

Contd...

Message 2

2 of 2 - Contd from previous..

Customer had another environment similar to this one, where everything was working. Since this non working environment was a test environment and not a production environment, Siebel Technical Support recommended copying the missing libray file \siebsrvr\lib\libgennewdb.so and \siebsrvr\DBTEMPL\sse_utf8.dbf from the working environment.

After the copy was done, customer restarted the siebel server and gateway and ran the GenNewDB and it worked successfully.

Recommendations:
-----------------------

Other recommendations made to the customer were as follows

1. Refer Alert 899: Maintenance Level Update Recommended for AIX 5L v5.1

According to the Alert 899, which refers Alert 851, Customers have to ensure Siebel deployment is running the latest C++ runtime libraries (6.0.0.12) or later, if they are on AIX 5L v5.1

2. Other recommendation made was to not copy missing files, if the environment is production. If missing files are noticed in production, then for long term stability, customers must consider reinstallign their Siebel environment.

Thank you




No comments:

Post a Comment