Search This Blog

SBL-SCB-00011: Failed to connect to pipe

Applies to:

Siebel System Software - Version: 7.7.2.2 [18356] and later   [Release: V7 and later ]
z*OBSOLETE: Microsoft Windows Server 2003
Product Release: V7 (Enterprise)
Version: 7.7.2.2 [18356]
Database: Oracle 9.2.0.6
Application Server OS: Microsoft Windows 2003 Server SP1
Database Server OS: Microsoft Windows 2003 Server SP1

This document was previously published as Siebel SR 38-3194443421.

Symptoms

Customer is experiencing a recurring issue with their Web Client environment. More times a day the following error is logged in the SCBroker log file.

SBL-SCB-00011: Failed to connect to pipe (SEBL_11_4248) on process 4248

At this moment no users can logon anymore and the ones which are logged on receive the error "The Siebel Server is experiencing problems".

By re-starting the EAFObjMgr_enu the issue gets resolved for the time being. But it's unclear why this happens. I attached a full .zip file of the \log directory. The SCBroker component has evtloglvl %=4. This is not true for other components including the EAFObjMgr_enu. At this stage I'm hessitant to increase the EAFObjMgr_enu log levels.

Number of users affected ± 40.



See attached the log files + evt files (nothing to report here).

Regards,

Cause

Change request 12-1GVIZVC

Solution

For the benefit of other users:

Customer encountered SCBroker hanging behavior as per Alert 1252. After dumping the hanging corresponding object manager process using adplus, the following call stack could be made reponsible for the hang:

A thread executing a workflow is dead locked in CSSUInboxTypeCacheMgr call:

sslcosa!osaWaitEvent+0x11
sslcsrcn!SrmSisMsgQ::GetMsg+0x17
sslcsrcn!SrmConn::GetResponeses+0x5a
sslcsrcn!SrmConn::SubmitSynch+0xa1
sslcsrcn!SrmConn::SendClientInfo+0xb3
sslcsrcn!SrmConn::Connect+0x307

Change request 12-1GVIZVC   has been created to develop a fix that is preventing the hang.


Applies to:

Siebel CRM - Version 8.1.1 SIA [21111] and later
Information in this document applies to any platform.

Symptoms

When the customer tries to login via the Web Client after the restart of Siebel Server and Web Server services, IE throws following error sometimes:
The server you are trying to access is either busy or experiencing difficulties. Please close the Web browser, open a new browser window, and try logging in again.

It will displays a login page if the customer press F5(refresh) button after the above error.


Cause

It was observed that Object Manager was neither crashed nor hang.
The SCBroker_xxx.log file logged following error but this Process ID 25780 was not logged on the enterprise log file:
SBL-SCB-00011: Failed to connect to pipe (SEBL_20_25780) on process 25780.
This Process ID was logged on the enterprise log file which was one generation earlier.

It was found that IE tried to connect to the Object Manager process which was one generation earlier. Because the Siebel Server and Web Server services were restarted while connecting the Web Client user and the customer used same IE process which has old session information.
Reproducible steps were found with following steps:
1. Launch IE with the URL like http://<web server>/callcenter_enu and login via the Web Client.

2. While connecting the Web Client user, restart the Siebel Server and Web Server services.

3. Press F5 button on the same IE process with above 1. Then IE throws following error:
The server you are trying to access is either busy or experiencing difficulties. Please close the Web browser, open a new browser window, and try logging in again.

4. Press F5 button again on the same IE process with above 1. Then login screen displays successfully.

Since this phenomenon is neither a crash nor hang of Object Manager, you can ignore the SBL-SCB-00011 error on the SCBroker_xxx.log file.

Solution

Please launch a new IE process and login via the Web Client after the restart of the Siebel Server and Web Server services.





Applies to:

Siebel System Software - Version: 7.7.2 [18325] and later   [Release: V7 and later ]
Information in this document applies to any platform.
Area(s):System Administration
Release(s):V7 (Enterprise), V7 (Professional), V8 (Enterprise), V8 (Professional)
Database(s):All Supported Databases
App Server OS(s):All Supported Platforms
Latest release tested against:V8 (Enterprise)
Keywords:hang, SCBroker, hung, SBL-SWP-00121, SBL-SCB-00011

This document was previously published as Siebel Alert 1252.

Description

The Siebel Connection Broker (SCBroker) is a server component that provides intraserver load balancing. SCBroker distributes session login requests across multiple instances of Application Object Managers (AOMs) running on a Siebel Server. SCBroker logic will always route the request to the AOM process that has the least number of running tasks. For details see Siebel Bookshelf version 8.0 > Siebel Deployment Planning Guide > Siebel Architecture Overview > About the Siebel Connection Broker.
 
Under certain scenarios, an AOM process can be in a hanging state and is not able to acknowledge connection requests from the SCBroker anymore. If this hanging AOM process happens to have the least number of running tasks, SCB will attempt to route new session requests to this AOM process repeatedly. This results in blockage of all new login requests on that Siebel Server.
 
Bug 10644728 has been logged to address the product enhancement request to have the SCBroker ignore this unresponsive AOM process.

Likelihood of Occurrence

This Alert is applicable if you are running Siebel Application Server 7.7, 7.8 , 8.0 and 8.1.

Possible Symptoms

The following criteria all needs to be met and these are the symptoms that you may encounter if you are experiencing this type of deadlock behavior:
 
  • Every login attempt to a certain AOM on the affected Siebel Server is prompted with the message:
 
SBL-SWP-00121: The server you are trying to access is either busy or experiencing difficulties. Please close the Web browser, open a new browser window, and try logging in again.
              It takes approximately 60 seconds until the server busy message appears. 
  • However session logins to other components are working without any problem.
 
  • The AOM component in the server admin screen shows up as running on the affected server.
 
  • No crashes for the AOM are reported in the enterprise logs. No core dump or crash.txt file is written on the affected server.
 
  • AOM tasks might be staying in the state “Handling Logon” or ”Relogin”.
 
  • The SCBroker log is continuously logging the following error message:
 
SBL-SCB-00011: Failed to connect to pipe (SEBL_0_pid) on process pid
 
where pid is the process id of the corresponding object manager process. This process id can be found in the enterprise log to associate it with a certain AOM.
 
  • The process with this pid can still be found in the process list of the operating system and corresponds to one of the AOM processes on the affected server.
 
  • NOTE: Applicable to Windows only. You may observe a crash.txt file, however there is no error message SBL-OSD-01000 being logged in the enterprise log and the crashing process is not restarted despite the AutoRestart parameter is set to true.

Workaround or Resolution

The workaround to get the SCBroker process operative again is to terminate the hanging AOM process manually. This can be done using the kill command on UNIX or the Task Manager on Windows.
 
Once the offending process has been terminated, SCBroker will continue normal distribution of new session requests across the remaining AOM processes.
 
To analyze the root cause that is responsible for the AOM hang, please log a new Service Request with Technical Support and provide the information as described in the following references:
 
 
 
Two scenarios that can result in SCBroker disruption and the corresponding resolution are already described in the following references:
 
  1. Document 473809.1 discuss Object Manager hang behaviors on UNIX Platforms due to calls to malloc memory management function in the crash handler code.
 
  1. Document 473854.1 discuss unexpected UNIX process signals can cause MT Server to shut down or hang.
 
NOTE: Applicable to Windows only and any changes made to the Windows registry should be performed only after taking appropriate back-ups:
 
  1. Check the registry key HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\AeDebug
 
  1. If the Auto key is set to 0, a message box will be displayed by Dr. Watson prior to postmortem debugging.
 
  1. In case Dr. Watson is enabled, this can cause the SCBroker to still route connection requests to the remaining fragment of the crashed process during the 5 minute period where Dr. Watson is waiting for the Visual notification being acknowledged.
 
  1. The Auto key must be set to 1 to avoid the pop up boxes in the future.


Applies to:

Siebel CRM - Version 7.8.2 [19213] to 8.1.1 SIA [21111] [Release V7 to V8]
z*OBSOLETE: Microsoft Windows Server 2003
Product Release: V7 (Professional) and later
Version: 7.8.2 [19213]
Database: Microsoft SQL Server 2005
Application Server OS: Microsoft Windows 2003 Server
Database Server OS: Microsoft Windows 2003 Server

This document was previously published as Siebel SR 38-3304776353.


*** Checked for Relevance on 11-Sep-2012 ***

Symptoms

SBL-NET-01023, SBL-SCB-00011, SBL-SSM-00004, SBL-SSM-00006
We have launched Siebel today. Users are seeing 4 problems:

1. Server busy or problem experienced browser error
2. White screen with progress bar
3. Siebel splash screen with progress bar (no login box)
4. Users get into the system and then get booted out.

When first launched, we had serious avaikability issues as above.
We then changed the following settings:

1. MAX TASKS params for all object managers from 20 to 100 (as recommended in PRR) except fo UK OM which was changd to 200.
2. MAX TASKS for connection broker from 1 to 2.

Since these changes users have been able to access, but within last 1 hour we have again seen the same issues.

Cause

Encryption for the communication between Siebel web server and Siebel servers is set to MSCRYPTO

Solution

Message 1

As per the client description their siebel web client became unavailable at certain times.
After investigations the symptoms were found to be as follows:
On the siebel application server there was a siebmtshmw process that became to use 100% CPU. This process did not became unresponsive but instead to became idle when no requests where processed it went to 100%CPU. On a 4 processor box when there were 4 siebmtshmw processes that went in this state the machine begun to respond very slow this resulting in dropping new sessions and overall log responses to incomming requests.
On the web server machine (IIS) there was present the same situation (the siebel extensions out of process begun to consume 100%CPU)
The thread that consumed CPU was identified from the performance monitor logs and then core dump where taken from these processes. The thread was a communication thread between the siebel web extensions (SWSE) and object manager (OM). The call stack for the corresponding thread was:

sslcnapi!compress_write+0xdb
sslcnapi!zlib_inflate+0x8d
sslcnapi!CSSSISConn::_DecodeRawMsg+0x158
sslcnapi!CSSSISConn::SisAsyncThreadMain+0x8d
sslcosd!OSDThreadStart+0x1c0 sslcosd!OSDNTThreadStart+0xc
msvcr70!beginthreadex+0xba
kernel32!GetModuleFileNameA+0xeb
...

The relevant line in the communication call stack is CSSSISConn::_DecodeRawMsg. The customer had the encryption for the communication set to MSCRYPTO. This encryption is not largely used or recommended by Siebel. After setting the encryption to NONE the problem did not re-occurred.

Resolution:

The customer choose to check if they need encrypted communication between the web server and the application servers and if this is the case use SSL instead.








Applies to:

Siebel Assignment Manager - Version: 7.7.2.6 SIA [18372] and later   [Release: V7 and later ]
Information in this document applies to any platform.

Symptoms

After encountering a cluster server crash which forced a restart of the application server it was no more possible to invoke Assignment Manager to Assign Service Requests.

Error reported includes:
[1] Could not route message to AsgnSrvr with registered key (null)
[2] no other server
[3] Stack trace:
Service [Synchronous Assignment Manager Requests].InvokeMethod(), Built-in function
BusComp [Service Request].BusComp_WriteRecord(), Line: 1568
From the logs:
ObjectMgr
SBL-SRB-00047: Could not route message to AsgnSrvr with registered key (null)

SRBroker
GenericLog GenericError 1 0 2008-07-15 07:53:48 (scbcomp.cpp (822) err=7100011 sys=0) SBL-SCB-00011: Failed to connect to pipe (SEBL_32_2460) on process 2460.

Cause


After a review of siebns.dat, the following section showed that the AsgnMgmt components  were 'Online' but not 'Enabled'
----------
[/enterprises/VFECRMPRD/servers/SIEBRPT01/component groups/AsgnMgmt]
Type=empty

[/enterprises/VFECRMPRD/servers/SIEBRPT01/component groups/AsgnMgmt/definition]
Type=empty

[/enterprises/VFECRMPRD/servers/SIEBRPT01/component groups/AsgnMgmt/definition/enable state]
Persistence=full
Type=string
Value="disabled"
Length=16

Details in the Gateway server caches provides statuses and availability of each server and it's components within the enterprise and this is what is used at startup.

The Cluster Server crash corrupted the siebns.dat and so after the restart, the corrupted values for the assignment components 'State' were being used.

Solution

The corrupted siebns.dat file was amended by resetting the assignment manager components 'enabled state' to 'enabled'.
Example:
From:
[/enterprises/VFECRMPRD/servers/SIEBRPT01/component groups/AsgnMgmt/definition/enable state]
Persistence=full
Type=string
Value="disabled"
Length=16
To:
[/enterprises/VFECRMPRD/servers/SIEBRPT01/component groups/AsgnMgmt/definition/enable state]
Persistence=full
Type=string
Value="enabled"
Length=16
 Once all the changes have been made, the system was restarted in the correct order to bring all components back  to an available state.
i.e.
a) Start Database
b) Start Gateway
c) Start Siebel Server





No comments:

Post a Comment

Subscribe to: Post Comments (Atom)