SBL-SEC-00102: Error loading the security adapter library

Applies to:

Siebel System Software - Version 8.0.0.1 [20408] to 8.2.2.2 SIA[23016] [Release V8]
Information in this document applies to any platform.
Product Release: V8 (Enterprise)
Version: 8.0 [20405]
Database: Oracle 10.1.0.4
Application Server OS: Red Hat Linux 4.0
Database Server OS: Sun Solaris 9

This document was previously published as Siebel SR 38-3467659813.


Symptoms

Hi, we're trying to setup a PoC of Siebel 8.0 on Linux Redhat 4.0 and had encountered issue while trying to config LDAP authentication and an error had ocurred in the OM log file. IBM LDAP 6.0 Client is installed and is using OID 10.1.2 as the LDAP Server.

We have repreatedly verified LDAPparameters many times and believed we have setup the right parmeters.

We got the following error in the log file:

SBL-SEC-00102 An error occurred loading the security adapter library. Please contact your system administrator for assistance.

SBL-DAT-00562 An error occurred loading the security adapter library. Please contact your system administrator for assistance


Btw, logfile and siebns are attached for your reference..

Cause

Change Request 12-1LDF7SJ

Solution

Message 1

Please assign this SR as soon as possible. This is becoming critical for the customer .

Many Thanks

Message 2

For the benefit of other readers

The customer was implementing LDAP authentication with Siebel 8.0 on Red Hat Linux 4.0 with Oracle Internet Directory.

After configuring the LDAP Adapter and restarting the Siebel Server the Object Manager startup would fail with the following errors:

GenericLog GenericError 1 0000000646fb2542:0 2007-09-27 20:38:39 (secmgr.cpp (2407) err=4587622 sys=0) SBL-SEC-00102: An error occurred loading the security adapter library. Please contact your system administrator for assistance.
ObjMgrSessionLog Error 1 0000000646fb2542:0 2007-09-27 20:38:39 (physmod.cpp (9244)) SBL-DAT-00562: An error occurred loading the security adapter library. Please contact your system administrator for assistance.
ObjMgrSessionLog Error 1 0000000646fb2542:0 2007-09-27 20:38:39 (model.cpp (5867)) SBL-DAT-00562: An error occurred loading the security adapter library. Please contact your system administrator for assistance.
ObjMgrCTLog Error 1 0000000646fb2542:0 2007-09-27 20:38:39 (ctxtmgr.cpp (4492)) SBL-SVC-00208: Please login first.

The customer confirmed that the IBM LDAP Client 6.0 was already installed. After further investigation it was found that the LDAP Client libraries needed to be added to the LD_LIBRARY_PATH set in /siebsrvr/siebenv.sh. siebenv.sh was modified as follows:

[....]

Message 3

[....]

if [ a${LD_LIBRARY_PATH} = ${LD_LIBRARY_PATH}a ] then LD_LIBRARY_PATH=${SIEBEL_ROOT}/lib:${SIEBEL_ROOT}/lib/odbc/merant:/opt/ibm/ldap/V6.0/lib:${MWHOME}/lib:${SQLANY}/lib:/usr/lib
else LD_LIBRARY_PATH=${SIEBEL_ROOT}/lib:${SIEBEL_ROOT}/lib/odbc/merant:/opt/ibm/ldap/V6.0/lib:${MWHOME}/lib:${SQLANY}/lib:/usr/lib:${LD_LIBRARY_PATH}
Fi

After making this change LDAP Authentication worked correctly. Change Request 12-1LDF7SJ was logged for a documentation defect.





Applies to:

Siebel CRM - Version 7.5.3.4 [16180] and later
Information in this document applies to any platform.
*** Checked for relevance on 29-JAN-2013 ***

Symptoms

Customer is trying to implement the LDAP security adapter (LDAPSecAdpt) for use with the Siebel application.  The application object manager is failing to bring up a login screen and the following error is appearing in the log file:

SBL-SEC-00102: Error loading the security adapter library. Please contact your system administrator.

The desired behavior is that the security adapter library should load successfully and a login page be presented to the user.

Cause

This behavior occurred because the server had to be physically rebooted after the installation of the IBM LDAP Client and/or GSKit software.

Solution

After some investigation, we determined that it is necessary to physically reboot the server after the installation of the IBM LDAP Client or GSKit software. Once this was done, everything worked correctly.

This requirement is not documented in the Siebel Bookshelf documentation.
Document Enhancement Request Bug:10575898/ 12-1UX6R4P to update the Security Guide with this information.




Applies to:

Siebel Financial Services CRM - Version 8.0.0.1 [20408] to 8.1.1.2[21215] [Release V8]
Information in this document applies to any platform.
***Checked for relevance on 03-Jan-2011***
***Checked for relevance on 23-Nov-2012***


Symptoms

We are trying to implement a Custom Security Adapter to perform application login authentication. We have downloaded the sample SDK from support web, compiled and deployed to our testing environment. We have also configured one of the AOM to test out the compiled security adapter. Unfortunately, we are unable to load the application login page. Based on the attached log file, the error message is as below:

"SBL-SEC-00102: An error occurred loading the security adapter library. Please contact your system administrator for assistance."

Cause

Incorrect setup/configuration on the custom security adapter named subsystem and configuration file or security adapter library compilation issue.

Solution


Verify the following is correct.
- Parameter setting in custom security adapter named subsystem
- Setting in the configuration file reference in ConfigFileName parameter in custom security adapter named subsystem
- LIBPATH environment variable setting

Setup the sdk custom adapter authentication and configuration in-house and successfully reproduce the issue when using the same compiler (g++) and makefile as customer.
Further research on these and found the issue on the sdk compilation. The makefile from customer is like this.

libsampsdk.so: secsamp.o secutil.o
g++ secutil.o secsamp.o -o libsampsdk.so

secsamp.o: secsamp.cpp secutil.h ssecurity.h
g++ -c secsamp.cpp

secutil.o: secutil.cpp secutil.h ssecurity.h
g++ -c secutil.cpp

Note that after compiling the secsamp.cpp and secutil.cpp with g++ -c swtich and option, it will create the .o file. Once this dependencies (secsamp.o and secutil.o) for the shared object secsamp.so have been created, there is need to invoke the linker using additional switch and option to create a dynamic library. So modify the makefile as below and recompile the sample code.

libsampsdk.so: secsamp.o secutil.o
g++ -fPIC -shared secutil.o secsamp.o -o libsampsdk.so

secsamp.o: secsamp.cpp secutil.h ssecurity.h
g++ -c secsamp.cpp

secutil.o: secutil.cpp secutil.h ssecurity.h
g++ -c secutil.cpp

Using this newly compiled library file, object manager and application succesfully loaded and login works fine.





pplies to:

Siebel Communications Service - Version 8.0.0.2 [20412] to 8.1.1.8 SIA [23012] [Release V8]
Information in this document applies to any platform.

Symptoms

Siebel version 8.0 application does not start when the LDAP security adaptor is configured at enterprise level, although the mandatory IBM LDAP client version 6.0 has been installed. The following error messages are logged as a result,
a) SBL-SEC-00102: An error occurred loading the security adapter library. Please contact your system administrator for assistance.
b) SBL-DAT-00562: An error occurred loading the security adapter library. Please contact your system administrator for assistance.

Cause

The Siebel security adaptor library aka. libsscfldap.so could not be loaded because of the installation and configuration (LD_LIBRARY_PATH) of the 64bit libraries that are part of the IBM LDAP client software.

Solution

Ensure that the LD_LIBRARY_PATH in the siebenv.sh shell script only references the 32-bit libraries of the IBM LDAP Client version 6.0 because this ensures that the Siebel modules, which are based on 32-bit processing - do not conflict with the 64-bit library versions of the IBM LDAP Client software - as documented in Siebel Bookshelf version 8.0 > Siebel Security Guide > Security Adapter Authentication > Installing LDAP Client Software > Configuring the siebenv.csh and siebenv.sh Scripts for the LDAP Client - where it states, "Siebel supports only the IBM 32-bit LDAP Client so you can use only the 32-bit LDAP Client libraries with your Siebel applications."

Please follow the following steps, also outlined in the Siebel Bookshelf reference,
1) Navigate to the Siebel server root installation directory
2) Open a text editor on your UNIX system e.g. vi
3) For both the shell script files; SIEBENV.SH and SIEBENV.CSH edit them and ensure that ONLY the location to the IBM LDAP Client 32bit libraries are referenced.

NOTE: The 64-bit libraries can be explicitly identified from the directory path naming convention - that is,
32-bit  =  /opt/IBM/ldap/V6.0/lib
64-bit  =  /optIBM/ldap/V6.0/lib64




Applies to:

Siebel CRM - Version 8.0.0.1 SIA [20408] and later
Information in this document applies to any platform.

Symptoms

Siebel Login Page fails to display

ERRORS
1) The server you are trying to access is either busy or experiencing difficulties. Please close the Web browser, open a new browser window, and try logging in again.[08:22:46]
2) SBL-SEC-00102: An error occurred loading the security adapter library. Please contact your system administrator for assistance.

Cause

The IBM LDAP Client installation on UNIX defaults to /opt/ibm/ldap/V6.0.
The environment variables in the SIEBENV.SH referenced the wrong directory for the IBM LDAP libraries - namely /opt/ibm/ldap/v6.0/lib as opposed to /opt/ibm/ldap/V6.0/lib.
Because the resolution of filename paths on UNIX systems is case sensitive the difference between v6.0 and V6.0 caused the error.
Bug 14651929 has been logged requesting that the Security Guide be updated accordingly under Security Adapter Authentication > Process of Installing and Configuring LDAP Client Software > Configuring the siebenv.csh and siebenv.sh Scripts for the LDAP Client.

Solution

Modify the SIEBENV.SH to reference /V6.0/lib as opposed to /v6.0/lib


Applies to:

Siebel CRM Call Center - Version: 8.1.1.5 [21229] and later   [Release: V8 and later ]
Information in this document applies to any platform.

Symptoms


=== ODM Issue Clarification ===
This is in relation to a 8.1.1.5 siebel enterprise environment deployed on a RHEL 5.5 linux platform (64-bit).


When accessing the EAIObjMgr_enu component fails when using an external Java based command line utility, which is accessing the EAI component via EAI_enu SWSE web server.

Looking at the EAIObjMgr component we can see mainly authentication errors like the following error messages:
"An error occurred loading the security adapter library. Please contact your system administrator for assistance.(SBL-DAT-00562)" and before this error ".....err=4587622....SBL-SEC-00102..."

Cause


=== ODM Cause Determination ===

Executed: strace against start_server and EAI OM PID. There was no evidence that <custom adapter>.so lib was loaded.

Executed: file <custom adapter>.so
This showed: ELF 64-bit LSB shared object, AMD x86-64, version 1 (SYSV), not stripped

Executed: ldd -v <custom adpater>.so
This showed:
>>>
      libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00002b536dd51000)
      libm.so.6 => /lib64/libm.so.6 (0x00002b536e051000)
      libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00002b536e2d4000)
      libc.so.6 => /lib64/libc.so.6 (0x00002b536e4e3000)
      /lib64/ld-linux-x86-64.so.2 (0x0000003d04c00000)

      Version information:
      ./cdetsnopassword.so:
              libc.so.6 (GLIBC_2.2.5) => /lib64/libc.so.6
...
<<<

=== ODM Cause Justification ===

Looking at the file and ldd output, we can see that 64-bit libraries are used:

Siebel product is only 32-bit and will only work with 32-bit libraries.

Solution


=== ODM Solution / Action Plan ===

Compile the custom security adapter as per SDK documented compile commands.
Also ensure that modifications are implemented as per published article: 1062497.1 - Changes to Siebel Custom Security Adapter SDK in Siebel 8.1.1
Also, if on a 64-bit operation system,  ensure to modify the compiler instructions to compile in 32-bit mode. There could be different ways this can be done pending on the OS vendor. For any clarification please consult with the OS vendor support.

In this specific case the following compiler instructions helped to resolve the issue.
g++ -m32 -c -fPIC secsamp.cpp
g++ -m32 -c -fPIC secutil.cpp
g++ -m32 -fPIC -shared secutil.o secsamp.o -o <custom security adapter>.so
Please note, Oracle  is providing the SDK and compiler instructions solely as a convenience to our customers and
Oracle  makes no warranties or representations of any kind regarding the
results customers may achieve by using the SDK. Siebel is not responsible
for any failure of any security adapter written using the SDK.

Check with file and ldd command (library dependency check on Linux) to ensure no 64-bit library is being used.
For example in this case the output after compiling in 32-bit mode confirmed that there was no 64-bit library involved.
$ ldd <custom security adapter>.so
linux-gate.so.1 =>  (0xffffe000)
libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0xf7e82000)
libm.so.6 => /lib/libm.so.6 (0xf7e5b000)
libgcc_s.so.1 => /lib/libgcc_s.so.1 (0xf7e4f000)
libc.so.6 => /lib/libc.so.6 (0xf7d09000)
/lib/ld-linux.so.2 (0x00473000)

Stop the siebel application server service.
Replaced this on the siebsrvr/lib
Start the siebel application server service

Test the siebel application authentication process.






SBL-SCB-00015: The component is down or not available

Applies to:

Siebel System Software - Version: 7.0.4 [14068] and later   [Release: V7 and later ]
Generic Windows
Area(s):System Administration
Release(s):V7 (Enterprise), V7 (Professional)
Database(s):All Supported Databases
App Server OS(s):HP-UX, Solaris, AIX
Latest release tested against:V7 (Enterprise)
Keywords:Solaris, accept, failed, shutdown, signal, SIGPIPE, core, SBL-SMI-00006, hang, logout, users

This document was previously published as Siebel Alert 1206.

Description

Process signals are used for the purpose of interrupting normal process execution and informing it about a certain error condition that needs some recovery activity by a process or by a certain thread of a process.
These signals could be generated by the operating system, or could be invoked via C/C++ code.
Examples:
  • A custom C++ application (library) is invoked from a Call Center Object Manager user or thread via the Siebel eScript SELib mechanism. The library connects to a process or files via named pipe mechanism. A networking problem or timeout occurs and the communication pipe is broken unexpectedly. The operating system issues a SIGPIPE signal to the process to inform it about the broken connection.
  • A custom C++ application (library) is invoked from a Call Center Object Manager user or thread via the Siebel eScript SELib mechanism. The C++ application contains code that sets a specified time it wants to perform an operation or wait on a particular resource. It uses a SIGALRM signal/handler routine to achieve this. After the time period is elapsed a SIGALRM process is triggered.
In both of these cases, the signals are intended to inform the thread which was performing a particular request to stop (interrupted) and then to execute a particular signal handler routine.
However, these signals could also intercepted by the listener thread (thread# 1) which is responsible for the startup and shutdown of the entire process. When such a signal is received, the process may shutdown resulting in all the users who are logged into this process to have to re-login.
Bug 10501186 has been logged to address this product defect.

Likelihood of Occurrence

You may encounter the behavior in this Alert if you are running the Siebel Business Application Server versions 7.7.x or 7.8.x on any UNIX platform.

Possible Symptoms

Below are different symptoms that may be reported if you are encountering this behavior:
  • The browser displays the error message for new users trying to login:
SBL-SWP-00121: The server you are trying to access is either busy or experiencing difficulties. Please close the Web browser, open a new browser window, and try logging in again.
  • The following errors are reported in the SCBroker log file:
GenericLog      GenericError    1       0       2005-11-14 10:50:14     (scbcomp.cpp (416) err=7100015 sys=0) SBL-SCB-00015: The component is down or not available on this server
GenericLog      GenericError    1       0       2005-11-14 10:50:14     (scbcomp.cpp (235) err=7100015 sys=0) SBL-SCB-00015: The component is down or not available on this server
GenericLog      GenericError    1       0       2005-11-14 11:06:18     (scbcomp.cpp (822) err=7100011 sys=0) SBL-SCB-00011: Failed to connect to pipe (SEBL_11_5316) on process 5316.
GenericLog      GenericError    1       0       2005-11-14 11:06:18     (scbcomp.cpp (416) err=7100011 sys=0) SBL-SCB-00011: Failed to connect to pipe (SEBL_11_5316) on process 5316.
  • Depending on the Siebel version you are using, you may encounter either a core dump or the process being frozen in the shutdown state. In the latter case you will need to kill it from the shell to get the process released.  Here is an example of an output from a process that is hung with the following call stack:
4017:   siebmtshmw /siebel/osprd/siebsrvr/admin/osPRDent.siebsrvr1.shm 11 6150
-----------------  lwp# 1 / thread# 1  --------------------
 7daa58f4 lwp_park (0, ffbfeae8, 0)
 7daa2b9c cond_wait_queue (ba25f0, 7dab8b48, 0, 0, 7d790000, 7dab8000) + d4
 7daa3114 cond_wait_common (0, ba25d8, ffbfeae8, 0, 0, 437afb47) + 1d8
 7daa35a4 _cond_timedwait (ba25f0, ba25d8, ffbfec18, 0, 0, 0) + 1f0
 7daa35d8 cond_timedwait (ba25f0, ba25d8, ffbfec18, 0, 0, 0) + 18
 7daa3618 pthread_cond_timedwait (ba25f0, ba25d8, ffbfec18, 10624c00, 0, 0) + c
 7fa1291c __1cMosaWaitEvent6Fpvi_i_ (ba25d0, 1388, 0, 0, ba25d8, 0) + 188
 00031600 __1cNsmiMainThreadIShutdown6M_v_ (695f38, a0, 1, 86070, 688, 400) + 50
 0002f7c8 __1cNsmiMainThreadIMainLoop6M_i_ (695f38, 1b7740, 0, 0, 7bd3080, 5e) + d00
 0002a754 wmain    (9, 12f100, 0, 0, 0, 0) + 274
 0005dd60 main     (ffbff174, 86070, ffbff19c, 7e157380, 74c, 400) + 94
  • The object manager’s listener thread session log file contains the following error:
SisnTcpIp      SisnSockError     1     0     2005-09-09 12:26:29     1: [LOCALTRANS-server] accept() failed during get conn request (error=9602, sys=4)
GenericLog     GenericError      1     0     2005-09-09 12:26:29     (smimtsrv.cpp (1538) err=2100006 sys=0) SBL-SMI-00006: Internal: SISListen failed
You can identify the listener thread session log file by searching for the pattern “ 1 “ in the header line of all log files by running the following command:
head -1 *.log | grep “ 1 “
The output should return something like the following:
1021 2005-10-05 20:55:19 0000-00-00 00:00:00 +0200 00000000 001 003f 0001 09 FINSObjMgr_deu 16434 23184 1

Workaround or Resolution

If you are running on Siebel Maintenance Release 7.7.2.x, the above behavior has been fixed in Siebel Fix Pack version 7.7.2.5.
For other Siebel versions, you should perform the following analysis:
  1. Check the SCBroker log file and confirm if it contains the SBL-SCB-00011 error.
  1. Check the Object Manager listener log file and confirm if it contains the SBL-SMI-00006: Internal: SISListen failed error.
  1. Check for the following:
    1. If the process is hanging, run the command below and see if the call stack matches the one documented in this Alert.
pstack <PID_OF_OM>
          or
    1. If the process has generated a core dump, run the command below and see if the thread# 1 is showing on the top of the stack trace as documented above.
pstack core
If the errors plus call stack matches all three scenarios, then you should apply the Siebel Fix Pack version 7.7.2.5 on top of Siebel Maintenance Release 7.7.2.
If you experience similar behaviors described in this Alert, but all the above 3 scenarios do not match, then please log a new service request. Provide all the relevant Siebel log files, pstack of hung processes or crashed processes (core file).
For additional information about troubleshooting Server components behaviors on UNIX, refer to Document 477520.1 and Document 478027.1.

Maintenance Release Number

Please click on the link below to view the current status of the Change Request and associated Fix Requests:
Bug 10501186 has been fixed on version 7.7.2.5 and 7.8.2.2




Applies to:

Siebel CRM - Version: 8.1 [21039] and later   [Release: V8 and later ]
Information in this document applies to any platform.
Server busy error with custom OM -ACSWMTObjMgr_enu

Symptoms


When try get the Siebel login by using custom component - customer Object Manager -ACSWMTObjMgr_enu ,we get following error

http://10.203.168.143/acswmt_enu/start.swe?" URL getting the following error message:
The server you are trying to access is either busy or experiencing difficulties. Please close the Web browser, open a new browser window, and try logging in again.[23:24:13]

In OM logs, these errors were found

 2009-10-11 23:07:37 Login failed for Login name : anonemp
 2009-10-11 23:07:39 (ctxtmgr.cpp (4504)) SBL-SVC-00208: Please login first

In SWSE logs, these errors where found

2009-10-11 22:53:16 ( (0) err=4653071 sys=0) SBL-SCB-00015: The component is down or not available on this server

2009-10-11 22:56:21 (smconn.cpp (283) err=1180866 sys=2321) SBL-NET-01218: The connection was refused by server hcmblysun004a. No component is listening on port 2321.

 2009-10-11 22:56:21 (ssmsismgr.cpp (773) err=3670019 sys=0) SBL-SSM-00003: Error opening SISNAPI connection.
 2009-10-11 22:56:21 (ssmsismgr.cpp (1761) err=3670022 sys=0) SBL-SSM-00006: Error while sending message to server.
 2009-10-11 22:56:21 (smconn.cpp (283) err=1180866 sys=2321) SBL-NET-01218: The connection was refused by server hcmblysun004a. No component is listening on port 2321.

Changes

Created custom Object manager -ACSWMTObjMgr_enu

Cause


Incorrect value for Siebel Repository File parameter


.

Solution


Setting the correct value for Siebel Repository File parameter resolved this issue.




Applies to:

Siebel System Software - Version: 7.8.2.2 [19219] and later   [Release: V7 and later ]
z*OBSOLETE: Microsoft Windows Server 2003
Product Release: V7 (Enterprise)
Version: 7.8.2.2 [19219]
Database: Oracle 9i
Application Server OS: Microsoft Windows 2003 Server SP1
Database Server OS: Sun Solaris 9

This document was previously published as Siebel SR 38-3035052921.

Symptoms

SBL-SVR-01015, SBL-SVR-01043, SBL-SMI-00033, SBL-NET-01034, SBL-NET-01033, SBL-ADM-01044, SBL-SCB-00015, SBL-SCB-00011Hi,

Each night a cold DB backup is taken on each environment. This is affecting a number of components on the Siebel server:

Transaction Processor
Transaction Merger
Transaction Router
Server Tables Cleanup

These components need to be restarted each morning as the backup is affecting them. I decided to use scripts to shutdown the siebel server 15 mins before backup is taken and another to restart it in the morning. I have tested these script on one environment and this works fine. On another environmnet the shutdown script appears to hang (i.e batch job does not complete). All the parameters are correct as the command would not run otherwise. I have attached the script and the log files at the time it executed (01:45). As you can see from the log file at 06:00 created after the startup script is executed, the server is still shutting down. Can you please inform me what the issue is?

Kind regards

Cause

Configuration/ Setup

Solution

Message 1

For the benefit of other readers:

The customer's script:

D:\sea78\test\siebsrvr\BIN\srvrmgr /g UKGateway /e siebtest /s SiebTest1 /u sadmin /p password /c "shutdown appserver SiebTest1"

In review of the customer's component log files, we could see that there was a problem with one or more components shutting down. As stated in SupportWeb SR 38-1147657941 [Siebel Servers stick in "Shutting down" state], the "shutdown appserver" command issued from the Server manager command-line is a synchronous process, thus the command will block until all tasks in the server have exited. There could be a problem stopping one or more components, caused either by the components themselves, or caused by the synchronous nature of the shutdown process (ie. one component is waiting on a response from another component, or the Siebel server - that may have already been shutdown).

The customer was recommended to use the Windows command, net stop. The net stop command is used as follows:

net stop <service>

For example:

net stop SiebSrvr

Which would stop the service called SiebSrvr.

Assuming you do not have any problem stopping your Siebel Server service using the Windows' Services utility, then you should not experience a problem with using the net stop command to stop the Siebel Server service. You can schedule the command using the Windows command, at.

This information resolved this for the customer.

Regards,

SBL-SCB-00011: Failed to connect to pipe

Applies to:

Siebel System Software - Version: 7.7.2.2 [18356] and later   [Release: V7 and later ]
z*OBSOLETE: Microsoft Windows Server 2003
Product Release: V7 (Enterprise)
Version: 7.7.2.2 [18356]
Database: Oracle 9.2.0.6
Application Server OS: Microsoft Windows 2003 Server SP1
Database Server OS: Microsoft Windows 2003 Server SP1

This document was previously published as Siebel SR 38-3194443421.

Symptoms

Customer is experiencing a recurring issue with their Web Client environment. More times a day the following error is logged in the SCBroker log file.

SBL-SCB-00011: Failed to connect to pipe (SEBL_11_4248) on process 4248

At this moment no users can logon anymore and the ones which are logged on receive the error "The Siebel Server is experiencing problems".

By re-starting the EAFObjMgr_enu the issue gets resolved for the time being. But it's unclear why this happens. I attached a full .zip file of the \log directory. The SCBroker component has evtloglvl %=4. This is not true for other components including the EAFObjMgr_enu. At this stage I'm hessitant to increase the EAFObjMgr_enu log levels.

Number of users affected ± 40.



See attached the log files + evt files (nothing to report here).

Regards,

Cause

Change request 12-1GVIZVC

Solution

For the benefit of other users:

Customer encountered SCBroker hanging behavior as per Alert 1252. After dumping the hanging corresponding object manager process using adplus, the following call stack could be made reponsible for the hang:

A thread executing a workflow is dead locked in CSSUInboxTypeCacheMgr call:

sslcosa!osaWaitEvent+0x11
sslcsrcn!SrmSisMsgQ::GetMsg+0x17
sslcsrcn!SrmConn::GetResponeses+0x5a
sslcsrcn!SrmConn::SubmitSynch+0xa1
sslcsrcn!SrmConn::SendClientInfo+0xb3
sslcsrcn!SrmConn::Connect+0x307

Change request 12-1GVIZVC   has been created to develop a fix that is preventing the hang.


Applies to:

Siebel CRM - Version 8.1.1 SIA [21111] and later
Information in this document applies to any platform.

Symptoms

When the customer tries to login via the Web Client after the restart of Siebel Server and Web Server services, IE throws following error sometimes:
The server you are trying to access is either busy or experiencing difficulties. Please close the Web browser, open a new browser window, and try logging in again.

It will displays a login page if the customer press F5(refresh) button after the above error.


Cause

It was observed that Object Manager was neither crashed nor hang.
The SCBroker_xxx.log file logged following error but this Process ID 25780 was not logged on the enterprise log file:
SBL-SCB-00011: Failed to connect to pipe (SEBL_20_25780) on process 25780.
This Process ID was logged on the enterprise log file which was one generation earlier.

It was found that IE tried to connect to the Object Manager process which was one generation earlier. Because the Siebel Server and Web Server services were restarted while connecting the Web Client user and the customer used same IE process which has old session information.
Reproducible steps were found with following steps:
1. Launch IE with the URL like http://<web server>/callcenter_enu and login via the Web Client.

2. While connecting the Web Client user, restart the Siebel Server and Web Server services.

3. Press F5 button on the same IE process with above 1. Then IE throws following error:
The server you are trying to access is either busy or experiencing difficulties. Please close the Web browser, open a new browser window, and try logging in again.

4. Press F5 button again on the same IE process with above 1. Then login screen displays successfully.

Since this phenomenon is neither a crash nor hang of Object Manager, you can ignore the SBL-SCB-00011 error on the SCBroker_xxx.log file.

Solution

Please launch a new IE process and login via the Web Client after the restart of the Siebel Server and Web Server services.





Applies to:

Siebel System Software - Version: 7.7.2 [18325] and later   [Release: V7 and later ]
Information in this document applies to any platform.
Area(s):System Administration
Release(s):V7 (Enterprise), V7 (Professional), V8 (Enterprise), V8 (Professional)
Database(s):All Supported Databases
App Server OS(s):All Supported Platforms
Latest release tested against:V8 (Enterprise)
Keywords:hang, SCBroker, hung, SBL-SWP-00121, SBL-SCB-00011

This document was previously published as Siebel Alert 1252.

Description

The Siebel Connection Broker (SCBroker) is a server component that provides intraserver load balancing. SCBroker distributes session login requests across multiple instances of Application Object Managers (AOMs) running on a Siebel Server. SCBroker logic will always route the request to the AOM process that has the least number of running tasks. For details see Siebel Bookshelf version 8.0 > Siebel Deployment Planning Guide > Siebel Architecture Overview > About the Siebel Connection Broker.
 
Under certain scenarios, an AOM process can be in a hanging state and is not able to acknowledge connection requests from the SCBroker anymore. If this hanging AOM process happens to have the least number of running tasks, SCB will attempt to route new session requests to this AOM process repeatedly. This results in blockage of all new login requests on that Siebel Server.
 
Bug 10644728 has been logged to address the product enhancement request to have the SCBroker ignore this unresponsive AOM process.

Likelihood of Occurrence

This Alert is applicable if you are running Siebel Application Server 7.7, 7.8 , 8.0 and 8.1.

Possible Symptoms

The following criteria all needs to be met and these are the symptoms that you may encounter if you are experiencing this type of deadlock behavior:
 
  • Every login attempt to a certain AOM on the affected Siebel Server is prompted with the message:
 
SBL-SWP-00121: The server you are trying to access is either busy or experiencing difficulties. Please close the Web browser, open a new browser window, and try logging in again.
              It takes approximately 60 seconds until the server busy message appears. 
  • However session logins to other components are working without any problem.
 
  • The AOM component in the server admin screen shows up as running on the affected server.
 
  • No crashes for the AOM are reported in the enterprise logs. No core dump or crash.txt file is written on the affected server.
 
  • AOM tasks might be staying in the state “Handling Logon” or ”Relogin”.
 
  • The SCBroker log is continuously logging the following error message:
 
SBL-SCB-00011: Failed to connect to pipe (SEBL_0_pid) on process pid
 
where pid is the process id of the corresponding object manager process. This process id can be found in the enterprise log to associate it with a certain AOM.
 
  • The process with this pid can still be found in the process list of the operating system and corresponds to one of the AOM processes on the affected server.
 
  • NOTE: Applicable to Windows only. You may observe a crash.txt file, however there is no error message SBL-OSD-01000 being logged in the enterprise log and the crashing process is not restarted despite the AutoRestart parameter is set to true.

Workaround or Resolution

The workaround to get the SCBroker process operative again is to terminate the hanging AOM process manually. This can be done using the kill command on UNIX or the Task Manager on Windows.
 
Once the offending process has been terminated, SCBroker will continue normal distribution of new session requests across the remaining AOM processes.
 
To analyze the root cause that is responsible for the AOM hang, please log a new Service Request with Technical Support and provide the information as described in the following references:
 
 
 
Two scenarios that can result in SCBroker disruption and the corresponding resolution are already described in the following references:
 
  1. Document 473809.1 discuss Object Manager hang behaviors on UNIX Platforms due to calls to malloc memory management function in the crash handler code.
 
  1. Document 473854.1 discuss unexpected UNIX process signals can cause MT Server to shut down or hang.
 
NOTE: Applicable to Windows only and any changes made to the Windows registry should be performed only after taking appropriate back-ups:
 
  1. Check the registry key HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\AeDebug
 
  1. If the Auto key is set to 0, a message box will be displayed by Dr. Watson prior to postmortem debugging.
 
  1. In case Dr. Watson is enabled, this can cause the SCBroker to still route connection requests to the remaining fragment of the crashed process during the 5 minute period where Dr. Watson is waiting for the Visual notification being acknowledged.
 
  1. The Auto key must be set to 1 to avoid the pop up boxes in the future.


Applies to:

Siebel CRM - Version 7.8.2 [19213] to 8.1.1 SIA [21111] [Release V7 to V8]
z*OBSOLETE: Microsoft Windows Server 2003
Product Release: V7 (Professional) and later
Version: 7.8.2 [19213]
Database: Microsoft SQL Server 2005
Application Server OS: Microsoft Windows 2003 Server
Database Server OS: Microsoft Windows 2003 Server

This document was previously published as Siebel SR 38-3304776353.


*** Checked for Relevance on 11-Sep-2012 ***

Symptoms

SBL-NET-01023, SBL-SCB-00011, SBL-SSM-00004, SBL-SSM-00006
We have launched Siebel today. Users are seeing 4 problems:

1. Server busy or problem experienced browser error
2. White screen with progress bar
3. Siebel splash screen with progress bar (no login box)
4. Users get into the system and then get booted out.

When first launched, we had serious avaikability issues as above.
We then changed the following settings:

1. MAX TASKS params for all object managers from 20 to 100 (as recommended in PRR) except fo UK OM which was changd to 200.
2. MAX TASKS for connection broker from 1 to 2.

Since these changes users have been able to access, but within last 1 hour we have again seen the same issues.

Cause

Encryption for the communication between Siebel web server and Siebel servers is set to MSCRYPTO

Solution

Message 1

As per the client description their siebel web client became unavailable at certain times.
After investigations the symptoms were found to be as follows:
On the siebel application server there was a siebmtshmw process that became to use 100% CPU. This process did not became unresponsive but instead to became idle when no requests where processed it went to 100%CPU. On a 4 processor box when there were 4 siebmtshmw processes that went in this state the machine begun to respond very slow this resulting in dropping new sessions and overall log responses to incomming requests.
On the web server machine (IIS) there was present the same situation (the siebel extensions out of process begun to consume 100%CPU)
The thread that consumed CPU was identified from the performance monitor logs and then core dump where taken from these processes. The thread was a communication thread between the siebel web extensions (SWSE) and object manager (OM). The call stack for the corresponding thread was:

sslcnapi!compress_write+0xdb
sslcnapi!zlib_inflate+0x8d
sslcnapi!CSSSISConn::_DecodeRawMsg+0x158
sslcnapi!CSSSISConn::SisAsyncThreadMain+0x8d
sslcosd!OSDThreadStart+0x1c0 sslcosd!OSDNTThreadStart+0xc
msvcr70!beginthreadex+0xba
kernel32!GetModuleFileNameA+0xeb
...

The relevant line in the communication call stack is CSSSISConn::_DecodeRawMsg. The customer had the encryption for the communication set to MSCRYPTO. This encryption is not largely used or recommended by Siebel. After setting the encryption to NONE the problem did not re-occurred.

Resolution:

The customer choose to check if they need encrypted communication between the web server and the application servers and if this is the case use SSL instead.








Applies to:

Siebel Assignment Manager - Version: 7.7.2.6 SIA [18372] and later   [Release: V7 and later ]
Information in this document applies to any platform.

Symptoms

After encountering a cluster server crash which forced a restart of the application server it was no more possible to invoke Assignment Manager to Assign Service Requests.

Error reported includes:
[1] Could not route message to AsgnSrvr with registered key (null)
[2] no other server
[3] Stack trace:
Service [Synchronous Assignment Manager Requests].InvokeMethod(), Built-in function
BusComp [Service Request].BusComp_WriteRecord(), Line: 1568
From the logs:
ObjectMgr
SBL-SRB-00047: Could not route message to AsgnSrvr with registered key (null)

SRBroker
GenericLog GenericError 1 0 2008-07-15 07:53:48 (scbcomp.cpp (822) err=7100011 sys=0) SBL-SCB-00011: Failed to connect to pipe (SEBL_32_2460) on process 2460.

Cause


After a review of siebns.dat, the following section showed that the AsgnMgmt components  were 'Online' but not 'Enabled'
----------
[/enterprises/VFECRMPRD/servers/SIEBRPT01/component groups/AsgnMgmt]
Type=empty

[/enterprises/VFECRMPRD/servers/SIEBRPT01/component groups/AsgnMgmt/definition]
Type=empty

[/enterprises/VFECRMPRD/servers/SIEBRPT01/component groups/AsgnMgmt/definition/enable state]
Persistence=full
Type=string
Value="disabled"
Length=16

Details in the Gateway server caches provides statuses and availability of each server and it's components within the enterprise and this is what is used at startup.

The Cluster Server crash corrupted the siebns.dat and so after the restart, the corrupted values for the assignment components 'State' were being used.

Solution

The corrupted siebns.dat file was amended by resetting the assignment manager components 'enabled state' to 'enabled'.
Example:
From:
[/enterprises/VFECRMPRD/servers/SIEBRPT01/component groups/AsgnMgmt/definition/enable state]
Persistence=full
Type=string
Value="disabled"
Length=16
To:
[/enterprises/VFECRMPRD/servers/SIEBRPT01/component groups/AsgnMgmt/definition/enable state]
Persistence=full
Type=string
Value="enabled"
Length=16
 Once all the changes have been made, the system was restarted in the correct order to bring all components back  to an available state.
i.e.
a) Start Database
b) Start Gateway
c) Start Siebel Server





SBL-SCB-00008: Component (sccobjmgrsmc_enu*) is unavailable on this server

Applies to:

Siebel CRM - Version: 7.8.2.8 SIA [19237] and later   [Release: V7 and later ]
Information in this document applies to any platform.

Goal


The following errors are happening in the scbroker files - err=7100008 sys=0) SBL-SCB-00008: El componente (ecommwirelessobjmgr_esn) no estM-CM-!
disponible en este servidor - why ?

Solution

An1:
Hi Clemente,

Thank you for taking my call earlier.

As discussed I believe your issue is related to the your VirtualServer section in your lbconfig.txt containing servers where OM's are not running.

The following 7.8 bookshelf reference explains how to configure your lbconfig.txt and eapps.cfg to cater for this scenario > Siebel System Administration Guide > Configuring the System Architecture > Configuring Siebel Server Load Balancing > Optimizing the Siebel Load-Balancing Performance.

Please review this section and modify your lbconfig.txt and eapps.cfg accordingly. I suggest you test this configuration on your pre-production environment initially.

If you have any further questions on this please let me know.



Yes, you only need servers using Application Object managers in these lines which in your case I believe is EAI and ecommwirelessobjmgr_esn.

As mentioned on the phone I strongly recommend that any changes to your eapps.cfg/lbconfig.txt files be tested in pre-production prior to rolling ou in production.



Applies to:

Siebel CRM - Version: 7.8.2.10 [19241] and later   [Release: V7 and later ]
Information in this document applies to any platform.

Goal

When attempting to load balance two new Siebel servers and deploy them in production,
the following errors occur in SCBroker log:

SBL-SCB-00008: Component (sccobjmgrsmc_enu*) is unavailable on this server.
SBL-SCB-00008: Component (sccobjmgrsmc_eng*) is unavailable on this server.

This is a production environment where they are trying to deploy 2 new load balanced servers. The existing servers are already load balanced and that works ok.
The difference is that the keepalive url is set to "/crmprod/SCCObjMgrSMC_eng" on the new LB server, in comparison with the CRMHDLB (the one that is working) where the keepalive uri is set to "/CRMPROD/scbroker".

Solution

After changing keepalive uri on the load balancer to point to SCBroker no error is generated in the log files. In this case the root cause was incorrectly configured uri for keepalive connection from the load balancer.

SBL-SCB-00007: Enterprise name ( %1) in the request is invalid.


Applies to:

Error Message Area:Server Infrastructure - SCB
Version:Siebel 8.0 and later

Purpose

This document is intended to provide cause and corrective action information about Siebel Error Message SBL-SCB-00007: Enterprise name ( %1) in the request is invalid.

Scope

This document is informational and intended for any user.

SBL-SCB-00007: Enterprise name ( %1) in the request is invalid.

Explanation

Enterprise name in connect string could be wrong.

Corrective Action

Check eapps.cfg configuration - make sure that enterprise name matches that of the server.




Applies to:

Product Release: V7 (Enterprise)
Version: 7.8.2.3 [19221]
Database: IBM DB2 8.2
Application Server OS: Microsoft Windows 2003 Server
Database Server OS: IBM AIX 5L 5.2

This document was previously published as Siebel SR 38-3365847407.

Symptoms

SBL-SCB-00007What is the error code SBL-SCB-00007. I was not able to retrieve anything from the search on this from the knowledge base. In our production environment we get a error GenericLog GenericError 1 0 2007-05-31 13:22:16 (scbcomp.cpp (235) err=7100007 sys=0) SBL-SCB-00007: Enterprise name ( sbldev_ent) in the request is invalid in the scbroker.log file, it is becoming a huge file. sblprod_ent is the enterprise name but not sure why and what is looking for sbldev_ent which is the name for the development enterprise. I have attached the eapps.cfg file for your reference. Please note we modifid the sections we use for production in the eapps.cfg file from Dev environment and placed it in production. We use F5 bigip to load balance the siebel app servers.

Thanks

Solution

Message 1

For the benefit of other readers:


The error code SBL-SCB-00007 means that the Enterprise name in connect string could be wrong. Then you must check eapps.cfg configuration and make sure that enterprise name matches that of the server.

However the eapps.cfg configuration file was configured properly and also any request to sbldev_ent was not found in the Siebel Web Server log files.

Siebel Technical Support suggested customer to install a network trace utility or sniffer tool such as ethereal on the Siebel server and trace the traffic to the SCBroker Port 2321 and after that customer was able identify that even after bringing down the entire qa environment, the port 2321 was getting traffic.

Customer worked with their network team to understand the source of the traffic and they were able to identify that the BigIP health monitor was not setup correctly to look at SBLQA_ENT and hence the errors on SCbroker for invalid enterprise name.

Regards


Applies to:

Siebel CRM - Version: 8.1.1 SIA [21111] and later   [Release: V8 and later ]
Information in this document applies to any platform.

Symptoms


Environment:
-------------------
Siebel 8.1.1 SIA [21111], HP-UX 11.11 Itanium (32-bit), Test environment

Statement of Issue:
-----------------------------
Unable to access the Siebel application using Thin client, giving server busy error message

Steps:
---------
1) Launch Internet explorer
2) Enter thin client URL & hit enter

Error:
------
SBL-SCB-00007: Enterprise name ( (null)) in the request is invalid.

Business Impact:
-------------------------
Unable to make use of Siebel application

Changes

Reconfigured the SWSE with incorrect SC Broker port number

Cause

Incorrect SC Broker port number mentioned while reconfiguring SWSE

Solution

1) Stop Web server
2) Take a back up of existing eapps.cfg & eapps_sia.cfg files
3) Replace the in correct SC Broker port number to correct one [find and replace all incorrect values]
4) Save files
5) Restart the web server

References

NOTE:493180.1 - SBL-SCB-00007