Ubuntu Linux Apache Error: [notice] child pid XXXX exit signal Segmentation fault (11)

I received the below error when troubleshooting an issue reported to me by a client regarding one of their intranet web sites not working properly. After some poking around I noticed that there were numerous segmentation faults being reported in the Apache error logs. I had previously set up the server to dump core files when there were segmentation faults within Apache so troubleshooting the error was fairly easy. It ended up being a connection issue with an AS400 that was down over the weekend at the company and I was able to verify this using the below information.

Load Apache Core Dump Into GNU Debugger On Ubuntu Linux:

The first thing that needed to take place was loading the Apache core dump using gdb as shown in the below example.

root@devserver:/etc# gdb /usr/sbin/apache2 /var/cache/apache2/core
GNU gdb 6.8-debian
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu"...
(no debugging symbols found)
warning: Can't read pathname for load map: Input/output error.
(no debugging symbols found)
Loaded symbols for /usr/sbin/apache2
Reading symbols from /lib/libpcre.so.3...(no debugging symbols found)...done.
Loaded symbols for /lib/libpcre.so.3
Reading symbols from /usr/lib/libaprutil-1.so.0...Reading symbols from /usr/lib/debug/usr/lib/libaprutil-1.so.0.2.12...done.
done.
Loaded symbols for /usr/lib/libaprutil-1.so.0
Reading symbols from /usr/lib/libapr-1.so.0...Reading symbols from /usr/lib/debug/usr/lib/libapr-1.so.0.2.12...done.
done.
Loaded symbols for /usr/lib/libapr-1.so.0
Reading symbols from /lib/libpthread.so.0...done.
Loaded symbols for /lib/libpthread.so.0
Reading symbols from /lib/libc.so.6...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /usr/lib/libldap_r-2.4.so.2...done.
Loaded symbols for /usr/lib/libldap_r-2.4.so.2
Reading symbols from /usr/lib/liblber-2.4.so.2...done.
Loaded symbols for /usr/lib/liblber-2.4.so.2
Reading symbols from /usr/lib/libdb-4.6.so...done.
Loaded symbols for /usr/lib/libdb-4.6.so
Reading symbols from /usr/lib/libpq.so.5...done.
Loaded symbols for /usr/lib/libpq.so.5
Reading symbols from /usr/lib/libmysqlclient_r.so.15...done.
Loaded symbols for /usr/lib/libmysqlclient_r.so.15
Reading symbols from /usr/lib/libsqlite3.so.0...done.
Loaded symbols for /usr/lib/libsqlite3.so.0
Reading symbols from /usr/lib/libexpat.so.1...done.
Loaded symbols for /usr/lib/libexpat.so.1
Reading symbols from /lib/libuuid.so.1...done.
Loaded symbols for /lib/libuuid.so.1
Reading symbols from /lib/librt.so.1...done.
Loaded symbols for /lib/librt.so.1
Reading symbols from /lib/libcrypt.so.1...done.
Loaded symbols for /lib/libcrypt.so.1
Reading symbols from /lib/libdl.so.2...done.
Loaded symbols for /lib/libdl.so.2
Reading symbols from /lib/ld-linux-x86-64.so.2...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /usr/lib/libsasl2.so.2...done.
Loaded symbols for /usr/lib/libsasl2.so.2
Reading symbols from /lib/libresolv.so.2...done.
Loaded symbols for /lib/libresolv.so.2
Reading symbols from /usr/lib/libgnutls.so.26...done.
Loaded symbols for /usr/lib/libgnutls.so.26
Reading symbols from /usr/lib/libtasn1.so.3...done.
Loaded symbols for /usr/lib/libtasn1.so.3
Reading symbols from /lib/libz.so.1...done.
Loaded symbols for /lib/libz.so.1
Reading symbols from /lib/libgcrypt.so.11...done.
Loaded symbols for /lib/libgcrypt.so.11
Reading symbols from /usr/lib/libssl.so.0.9.8...done.
Loaded symbols for /usr/lib/libssl.so.0.9.8
Reading symbols from /usr/lib/libcrypto.so.0.9.8...done.
Loaded symbols for /usr/lib/libcrypto.so.0.9.8
Reading symbols from /usr/lib/libkrb5.so.3...done.
Loaded symbols for /usr/lib/libkrb5.so.3
Reading symbols from /lib/libcom_err.so.2...done.
Loaded symbols for /lib/libcom_err.so.2
Reading symbols from /usr/lib/libgssapi_krb5.so.2...done.
Loaded symbols for /usr/lib/libgssapi_krb5.so.2
Reading symbols from /lib/libnsl.so.1...done.
Loaded symbols for /lib/libnsl.so.1
Reading symbols from /lib/libm.so.6...done.
Loaded symbols for /lib/libm.so.6
Reading symbols from /lib/libgpg-error.so.0...done.
Loaded symbols for /lib/libgpg-error.so.0
Reading symbols from /usr/lib/libk5crypto.so.3...done.
Loaded symbols for /usr/lib/libk5crypto.so.3
Reading symbols from /usr/lib/libkrb5support.so.0...done.
Loaded symbols for /usr/lib/libkrb5support.so.0
Reading symbols from /lib/libkeyutils.so.1...done.
Loaded symbols for /lib/libkeyutils.so.1
Reading symbols from /lib/libnss_compat.so.2...done.
Loaded symbols for /lib/libnss_compat.so.2
Reading symbols from /lib/libnss_nis.so.2...done.
Loaded symbols for /lib/libnss_nis.so.2
Reading symbols from /lib/libnss_files.so.2...done.
Loaded symbols for /lib/libnss_files.so.2
Reading symbols from /lib/libnss_mdns4_minimal.so.2...done.
Loaded symbols for /lib/libnss_mdns4_minimal.so.2
Reading symbols from /lib/libnss_dns.so.2...done.
Loaded symbols for /lib/libnss_dns.so.2
Reading symbols from /usr/lib/apache2/modules/mod_alias.so...done.
Loaded symbols for /usr/lib/apache2/modules/mod_alias.so
Reading symbols from /usr/lib/apache2/modules/mod_auth_basic.so...done.
Loaded symbols for /usr/lib/apache2/modules/mod_auth_basic.so
Reading symbols from /usr/lib/apache2/modules/mod_auth_pam.so...done.
Loaded symbols for /usr/lib/apache2/modules/mod_auth_pam.so
Reading symbols from /lib/libpam.so.0...done.
Loaded symbols for /lib/libpam.so.0
Reading symbols from /usr/lib/apache2/modules/mod_authn_file.so...done.
Loaded symbols for /usr/lib/apache2/modules/mod_authn_file.so
Reading symbols from /usr/lib/apache2/modules/mod_authz_default.so...done.
Loaded symbols for /usr/lib/apache2/modules/mod_authz_default.so
Reading symbols from /usr/lib/apache2/modules/mod_authz_groupfile.so...done.
Loaded symbols for /usr/lib/apache2/modules/mod_authz_groupfile.so
Reading symbols from /usr/lib/apache2/modules/mod_authz_host.so...done.
Loaded symbols for /usr/lib/apache2/modules/mod_authz_host.so
Reading symbols from /usr/lib/apache2/modules/mod_authz_user.so...done.
Loaded symbols for /usr/lib/apache2/modules/mod_authz_user.so
Reading symbols from /usr/lib/apache2/modules/mod_autoindex.so...done.
Loaded symbols for /usr/lib/apache2/modules/mod_autoindex.so
Reading symbols from /usr/lib/apache2/modules/mod_cgi.so...done.
Loaded symbols for /usr/lib/apache2/modules/mod_cgi.so
Reading symbols from /usr/lib/apache2/modules/mod_dav.so...done.
Loaded symbols for /usr/lib/apache2/modules/mod_dav.so
Reading symbols from /usr/lib/apache2/modules/mod_dav_svn.so...done.
Loaded symbols for /usr/lib/apache2/modules/mod_dav_svn.so
Reading symbols from /usr/lib/libsvn_repos-1.so.1...done.
Loaded symbols for /usr/lib/libsvn_repos-1.so.1
Reading symbols from /usr/lib/libsvn_fs-1.so.1...done.
Loaded symbols for /usr/lib/libsvn_fs-1.so.1
Reading symbols from /usr/lib/libsvn_fs_fs-1.so.1...done.
Loaded symbols for /usr/lib/libsvn_fs_fs-1.so.1
Reading symbols from /usr/lib/libsvn_fs_base-1.so.1...done.
Loaded symbols for /usr/lib/libsvn_fs_base-1.so.1
Reading symbols from /usr/lib/libsvn_fs_util-1.so.1...done.
Loaded symbols for /usr/lib/libsvn_fs_util-1.so.1
Reading symbols from /usr/lib/libsvn_delta-1.so.1...done.
Loaded symbols for /usr/lib/libsvn_delta-1.so.1
Reading symbols from /usr/lib/libsvn_subr-1.so.1...done.
Loaded symbols for /usr/lib/libsvn_subr-1.so.1
Reading symbols from /usr/lib/apache2/modules/mod_authz_svn.so...done.
Loaded symbols for /usr/lib/apache2/modules/mod_authz_svn.so
Reading symbols from /usr/lib/apache2/modules/mod_deflate.so...done.
Loaded symbols for /usr/lib/apache2/modules/mod_deflate.so
Reading symbols from /usr/lib/apache2/modules/mod_dir.so...done.
Loaded symbols for /usr/lib/apache2/modules/mod_dir.so
Reading symbols from /usr/lib/apache2/modules/mod_env.so...done.
Loaded symbols for /usr/lib/apache2/modules/mod_env.so
Reading symbols from /usr/lib/apache2/modules/mod_fastcgi.so...done.
Loaded symbols for /usr/lib/apache2/modules/mod_fastcgi.so
Reading symbols from /usr/lib/apache2/modules/mod_jk.so...done.
Loaded symbols for /usr/lib/apache2/modules/mod_jk.so
Reading symbols from /usr/lib/apache2/modules/mod_mime.so...done.
Loaded symbols for /usr/lib/apache2/modules/mod_mime.so
Reading symbols from /usr/lib/apache2/modules/mod_negotiation.so...done.
Loaded symbols for /usr/lib/apache2/modules/mod_negotiation.so
Reading symbols from /usr/lib/apache2/modules/libphp5.so...done.
Loaded symbols for /usr/lib/apache2/modules/libphp5.so
Reading symbols from /lib/libbz2.so.1.0...done.
Loaded symbols for /lib/libbz2.so.1.0
Reading symbols from /usr/lib/libxml2.so.2...done.
Loaded symbols for /usr/lib/libxml2.so.2
Reading symbols from /usr/lib/apache2/modules/mod_proxy.so...done.
Loaded symbols for /usr/lib/apache2/modules/mod_proxy.so
Reading symbols from /usr/lib/apache2/modules/mod_proxy_ajp.so...done.
Loaded symbols for /usr/lib/apache2/modules/mod_proxy_ajp.so
Reading symbols from /usr/lib/apache2/modules/mod_proxy_balancer.so...done.
Loaded symbols for /usr/lib/apache2/modules/mod_proxy_balancer.so
Reading symbols from /usr/lib/apache2/modules/mod_proxy_connect.so...done.
Loaded symbols for /usr/lib/apache2/modules/mod_proxy_connect.so
Reading symbols from /usr/lib/apache2/modules/mod_proxy_ftp.so...done.
Loaded symbols for /usr/lib/apache2/modules/mod_proxy_ftp.so
Reading symbols from /usr/lib/apache2/modules/mod_proxy_html.so...done.
Loaded symbols for /usr/lib/apache2/modules/mod_proxy_html.so
Reading symbols from /usr/lib/apache2/modules/mod_proxy_http.so...done.
Loaded symbols for /usr/lib/apache2/modules/mod_proxy_http.so
Reading symbols from /usr/lib/apache2/modules/mod_python.so...done.
Loaded symbols for /usr/lib/apache2/modules/mod_python.so
Reading symbols from /usr/lib/libpython2.6.so.1.0...done.
Loaded symbols for /usr/lib/libpython2.6.so.1.0
Reading symbols from /lib/libutil.so.1...done.
Loaded symbols for /lib/libutil.so.1
Reading symbols from /usr/lib/apache2/modules/mod_setenvif.so...done.
Loaded symbols for /usr/lib/apache2/modules/mod_setenvif.so
Reading symbols from /usr/lib/apache2/modules/mod_ssl.so...done.
Loaded symbols for /usr/lib/apache2/modules/mod_ssl.so
Reading symbols from /usr/lib/apache2/modules/mod_status.so...done.
Loaded symbols for /usr/lib/apache2/modules/mod_status.so
Reading symbols from /usr/lib/php5/20060613/adodb.so...done.
Loaded symbols for /usr/lib/php5/20060613/adodb.so
Reading symbols from /usr/lib/php5/20060613/curl.so...done.
Loaded symbols for /usr/lib/php5/20060613/curl.so
Reading symbols from /usr/lib/libcurl.so.4...done.
Loaded symbols for /usr/lib/libcurl.so.4
Reading symbols from /usr/lib/libidn.so.11...done.
Loaded symbols for /usr/lib/libidn.so.11
Reading symbols from /usr/lib/php5/20060613/gd.so...done.
Loaded symbols for /usr/lib/php5/20060613/gd.so
Reading symbols from /usr/lib/libgd.so.2...done.
Loaded symbols for /usr/lib/libgd.so.2
Reading symbols from /usr/lib/libt1.so.5...done.
Loaded symbols for /usr/lib/libt1.so.5
Reading symbols from /usr/lib/libfreetype.so.6...done.
Loaded symbols for /usr/lib/libfreetype.so.6
Reading symbols from /usr/lib/libX11.so.6...done.
Loaded symbols for /usr/lib/libX11.so.6
Reading symbols from /usr/lib/libXpm.so.4...done.
Loaded symbols for /usr/lib/libXpm.so.4
Reading symbols from /usr/lib/libpng12.so.0...done.
Loaded symbols for /usr/lib/libpng12.so.0
Reading symbols from /usr/lib/libjpeg.so.62...done.
Loaded symbols for /usr/lib/libjpeg.so.62
Reading symbols from /usr/lib/libfontconfig.so.1...done.
Loaded symbols for /usr/lib/libfontconfig.so.1
Reading symbols from /usr/lib/libxcb.so.1...done.
Loaded symbols for /usr/lib/libxcb.so.1
Reading symbols from /usr/lib/libXau.so.6...done.
Loaded symbols for /usr/lib/libXau.so.6
Reading symbols from /usr/lib/libXdmcp.so.6...done.
Loaded symbols for /usr/lib/libXdmcp.so.6
Reading symbols from /usr/lib/php5/20060613/ldap.so...done.
Loaded symbols for /usr/lib/php5/20060613/ldap.so
Reading symbols from /usr/lib/php5/20060613/mssql.so...done.
Loaded symbols for /usr/lib/php5/20060613/mssql.so
Reading symbols from /usr/lib/libsybdb.so.5...done.
Loaded symbols for /usr/lib/libsybdb.so.5
Reading symbols from /usr/lib/php5/20060613/mysql.so...done.
Loaded symbols for /usr/lib/php5/20060613/mysql.so
Reading symbols from /usr/lib/php5/20060613/mysqli.so...done.
Loaded symbols for /usr/lib/php5/20060613/mysqli.so
Reading symbols from /usr/lib/php5/20060613/odbc.so...done.
Loaded symbols for /usr/lib/php5/20060613/odbc.so
Reading symbols from /usr/lib/libodbc.so.1...done.
Loaded symbols for /usr/lib/libodbc.so.1
Reading symbols from /usr/lib/libltdl.so.7...done.
Loaded symbols for /usr/lib/libltdl.so.7
Reading symbols from /usr/lib/php5/20060613/pdo.so...done.
Loaded symbols for /usr/lib/php5/20060613/pdo.so
Reading symbols from /usr/lib/php5/20060613/pdo_dblib.so...done.
Loaded symbols for /usr/lib/php5/20060613/pdo_dblib.so
Reading symbols from /usr/lib/php5/20060613/pdo_mysql.so...done.
Loaded symbols for /usr/lib/php5/20060613/pdo_mysql.so
Reading symbols from /usr/lib/php5/20060613/pdo_odbc.so...done.
Loaded symbols for /usr/lib/php5/20060613/pdo_odbc.so
Reading symbols from /usr/lib/php5/20060613/pdo_pgsql.so...done.
Loaded symbols for /usr/lib/php5/20060613/pdo_pgsql.so
Reading symbols from /usr/lib/php5/20060613/pgsql.so...done.
Loaded symbols for /usr/lib/php5/20060613/pgsql.so
Reading symbols from /usr/lib/php5/20060613/snmp.so...done.
Loaded symbols for /usr/lib/php5/20060613/snmp.so
Reading symbols from /usr/lib/libnetsnmp.so.15...done.
Loaded symbols for /usr/lib/libnetsnmp.so.15
Reading symbols from /usr/lib/php5/20060613/xsl.so...done.
Loaded symbols for /usr/lib/php5/20060613/xsl.so
Reading symbols from /usr/lib/libexslt.so.0...done.
Loaded symbols for /usr/lib/libexslt.so.0
Reading symbols from /usr/lib/libxslt.so.1...done.
Loaded symbols for /usr/lib/libxslt.so.1
Reading symbols from /lib/libgcc_s.so.1...done.
Loaded symbols for /lib/libgcc_s.so.1
Reading symbols from /usr/lib/gconv/ISO8859-1.so...done.
Loaded symbols for /usr/lib/gconv/ISO8859-1.so
Reading symbols from /usr/lib/gconv/CP1252.so...done.
Loaded symbols for /usr/lib/gconv/CP1252.so
Reading symbols from /opt/ibm/iSeriesAccess/lib64/libcwbodbc.so...done.
Loaded symbols for /opt/ibm/iSeriesAccess/lib64/libcwbodbc.so
Reading symbols from /usr/lib/libcwbcore.so...done.
Loaded symbols for /usr/lib/libcwbcore.so
Reading symbols from /usr/lib/libodbcinst.so.1...done.
Loaded symbols for /usr/lib/libodbcinst.so.1
Reading symbols from /usr/lib/libstdc++.so.6...done.
Loaded symbols for /usr/lib/libstdc++.so.6
Core was generated by `/usr/sbin/apache2 -k start'.
Program terminated with signal 11, Segmentation fault.
[New process 3140]
#0 0x00007f03f2735f71 in PiNlStrFile::loadup () from /usr/lib/libcwbcore.so
(gdb)

Once loaded use the “bt” or “bt full”, which stands for backtrace and will display the stack, command to display output from the Apache core file just loaded into GDB or the GNU Debugger as shown in the below example. In this example we issue the bt command only since it provides enough output to locate the issue.

Issue Backtrace Command Once Core File Loaded In GDB:

(gdb) bt
#0 0x00007f988cdc3f71 in PiNlStrFile::loadup() () from /usr/lib/libcwbcore.so
#1 0x00007f988cdc4402 in PiNlStrFile::gets(unsigned int, char*, unsigned long) () from /usr/lib/libcwbcore.so
#2 0x00007f988cfcf9b9 in ERROR_LIST_INFO::storeTheError(unsigned int, __va_list_tag*) ()
from /opt/ibm/iSeriesAccess/lib64/libcwbodbc.so
#3 0x00007f988cfd037e in ERROR_LIST_INFO::vstoreError(unsigned int, ...) () from /opt/ibm/iSeriesAccess/lib64/libcwbodbc.so
#4 0x00007f988cff3fe5 in ScopeDelSystem::~ScopeDelSystem() () from /opt/ibm/iSeriesAccess/lib64/libcwbodbc.so
#5 0x00007f988cff417a in CONNECT_INFO::fullconnect() () from /opt/ibm/iSeriesAccess/lib64/libcwbodbc.so
#6 0x00007f988cff7369 in CONNECT_INFO::attemptConnection() () from /opt/ibm/iSeriesAccess/lib64/libcwbodbc.so
#7 0x00007f988cff7f43 in CONNECT_INFO::sqlConnect(wchar_t*, unsigned int, wchar_t*, unsigned int, wchar_t*, unsigned int) ()
from /opt/ibm/iSeriesAccess/lib64/libcwbodbc.so
#8 0x00007f988cff8288 in cow_SQLConnect(void*, wchar_t*, short, wchar_t*, short, wchar_t*, short) ()
from /opt/ibm/iSeriesAccess/lib64/libcwbodbc.so
#9 0x00007f988cfde147 in SQLConnect () from /opt/ibm/iSeriesAccess/lib64/libcwbodbc.so
#10 0x00007f988f8d0129 in SQLConnect () from /usr/lib/libodbc.so.1
#11 0x00007f988fb26845 in odbc_sqlconnect () from /usr/lib/php5/20060613/odbc.so
#12 0x00007f988fb2f128 in odbc_do_connect () from /usr/lib/php5/20060613/odbc.so
#13 0x00007f9895f8a2ab in ?? () from /usr/lib/apache2/modules/libphp5.so
#14 0x00007f9895f85fcc in execute () from /usr/lib/apache2/modules/libphp5.so
#15 0x00007f9895f61e66 in zend_execute_scripts () from /usr/lib/apache2/modules/libphp5.so
#16 0x00007f9895f1cee3 in php_execute_script () from /usr/lib/apache2/modules/libphp5.so
#17 0x00007f9895fd3e5d in ?? () from /usr/lib/apache2/modules/libphp5.so
#18 0x00007f989eb730d0 in ap_run_handler () from /usr/sbin/apache2
#19 0x00007f989eb76a18 in ap_invoke_handler () from /usr/sbin/apache2 #20 0x00007f989eb84578 in ap_process_request () from /usr/sbin/apache2
#21 0x00007f989eb81458 in ?? () from /usr/sbin/apache2
#22 0x00007f989eb7ac68 in ap_run_process_connection () from /usr/sbin/apache2
#23 0x00007f989eb88fcb in ?? () from /usr/sbin/apache2
#24 0x00007f989eb8929a in ?? () from /usr/sbin/apache2
#25 0x00007f989eb89f24 in ap_mpm_run () from /usr/sbin/apache2
#26 0x00007f989eb5f350 in main () from /usr/sbin/apache2
(gdb)

So from the output above I was able to determine that there was a SQL connection issue via the libcwbodbc.so library file which happens to be the ODBC driver from IBM used to access a DB2 database on the IBM AS400. Once the AS400 DB2 database was back online the site was working again without issue.