Ubuntu Linux Apache Error: [notice] child pid XXXX exit signal Segmentation fault (11)

I received the below error when troubleshooting an issue reported to me by a client regarding one of their intranet web sites not working properly. After some poking around I noticed that there were numerous segmentation faults being reported in the Apache error logs. I had previously set up the server to dump core files when there were segmentation faults within Apache so troubleshooting the error was fairly easy. It ended up being a connection issue with an AS400 that was down over the weekend at the company and I was able to verify this using the below information.

Load Apache Core Dump Into GNU Debugger On Ubuntu Linux:

The first thing that needed to take place was loading the Apache core dump using gdb as shown in the below example.

Once loaded use the “bt” or “bt full”, which stands for backtrace and will display the stack, command to display output from the Apache core file just loaded into GDB or the GNU Debugger as shown in the below example. In this example we issue the bt command only since it provides enough output to locate the issue.

Issue Backtrace Command Once Core File Loaded In GDB:

So from the output above I was able to determine that there was a SQL connection issue via the libcwbodbc.so library file which happens to be the ODBC driver from IBM used to access a DB2 database on the IBM AS400. Once the AS400 DB2 database was back online the site was working again without issue.