• Home »
  • Errors »
  • CriticalMailer ::Exception:: FATAL C53300 Msorry, too many clients already Fproc.c L276 RInitProcess

CriticalMailer ::Exception:: FATAL C53300 Msorry, too many clients already Fproc.c L276 RInitProcess

I was tasked with troubleshooting a Ruby Rufus Task Scheduler issue today so I started with reviewing the log files. Upon digging into the logs I noticed numerous errors which are expanded below. The primary error appeared to be a connection issue from Ruby to PostgreSQL as it was complaining about too many connections. While the server I was looking into is actively used in a development environment the connections limit of 100 should have been more than enough to handle the load. After looking into the issue more I found out that some large Postgres queries from within the Ruby code were causing the issue and once those were optimized things started working properly. Below I describe the errors noticed via the Ruby logs, steps taken to troubleshoot the errors, and final resolution.

Ruby Rufus Task Scheduler Logs Error:

ruby

  1. CriticalMailer.deliver_successful_user_logins_daily()::Exception:: FATAL  C53300  Msorry, too many clients already        Fproc.c L276    RInitProcess

Ruby Rufus Task Scheduler Logs Error Details:

ruby

  1. [DEV]CriticalMailer.deliver_successful_user_logins_daily()::Exception:: FATAL  C53300  Msorry, too many clients already        Fproc.c L276    RInitProcess
  2. /usr/local/lib/ruby/gems/1.8/gems/postgres-pr-0.5.1/lib/postgres-pr/connection.rb:81:in `initialize'
  3. /usr/local/lib/ruby/gems/1.8/gems/postgres-pr-0.5.1/lib/postgres-pr/connection.rb:55:in `loop'
  4. /usr/local/lib/ruby/gems/1.8/gems/postgres-pr-0.5.1/lib/postgres-pr/connection.rb:55:in `initialize'
  5. /usr/local/lib/ruby/gems/1.8/gems/postgres-pr-0.5.1/lib/postgres-pr/postgres-compat.rb:23:in `new'
  6. /usr/local/lib/ruby/gems/1.8/gems/postgres-pr-0.5.1/lib/postgres-pr/postgres-compat.rb:23:in `initialize'
  7. /home/web/project/vendor/rails/activerecord/lib/active_record/connection_adapters/postgresql_adapter.rb:814:in `connect'
  8. /home/web/project/vendor/rails/activerecord/lib/active_record/connection_adapters/postgresql_adapter.rb:814:in `connect'
  9. /home/web/project/vendor/rails/activerecord/lib/active_record/connection_adapters/postgresql_adapter.rb:251:in `initialize'
  10. /home/web/project/vendor/rails/activerecord/lib/active_record/connection_adapters/postgresql_adapter.rb:37:in `new'
  11. /home/web/project/vendor/rails/activerecord/lib/active_record/connection_adapters/postgresql_adapter.rb:37:in `postgresql_connection'
  12. /home/web/project/vendor/rails/activerecord/lib/active_record/connection_adapters/abstract/connection_specification.rb:292:in `send'
  13. /home/web/project/vendor/rails/activerecord/lib/active_record/connection_adapters/abstract/connection_specification.rb:292:in `connection='
  14. /home/web/project/vendor/rails/activerecord/lib/active_record/connection_adapters/abstract/connection_specification.rb:260:in `retrieve_connection'
  15. /home/web/project/vendor/rails/activerecord/lib/active_record/connection_adapters/abstract/connection_specification.rb:78:in `connection'
  16. /home/web/project/vendor/rails/activerecord/lib/active_record/base.rb:2693:in `quoted_table_name'
  17. /home/web/project/vendor/rails/activerecord/lib/active_record/base.rb:1469:in `construct_finder_sql'
  18. /home/web/project/vendor/rails/activerecord/lib/active_record/base.rb:1341:in `find_every'
  19. /home/web/project/vendor/rails/activerecord/lib/active_record/base.rb:536:in `find'
  20. /home/web/project/app/models/compliance.rb:161:in `successful_user_logins'
  21. /home/web/project/app/models/critical_mailer.rb:104:in `successful_user_logins_daily'
  22. /home/web/project/vendor/rails/actionmailer/lib/action_mailer/base.rb:446:in `__send__'
  23. /home/web/project/vendor/rails/actionmailer/lib/action_mailer/base.rb:446:in `create!'
  24. /home/web/project/vendor/rails/actionmailer/lib/action_mailer/base.rb:439:in `initialize'
  25. /home/web/project/vendor/rails/actionmailer/lib/action_mailer/base.rb:383:in `new'
  26. /home/web/project/vendor/rails/actionmailer/lib/action_mailer/base.rb:383:in `method_missing'
  27. /home/web/project/config/jobs/reporting.rb:65
  28. /home/web/project/vendor/gems/rufus-scheduler-2.0.1/lib/rufus/sc/jobs.rb:136:in `call'
  29. /home/web/project/vendor/gems/rufus-scheduler-2.0.1/lib/rufus/sc/jobs.rb:136:in `trigger'
  30. /home/web/project/vendor/gems/rufus-scheduler-2.0.1/lib/rufus/sc/scheduler.rb:331:in `call'
  31. /home/web/project/vendor/gems/rufus-scheduler-2.0.1/lib/rufus/sc/scheduler.rb:331:in `trigger_job'
  32. /home/web/project/vendor/gems/rufus-scheduler-2.0.1/lib/rufus/sc/scheduler.rb:331:in `initialize'
  33. /home/web/project/vendor/gems/rufus-scheduler-2.0.1/lib/rufus/sc/scheduler.rb:331:in `new'
  34. /home/web/project/vendor/gems/rufus-scheduler-2.0.1/lib/rufus/sc/scheduler.rb:331:in `trigger_job'
  35. /home/web/project/vendor/gems/rufus-scheduler-2.0.1/lib/rufus/sc/jobs.rb:121:in `send'
  36. /home/web/project/vendor/gems/rufus-scheduler-2.0.1/lib/rufus/sc/jobs.rb:121:in `trigger'
  37. /home/web/project/vendor/gems/rufus-scheduler-2.0.1/lib/rufus/sc/jobs.rb:334:in `trigger_if_matches'
  38. /home/web/project/vendor/gems/rufus-scheduler-2.0.1/lib/rufus/sc/jobqueues.rb:136:in `trigger_matching_jobs'
  39. /home/web/project/vendor/gems/rufus-scheduler-2.0.1/lib/rufus/sc/jobqueues.rb:136:in `each'
  40. /home/web/project/vendor/gems/rufus-scheduler-2.0.1/lib/rufus/sc/jobqueues.rb:136:in `trigger_matching_jobs'
  41. /home/web/project/vendor/gems/rufus-scheduler-2.0.1/lib/rufus/sc/scheduler.rb:280:in `cron_step'
  42. /home/web/project/vendor/gems/rufus-scheduler-2.0.1/lib/rufus/sc/scheduler.rb:266:in `step'
  43. /home/web/project/vendor/gems/rufus-scheduler-2.0.1/lib/rufus/sc/scheduler.rb:351:in `start'
  44. /home/web/project/vendor/gems/rufus-scheduler-2.0.1/lib/rufus/sc/scheduler.rb:349:in `loop'
  45. /home/web/project/vendor/gems/rufus-scheduler-2.0.1/lib/rufus/sc/scheduler.rb:349:in `start'
  46. /home/web/project/vendor/gems/rufus-scheduler-2.0.1/lib/rufus/sc/scheduler.rb:348:in `initialize'
  47. /home/web/project/vendor/gems/rufus-scheduler-2.0.1/lib/rufus/sc/scheduler.rb:348:in `new'
  48. /home/web/project/vendor/gems/rufus-scheduler-2.0.1/lib/rufus/sc/scheduler.rb:348:in `start'
  49. (eval):121
  50. (eval):25:in `start'
  51. (eval):100
  52. ./script/runner:3:in `eval'
  53. /home/web/project/vendor/rails/railties/lib/commands/runner.rb:45
  54. ./script/runner:3:in `require'
  55. ./script/runner:3
  56. Exception in UtilityHelper.self.safe_send_email FATAL   C53300  Msorry, too many clients already        Fproc.c L276    RInitProcess
  57. Exception in UtilityHelper.self.safe_send_email FATAL   C53300  Msorry, too many clients already        Fproc.c L276    RInitProcess
  58. Exception in UtilityHelper.self.safe_send_email FATAL   C53300  Msorry, too many clients already        Fproc.c L276    RInitProcess

Raise The Connection Limit In PostgreSQL:

The exceptions above were thrown from a daily report that is run at night in what should be a low traffic time. To get past the above error and work towards finding final resolution I initially raised the maximum allowed conenctions by PostgreSQL from the default of 100 to 1000. To do this modify the postgresql.conf file typically located in the /var/lib/pgsql/data directory. The new line would look similar to the below.

Modify PostgreSQL max_connections Variable:

code

  1. max_connections = 100                   # (change requires restart)

As noted in the configuration file you are required to restart PostgreSQL ater modifying max_connections so issue the below command to restart postgresql from a Linux shell. The below example is from CentOS Linux however should be similar on most distrobution.

Restart PostgreSQL On CentOS Linux:

bash

  1. [root@dev ~]# /etc/init.d/postgresql restart
  2. Stopping postgresql service: [  OK  ]
  3. Starting postgresql service: [  OK  ]

Once restarted I set the report causing the issues to run and sure enough it ran without issue this time. While the report was running I watched the PostgreSQL logs by using the below command.

Analyze PostgreSQL Logs In Real Time:

bash

  1. [root@dev ~]# tail -f /var/lib/pgsql/data/pg_log/postgresql-Tue.log
  2. 2010-02-02 12:09:31.908 CSTLOG:  statement: INSERT INTO "mgmt_queries" ("cycle_num", "device_id", "poll_rate", "timestamp_server", "mgmt_cmd_id", "timestamp_device") VALUES(2, 221, 30, '2010-02-02 18:09:31.907087', NULL, '2010-02-02 18:11:45.000000')
  3. 2010-02-02 12:09:31.909 CSTLOG:  duration: 0.528 ms
  4. 2010-02-02 12:09:31.909 CSTLOG:  statement: SELECT currval('mgmt_queries_id_seq')
  5. 2010-02-02 12:09:31.909 CSTLOG:  duration: 0.079 ms
  6. 2010-02-02 12:09:31.909 CSTLOG:  statement: COMMIT
  7. 2010-02-02 12:09:31.910 CSTLOG:  duration: 0.424 ms
  8. 2010-02-02 12:09:31.910 CSTLOG:  statement: SELECT * FROM "device_latest_queries"     WHERE ("device_latest_queries"."id" = 221)
  9. 2010-02-02 12:09:31.910 CSTLOG:  duration: 0.207 ms
  10. 2010-02-02 12:09:31.911 CSTLOG:  statement: SELECT * FROM "mgmt_cmds"     WHERE (pending_on_ack = true and originator = 'server' and pending = true and device_id = 221)  ORDER BY timestamp_initiated DESC LIMIT 1
  11. 2010-02-02 12:09:31.911 CSTLOG:  duration: 0.365 ms

Please note that your log files are more than likely named something different and will depend on what you have configured in postgresql.conf. The main thing I would suggest you look out for in this scenario is the “duration” lines which describe, in milliseconds, how long a query takes to run. In doing this I noticed several queries from the report code that were taking seconds to run which should never be the case if possible. Once we optimized these queries and I set max_connections back to the default of 100 things started working properly again.

PostgreSQL 8.4 Official Documentation – Volume I. The SQL Language (Paperback)


List Price: $34.99 USD
New From: $3.24 USD In Stock
Used from: $3.25 USD In Stock

PostgreSQL 8.4 Official Documentation – Volume II. Server Administration (Paperback)


List Price: $36.99 USD
New From: $36.95 USD In Stock
Used from: $10.48 USD In Stock

Share