Hi, I have a ruby script inside a Rails project. The script includes environment.rb so that Rails gets loaded etc. It then listens to data on a TCP socket of another server, parsing it to keep a mysql database up to date.
I'm using the daemons gem to be able to start and stop the process easily. Iv had the script running in production with no problems for over a day but now it has gone stale. The Daemons control file says the process is still running, the process is showing up when I do a ps -ef and monit also thinks the process is running fine. However the database is not getting updated (if I run the script again in another process the database updated properly).
I have tried using GDB to debug what the issue with the process is by running sudo gdb /usr/local/bin/ruby 25395, the output of this is
GNU gdb 6.8-debian
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu"...
/usr/local/bin/ruby: No such file or directory.
Attaching to process 25395
Reading symbols from /usr/bin/ruby1.8...(no debugging symbols found)...done.
Reading symbols from /usr/lib/libruby1.8.so.1.8...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libruby1.8.so.1.8
Reading symbols from /lib/libpthread.so.0...
(no debugging symbols found)...done.
[Thread debugging using libthread_db enabled]
[New Thread 0x2b7decb08d30 (LWP 25395)]
Loaded symbols for /lib/libpthread.so.0
Reading symbols from /lib/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libdl.so.2
Reading symbols from /lib/libcrypt.so.1...
(no debugging symbols found)...done.
Loaded symbols for /lib/libcrypt.so.1
Reading symbols from /lib/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/libm.so.6
Reading symbols from /lib/libc.so.6...
(no debugging symbols found)...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /usr/lib/ruby/1.8/x86_64-linux/thread.so...
(no debugging symbols found)...done.
Loaded symbols for /usr/lib/ruby/1.8/x86_64-linux/thread.so
Reading symbols from /usr/lib/ruby/1.8/x86_64-linux/etc.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/ruby/1.8/x86_64-linux/etc.so
Reading symbols from /usr/lib/ruby/1.8/x86_64-linux/stringio.so...
(no debugging symbols found)...done.
Loaded symbols for /usr/lib/ruby/1.8/x86_64-linux/stringio.so
Reading symbols from /usr/lib/ruby/1.8/x86_64-linux/syck.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/ruby/1.8/x86_64-linux/syck.so
Reading symbols from /lib/libnss_compat.so.2...
(no debugging symbols found)...done.
Loaded symbols for /lib/libnss_compat.so.2
Reading symbols from /lib/libnsl.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libnsl.so.1
Reading symbols from /lib/libnss_nis.so.2...
(no debugging symbols found)...done.
Loaded symbols for /lib/libnss_nis.so.2
Reading symbols from /lib/libnss_files.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libnss_files.so.2
Reading symbols from /usr/lib/ruby/1.8/x86_64-linux/socket.so...
(no debugging symbols found)...done.
Loaded symbols for /usr/lib/ruby/1.8/x86_64-linux/socket.so
Reading symbols from /usr/lib/ruby/1.8/x86_64-linux/nkf.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/ruby/1.8/x86_64-linux/nkf.so
Reading symbols from /usr/lib/ruby/1.8/x86_64-linux/bigdecimal.so...
(no debugging symbols found)...done.
Loaded symbols for /usr/lib/ruby/1.8/x86_64-linux/bigdecimal.so
Reading symbols from /usr/lib/ruby/1.8/x86_64-linux/iconv.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/ruby/1.8/x86_64-linux/iconv.so
Reading symbols from /usr/lib/ruby/1.8/x86_64-linux/strscan.so...
(no debugging symbols found)...done.
Loaded symbols for /usr/lib/ruby/1.8/x86_64-linux/strscan.so
Reading symbols from /usr/lib/ruby/gems/1.8/gems/hpricot-0.8.1/lib/fast_xs.so...done.
Loaded symbols for /usr/lib/ruby/gems/1.8/gems/hpricot-0.8.1/lib/fast_xs.so
Reading symbols from /usr/lib/ruby/gems/1.8/gems/json-1.1.9/ext/json/ext/parser.so...done.
Loaded symbols for /usr/lib/ruby/gems/1.8/gems/json-1.1.9/ext/json/ext/parser.so
Reading symbols from /usr/lib/ruby/gems/1.8/gems/json-1.1.9/ext/json/ext/generator.so...done.
Loaded symbols for /usr/lib/ruby/gems/1.8/gems/json-1.1.9/ext/json/ext/generator.so
Reading symbols from /usr/lib/ruby/1.8/x86_64-linux/racc/cparse.so...done.
Loaded symbols for /usr/lib/ruby/1.8/x86_64-linux/racc/cparse.so
Reading symbols from /usr/lib/ruby/gems/1.8/gems/tmail-1.2.3.1/ext/tmailscanner/tmail/tmailscanner.so...done.
Loaded symbols for /usr/lib/ruby/gems/1.8/gems/tmail-1.2.3.1/ext/tmailscanner/tmail/tmailscanner.so
Reading symbols from /usr/lib/ruby/1.8/x86_64-linux/openssl.so...done.
Loaded symbols for /usr/lib/ruby/1.8/x86_64-linux/openssl.so
Reading symbols from /lib/libssl.so.0.9.8...done.
Loaded symbols for /lib/libssl.so.0.9.8
Reading symbols from /lib/libcrypto.so.0.9.8...done.
Loaded symbols for /lib/libcrypto.so.0.9.8
Reading symbols from /lib/libz.so.1...done.
Loaded symbols for /lib/libz.so.1
Reading symbols from /usr/lib/ruby/1.8/x86_64-linux/digest.so...done.
Loaded symbols for /usr/lib/ruby/1.8/x86_64-linux/digest.so
Reading symbols from /usr/lib/ruby/1.8/x86_64-linux/fcntl.so...done.
Loaded symbols for /usr/lib/ruby/1.8/x86_64-linux/fcntl.so
Reading symbols from /usr/lib/ruby/gems/1.8/gems/mysql-2.7/lib/mysql.so...done.
Loaded symbols for /usr/lib/ruby/gems/1.8/gems/mysql-2.7/lib/mysql.so
Reading symbols from /usr/lib/libmysqlclient.so.15...done.
Loaded symbols for /usr/lib/libmysqlclient.so.15
0x00002b7dec86df10 in read ()
from /lib/libc.so.6
This would suggest that that the issue is with the mysql connection. It is possible that there may not have been any data for a few hours. I believe that this could cause the process to be disconnected from Mysql? But if this did happen, I would have though that when the script did start getting data again, an exception would have been thrown which would have stopped the script from running, prompting monit to restart it.