Processing, Analyzing & Visualizing

Troubleshooting

Log Files

/data/elsa/log/node.log

This is the main ELSA log on each log node. It will contain any errors or information regarding the recording and indexing of logs. If no new logs are coming in, this is the first log file to check.

/var/log/apache2/error.log

This log can be named differently or be in /var/log/httpd. It is the standard Apache log file which will be the first place to check if any “Query Failed” error messages appear on the web interface. Errors only show up here if they are major enough to break the underlying ELSA code. Typically, these kinds of errors are connectivity or permissions related.

/data/elsa/log/web.log

This is the main ELSA log for the web interface. It has information on any ELSA-specific actions initiated from the web interface. If queries are not returning the results you expect, check this log.

/data/elsa/log/syslog-ng.log

Syslog-NG’s internal log file will give low-level debugging info like raw message rates. It should generally only be needed when you’re not sure that a node is receiving logs.

/data/elsa/log/query.log

This file contains the query log generated by the Sphinx searchd daemon. It should not normally be needed, but can be a good place to get a feel for what queries the system is running and how long they are taking.

/data/elsa/log/searchd.log

This is the Sphinx searchd daemon log and will contain info on index rotation.

Common Troubleshooting Symptoms

Symptom Resolution
Chronic warnings in the web UI for “couldn’t connect to MySQL” This can be caused if a web frontend has issues and the MySQL server decides it no longer wishes to speak to the server because of too many dropped connections. To fix it, you need to log into the node referred to in the message and issue:mysqladmin flush-hosts which will cause the MySQL daemon to once again accept connections from the “flaky” frontend.
“Query Failed” red error message This is a low-level error indicating that there is a connectivity or permissions problem between the web server and the MySQL/Sphinx daemons on the node. It will also show up in the node.log as “No nodes available.” You can verify database connectivity by manually running: mysql -h<node IP> -uelsa -p syslog and mysql -h<node IP> -P9306. If both work, then the problem may be something more specific with either MySQL or Sphinx. To troubleshoot that, run tcpdump -i eth0 -n -s0 -X “port 3306 or port 9306″ and watch the traffic to see what’s occurring when you run a query.