It's nice to be important, but it's more important to be nice.
The more monitoring you have, the more trouble you find. If you don’t have monitoring, trouble finds you.
Splunk. Has worked very well for us.
Pricey, but when you don't have time to *design* something, you just throw a full-text search engine at your app logs.
I think we're starting to talk about a more open alternative?
I'll ask today. Remind me if I don't.
theres something like logstash, which I again instantly forgot about since its also done in some bizare language running in the java interperter.
logstash is one of the alternatives we were talking about, yes. I doubt we did a full evaluation yet.
One of the most powerful things we do with Splunk is the "transaction" filter. I don't see any direct replacement with logstash... this is statically configured and seems to have limitations, but it's somewhere in the ballpark: http://logstash.net/docs/1.4.2/filters/multiline
not quite close enough, transaction is an ad-hoc query: http://docs.splunk.com/Documentation/Splunk/6.2.1/SearchReference/Transaction
probably one would want something like that:
this is the logstash alternative:
(the credativ guys work with it)
Another tool /me wouldn't use... but may be interesting ;-)
I like very much this one:
(have a look at the crazy videos ;-)
It uses a pimped collectd as some of the data sources.
OK, for one thing, I hadn't correctly understood how logstash fits together with the rest of its ecosystem. Logstash is like splunkforwarder, I guess- it's a piece of low-level plumbing.
The querying all happens in elasticsearch: http://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-queries.html
Not similar at all, but I do use this for viewing / checking combined logs from systems:
Epylog - https://fedorahosted.org/epylog/
It lets you boil down combined syslogs from multiple systems, and get rolled up reports on logins, and a free-for-all report of anything that was not parsed in an email.
It takes quite a bit of time to get what they call the "weeder" to build up to rid yourself of the background noise from the reports, and it does take ocasional changes to regexes on lines to account for some daemons changed log output for warnings, etc, but I find it worthwhile. Once you have your "don't care" lines out of the report, you will be left with the ones to either investigate and act on or just add to the don't care if they turn out to be a more of just miss-placed info output.
You can set up your own roll up reports, but I have not played with that as of yet.
Not the splunk way - there are no "don't care" records, you throw everything in the big index and figure out how to query it later.
Been talking to some former coworkers (now at NYTimes, ghod help them) and one of the Splunk alternatives they are looking at is Sumologic:
(^^ one of my litmus tests for a Splunk replacement)
When you go to this site, you may be treated to the following popup:
This page ws unable to display a Google Maps element. The provided Google API key is invalid or this site is not authorized to use it. Error Code:
So, they hope to supplant Splunk, but...?
Evidently their Big Data got so big it popped out of the screen.
(By the way, we ended up just buying a bigger Splunk license.)
Wow, this is funny and pathetic at the same time. I found this on the web site of a fairly popular piece of software today:
$i_understand_file_permissions_and_how_to_fix_them = false;
$i_understand_file_permissions_and_how_to_fix_them = true;
Now you have the button, the following explains the issues in detail."
Hmmm... someone has obviously been torn between wanting to help people, and not wanting to deal with the same stupid problem repeatedly.
LOL. The real problem is using anything written in PHP ;)
Yeah, I'm trying to migrate us to rails.