That means live tail and search of current and recent logs is a necessary feature, alerts are important, and graphs could be nice to have. And we want to hook right to the unix standard logging facilities rather than have another piece of code to wonder about.
While there are a lot of ways to run a centralized log yourself, now we have to monitor and maintain that server as well, and we're not log experts. It would be better to throw a reasonable amount of money at the problem. Looking around most hosted log solutions are heavy and analytics focused without live tailing. Only two really have that as a key feature: Logentries and Papertrail.
After initial reading and questioning evaluation, Logentries was the clear winner. It has live tail, it has ways to highlight and tag them, search them, graphically interact with them, it has dashboards that hook to these things and a bunch of integrations. Papertrail has a live tail you can search, and it can trigger alerts off the searches, and that's it. Papertrail also gives you about half the log volume and log retention per dollar that Logentries does.
Once the rubber meets the road though Logentries falls down. Live tail, as I was evaluating, lost some of the messages. Not many, and they were in the non-live log, but they were not in the live tail. Interacting with the UI it worked, but it could be fiddly, the sort of "if you click A and then B it gets a little confused" thing that shows up in so many JS heavy websites. The support staff was not able to diagnose that my initial connection problem might be because selinux was preventing the connection. The docs are sometimes a bit out of date (though generally pretty good). And the killer is that the live tail issues were known by support, but not in the incident log, so how many times has this sort of thing happened recently? Who can say.
Papertrail didn't grow any more features, but it performed flawlessly. Live tail chugged along, completely searchable. All of the UI features work smoothly and intuitively and have help links embedded right there. When I had the selinux problem I had solved it before they were awake but they knew all about how that happens and sent me the commands to fix it, which are also in their docs. Papertrail status includes (not many) slowdowns and multiple updates and goes back for years.
Papertrail is only one tool but it's one sharp well kept tool. Logentries is trying to do a lot of things, but they're not sharp. We're going to pay more, and get fewer things, but I think we're going to get more in the end.