August 14, 2015

Papertrail vs. Logentries - One great tool beats a bag of OK tools

Recently I've been looking into a centralized logging solution for us so that we can lock down the servers but still see what is going on. Our analytics needs are not huge - we have a low number of valuable customers so we don't need to analyze behavior, we can ask them. But we do need to be able to solve problems for them immediately.

That means live tail and search of current and recent logs is a necessary feature, alerts are important, and graphs could be nice to have. And we want to hook right to the unix standard logging facilities rather than have another piece of code to wonder about.

While there are a lot of ways to run a centralized log yourself, now we have to monitor and maintain that server as well, and we're not log experts. It would be better to throw a reasonable amount of money at the problem. Looking around most hosted log solutions are heavy and analytics focused without live tailing. Only two really have that as a key feature: Logentries and Papertrail.

After initial reading and questioning evaluation, Logentries was the clear winner. It has live tail, it has ways to highlight and tag them, search them, graphically interact with them, it has dashboards that hook to these things and a bunch of integrations. Papertrail has a live tail you can search, and it can trigger alerts off the searches, and that's it. Papertrail also gives you about half the log volume and log retention per dollar that Logentries does.

Once the rubber meets the road though Logentries falls down. Live tail, as I was evaluating, lost some of the messages. Not many, and they were in the non-live log, but they were not in the live tail. Interacting with the UI it worked, but it could be fiddly, the sort of "if you click A and then B it gets a little confused" thing that shows up in so many JS heavy websites. The support staff was not able to diagnose that my initial connection problem might be because selinux was preventing the connection. The docs are sometimes a bit out of date (though generally pretty good). And the killer is that the live tail issues were known by support, but not in the incident log, so how many times has this sort of thing happened recently? Who can say.

Papertrail didn't grow any more features, but it performed flawlessly. Live tail chugged along, completely searchable. All of the UI features work smoothly and intuitively and have help links embedded right there. When I had the selinux problem I had solved it before they were awake but they knew all about how that happens and sent me the commands to fix it, which are also in their docs. Papertrail status includes (not many) slowdowns and multiple updates and goes back for years.

Papertrail is only one tool but it's one sharp well kept tool. Logentries is trying to do a lot of things, but they're not sharp. We're going to pay more, and get fewer things, but I think we're going to get more in the end.

July 30, 2015

Everywhere I've Worked that Worked was agile, mostly unrelated to Agile

Agile is at this point a tired buzzword. Too often it means "we run around a lot and generate Agile artifacts at a terrific rate" without actually producing useful product.

Thinking back I realize that everywhere I've ever worked that actually shipped things was using an agile process in the small a sense - cycles of customer feedback, planning, and work. Back before Agile, when BBN was trying to sell it to a government suspicious of any lifecycle other than plan/do/realize it all sucks and throw it out/repeat we called it spiral, where you spiraled out through successive cycles.

The next company didn't have the feedback cycle and just did stuff. That was fast, but it failed. After that I went to another which had the planning cycle, but no feedback. Work got done but it was crushed by the market. Forum Systems had both customer input and a planning cycle but again no Agile, just all the useful pieces, and that product worked (though the company had a rough ride for other reasons).

After that my next company had Agile with a capital A, but could not prioritize worth a damn or believe schedule estimates they didn't like. That failed (big surprise). Then Tripadvisor was a super agile cycle with weekly releases, prioritization, and planning, with great feedback via analytics, and that shipped an amazing amount of product updates that really worked. Again, TA has been working like that since before Agile got its A.

The current company is the first I've been at that uses actual Agile methods in an actually agile way. I suspect that compared to industry the ratio of functional to nonfunctional processes in companies I've been at is fairly high, and the ratio of capital A Agile to actually agile is fairly low.

December 19, 2014

Interesting paper on the system issues in Machine Learning

Not just a snappy title. Glue code, pipeline jungles, crazy problems familiar to any big data processing system and unique to ones where you're stuffing the data into a model.

Machine Learning: The High Interest Credit Card of Technical Debt

October 4, 2014

Optional Syntax

Optional syntax is a popular feature in recent languages (or really recently popular languages, some of them have been percolating for a long time). Some languages also make parenthesis and commas optional for invocation allowing for "function arg arg arg" and sometimes even "object method arg arg arg" syntax. Optional end of line marks show up couple of places. So far all of the cases where I have seen optional constructs like this include at least one gotcha case where omitting or including punctuation are both meaningful but different.

I'm not a big fan. Optional syntax introduces a choice which allows me to avoid a cost. But the cost is tiny - typing parentheses and other delimiters takes very little time, and reading a language which uses delimiters is not hard. And the cost of making and in that one gotcha case mis-making a choice is notable. Programming productivity is limited by how many good choices you can make in a day, not by time or lines of code. I would prefer not to spend my choices on punctuation.

The end result, familiar to any C++ shop, is that you have to decide on what parts of the language you are going to use in your company so that you can stop making pointless choices and keep your code uniform and understandable. Only in this case it's not libraries or template tricks, but commas.

The argument I usually see for this sort of thing is that it allows the programmer to be expressive, which is a comparison of coding with painting, music or creative writing which is more harmful than useful. You've probably seen some code written in a James Joyce style and do not consider it a masterpiece like Ulysses.

There are a few recently languages that have gone the other way. Python (not that recent any more I suppose) goes somewhat down the line of syntax more rigid than the standard with its significant whitespace, and Go goes even farther which even formatting being part of the language. After initial suspicion about that I very much enjoyed Python and I suspect Go might be similarly satisfying although I haven't yet had the chance to work with it.

July 11, 2014

From the Department of Marketing

Finally an explanation of why the term "dynamic programming" doesn't seem to make any sense: ...the origin of the notably inapt term dynamic programming for a highly useful method of memoization...