Everything Not Nailed Down

The problem and the solution is people

2011-05-05T17:30:00.003-04:00

This page starts out in a humorous vein but the more you read the more you nod.

Characterizing people as non-linear, first-order components in software development

"The fundamental characteristics of “people” have a first-order effect on software development, not a lower-order effect. Consequently, understanding this first-order effect should become a first-order research agenda item, and not neglected as a second-order item. I suggest that this field of study become a primary area in the field “software engineering” for the next 20-50 years."

Data Stampede!

2011-04-05T20:37:00.003-04:00

During your CS education, and even working on small problems in your real working life, you get the idea that data is like a school of fish. All the same, all going in the same direction, shimmering and moving as one. Every now and then one fish will go in the wrong direction because it's a bad fish, or a piece of seaweed. Anyway it's not a real valuable fish so you just drop it.

Real world data isn't like that. It's more like a cattle drive. It's wandering all over, trying to get away, breaking its leg, suddenly all showing up at the same time and trying to trample you, spooking at loud noises. It smells bad, some of it isn't going to fetch much when you get it to Chicago, it makes weird noises all the time, and frankly it's dumb as a sack of cheese.

But it's all data. Unlike the school of fish where the ringers are thrown in and can be disregarded, everything in the herd is pretty much a cow. You want to get them all to market. Only with great regret will you leave one behind... so you have to do the work of rounding up stragglers and checking brands.

Why the shower is the most billable time of the day

2011-02-08T11:34:00.004-05:00

Not that I've worked anywhere with a concept of billable vs. non-billable time in a long while, thank goodness.

It is a pretty common experience for the morning shower to be the most productive time of the day. The general explanation I've heard is that the subconscious has been working all night on your problems, but the other morning in the shower I became convinced that that is only a small part of the reason if it's any part at all.

I think the main value of the shower is the limited number of things to think about. By the time you've got a job you've had so many showers that the actual thing you're there for is completely automatic and requires little intervention from the mind. At the same time there are no distractions - you can't read, answer your phone, write an email, and so on. If you're like me you can't even see that well because your glasses are somewhere else, and of course you can't hear the rest of the house over the shower.

I submit that it is because of the lack of distractions that the shower is so productive. While in theory you can think at your desk the email program is always a click away and your coworkers might come by any moment and even if you are not actively thinking about them your mind knows it might think about them. All that is left to do is think, and whatever you think about will get your full attention, which almost never happens.

By this reasoning you should be able to get shower levels of productivity at work when the power goes out and all your coworkers are at lunch. Or you can become very good at actually giving things your real full attention for extended periods of time, but that turns out to be very hard. Probably easier to cut the power when your coworkers are out.

On airlines and customer service

2011-01-21T15:49:00.003-05:00

If you were an airline and you got a booking from a traveler booking on a couple weeks notice, buying a fairly expensive ticket due to short notice, would you:

a) try to be nice to a possible valued customer or

b) ask for a $30 fee for getting a seat assignment before check-in?

presumably British airways chooses "b" because they're worried that with such a short trip I won't pay them a baggage fee? Or maybe there's also a carry-on fee when I get to the airport.

By next year cabins will be unpressurized and there will be a mask fee. Think of the revenue stream!

slides on django's warts

2010-09-08T18:06:00.004-04:00

It's been a while now since django, and ours was very non-standard, but even so we ran into a bunch of the problems in this set of slides:

why django sucks and how we can fix it

UUIDs, apps that exist but you can't use them, needing to extend the auth system... all very familiar. Hopefully these suggestions will be taken to heart.

Increased Process Failure Speed

2010-09-07T12:42:00.005-04:00

Agile and associated processes (Scrum, TDD, etc) seem to have gone from small p good ideas about process to capital P you must follow this Processes more quickly than previous good ideas turned bad implementations. Is this because of the bad-idea-acceleration of the internet, or solar rays, or is the whole impression that it happened faster just a misguided impression on my part?

Either way, actual good development seems to look about the same as it always has: prioritize, iterate with feedback, don't lose sight of the goal.

Life as the third level compiler

2010-08-03T17:02:00.003-04:00

I was in a meeting today thinking about my place in the hierarchy of compilers.

Below me there is the Java compiler, or sometimes the python interpreter. They are seriously cranky to work with, you have to talk to them in a very exact way and they are pedants like you would not believe.

Above me is the project requirements compiler. Really in our environment she's an interpreter, which continuously turns the general ideas of things we want to do into more narrowly specified requirements. Above her are the business interpreters that turn where we want to go into projects we want to do.

I'd prefer my interface to be more like the layers above me - flexible and easy to debug - than like the layer below me which is brittle, picky, and humorless. There's a constant tendency to become like the things you work with to fight against.

Especially in pedantry - is there a group more pedantic than programmers?

From the department of I wish this were true

2010-07-29T15:37:00.001-04:00

Steve Yegge: Wikileaks To Leak 5000 Open Source Java Projects With All That Private/Final Bullshit Removed

Language Drift

2010-05-28T11:17:00.002-04:00

"Hacker" moved from its original meaning of someone who made cool technical stuff to mean someone who broke into systems, at least in the general language. Then programmers took it back to mean someone who made cool programs again (though not really physical stuff without a qualifier, like "hardware hacker"). Now it seems to be being taken to mean someone who does intense technical work for the purpose of making a boatload of money. (for example Y Combinator's site is called "Hacker News" meaning "News for Technically Minded Entrepreneurs").

The Software Lathe

2010-05-11T14:59:00.002-04:00

When creating a piece of software I often use a process that is a lot like running a lathe to create the fancy post on the end of a banister. I start out with the gross outlines of what I want, a bunch of empty functions and TODO notes about what goes in them. This is like starting out with a piece of wood and making some pencil marks on it. Then I start running the lathe and carving.

Initially I don't get any piece right, I just go for broad shapes. Keeping moving is what's important here - if you don't understand it all immediately TODO it and move on. At some point I start being able to run the code and that's when the lathe really starts spinning. At the beginning I carve off a TODO and replace it with some code and two more, but eventually I get down to the finished details first on one part of the code and then on the whole thing. Of course the lathe metaphor breaks down in that sometimes a section is finished and then refactored into something totally different, but hey, this metaphor has some TODO in it.

Today my project of the past couple of weeks got has really started spinning at speed, after a long time of pencil marks and slow carving.

From the department of rotary dials on cellphones

2010-05-03T16:43:00.003-04:00

Someone has created Spring in python. Complete with its XML (or yml) configuration files, decorators and so on.

I'm thinking that a reimplementation of the reason so many people got disgusted with Java is not going to catch on in one of the languages they fled to, but maybe I'm too idealistic.

http://springpython.webfactional.com/node/6

grep "clue" -r .

2010-04-22T13:01:00.002-04:00

I understand that there are programmers who work without grep.

How is that even possible? I can no more conceive of that than cooking without fire.

Fancy Tricks

2010-04-21T17:40:00.004-04:00

Along with being a programmer I study a couple of sword martial arts, with the one I'm most experienced in being German Longsword of the 14th-16th century. Anyway, in longsword there are a lot of complicated things you can do, like for example this... and things even more complex.

They'd be very very hard to pull off in an actual fight. But I read a thought recently on martial arts that suggested the reason you learn the complicated things and practice them is not because you expect to do them ever, it's because in doing that you continue to hone the principles that make the simple things work. Most of the time you do those simple things, but they'll be better because you've practiced the weird stuff and reinforced the principles that underlie everything.

I liked that idea. Programming, I think, is the same way. 99% of the time you shove things in a list or a hash or you use an object directly through it's only interface or whatever. But having done some of the weird stuff not only means you've done that, it also builds up your understanding of the underlying principles so that you're better at the simple stuff.

Nonlanguage Languages

2010-04-03T16:52:00.001-04:00

There are a bunch of programming annoyances which I've slowly come to realize are a single programming annoyance. Things like:

- make files, ant files, foo build system files

- web templating systems

- XML configured frameworks

Underneath the covers they are all people writing a programming language, without knowing how to do that or even understanding that that is what they're doing. Putting together your process in the resulting pidgin is always painful as the task grows past nontrivial.

These days some people are writing these things as extensions of real languages rather than as cobbled piles of data - my impression of ruby is that it is crackerjack tool for writing domain languages - but makefiles will undoubtedly remain with us for nigh unto forever, and XML frameworks will probably have to be staked and beheaded a hundred times before they finally croak.

Possibly this is part of our punishment after we ate the fruit of the knowledge of deadlines and markets and were ejected from the Garden of Lisp.

DVCS - I can has branching?

2010-01-12T15:10:00.004-05:00

Having used git for a very short amount of time at my brief job-that-died and then gone back to SVN at TA I already want some of the features back. SVN, specifically SVN's gruesome branching and merging, is just not enough for modern development.

Mind you, I don't give a flying whatsis about the "D" in "DVCS" - I don't really need to have multiple distributed sources of truth. What I need is to be able to freely branch and merge from branch to branch. Where by "be able to" I mean "not recoil in pain looking at the steps to do these things".

The DVCSs seem to have branching as fundamental as compared to the older VCS generation where branching is something that is supported because sometimes you have to do it.

In modern development, sometimes is all the time.

I suspect I would probably prefer another DVCS to git - git has that special unix-culture syntax that I hate so much - but I'd take git if it were offered to me and never look back.

Throw me the Idol, I'll Throw you the Whip

2010-01-04T14:59:00.003-05:00

So at my current job, I'm working on a quite profitable big website that has been written in Java with javascript and velocity templates and css and html continuously for the last seven years. Given that I'm relatively pleased with the state of the codebase. It's not totally nuts, and while I don't agree with the code formatting conventions they're mostly followed.

However it's huge and has a large number of interlocking parts and it's making money all the time so don't break it. That said, there is nothing really preventing me from breaking it beyond some automated tests and my desire not to. And the corollary to it's making money all the time is that's because we're making it better all the time, so there's always a change to go in.

The whole thing is a lot like archaeology - not real archaeology, but Indiana Jones archaeology where you have to figure out how to get the idol out of the deadly trap or perhaps ancient weather control machine or maybe it's an alien time capsule, without setting off the thing and causing the end of the movie.

The Worst Interface Sin

2009-12-31T13:20:00.002-05:00

One of the small bugs I squashed last week was one rogue link to vacation rentals on the Spanish site. We don't have vacation rentals in Spanish yet so the link would bounce and go to hotels. Faking out the user is terrible user experience.

But that's a very small version of how the grocery store was screwing with me today. I weakened in my anti automatic checkout stance and tried it again. First I put my own bags in the bagging area. It asked as I started "are you using your own bags?" so I pressed yes and it beeped. Then nothing happened. Basically the are you using your own bags button beeps and then does nothing until you take your own bags out. Awesome.

I put them on the floor eventually and started stacking things in the bagging area to transfer them when I finished. I got to the parsley, and rather than looking up the number on the tiny twist inside the bag, I looked up parsley through the interface, which was easy. Unfortunately, on pressing "flat parsley" it responded that it didn't know what I meant. What was the button for then? And extra exciting, this tripped the "unknown item" flag which meant that after rooting through the bag and looking up the number I was in need manager mode and could not finish and pay.

Nice interface design! I'll be sure to use a real human in the future.

Shackld to the UNIX Past

2009-12-22T15:13:00.002-05:00

Just because way back in the mists of time "password" was too long to be put into the ancestral operating systems and languages... is no reason to have functions in your modern language named "prompt_user_passwd" or anything else with "passwd". Embrace language, idts!

Scripting Productivity

2009-11-24T15:15:00.003-05:00

Now that I'm back in a Big Build environment which takes a while to rebuild and bounce a test server to test your change, I look back on my time using python/django and ask:

Is the productivity gain commonly reported in scripting languages because the language is lighter weight, or is it just because your feedback loop is supertight so you can maintain a laser like focus on the problem at hand?

Maybe if you count the time writing blog posts while rebuilding as productive you're just as productive in a compiled language.

Would you like some 500 CHEESE ERROR with your 200 WHINE OK?

2009-11-23T16:43:00.003-05:00

As is typical of environments everywhere, new work has shiny new laptops with snow leopard and thus python 2.6 on desks, but the dev servers have an old and stable system with only 2.4.3.

As I was complaining about python 2.4's HTTP protocol support is not the most sparkling. But amazingly it doesn't even provide a way to extract an error code from the response. It's too busy pretending the response is a file to add an accessor for the first piece of data you get.

Instead you have to write a special opener which overrides default error handling on non-200 codes to do something useful with the error code so you can find it later.

Thankfully later pythons realized that when you represent a thing with know fields you should provide access to the fields.

I mean, who makes that sort of design? Someone who has never actually made an HTTP request that was malformed - someone who hasn't actually used HTTP to do anything.

Why the Title?

2009-09-15T20:18:00.000-04:00

Why "Everything Not Nailed Down"?

Because when you get into a new language or framework or programming tool,
you take everything not nailed down...
and then pry up the rest.

You do this in service of some requirements
which are never nailed down themselves
and then management pries them up.

Prybar Accessibility

2009-09-08T10:16:00.001-04:00

Writing about the timeout problem and how I'd encountered it in python and in java.

The difference is that in python I solved it in a day of moderately messy hacking, reusing most of the python library internals. In java we looked over the problem and determined we'd have to basically redo HTTP from sockets to fix it, and gave up.

Python was accessible to my prybar. When one class didn't work I could subclass another and then munge its internals. Java made everything private, and final, and private final double-do-not-touch. The only way to change functionality on the HTTPConnection was to replace the class, but that was not possible because it was in the protected java libraries that java will complain about if you swap in your own class file. And of course there were probably legal reasons that wasn't possible either.

When I was just learning about object orientation and making things private and so on I thought it was the bee's knees: now no one could go into my beautiful code and mess it up. I could create a perfect little sealed black box of functionality.

Now I'm more experienced and I've worked with a lot of code and written a lot of code and look back at my previous self as hopelessly naive.

First, most (for values of most close to all) of the code I've worked with has needed fixing in ways the original author did not expect. Sometimes the original author wasn't even qualified to expect anything, since they were writing code for a spec or some other abstract reason that didn't provide any real-world pressures.

Second, my own code is no different.

Finally, why do I care about defending my code? I should provide a good API that tells people how I expect it to be used, but if they stray out of that it's not my problem. It's even good, they might then understand the code enough to help me with it.

Django Model Trick: Versioning

2009-09-02T00:01:00.001-04:00

In our configuration manager we wanted to have configuration objects with complete version histories, so that the user could go back into past versions and see who had put what in what state. Basic version control stuff. But we also wanted to not have a giant extra set of versioning tables or a situation where updating object A meant having to change objects B-Z to have the new reference to the new version of A.

So I developed the following scheme:

Objects had their own primary key (a UUID) and also a key representing their lineage. When a new model object was created, they were both set to the same UUID - that indicated the "current working version".

To update an object, I took two steps:

save the before state as a new object, with the same lineage UUID and a new primary key UUID (so a new object as far as django knew)
update fields in the working version object, leaving its keys unchanged

This way every other object with a foreign key to the working version was still pointing at the current working version, but the old versions could sit around in the same table, easily findable with a simple query for the lineage UUID.

We had one more operation, which was to create a coherent, stable version - a bit like tagging with an actual copy of the objects, or branching (but we did not allow changes on this branch). To do that took three steps:

create a special object that tracked this version with its own UUID key
make a copy of every object to be in this version as above - new primary key, same lineage key - and give it a foreign key to the version object as well
for all of those copies, change their foreign keys to current working objects to be keys to objects in this stable version

This is not much code because the key changes just require keeping a map and then iterating through all fields in all objects and asking which ones are foreign keys, then updating according to the map.

There was of course a lot of subsidiary code to handle things like revert to version, history retrieval, and so on, but I think those are pretty self-explanatory after the core versioning trick is revealed.

If I were to do it over again now I'd investigate projects like django reversion to see if they meet the need, but there was nothing available a couple of years ago that did what we wanted.

Long Interviews are Good

2009-08-31T17:52:00.003-04:00

I realized this afternoon after a long interview day that I like longer interviews.

Not that I love the process - it's not easy for anyone involved!

But I like to talk to companies who treat interviewing as a serious task that requires serious effort.

My previous company had a day-long interview process that included a technical presentation and some code review (let's look over this canned code together and tell me what you see in it) as well as board coding and problems and the standard tell me about your projects questions. That was a good interview day.

The most important element in any company is the people. Selecting those people is hard, but it's good to find places that try hard at it rather than throwing up their hands and saying "it's hard, so we'll just do a couple hours and decide something".

Why Django: Restrictions are Freeing

2009-08-29T18:23:00.002-04:00

So why did I choose django to implement our project in the first place? Though django was the best documented, there were other options available even two years ago - pylons, turbogears, and a number of even more lightweight frameworks. Both of those offered a more flexible ORM layer and are more set up to mess around with.

Django is very much a "here's the way we're going to do things" framework. It comes wired together and while you can substitute the pieces it can be a lot of work. This was exactly what we needed for our project because we were trying to build a configuration tool, not a web framework.

In the early stages of a project you have to make a lot of decisions. If you can get some of them out of the way by leaning on a good framework, that leaves a lot more decision power free to work on things that are more relevant to the project.

In time we had to break out of some of the django restrictions, and work with others. I wrote some custom SQL utilities for upgrade that worked outside the ORM, and disabled constraints. Todd wrote the job manager.

Working within a fairly restrictive framework allowed us to pick our battles rather than fight them all. It was important that it was possible to break out of django - being in python is a big help here, because it enabled us to use the internals when we needed to bend the externals a little. I've certainly seen frameworks that really don't have any way to break out of them beyond defined plugin architectures which are never quite what you want.

But if you do have the power to break the shell when you need to, a cozy little egg is a good place to grow.

TED on motivation

2009-08-27T15:31:00.002-04:00

One of the benefits of not being employed at the moment is having the time to catch up on a bunch of information sources I'd heard about but hadn't gotten around to. This is a talk at TED about motivational techniques which was interesting from a programming as work viewpoint. If you don't have time to watch it I'll give you a capsule summary in the comments.

XMLRPC API Design

2009-08-26T10:33:00.005-04:00

In between the central mothership and the agents we used XMLRPC to communicate. There's not much to say about XMLRPC - it's pretty well supported and much lighter weight than SOAP, while preserving the whole look of an API call for easy integration as opposed to say a REST API.

The interesting thing about our XMLRPC wasn't so much what it did, which was pushing data and causing state changes, but the support needed because this was a real world networking API. Anyone could be down or unreachable at any point. Which meant:

Calls had to have a confirmation
Failed confirmation lead to a retry cycle of configurable but limited length
Calls had to be idempotent in case only the confirmation was lost
There had to be status calls to check what actually happened
The whole interaction state machine needed to have states for partial and failed communication
The user had to be shown what had gone wrong, and offered ways to fix it

This made the automated code running these communications the most complicated code in the application, with the most complicated set of tests (test good state, failed state, partial failed state, Rhode Island state, and so on). There were several interlocking state machines going on - each individual change pushed to a single agent had its own state tracked, and the overall push to multiple agents was also tracked. Since these were multithreaded and multiprocess (apache again!), the states had to be tracked in the database.

I'd recommend avoiding all that if at all possible, but if your application is automagically changing remote state you probably have to go there like we did.

Timeouts: the Real World is Messy

2009-08-25T14:34:00.003-04:00

For communication between processes in our project we used XMLRPC, because it's lightweight and well supported in python.

But not, as it turns out, well enough supported. And it's not well enough supported in a very standard way: it does not give you a quick easy way to make a timeout. This is never a problem in the imaginary language development world, but once you get into the real world you need to have a better default timeout than whatever happens to be baked in.

The python XMLRPC stuff does let you specify your transport so you can use whatever. Unfortunately we were on python 2.4, the timeouts were only added in to HTTP connection in 2.6. So I ended up subclassing all the way down the object stack to make a timeout transport that would take a timeout HTTP connection so I could have a timeout XMLRPC connection.

Then I had to do it all over again to put a timeout into urlopen which we used when the agents fetched files from the main server.

In a more modern python they've put timeouts into most of those things, so all you'd have to do would be to get the timeout enabled connection in as the XMLRPC transport. Which is good because putting a timeout into the 2.4 version was about as pleasant as you would expect in a set of libraries that has both urllib and urllib2.

The weird thing is that this seems to be a required step in language development. The socket guys write settable timeouts, and then the HTTP guys don't expose them until later versions. In python that's 2.6. In Java it was 1.5, and I remember the day when we moved to that version and hallelujah, timeouts! One thing to say though is that ugly as getting timeout into python was, it was a lot easier than doing it in Java because the class internals were accessible to my python subclasses.

Anyway if in the future you're writing a connection type on top of sockets - think of the real world where connections aren't perfect and expose the timeouts for use in your first version.

Django Session Trick: Confirm Import

2009-08-24T18:55:00.000-04:00

There are probably other ways to do this trick but this one was mine.

One of the features our configuration app had was the ability to import and export one or multiple configuration objects (django model objects) together.

How to get these files is pretty obvious: the same code that can give you manage.py dumpdata gives you free export, just get a serializer out of django.core.serializers, give it a list of objects and if you're feeling nice an indent level to pretty it all up, and off you go.

Getting objects for import isn't much more complicated, just get a deserializer and give it your file, now you have a nice pile of objects. We actually tarted up the deserializer to modify some of the fields so it would note for example "imported by Andy" instead of whatever its last editor before export was.

But you always want to give the user a chance to back out. So once we deserialized the objects we did not save them, instead we displayed them in a summary list, noting things like their names and which ones would overwrite already existing stuff and so on. If the user said go ahead and import, then we'd really do the import.

Of course that's a separate HTTP request, so they can't have the objects we just used to make the import report page. We could have stuck them in the database labeled temporary, but that just means we'd need a temporary table cleaning job and a special status on every object and so on. What a pain. We could stash the file in a file cache but that has the same problem as the database.

Sessions seemed like the obvious place to stash temporary objects, but we couldn't just stick arbitrary django model objects inside the django session - maybe you can now, but in 0.96 it just was not happening. But we could stick the contents of the file in there, it's just a string! So... on first upload, generate the objects for display and stick the file contents in the session. If they say yes, reimport from the session and save the objects. If they say no, scrub that variable out of the session for cleanliness. If they ignore it and go do something else, the session will get cleaned up by the lovely session code.

Code Parenting

2009-08-22T18:40:00.000-04:00

This comes from a discussion with some good programmers I know. We were kvetching about code ownership, how it gets in the way but is better than people just writing stuff and then abandoning it to be fixed by others. Code ownership shouldn't mean that I can't fix bugs in your code. But lack of ownership also shouldn't mean that I just ignore my bugs until you fix them.

I think code parenting is better. It's your code and you are responsible to fix it, but it can take lessons from other people. Or they can adopt it as their own.

Django Model Hack: Breaking Constraints

2009-08-21T11:13:00.001-04:00

This isn't a trick, it's a terrible hack. But it uses such a tiny bit of code that it was never worth replacing it with a long, well structured version. Anyway:

One of the things we had in our configuration tool was the ability to import and export configuration objects, both in batches and one at a time.

Of course pulling single objects in a set of objects with foreign keys means that you won't have their dependencies. This we allowed deliberately to let the users export only what they wanted - maybe they wanted to export an item, bring it into another system, and then hook it up differently. Alternately, you might import a couple of objects separately that depend on each other - it's easier if you don't have to import them in a specific order, but can let the keys dangle until you bring in the dependencies.

Django, as it turns out, has no problem with this sort of thing. You pull in the new object, and the foreign keys that don't work just show up in forms as not set yet, so you have to set them before you can save again. We additionally wrote a validation to keep track of foreign keys that had been broken by import/export and not filled in yet.

When we were first messing around with this we were using the easy sqlite database, which also had no problem with this sort of thing because it doesn't have real constraints. When we moved up to postgres, it rightly refused to save foreign keys that didn't exist yet.

So I put in a terrible hack (this was built for 0.96, so you may have to hack the hack if you want it) to prevent django from generating constraints:


from django.db import backend
#strip the constraints from whatever backend you are using
backend.supports_constraints = False
from django.core import management
management._get_sql_model_create_original = management._get_sql_model_create
#if there are known models certain constraints escape the simple supports
#so we need to add a little more ugliness
management._get_sql_model_create = lambda model, known_models=None : \
    management._get_sql_model_create_original(model)

This is a bit heinous - actually I don't even remember that last ugly part, one of my co-conspirators must have added it - but it got the job done. This was imported from our base models file so that it would go into effect before django created any models.

Django Model Trick: Cross Validation

2009-08-20T10:29:00.001-04:00

While django validators provide you with excellent field validation tools to keep your data on the straight and narrow, there may be conflicts between objects or between fields in an object where it is not clear who should change.

Conflicts between fields have been somewhat handled since the django 0.96 that we were using, see the description in the docs, but this is still a limited to one object solution.

Our validation plan was to use field validation to disallow bad data, and to save data which was individually good but had a conflict with other data in this or another object, but show an error on the object's page.

So if I edit object foo so that it now has a conflict with object bar, when the edit page comes up it will say something like "foo can't live with bar" and on bar's page it says "bar can't live with foo". Now I must go to one or the other and resolve the conflict. In this way I can make edits which break the system and clean up later, rather than having to resolve a problem of edit ordering.

The actual code for this was fairly simple. Each model had two lists: "errors" and "warnings", warnings being things which were unlikely to be a good idea but not actually disallowed. The items in each list were functions taking a single argument expected to be the model object they were attached to, and returning a list of error strings. Through the magic of python duck typing, some validation functions could be reused on multiple objects. An example validation:


def example_validation(model_obj):
    error_list = []
    for other in model_obj.__class__.objects.all():
        if sound_alike(other.name, model_obj.name):
            error_list.append(
                '%s sounds like %s' % (other.name,
                             model_obj.name))
    return error_list

In the actual app instead of generating a plain string we generated a string which had built-in object links, so you could go to the edit page for anything mentioned in the string by clicking its name.

Once we ran all the validations, we had a big list of errors which we displayed on top of the object's page. In our configuration app, when you wanted to actually make changes take effect in the distributed system instead of just messing around with potential configurations, you had to resolve all of the errors before you could proceed. For this there was a central page which would give a complete list of errors (this is where links in the error descriptions became a necessity).

More description on making changes happen in the system has to wait until I can figure out how to describe our versioning scheme without a whiteboard.

Python Syntax Musings

2009-08-19T12:08:00.004-04:00

I'm learning a bit about haskell because why not, and working through this new to me syntax sends my mind back to think about python syntax. Maybe it's the rose_colored_glasses but there are only a few times in my python journey when I've really been annoyed by the things I had to write:


' '.join(list_of_strings)

def horrible_betrayal_using_defaults(innocent_map={}):
    share_state_with_strangers()

def default_workaround(map=None):
    if map is None:
        map = {}
    sigh()

And of course the way global works is not really friendly either. But that's about it. Some things like the way print works (before the new world order) are weird but not actually annoying to me personally.

That's not bad. Python for me has a very low impedance between idea and expression.

Just Plain Weird Django: Long Running Jobs

2009-08-17T21:50:00.006-04:00

Wanting to have long running background jobs is probably somewhat less weird than some of the other things we did. This was another Todd Rowell special, so if my description of it is a bit fuzzy it's because A) I'm not Todd and 2) since the company saw fit to lay us off I can't just throw the code at you. It's too long for a blog anyway.

The basic idea is that some parts of configuration needed to be broken out of the request/response timeframe: for example "contact these 20 other servers over a network that may be a little flakey, tell them to do something long, and wait for them to report back".

So for those tasks we needed long running jobs. So we (for Todd values of we) built a job manager. The job manager API allowed us to pass it pretty much any function that was written to conform to the job requirements, which were: when called, do your work, then tell me when to call you again (if ever).

The main job manager thread would actually embed each job in its own separate job thread rather than attempting to carry out all computation itself. It was only responsible for starting those threads initially and on restart. To be stable through restarts (planned and accidental) it kept a file of pickled jobs. Finally the job manager logged information about its jobs and had other job debugging utilities.

An interesting wrinkle came up once we moved the system onto apache mod_python in preparation for real use. Apache of course starts python processes as it likes. If each process starts a job manager, chaos ensues, which is something we should have put together from the pieces we knew before going there but that's development for you. This resulted in some quick rearchitecting to let the job manager run either inside the main django server or as a separate headless django which accepted the job manager's calls via XMLRPC.

Which underscores that messing with threads is a bit more difficult than the standard django - actually a lot more since django makes its standard stuff so easy! - but if you have to go there, you have to go there. It's worth biting the bullet and having a quality piece of code managing these jobs instead of just spawning a special thread each time you need to run one.

Just Plain Weird Django: Headless Django

2009-08-17T11:17:00.004-04:00

One of the weirder aspects of our system was that even a single installation was not one django. Instead it was one django on the mothership providing the user interface and an agent on each managed server, also built on django.

These agents were headless: they had no human interface. They received XMLRPC calls from the mothership and responded appropriately. I'm not going to go into the design of the XMLRPC calls here, they were simple calls but their application was a bit complicated since real networks can of course lose any call at any time. The XMLRPC stuff started with Graham Binn's work and was then extensively modified for our particular use.

By basing the agents on django any code we used in the mothership could be used in the agents as well. Originally we had the idea that we might put a UI on the agents for debugging, but using the debug page saving and good logging made that pretty much unnecessary. One of the agents XMLRPC calls was to ask it to bundle up its logs and saved debug pages and return them to the mothership, and from there we could download and view the information.

I don't recommend headless django until you have a real logging framework in place and debug page capture, but once you have those you can use django for your machine interface needs as well as your human interface needs.

You might also want a persistent job engine to decouple long processing from the request-response pattern of django, but talking about ours will have to wait for another post.

Django Settings Trick: Override Files

2009-08-16T15:02:00.004-04:00

The problem we needed to solve was this: each developer wants to mess with their settings file without changing the project default settings file under version control. Additionally, the production system needs to be able to have different behavior than the default system, and it would be nice if the installer of the system could customize it with a small file rather than letting them touch the main settings file.

The solution is to have the main settings.py file import from a number of secondary files which may exist or not and override the main settings. You can do this by sticking sections of this nature at the end of your settings.py:


import sys
try:
   local_settings_path = '/etc/our_django/'
   if local_settings_path not in sys.path:
       sys.path.append(local_settings_path)
   from local_settings import *
except ImportError:
   pass

In this way any settings in the file /etc/our_django/local_settings.py will import into the main settings file and replace any prior assignments. If there is no local_settings.py then that's not a problem either, the import error is suppressed. And of course anything that comes after this in the main settings file cannot be overridden by it as python chews through (swallows?) the file. The sys path munging step is only required if local_settings.py lives separately from your main code.

We ended up having a couple of separate settings override files that could be used:

a database settings file, because that was most often messed with by developers
an installation settings file for system defaults appropriate to one installation
a local settings file for messing around with

Our install process took the checked in settings file and then installed appropriate database and installation settings files into the installed django project directory. If someone maintaining the system needed to mess with its behavior, they were instructed to make changes to their local settings file only, which lived somewhere in /etc like a normal settings file. My own local settings file would often look like:


ENABLE_DEV_MODE_LOGGING=True
ENABLE_DANGEROUS_UTILS=True

As you can see we loved to expand beyond the basic django settings.

Some similar settings tricks are floating around the web, such as the options on the django wiki, but this optional override files version is different enough to be worth a separate mention.

We had an additional settings safeguard by also having settings unit tests which would complain if settings were in an inappropriate state - for example the main settings.py should not be checked in with our test setting FAIL_HALF_OF_ALL_REQUESTS set.

No really, we had a test setting that basically did that for testing the effect of a bad network on our XMLRPC traffic. But that's a post for another day.

Saturday I Get to Rant (XML Config Edition)

2009-08-15T21:32:00.006-04:00

After writing a little bit about django's settings file not being in XML I'm left with half a rant ready to go and I've done enough for this week so I get to let it out.

I've been exposed to that complicated XML config file world back in Java land. I don't mind the Java language (except for generics, but that's another rant) but it has some very, very stupid friends.

A lot of the time creeping XML seems to come from a deep seated desire to create behavior that doesn't have to be programmed. Configuring somehow doesn't count as programming, even when it's thousands of lines. Which is good, because XML doesn't count as a programming language. Unfortunately the first statement there is just wishful thinking: complex system configuration is so close to programming that they're wearing the same pants. And doing a programming task in XML is like knitting with mittens on.

And of course it comes with a bunch of other problems, like separating pieces of object behavior so you can't learn both what fields it has and how it behaves without finding both the code and the magic XML section. Separating pieces of object behavior into different files drives me up two walls at once.

Finally, each XML config file format is effectively a miniature programming language designed by someone who didn't know they were designing one, with predictably wonderful results.

XML config defenders will talk about how you can validate the config file with a DTD or schema or whatever, but this is a big glass of snake oil. Sure, you can validate the format of the config file, and field values, but as soon as one field relates to another you're going to have to validate it with real code anyway. XML defenders might also talk about how XML is human readable. I'm not sure what the proper response to those people is. Probably we should make them sing their own config files - if you can read it, shouldn't you be able to sing it?

Anyway django configuration is just about my style. Generally items are just named values, or lists of values, and on the outside chance you need something more complicated, you can unleash a real programming language.

There's another related rant for another time about the whole XML web hellworld that is the WS* specifications for SOAP and associated perversions, but I should limit myself to one per week.

Django Settings Trick: Time Zone

2009-08-15T14:13:00.006-04:00

Since django settings are just another python file (Which was one of the reasons I originally chose django - I've been to "pretend you can program in XML" land and it's a hellish blasted wasteland where angle brackets prey upon the weak.) there's no reason you have to put up with the django hardwired timezone. Here's some code that works under Debian linux to pull out the system timezone. We just tossed similar code right in settings.py:


def get_time_zone():
    try:
        tzfile = open('/etc/timezone')
        for line in tzfile:
            if line:
                return line.strip()
    except:
        return 'GMT'
    return 'GMT'
TIME_ZONE = get_time_zone()

If you're not using a linux that writes the timezone into /etc/timezone so agreeably, you'll have to do a little research to learn how to get your time zone string, but the concept stands. Remember to check that you're delivering it in the format django expects, documented in the django docs.

Don't forget to handle exceptions gracefully, and give some indication of errors (which I stripped from this code for space).

Django Debugging Trick: Save the Page!

2009-08-15T10:06:00.010-04:00

I wish I could say this one was my idea, but it was built by Todd Rowell, another member of our team. Still, it's my blog so I get to write it down.

When you first start working with django one of the best parts is the incredibly detailed debug page that shows up whenever you inevitably do something wrong.

Of course, your users aren't going to appreciate seeing that page, so on a real installation of your system you turn off DEBUG and then you'll never see that beautiful page again.

Unless of course you generate it and save it out on every error with middleware.

We had a piece of debug saving middleware with a process_exception method that roughly looked like:


from django.views import debug
def process_exception(self, request, exception):
    if exception.__class__ is http.Http404:
        response = debug.technical_404_response(
                   request, exception)
    else:
        response = debug.technical_500_response(
                   request, *sys.exc_info())
    save_page(response._get_content())
    return None

Where save_page took care of putting it in the proper directory for the installation, doing a rotation so only the last N debug pages would be saved, naming, and so on.

The return value is None because we don't actually want to interfere with regular exception processing (in this case a normal 404 or 500 page) we just want to write the exception down as it goes by. Returning None keeps things rolling.

See the django docs on middleware for more about that and on how to install middleware and all your other "what is middleware" needs.

Django Test Trick: Fixture Migration the Easy Way

2009-08-14T20:12:00.001-04:00

This is not a way to do real model migration in the database. There are various projects underway for that; we had our own home rolled version, but I'm not in the mood to talk about it now.

However when you're running like mad you do probably keep changing what fields are in your models. This is 100% double plus true if you are running to keep up with a larger team of programmers who need your tool to configure their rapidly evolving applications.

And if you're running properly, you have a bunch of test fixture files which hold useful test data which probably won't even load when you're finished with today's changes.

You can hand-edit the fixtures, or write scripts to edit the fixtures, and I got pretty good at that. But sometimes you're adding new relations which are tricky enough that it the best tool to make a test fixture is the application you're already building. The trick is a simple two-stage process.

When you add your new fields, you want to make them blank=True and null=True.
Zero your database, load your old test fixture, which will load fine because it has no data and you demand none.
Run around your test data doing the setup you need.
Export your test data again.
Go back to your models and remove the blank and null arguments.
Now your models are as you want them, and your test data will load.

Also speaking from experience: do your field changes one or two at a time, and run your unit tests after each one. Nothing is worse than sorting through ten different model field changes causing errors at once.

Django Model Trick: UUIDs

2009-08-14T14:34:00.002-04:00

One of the most important model tricks we used in our application was UUIDs. We were stuck on python 2.4 so we had to copy the UUID code in from later pythons, but now you can have it with even less work.

Django gives you automatically incrementing integer IDs for everything, which is possibly the worst ID scheme known to man. The only thing that recommends it is that integers are very small. But since everyone starts at the same place, they're only unique for one spin of the database... and that's not good enough.

In our configuration application model ID fields were something like:


uuid = models.CharField(max_length=36, primary_key=True,
      default=make_uuid, editable=False)

where make_uuid is:


def make_uuid():
    return str(uuid.uuid4())

What does this give you for your jumbo-size keys? Freedom to toss objects around like confetti!

If I make a test fixture, and my coworker makes a test fixture, and we want to load them on the same machine to run a joint test? There are no ID conflicts.

If I make a useful data setup and want to export it and give it to someone else, it will not overwrite the partial setup she already has.

You can also give me an object A, then later modify it to A' and send me that exported object... and when I import it, you know it will overwrite A because that's the ID that matches.

There are also some other nice features of UUID keys, for example: if I get the object ID for a foo when I wanted a bar, it won't resolve to a foo by coincidence because the foo and bar ID spaces are not going to overlap.

UUIDs - because your object IS a special snowflake.

Why Weird?

2009-08-13T23:50:00.002-04:00

The reason we ended up doing things that could be considered weird django was because we were building an atypical app. Where the typical django application, if there is such a thing, is a single installation of a service sitting on top of its database, ours was meant as a configuration tool for a distributed system.

This meant some strange things for a django app:

We expected a large number of separate installations, sometimes wanting to exchange data
Our application would have a relatively small number of users
Configuration state objects needed to be versioned
We had to take configuration state and push it out to set of distributed agents which would do the actual configuration
The agents needed to receive data via XMLRPC, but should share code with the mothership (we used a no-UI django there)
We needed to have long-running jobs to communicate with the agents
We needed to generate configuration files for the agents to give to the actual running programs
We wanted some bigger units in our unit tests to test inter-process communication
The django admin was right out
There are probably some more strange requirements but this is more than enough bullet points

We also had to do some tricks because building a real product we were stuck on stable django, 0.96 at the time, with all its warts. I'm not going to talk about those, most of them have been fixed in django 1.0 and 1.1.

And I'm not going to talk domain specifics: though the company laid off 3/4 of the team as it ran low on money (myself included), 1/4 is still there struggling to finish requirements with a quarter of the original manpower. At least until he goes mad.

And frankly the domain specifics would bore a rhino to death.

Weird Django Tricks (Why Start a Blog?)

2009-08-13T23:30:00.001-04:00

Went to the Cambridge django meetup yesterday, saw a presentation on Schedr, which is a neat app if you go to UMass and maybe in the near future a neat app for many other schools.

Anyway seeing it and some of its code reminded me just how... weird... the django application I (and a few others) spent the past couple of years building, and I decided that it would be unfortunate to let these weird tricks die. So, here's an instant blog to record these django tricks in, and perhaps other programming subjects as the mood strikes me.