Just Plain Weird Django: Long Running Jobs

Wanting to have long running background jobs is probably somewhat less weird than some of the other things we did. This was another Todd Rowell special, so if my description of it is a bit fuzzy it's because A) I'm not Todd and 2) since the company saw fit to lay us off I can't just throw the code at you. It's too long for a blog anyway.

The basic idea is that some parts of configuration needed to be broken out of the request/response timeframe: for example "contact these 20 other servers over a network that may be a little flakey, tell them to do something long, and wait for them to report back".

So for those tasks we needed long running jobs. So we (for Todd values of we) built a job manager. The job manager API allowed us to pass it pretty much any function that was written to conform to the job requirements, which were: when called, do your work, then tell me when to call you again (if ever).

The main job manager thread would actually embed each job in its own separate job thread rather than attempting to carry out all computation itself. It was only responsible for starting those threads initially and on restart. To be stable through restarts (planned and accidental) it kept a file of pickled jobs. Finally the job manager logged information about its jobs and had other job debugging utilities.

An interesting wrinkle came up once we moved the system onto apache mod_python in preparation for real use. Apache of course starts python processes as it likes. If each process starts a job manager, chaos ensues, which is something we should have put together from the pieces we knew before going there but that's development for you. This resulted in some quick rearchitecting to let the job manager run either inside the main django server or as a separate headless django which accepted the job manager's calls via XMLRPC.

Which underscores that messing with threads is a bit more difficult than the standard django - actually a lot more since django makes its standard stuff so easy! - but if you have to go there, you have to go there. It's worth biting the bullet and having a quality piece of code managing these jobs instead of just spawning a special thread each time you need to run one.