Wanting to have long running background jobs is probably somewhat less weird than some of the other things we did. This was another Todd Rowell special, so if my description of it is a bit fuzzy it's because A) I'm not Todd and 2) since the company saw fit to lay us off I can't just throw the code at you. It's too long for a blog anyway.
The basic idea is that some parts of configuration needed to be broken out of the request/response timeframe: for example "contact these 20 other servers over a network that may be a little flakey, tell them to do something long, and wait for them to report back".
So for those tasks we needed long running jobs. So we (for Todd values of we) built a job manager. The job manager API allowed us to pass it pretty much any function that was written to conform to the job requirements, which were: when called, do your work, then tell me when to call you again (if ever).
The main job manager thread would actually embed each job in its own separate job thread rather than attempting to carry out all computation itself. It was only responsible for starting those threads initially and on restart. To be stable through restarts (planned and accidental) it kept a file of pickled jobs. Finally the job manager logged information about its jobs and had other job debugging utilities.
An interesting wrinkle came up once we moved the system onto apache mod_python in preparation for real use. Apache of course starts python processes as it likes. If each process starts a job manager, chaos ensues, which is something we should have put together from the pieces we knew before going there but that's development for you. This resulted in some quick rearchitecting to let the job manager run either inside the main django server or as a separate headless django which accepted the job manager's calls via XMLRPC.
Which underscores that messing with threads is a bit more difficult than the standard django - actually a lot more since django makes its standard stuff so easy! - but if you have to go there, you have to go there. It's worth biting the bullet and having a quality piece of code managing these jobs instead of just spawning a special thread each time you need to run one.
August 17, 2009
Just Plain Weird Django: Headless Django
One of the weirder aspects of our system was that even a single installation was not one django. Instead it was one django on the mothership providing the user interface and an agent on each managed server, also built on django.
These agents were headless: they had no human interface. They received XMLRPC calls from the mothership and responded appropriately. I'm not going to go into the design of the XMLRPC calls here, they were simple calls but their application was a bit complicated since real networks can of course lose any call at any time. The XMLRPC stuff started with Graham Binn's work and was then extensively modified for our particular use.
By basing the agents on django any code we used in the mothership could be used in the agents as well. Originally we had the idea that we might put a UI on the agents for debugging, but using the debug page saving and good logging made that pretty much unnecessary. One of the agents XMLRPC calls was to ask it to bundle up its logs and saved debug pages and return them to the mothership, and from there we could download and view the information.
I don't recommend headless django until you have a real logging framework in place and debug page capture, but once you have those you can use django for your machine interface needs as well as your human interface needs.
You might also want a persistent job engine to decouple long processing from the request-response pattern of django, but talking about ours will have to wait for another post.
These agents were headless: they had no human interface. They received XMLRPC calls from the mothership and responded appropriately. I'm not going to go into the design of the XMLRPC calls here, they were simple calls but their application was a bit complicated since real networks can of course lose any call at any time. The XMLRPC stuff started with Graham Binn's work and was then extensively modified for our particular use.
By basing the agents on django any code we used in the mothership could be used in the agents as well. Originally we had the idea that we might put a UI on the agents for debugging, but using the debug page saving and good logging made that pretty much unnecessary. One of the agents XMLRPC calls was to ask it to bundle up its logs and saved debug pages and return them to the mothership, and from there we could download and view the information.
I don't recommend headless django until you have a real logging framework in place and debug page capture, but once you have those you can use django for your machine interface needs as well as your human interface needs.
You might also want a persistent job engine to decouple long processing from the request-response pattern of django, but talking about ours will have to wait for another post.
August 16, 2009
Django Settings Trick: Override Files
The problem we needed to solve was this: each developer wants to mess with their settings file without changing the project default settings file under version control. Additionally, the production system needs to be able to have different behavior than the default system, and it would be nice if the installer of the system could customize it with a small file rather than letting them touch the main settings file.
The solution is to have the main settings.py file import from a number of secondary files which may exist or not and override the main settings. You can do this by sticking sections of this nature at the end of your settings.py:
In this way any settings in the file /etc/our_django/local_settings.py will import into the main settings file and replace any prior assignments. If there is no local_settings.py then that's not a problem either, the import error is suppressed. And of course anything that comes after this in the main settings file cannot be overridden by it as python chews through (swallows?) the file. The sys path munging step is only required if local_settings.py lives separately from your main code.
We ended up having a couple of separate settings override files that could be used:
Our install process took the checked in settings file and then installed appropriate database and installation settings files into the installed django project directory. If someone maintaining the system needed to mess with its behavior, they were instructed to make changes to their local settings file only, which lived somewhere in /etc like a normal settings file. My own local settings file would often look like:
As you can see we loved to expand beyond the basic django settings.
Some similar settings tricks are floating around the web, such as the options on the django wiki, but this optional override files version is different enough to be worth a separate mention.
We had an additional settings safeguard by also having settings unit tests which would complain if settings were in an inappropriate state - for example the main settings.py should not be checked in with our test setting FAIL_HALF_OF_ALL_REQUESTS set.
No really, we had a test setting that basically did that for testing the effect of a bad network on our XMLRPC traffic. But that's a post for another day.
The solution is to have the main settings.py file import from a number of secondary files which may exist or not and override the main settings. You can do this by sticking sections of this nature at the end of your settings.py:
import sys
try:
local_settings_path = '/etc/our_django/'
if local_settings_path not in sys.path:
sys.path.append(local_settings_path)
from local_settings import *
except ImportError:
pass
In this way any settings in the file /etc/our_django/local_settings.py will import into the main settings file and replace any prior assignments. If there is no local_settings.py then that's not a problem either, the import error is suppressed. And of course anything that comes after this in the main settings file cannot be overridden by it as python chews through (swallows?) the file. The sys path munging step is only required if local_settings.py lives separately from your main code.
We ended up having a couple of separate settings override files that could be used:
- a database settings file, because that was most often messed with by developers
- an installation settings file for system defaults appropriate to one installation
- a local settings file for messing around with
Our install process took the checked in settings file and then installed appropriate database and installation settings files into the installed django project directory. If someone maintaining the system needed to mess with its behavior, they were instructed to make changes to their local settings file only, which lived somewhere in /etc like a normal settings file. My own local settings file would often look like:
ENABLE_DEV_MODE_LOGGING=True
ENABLE_DANGEROUS_UTILS=True
As you can see we loved to expand beyond the basic django settings.
Some similar settings tricks are floating around the web, such as the options on the django wiki, but this optional override files version is different enough to be worth a separate mention.
We had an additional settings safeguard by also having settings unit tests which would complain if settings were in an inappropriate state - for example the main settings.py should not be checked in with our test setting FAIL_HALF_OF_ALL_REQUESTS set.
No really, we had a test setting that basically did that for testing the effect of a bad network on our XMLRPC traffic. But that's a post for another day.
August 15, 2009
Saturday I Get to Rant (XML Config Edition)
After writing a little bit about django's settings file not being in XML I'm left with half a rant ready to go and I've done enough for this week so I get to let it out.
I've been exposed to that complicated XML config file world back in Java land. I don't mind the Java language (except for generics, but that's another rant) but it has some very, very stupid friends.
A lot of the time creeping XML seems to come from a deep seated desire to create behavior that doesn't have to be programmed. Configuring somehow doesn't count as programming, even when it's thousands of lines. Which is good, because XML doesn't count as a programming language. Unfortunately the first statement there is just wishful thinking: complex system configuration is so close to programming that they're wearing the same pants. And doing a programming task in XML is like knitting with mittens on.
And of course it comes with a bunch of other problems, like separating pieces of object behavior so you can't learn both what fields it has and how it behaves without finding both the code and the magic XML section. Separating pieces of object behavior into different files drives me up two walls at once.
Finally, each XML config file format is effectively a miniature programming language designed by someone who didn't know they were designing one, with predictably wonderful results.
XML config defenders will talk about how you can validate the config file with a DTD or schema or whatever, but this is a big glass of snake oil. Sure, you can validate the format of the config file, and field values, but as soon as one field relates to another you're going to have to validate it with real code anyway. XML defenders might also talk about how XML is human readable. I'm not sure what the proper response to those people is. Probably we should make them sing their own config files - if you can read it, shouldn't you be able to sing it?
Anyway django configuration is just about my style. Generally items are just named values, or lists of values, and on the outside chance you need something more complicated, you can unleash a real programming language.
There's another related rant for another time about the whole XML web hellworld that is the WS* specifications for SOAP and associated perversions, but I should limit myself to one per week.
I've been exposed to that complicated XML config file world back in Java land. I don't mind the Java language (except for generics, but that's another rant) but it has some very, very stupid friends.
A lot of the time creeping XML seems to come from a deep seated desire to create behavior that doesn't have to be programmed. Configuring somehow doesn't count as programming, even when it's thousands of lines. Which is good, because XML doesn't count as a programming language. Unfortunately the first statement there is just wishful thinking: complex system configuration is so close to programming that they're wearing the same pants. And doing a programming task in XML is like knitting with mittens on.
And of course it comes with a bunch of other problems, like separating pieces of object behavior so you can't learn both what fields it has and how it behaves without finding both the code and the magic XML section. Separating pieces of object behavior into different files drives me up two walls at once.
Finally, each XML config file format is effectively a miniature programming language designed by someone who didn't know they were designing one, with predictably wonderful results.
XML config defenders will talk about how you can validate the config file with a DTD or schema or whatever, but this is a big glass of snake oil. Sure, you can validate the format of the config file, and field values, but as soon as one field relates to another you're going to have to validate it with real code anyway. XML defenders might also talk about how XML is human readable. I'm not sure what the proper response to those people is. Probably we should make them sing their own config files - if you can read it, shouldn't you be able to sing it?
Anyway django configuration is just about my style. Generally items are just named values, or lists of values, and on the outside chance you need something more complicated, you can unleash a real programming language.
There's another related rant for another time about the whole XML web hellworld that is the WS* specifications for SOAP and associated perversions, but I should limit myself to one per week.
Django Settings Trick: Time Zone
Since django settings are just another python file (Which was one of the reasons I originally chose django - I've been to "pretend you can program in XML" land and it's a hellish blasted wasteland where angle brackets prey upon the weak.) there's no reason you have to put up with the django hardwired timezone. Here's some code that works under Debian linux to pull out the system timezone. We just tossed similar code right in settings.py:
If you're not using a linux that writes the timezone into /etc/timezone so agreeably, you'll have to do a little research to learn how to get your time zone string, but the concept stands. Remember to check that you're delivering it in the format django expects, documented in the django docs.
Don't forget to handle exceptions gracefully, and give some indication of errors (which I stripped from this code for space).
def get_time_zone():
try:
tzfile = open('/etc/timezone')
for line in tzfile:
if line:
return line.strip()
except:
return 'GMT'
return 'GMT'
TIME_ZONE = get_time_zone()
If you're not using a linux that writes the timezone into /etc/timezone so agreeably, you'll have to do a little research to learn how to get your time zone string, but the concept stands. Remember to check that you're delivering it in the format django expects, documented in the django docs.
Don't forget to handle exceptions gracefully, and give some indication of errors (which I stripped from this code for space).
Subscribe to:
Posts (Atom)
Posts
