Wanting to have long running background jobs is probably somewhat less weird than some of the other things we did. This was another Todd Rowell special, so if my description of it is a bit fuzzy it's because A) I'm not Todd and 2) since the company saw fit to lay us off I can't just throw the code at you. It's too long for a blog anyway.
The basic idea is that some parts of configuration needed to be broken out of the request/response timeframe: for example "contact these 20 other servers over a network that may be a little flakey, tell them to do something long, and wait for them to report back".
So for those tasks we needed long running jobs. So we (for Todd values of we) built a job manager. The job manager API allowed us to pass it pretty much any function that was written to conform to the job requirements, which were: when called, do your work, then tell me when to call you again (if ever).
The main job manager thread would actually embed each job in its own separate job thread rather than attempting to carry out all computation itself. It was only responsible for starting those threads initially and on restart. To be stable through restarts (planned and accidental) it kept a file of pickled jobs. Finally the job manager logged information about its jobs and had other job debugging utilities.
An interesting wrinkle came up once we moved the system onto apache mod_python in preparation for real use. Apache of course starts python processes as it likes. If each process starts a job manager, chaos ensues, which is something we should have put together from the pieces we knew before going there but that's development for you. This resulted in some quick rearchitecting to let the job manager run either inside the main django server or as a separate headless django which accepted the job manager's calls via XMLRPC.
Which underscores that messing with threads is a bit more difficult than the standard django - actually a lot more since django makes its standard stuff so easy! - but if you have to go there, you have to go there. It's worth biting the bullet and having a quality piece of code managing these jobs instead of just spawning a special thread each time you need to run one.
Showing posts with label python. Show all posts
Showing posts with label python. Show all posts
August 17, 2009
Just Plain Weird Django: Headless Django
One of the weirder aspects of our system was that even a single installation was not one django. Instead it was one django on the mothership providing the user interface and an agent on each managed server, also built on django.
These agents were headless: they had no human interface. They received XMLRPC calls from the mothership and responded appropriately. I'm not going to go into the design of the XMLRPC calls here, they were simple calls but their application was a bit complicated since real networks can of course lose any call at any time. The XMLRPC stuff started with Graham Binn's work and was then extensively modified for our particular use.
By basing the agents on django any code we used in the mothership could be used in the agents as well. Originally we had the idea that we might put a UI on the agents for debugging, but using the debug page saving and good logging made that pretty much unnecessary. One of the agents XMLRPC calls was to ask it to bundle up its logs and saved debug pages and return them to the mothership, and from there we could download and view the information.
I don't recommend headless django until you have a real logging framework in place and debug page capture, but once you have those you can use django for your machine interface needs as well as your human interface needs.
You might also want a persistent job engine to decouple long processing from the request-response pattern of django, but talking about ours will have to wait for another post.
These agents were headless: they had no human interface. They received XMLRPC calls from the mothership and responded appropriately. I'm not going to go into the design of the XMLRPC calls here, they were simple calls but their application was a bit complicated since real networks can of course lose any call at any time. The XMLRPC stuff started with Graham Binn's work and was then extensively modified for our particular use.
By basing the agents on django any code we used in the mothership could be used in the agents as well. Originally we had the idea that we might put a UI on the agents for debugging, but using the debug page saving and good logging made that pretty much unnecessary. One of the agents XMLRPC calls was to ask it to bundle up its logs and saved debug pages and return them to the mothership, and from there we could download and view the information.
I don't recommend headless django until you have a real logging framework in place and debug page capture, but once you have those you can use django for your machine interface needs as well as your human interface needs.
You might also want a persistent job engine to decouple long processing from the request-response pattern of django, but talking about ours will have to wait for another post.
August 16, 2009
Django Settings Trick: Override Files
The problem we needed to solve was this: each developer wants to mess with their settings file without changing the project default settings file under version control. Additionally, the production system needs to be able to have different behavior than the default system, and it would be nice if the installer of the system could customize it with a small file rather than letting them touch the main settings file.
The solution is to have the main settings.py file import from a number of secondary files which may exist or not and override the main settings. You can do this by sticking sections of this nature at the end of your settings.py:
In this way any settings in the file /etc/our_django/local_settings.py will import into the main settings file and replace any prior assignments. If there is no local_settings.py then that's not a problem either, the import error is suppressed. And of course anything that comes after this in the main settings file cannot be overridden by it as python chews through (swallows?) the file. The sys path munging step is only required if local_settings.py lives separately from your main code.
We ended up having a couple of separate settings override files that could be used:
Our install process took the checked in settings file and then installed appropriate database and installation settings files into the installed django project directory. If someone maintaining the system needed to mess with its behavior, they were instructed to make changes to their local settings file only, which lived somewhere in /etc like a normal settings file. My own local settings file would often look like:
As you can see we loved to expand beyond the basic django settings.
Some similar settings tricks are floating around the web, such as the options on the django wiki, but this optional override files version is different enough to be worth a separate mention.
We had an additional settings safeguard by also having settings unit tests which would complain if settings were in an inappropriate state - for example the main settings.py should not be checked in with our test setting FAIL_HALF_OF_ALL_REQUESTS set.
No really, we had a test setting that basically did that for testing the effect of a bad network on our XMLRPC traffic. But that's a post for another day.
The solution is to have the main settings.py file import from a number of secondary files which may exist or not and override the main settings. You can do this by sticking sections of this nature at the end of your settings.py:
import sys
try:
local_settings_path = '/etc/our_django/'
if local_settings_path not in sys.path:
sys.path.append(local_settings_path)
from local_settings import *
except ImportError:
pass
In this way any settings in the file /etc/our_django/local_settings.py will import into the main settings file and replace any prior assignments. If there is no local_settings.py then that's not a problem either, the import error is suppressed. And of course anything that comes after this in the main settings file cannot be overridden by it as python chews through (swallows?) the file. The sys path munging step is only required if local_settings.py lives separately from your main code.
We ended up having a couple of separate settings override files that could be used:
- a database settings file, because that was most often messed with by developers
- an installation settings file for system defaults appropriate to one installation
- a local settings file for messing around with
Our install process took the checked in settings file and then installed appropriate database and installation settings files into the installed django project directory. If someone maintaining the system needed to mess with its behavior, they were instructed to make changes to their local settings file only, which lived somewhere in /etc like a normal settings file. My own local settings file would often look like:
ENABLE_DEV_MODE_LOGGING=True
ENABLE_DANGEROUS_UTILS=True
As you can see we loved to expand beyond the basic django settings.
Some similar settings tricks are floating around the web, such as the options on the django wiki, but this optional override files version is different enough to be worth a separate mention.
We had an additional settings safeguard by also having settings unit tests which would complain if settings were in an inappropriate state - for example the main settings.py should not be checked in with our test setting FAIL_HALF_OF_ALL_REQUESTS set.
No really, we had a test setting that basically did that for testing the effect of a bad network on our XMLRPC traffic. But that's a post for another day.
August 15, 2009
Django Settings Trick: Time Zone
Since django settings are just another python file (Which was one of the reasons I originally chose django - I've been to "pretend you can program in XML" land and it's a hellish blasted wasteland where angle brackets prey upon the weak.) there's no reason you have to put up with the django hardwired timezone. Here's some code that works under Debian linux to pull out the system timezone. We just tossed similar code right in settings.py:
If you're not using a linux that writes the timezone into /etc/timezone so agreeably, you'll have to do a little research to learn how to get your time zone string, but the concept stands. Remember to check that you're delivering it in the format django expects, documented in the django docs.
Don't forget to handle exceptions gracefully, and give some indication of errors (which I stripped from this code for space).
def get_time_zone():
try:
tzfile = open('/etc/timezone')
for line in tzfile:
if line:
return line.strip()
except:
return 'GMT'
return 'GMT'
TIME_ZONE = get_time_zone()
If you're not using a linux that writes the timezone into /etc/timezone so agreeably, you'll have to do a little research to learn how to get your time zone string, but the concept stands. Remember to check that you're delivering it in the format django expects, documented in the django docs.
Don't forget to handle exceptions gracefully, and give some indication of errors (which I stripped from this code for space).
Django Debugging Trick: Save the Page!
I wish I could say this one was my idea, but it was built by Todd Rowell, another member of our team. Still, it's my blog so I get to write it down.
When you first start working with django one of the best parts is the incredibly detailed debug page that shows up whenever you inevitably do something wrong.
Of course, your users aren't going to appreciate seeing that page, so on a real installation of your system you turn off DEBUG and then you'll never see that beautiful page again.
Unless of course you generate it and save it out on every error with middleware.
We had a piece of debug saving middleware with a process_exception method that roughly looked like:
Where save_page took care of putting it in the proper directory for the installation, doing a rotation so only the last N debug pages would be saved, naming, and so on.
The return value is None because we don't actually want to interfere with regular exception processing (in this case a normal 404 or 500 page) we just want to write the exception down as it goes by. Returning None keeps things rolling.
See the django docs on middleware for more about that and on how to install middleware and all your other "what is middleware" needs.
When you first start working with django one of the best parts is the incredibly detailed debug page that shows up whenever you inevitably do something wrong.
Of course, your users aren't going to appreciate seeing that page, so on a real installation of your system you turn off DEBUG and then you'll never see that beautiful page again.
Unless of course you generate it and save it out on every error with middleware.
We had a piece of debug saving middleware with a process_exception method that roughly looked like:
from django.views import debug
def process_exception(self, request, exception):
if exception.__class__ is http.Http404:
response = debug.technical_404_response(
request, exception)
else:
response = debug.technical_500_response(
request, *sys.exc_info())
save_page(response._get_content())
return None
Where save_page took care of putting it in the proper directory for the installation, doing a rotation so only the last N debug pages would be saved, naming, and so on.
The return value is None because we don't actually want to interfere with regular exception processing (in this case a normal 404 or 500 page) we just want to write the exception down as it goes by. Returning None keeps things rolling.
See the django docs on middleware for more about that and on how to install middleware and all your other "what is middleware" needs.
August 14, 2009
Django Test Trick: Fixture Migration the Easy Way
This is not a way to do real model migration in the database. There are various projects underway for that; we had our own home rolled version, but I'm not in the mood to talk about it now.
However when you're running like mad you do probably keep changing what fields are in your models. This is 100% double plus true if you are running to keep up with a larger team of programmers who need your tool to configure their rapidly evolving applications.
And if you're running properly, you have a bunch of test fixture files which hold useful test data which probably won't even load when you're finished with today's changes.
You can hand-edit the fixtures, or write scripts to edit the fixtures, and I got pretty good at that. But sometimes you're adding new relations which are tricky enough that it the best tool to make a test fixture is the application you're already building. The trick is a simple two-stage process.
However when you're running like mad you do probably keep changing what fields are in your models. This is 100% double plus true if you are running to keep up with a larger team of programmers who need your tool to configure their rapidly evolving applications.
And if you're running properly, you have a bunch of test fixture files which hold useful test data which probably won't even load when you're finished with today's changes.
You can hand-edit the fixtures, or write scripts to edit the fixtures, and I got pretty good at that. But sometimes you're adding new relations which are tricky enough that it the best tool to make a test fixture is the application you're already building. The trick is a simple two-stage process.
- When you add your new fields, you want to make them blank=True and null=True.
- Zero your database, load your old test fixture, which will load fine because it has no data and you demand none.
- Run around your test data doing the setup you need.
- Export your test data again.
- Go back to your models and remove the blank and null arguments.
- Now your models are as you want them, and your test data will load.
Django Model Trick: UUIDs
One of the most important model tricks we used in our application was UUIDs. We were stuck on python 2.4 so we had to copy the UUID code in from later pythons, but now you can have it with even less work.
Django gives you automatically incrementing integer IDs for everything, which is possibly the worst ID scheme known to man. The only thing that recommends it is that integers are very small. But since everyone starts at the same place, they're only unique for one spin of the database... and that's not good enough.
In our configuration application model ID fields were something like:
where make_uuid is:
What does this give you for your jumbo-size keys? Freedom to toss objects around like confetti!
If I make a test fixture, and my coworker makes a test fixture, and we want to load them on the same machine to run a joint test? There are no ID conflicts.
If I make a useful data setup and want to export it and give it to someone else, it will not overwrite the partial setup she already has.
You can also give me an object A, then later modify it to A' and send me that exported object... and when I import it, you know it will overwrite A because that's the ID that matches.
There are also some other nice features of UUID keys, for example: if I get the object ID for a foo when I wanted a bar, it won't resolve to a foo by coincidence because the foo and bar ID spaces are not going to overlap.
UUIDs - because your object IS a special snowflake.
Django gives you automatically incrementing integer IDs for everything, which is possibly the worst ID scheme known to man. The only thing that recommends it is that integers are very small. But since everyone starts at the same place, they're only unique for one spin of the database... and that's not good enough.
In our configuration application model ID fields were something like:
uuid = models.CharField(max_length=36, primary_key=True,
default=make_uuid, editable=False)
where make_uuid is:
def make_uuid():
return str(uuid.uuid4())
What does this give you for your jumbo-size keys? Freedom to toss objects around like confetti!
If I make a test fixture, and my coworker makes a test fixture, and we want to load them on the same machine to run a joint test? There are no ID conflicts.
If I make a useful data setup and want to export it and give it to someone else, it will not overwrite the partial setup she already has.
You can also give me an object A, then later modify it to A' and send me that exported object... and when I import it, you know it will overwrite A because that's the ID that matches.
There are also some other nice features of UUID keys, for example: if I get the object ID for a foo when I wanted a bar, it won't resolve to a foo by coincidence because the foo and bar ID spaces are not going to overlap.
UUIDs - because your object IS a special snowflake.
August 13, 2009
Why Weird?
The reason we ended up doing things that could be considered weird django was because we were building an atypical app. Where the typical django application, if there is such a thing, is a single installation of a service sitting on top of its database, ours was meant as a configuration tool for a distributed system.
This meant some strange things for a django app:
And I'm not going to talk domain specifics: though the company laid off 3/4 of the team as it ran low on money (myself included), 1/4 is still there struggling to finish requirements with a quarter of the original manpower. At least until he goes mad.
And frankly the domain specifics would bore a rhino to death.
This meant some strange things for a django app:
- We expected a large number of separate installations, sometimes wanting to exchange data
- Our application would have a relatively small number of users
- Configuration state objects needed to be versioned
- We had to take configuration state and push it out to set of distributed agents which would do the actual configuration
- The agents needed to receive data via XMLRPC, but should share code with the mothership (we used a no-UI django there)
- We needed to have long-running jobs to communicate with the agents
- We needed to generate configuration files for the agents to give to the actual running programs
- We wanted some bigger units in our unit tests to test inter-process communication
- The django admin was right out
- There are probably some more strange requirements but this is more than enough bullet points
And I'm not going to talk domain specifics: though the company laid off 3/4 of the team as it ran low on money (myself included), 1/4 is still there struggling to finish requirements with a quarter of the original manpower. At least until he goes mad.
And frankly the domain specifics would bore a rhino to death.
Weird Django Tricks (Why Start a Blog?)
Went to the Cambridge django meetup yesterday, saw a presentation on Schedr, which is a neat app if you go to UMass and maybe in the near future a neat app for many other schools.
Anyway seeing it and some of its code reminded me just how... weird... the django application I (and a few others) spent the past couple of years building, and I decided that it would be unfortunate to let these weird tricks die. So, here's an instant blog to record these django tricks in, and perhaps other programming subjects as the mood strikes me.
Anyway seeing it and some of its code reminded me just how... weird... the django application I (and a few others) spent the past couple of years building, and I decided that it would be unfortunate to let these weird tricks die. So, here's an instant blog to record these django tricks in, and perhaps other programming subjects as the mood strikes me.
Subscribe to:
Posts (Atom)
Posts
