[ironic_monkey]

the cure for the hammering is batch processing. this can only be achieved if we reboot the scraper system completely; drop the xml scrapers, then refine the python APIs to allow batching (i am assuming the servers supports it). i did this in the compatible way currently to aid the transition process (and minimize the impact in other parts of the code). at some point the pain needs to be absorbed, but i think it's wise to shake out the bugs from the generic stuff using the current implementation before taking this further.

it would also be very useful if i wasn't the one doing this, you guys maintain this now so you should have a firm understanding of what's going on. that doesn't mean i won't help though!