Sporkmonger

FeedTools 0.2.18 is almost done. Mostly it’s bugfixes, but there’s some gentle reorganization of the API as well. I’m going to be splitting a lot of the functions currently in the FeedTools module out into their own helper modules. Testing will now be done through a helper method that Jens Kraemer wrote. Jens also rewrote all the unit tests for Mark Pilgrim’s test cases so that they would point to the local feed instead of the remote one. I need to do a bit of work still to take better advantage of the potential speedup involved here, but that won’t likely come until 0.2.19.

The most significant improvement is likely going to be performance.

1. There were some issue with multithreading causing connections to the database to get way out of hand. I put an interim solution in place, but I’m going to see what it would take to create some sort of a queueing system that would allow http requests to be done concurrently while requests to the database remain in a single thread. Or really, whatever technique gives us the ability to request 5 feeds at once without killing the database dead as a doornail. More investigation required.
2. There’s a stack overflow in the current code because I did a really shoddy job of implementing some of the timestamp estimation code. It was overflowing, then hiding the error and defaulting to the standard behavior. In many cases, it’d take 10 seconds and up to parse a feed where this was occurring. It would always produce the correct output though, so I failed to notice it. One of those times when test cases fail you.
3. Timeouts are now handled more effectively. If you get a timeout, it will revert to the cache if possible, and will revert more quickly. In many cases, it took as long as 60 seconds to fail before.

There were a couple of other bugs that got fixed as well. Should hopefully be done later today. I still have to properly implement base64 encoding for Atom. Jens already did most of the work for me on that one though. (Big thank you to him, by the way. Much appreciated.)

Update: Annnnnd it’s up.