Sporkmonger

If I have my way, desktop feed readers will just turn out to be a sort of thin-client for interfacing with a web-based feed reading proxy of sorts, which would itself, be a normal web-based feed reader. All of the majorly nifty aggregation features such as filtering and what-have-you would reside within the web-based proxy, while the user gets to choose how they actually interface with that information. They could use the web-based aggregator as their actual client, or they could point NNW at the web-based aggregator’s republished feeds (in your choice of Atom, RSS, or heck, !okay/news, if you prefer). You get all the advantages of the desktop client, plus all of the advantages of the web-based client all at once. And with OPML/XOXO, it’s not even difficult to set up. Just a matter of 30 seconds worth of importing.

In case you were wondering about my cryptic and nonsensical .plan thing over there.

For the record, I think NewsGator is, by far, the best-positioned RSS company around and I’m curious what they’ll do with that positioning.

I’m only doing a comparison of parsers here, not feed readers or parsers embedded within feed readers that aren’t available as a separate download, although I suppose… I’m really using a very loose interpretation of the word “parser” here.

Or more accurately, a list of things which claim to parse feeds, but generally do a bloody terrible job of it, however, there’s a few on the list that actually manage to do reasonably well.

I will try to keep this list up to date, so let me know if I’ve missed a parser or if the links to any of the parsers on the list go dead. Feel free to argue with my ratings, since they’re completely subjective, or suggest a rating for the ones I’ve marked “Not Sure”, but be aware that I consider any parser that fails to parse Atom (or similarly, if it fails to parse RSS) to be “poor” unless the author also wrote an adequate sibling parser that does parse Atom.

So, I’m curious, how many parsers/feed readers show an incorrect date for this feed or just blow up for that matter?

So far in my testing, NetNewsWire has been the only parser that correctly displays the date given in the feed. FeedTools can’t parse the date and reverts to the current date and time instead. So does Google Reader and Bloglines, as well as most of the online feed readers I tried it out with.

The Feed Validator does helpfully warn me of an “implausible date”, but the feed is perfectly valid Atom 1.0, so far as I can tell.

I’m honestly not too worried about the issue of parsers not being able to handle the concept of 2038 when it roles around. By 2038, the concept of feeds will likely seem utterly obsolete. But I can’t help but wonder if some parsers will end up tossing an exception on this feed. I couldn’t find anything in Mark Pilgrim ’s UFP test suite for dates being out of range. The largest date among the tests seems to be some time in 2004.

FeedTools 0.2.18 is almost done. Mostly it’s bugfixes, but there’s some gentle reorganization of the API as well. I’m going to be splitting a lot of the functions currently in the FeedTools module out into their own helper modules. Testing will now be done through a helper method that Jens Kraemer wrote. Jens also rewrote all the unit tests for Mark Pilgrim’s test cases so that they would point to the local feed instead of the remote one. I need to do a bit of work still to take better advantage of the potential speedup involved here, but that won’t likely come until 0.2.19.

The most significant improvement is likely going to be performance.

1. There were some issue with multithreading causing connections to the database to get way out of hand. I put an interim solution in place, but I’m going to see what it would take to create some sort of a queueing system that would allow http requests to be done concurrently while requests to the database remain in a single thread. Or really, whatever technique gives us the ability to request 5 feeds at once without killing the database dead as a doornail. More investigation required.
2. There’s a stack overflow in the current code because I did a really shoddy job of implementing some of the timestamp estimation code. It was overflowing, then hiding the error and defaulting to the standard behavior. In many cases, it’d take 10 seconds and up to parse a feed where this was occurring. It would always produce the correct output though, so I failed to notice it. One of those times when test cases fail you.
3. Timeouts are now handled more effectively. If you get a timeout, it will revert to the cache if possible, and will revert more quickly. In many cases, it took as long as 60 seconds to fail before.

There were a couple of other bugs that got fixed as well. Should hopefully be done later today. I still have to properly implement base64 encoding for Atom. Jens already did most of the work for me on that one though. (Big thank you to him, by the way. Much appreciated.)

Update: Annnnnd it’s up.

One of the big advantages that Mark Pilgrim’s UFP has over FeedTools is in its *_detail methods. The UFP basically gives you a bunch more information without having to dive into XPath or some such. In a couple of obvious places, like the author field, FeedTools also does this, but it used to do it with an interesting trick.

I really didn’t like the idea of having two different methods to retrieve what was essentially the same piece of information. But I also didn’t like forcing people who were used to element names from RSS being stuck using a more unfamiliar Atom-like class structure.

So awhile back, I made it so that there were these special funky classes, like the FeedTools::Author class, that were basically a structure containing the author’s name, email, url, and the raw data in the case of an RSS feed. So you could write:

1
2
3
4
5
6
7
8
9

feed = FeedTools::Feed.open(
  'http://sporkmonger.com/xml/atom10/feed.xml')
feed.author.name
=> 'Bob Aman'
feed.author.email
=> 'bob@sporkmonger.com'
feed.author.url
=> 'http://sporkmonger.com/'

But… you could also write:

1
2
3
4
5

feed = FeedTools::Feed.open(
  'http://sporkmonger.com/xml/atom10/feed.xml')
feed.author
=> 'Bob Aman'

Because I had used method_missing to relay any missing methods on the class to the name object, and overwritten the inspect method and a few others to do the same.

I eventually removed it because the code was quite repetitive (though I know how to fix that at this point), but also because I was worried that someone might actually need one of those overwritten methods for some reason, and also because I thought ‘chameleon classes’ might be surprising to some people. Actually, quite possibly very surprising. And you know, we supposedly want ‘least surprise.’

Now, the reason I’m bringing this up is because I wanted to get more information attached to the content element. I could change the API and have the content method return a class instead of a string, I could leave it the way it is and go with the Mark Pilgrim method, or I could return to the ‘chameleon classes’ only better written the second time round.

My personal preference is actually the shape changing niftiness, despite my reservations. Aesthetically, I think it’s the nicest looking solution, because it keeps the API much, much closer to the names of the elements in the respective syndication formats. Remember, our fearless leaders are of the opinion that beauty and productivity go hand in hand. But I worry, so I’m just going to ask you guys, what do you prefer, and why?

Oh yeah, and I’m strongly considering following Mark’s example and using ‘href’ everywhere for consistency.