Yahoo Pipes--RSS without Server Side Scripts

Filed: Mon, Feb 19 2007 under Programming|| Tags: json pipes rss javascript

Something important happened when Yahoo released their new pipes service. Beyond the ability to mash up different internet resources, pipes made it possible for every web-publisher to include RSS feeds on their pages without the need of server side scripts! From conveniently publishing your delicious bookmarks, to making your own launch page, this new service heralds a major, major shift in web publishing.

This article delves into how to create and integrate yahoo-pipes RSS into your web page. If you'd like a simple plug-in solution check out Kent Brewster's Badger. If you'd like to know how this works so you can REALLY exploit the full power of this new tool, read on…

Introduction

Yahoo Pipes is a service which lets you take information on the web, process it, then format it in interesting ways. At it's most simplest, we can ask yahoo pipes to retrieve a RSS feed and then just display it for us. Here's a simple pipe which will display this site's rss feed. All it does is accept a RSS URL with the name of feed, retrieves it, and displays it.

If you scroll down to the end of the feed page you'll see the traditional RSS label for the feed, the same ol' RSS label which requires a web-server to retrieve and process before it can be sent on to the browser. But beside that RSS link is an option to retrieve the pipe's data as JSON and JSON can be directly retrieved by the web browser -- the only server you need is Yahoo.

I've already created, saved, and published a generic feed pipe you can use. The URL for this is as follows:

http://pipes.yahoo.com/pipes/9oyONQzA2xGOkM4FqGIyXQ/run?&_render=json&_callback=piper&feed=

All you have to do is append the RSS feed URL to the end of the pipe URL above. For instance, if you wanted the feed for del.icio.us's popular javascript with the url of http://del.icio.us/rss/popular/javascript then the pipes URL becomes:

http://pipes.yahoo.com/pipes/9oyONQzA2xGOkM4FqGIyXQ/run?&_render=json&_callback=piper&feed=http://del.icio.us/rss/popular/javascript

The getFeed Function

To use this URL inside your web page you'll need to create a small function.

function getFeed(feed) {
var newScript = document.createElement('script');
    newScript.type = 'text/javascript';
    newScript.src = 'http://pipes.yahoo.com/pipes/9oyONQzA2xGOkM4FqGIyXQ/run?&_render=json&_callback=piper&feed='+feed;
document.getElementsByTagName("head")[0].appendChild(newScript);
}

This function creates a new <script> element, sets its attributes to text/javascript and the source to pipes.yahoo.com and appends your RSS feed URL. When the new element is appended to the page's <head> section the script is automatically loaded.

To load our delicious rss feed we'd just call our new function like this...

getFeed('http://del.icio.us/rss/popular/javascript');

The Data Structure

In the pipes.yahoo.com URL we specify a _callback value of piper. What this does is tell the feed to call the javascript function piper and pass it the feed data. So you'll need to create your own function now and name it piper(feed). feed will be the data as a javascript object. Feed will have the following elements (This isn't a complete list, just the important items)…

feed.value.items[].link             -- Link of current item
feed.value.items[].title            -- Title of current item
feed.value.items[].description      -- description of current item

Note that this structure is FLUID! For 90% of all RSS feeds these three items will serve you well, but some feeds like youTube have different structure to contain screenshots and other elements. YouTube in particular extends the object out with the enclosure attribute. So to get a title off youTube you'll need to use feed.value.items[].enclosure.title.

If your feed isn't showing up, just call the pipe feed manually and sort out the structure for yourself. Json isn't terribly hard to understand...

"title": "sunset - Everyone's Tagged Photos"
"link": "http://www.flickr.com/photos/tags/sunset/" 
"description": "A feed of sunset - Everyone's Tagged Photos"
"modified": "2007-01-29T16:45:24Z"
"generator": "http://www.flickr.com/"
"items": [ { "title": "IMG_0697"
             "link": "http://www.flickr.com/photos/32655671@N00/373458148/"
             "media": { "m":"http://farm1.static.flickr.com/123/373458148_70dbabf167_m.jpg" }
             "date_taken": "2006-07-22T22:51:18-08:00"
             "description": "Long description -- contians HTML"
             "published": "2007-01-29T16:45:24Z"
             "author": "nobody@flickr.com (mae2007)"
             "tags": "africa sunset water" 
           }
          ]

Assuming this data was passed to piper(feed) then feed.title would be sunset - Everyone's Tagged Photos. feed.link would have the link. feed.items[0].title would be the title of the first item. feed.items[1].link would have the link of the second item. feed.items[0].media.m would contain the url for the media in the first item. Once you get used to it JSON is one of the easiest datastructures around to work with. If you encounter a wonky feed just calling up the json data manually should let you know how you'll need to modify your function.

Processing the Data

To process this we just have to loop through the items array and output the link, title and description. Here's a simple piper function. You'll of course want to modify it to fit into your blog.

function piper(feed) {
   var tmp='';
   for (var i=0; i<feed.value.items.length; i++) {
      tmp+='<a href="'+feed.value.items[i].link+'">';
      tmp+=feed.value.items[i].title+'</a><br>';
      if (feed.value.items[i].description) {
         tmp+=feed.value.items[i].description;
      }
      tmp+='<hr>';
   }
   document.getElementById('rssLayer').innerHTML=tmp;
}

To make this work we'll need a division named rssLayer somewhere on the page. This is where piper will output the RSS feed.

The Big Picture

Here's a complete template so you can see the big picture.

<html>
   <head>
      <script type="text/javascript">
         function getFeed(feed) {
            var newScript = document.createElement('script');
                newScript.type = 'text/javascript';
                newScript.src = 'http://pipes.yahoo.com/pipes/9oyONQzA2xGOkM4FqGIyXQ/run?&_render=json&_callback=piper&feed='+feed;
            document.getElementsByTagName("head")[0].appendChild(newScript);
         }     
         function piper(feed) {
            var tmp='';
            for (var i=0; i<feed.value.items.length; i++) {
               tmp+='<a href="'+feed.value.items[i].link+'">';
               tmp+=feed.value.items[i].title+'</a><br>';
               if (feed.value.items[i].description) {
                  tmp+=feed.value.items[i].description;
               }
               tmp+='<hr>';
            }
            document.getElementById('rssLayer').innerHTML=tmp;
         }
      </script>
   </head>
   <body onLoad='getFeed("http://del.icio.us/rss/popular/javascript")';>
      <div id='rssLayer'></div>
   </body>
</html>

We start the process in the <body>'s onLoad event. When the web page has been fully loaded, getFeed will be called with the feed URL. getFeed will make the JSON call for us. When the data arrives it will call the piper function and feed will contain all the feed data as a javascript object. Piper will process the data and then stuff it into a division named rssLayer.

Assuming you apply some basic styles to the rssLayer division you're fully good to go! You can now access any feed on the net even if you don't have access to a server-side scripting language.

Dynamic Feeds

The HTML example above is actually pretty powerful. If you add a few links in the <body> you can dynamically change the feed. For instance…

<A HREF="" onclick='getFeed("http://reddit.com/.rss"); return false;'>Reddit</A><BR>
<A HREF="" onclick='getFeed("http://del.icio.us/rss/popular/javascript"); return false;'>Delicious</A><BR>
<A HREF="" onclick='getFeed("http://www.dzone.com/feed/frontpage/rss.xml"); return false;'>DZone</A><BR>

Waiting for User Action

This will start the retreival process and the new feed will be placed in the rssLayer. Here's an example for the fun of it all. (This example has been modified to show only the first three items).

Again, just to stress how cool this is, this web page DIRECTLY contacted yahoo, requested the information, processed it, and displayed it. If my web host didn't offer me ANY server-side scripting services at all, the feed example above would still work exactly as it does now.

IE7 Bug

IE7 has a bug which will basically crash the page if a script is nested so that it is not a direct child of the parent. That's pretty confusing but it basically boils down to this…

<body>
   <div>
      <script>something which appends something to body</script>
   </div>
</body>

Will crash the page in IE7, while the following will not…

<body>
   <div>
   </div>
   <script>something which appends something to body</script>
</body>

You can find more information on this bug at Microsoft and there's a nice little writup on the bug on asp.net.

So if your script is working fine in firefox but IE7 is crashing with an "operation aborted" message be mindful of where your script is in relation to what it's modifying. For the most part however, as long as you keep getFeed in the <head> section you should have no problems of this nature.

Older Browsers

As you might expect, older browsers don't necessarily like the appendChild method so you might want to do a little testing to see if the browser can handle DOM appends

Piper? Ugh!

In the URL we use to call pipes.yahoo.com, the function JSON calls is set as piper thanks to: _callback=piper in the url. You can change piper to be any function name you want. You can have one request call piper, and another request call a more sensibly named _callback=ProcessDiggFeed and that JSON will call the function ProcessDiggFeed(feed). This is one of the most important consideration if you decide to carry more than one feed on your page.

An endless sea of possibilities

I really can't stress just how revolutionary yahoo pipes are. Before pipes we were looking at years if not decades before JSON replaced RSS, now, thanks to yahoo, you can get any feed as JSON data direct from yahoo pipes. This means you can make your own start page ala Google, you can make your own sidebars with the stuff that interests you. Create a recommended sub-menu in your del.icio.us bookmarks and clip interesting sites you find on the net and have them instantly show up on the sidebar of your personal site.

Remember, RSS isn't just news anymore, it's a resource able to give you the Weather, Sports scores, even search results. For instance if you're interested in model trains you can enter model trains on many search engines and get an RSS feed that will track whenever new resources are added to the list (Digg is especially useful here). You can show the last few pictures you uploaded to Flickr, or the latest free books uploaded to Project Gutenberg.

The list is pretty much endless. This revolution quite simply makes the entire internet a resource you can draw on to improve your blog.

Kent Brewster already created a badging service to turn any feed into a badge you can drop on your site, this is just the first of many new and unexpected twists and turns that pipes affords. It's not often you can point to something on the web as an increment in its version counter but if we were on web 2.0 before, thanks to pipes we're at least 2.5 now, if not 3.

As you begin playing around with server-less RSS feeds, directly integrated into your web pages I think you'll come to agree with that assessment.