OKN Yahoo Pipes presentation

February 2nd, 2009

At the last Open Knowledge Network event on Friday 30th Jan 2009, I (Mark McAulay) gave a brief presentation on Yahoo Pipes. I promised to post the presentation for reference so here it is.

Introduction to Yahoo Pipes

What is/are Yahoo Pipes?

According to the Yahoo website, “Pipes is a powerful composition tool to aggregate, manipulate, and mashup content from around the web.”

Launched in Febuary of 2007, there was a good bit of initial excitement that surrounded the launch, and although it’s grown quite a bit in terms of functionality, for one reason or another it’s pretty much remained a “i must get round to having a look at that” item for most web people which I think is a real shame. Hopefully, for some of you, this talk will give you that “must get round to” moment and furnish you with a basic overview of what it’s all about.

If I were to tell you that you could generate dynamic content including, text, images, video and audio on-the-fly without having to do any real programming, would that interest you?

If you want to aggregate content from various sources and use that content for another purpose then pipes is well worth a look. Another way to think about this was brought up on readwriteweb.com where it was concluded that with pipes, “the web essentially becomes a giant database that can be queried and remixed in any number of ways”.

What kinds of things can you do with pipes?

Well, there are a lot of things you can do with pipes, I’ve discovered several things myself that I don’t yet know about while preparing to give this talk. As this is an introduction, we’ll cover the basics and let those of you with an interest discover the rest for yourselves later.

  1. Display an RSS feed’s items on your web page without any programming using a “pipes badge”
  2. Create your own custom RSS feed pulling data from various sources, filter it and output more relevant results to display on a site/app
  3. Create custom search results by mashing up google results, yahoo results and msn results, or indeed any search engine you care to use.
  4. Pipes is not limited to text based results, it’s possible to return images, MP3 tracks (that includes podcasts, yes) and video.
  5. Possible to “clone” existing pipes and tinker with them to alter the output.
  6. You can even make a pipe from other pipes and you can take this as far as you want really.
  7. You can use pipes to power flash applications as well thanks to yahoo setting up an open crossdomain.xml file at pipes.yahooapis.com.

Who’s using pipes + for what

  1. Blondie’s website is using pipes to deal with her news RSS feed.
  2. There’s a US government dept using a pipe for aggregated news (not very interesting though!)
  3. A cycling website makes use of a pipe for their news feed.
  4. Any PHP people will have come across the CakePHP framework, they use pipes for their “super feed” here: http://cakephp.org/planet


To begin with, we’ll look at the yahoo pipes interface. It is a bit daunting when you first look at it but once you’ve dragged your first object onto stage and connected it to something else you soon get the hang of it. The list down the left hand side contains all the objects that are available to you. I’m not going to go through them all in detail, largely because I haven’t used some of them myself yet and also because of the time it would take.


I’ve prepared pipes built by myself and some other pipes created by other people. [Some of the custom pipes used have not been published for a number of reasons...sorry!]

  1. generate a basic RSS feed using the East Coast Interactive feed
  2. more advanced RSS feed – OKN-coverage [not published]
  3. display images by keyword from flickr – Photos from Flickr
  4. display picassa web albums by userID
  5. search engine mashup – search – highlander edition[not published]
  6. display last 3 podcasts from directory (loads of inputs, hence slow load time)
  7. display video from vimeo by tag
  8. multi pipe mashup with okn-coverage and highlander[not published]

In case you’re wondering how to make use of this content, yahoo have made several options available. Route 1 is to use a pipes badge which can be scaled and generated by clicking the “Get as a Badge” button. Depending on which service you want to embed your badge in, there are a few buttons that will give you specific outputs. I’m a more manual guy so I choose the “embed” button to generate me some JavaScript.

Another way to deal with using the output is to have pipes generate an RSS feed of your mashup by clicking on the “Get as RSS” button and being a “do stuff manually kind of guy”, this is the method I used live on the night of the presentation to embed some feed content as html into the OKN website by the power of CFML (you didn’t think I’d miss an opportunity to speak about the mighty ColdFusion markup language did you?) So, in proper “here’s one I made earlier” fashion all we do is use the super handy CFFEED tag to go and read the generated RSS feed and then use a loop to return the items, displaying them in a list.

<cffeed action=”read” source=”http://pipes.yahoo.com/pipes/pipe.run?_id=CKvAv4Ht3RG7trrlpQt1Yg&_render=rss” name=”OKNCoverage” />
<cfloop from=”1″ to=”#ArrayLen(OKNCoverage.item)#” index=”i”>
<cfoutput><li><a href=”#OKNCoverage.item[i].link#” title=”#OKNCoverage.item[i].title#”>#OKNCoverage.item[i].title#</a></li></cfoutput>

Job done…now we have an always up to date list of what’s being said about OKN from many locations on the internet. This can be viewed here

You’ve probably noticed there are more options including the ability to add it to your yahoo or google page, return the data in a JSON format and you can even add it directly to your RSS aggregator of choice. Additionaly you can use yahoo’s alert service to notify you by email or sms when a new item is picked up in your feed giving you the ability to be super nosey. The last item on the list gives you some form of PHP back which may be of interest to some of you. Presumably it generates the php required to output an RSS feed with php on to a page…personaly I’m happy using my 4 lines of CFML but each to their own!


Many interesting things can be done with pipes as you can see. best thing to do is to head over to the website (http://pipes.yahoo.com) and give it a go yourselves, there is a basic tutorial on the homepage and once you’re bored with that, the next step is to hit the “Browse” link and start wading through all the pipes people have built. You can dissect how they work or clone them and re-engineer them to suit your purposes.

- There is an iPhone interface for pipes at http://iphone.pipes.yahoo.com/pipes/

I’ve been speaking to fellow OKN instigator Jack Keenan about doing some kind of pipes/max/msp mashup using particular keywords found in feeds to trigger audio/video etc. When we get that done, we’ll maybe show it at a future OKN. I’d also be interested to hear any other ideas you may have so feel free to comment.

Entry Filed under: ECI, presentations, web talk

2 Comments Add your own

  • 1. Phil Thompson  |  February 11th, 2009 at 4:46 pm

    Hey Mark,
    I’m still getting to grips with how I can get the most out of pipes…I was so glad to see your presentation at the open Knowledge network – it came at just the right time…I’ve only just started to get my head around the power / flexibility of RSS syndication – and now this is going to be perfect to replicate multiple feeds in various places…

    As you know, I work for Creative Cultures – and I think this might be the perfect way for me to easily apply many dynamic inputs into our now aging site! (my spring clean is long overdue!)

    thanks so much – if there is anything Creative Cultures can do for you, please just give me a tinkle! info@creativecultures.co.uk

    yours, Phil.

  • 2. Mark  |  February 25th, 2009 at 10:54 am

    Hey Phil,
    Thanks for the feedback. Glad to hear that people are getting something from the talk. I tried emailing you but got bounced. Will be in touch soon enough i’d imagine though.
    once again, thanks for the kind words.

Leave a Comment


Required, hidden

Some HTML allowed:
<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

Trackback this post  |  Subscribe to the comments via RSS Feed


March 2019
« Feb    


Most Recent Posts