web talk

OKN Yahoo Pipes presentation

At the last Open Knowledge Network event on Friday 30th Jan 2009, I (Mark McAulay) gave a brief presentation on Yahoo Pipes. I promised to post the presentation for reference so here it is.

Introduction to Yahoo Pipes

What is/are Yahoo Pipes?

According to the Yahoo website, “Pipes is a powerful composition tool to aggregate, manipulate, and mashup content from around the web.”

Launched in Febuary of 2007, there was a good bit of initial excitement that surrounded the launch, and although it’s grown quite a bit in terms of functionality, for one reason or another it’s pretty much remained a “i must get round to having a look at that” item for most web people which I think is a real shame. Hopefully, for some of you, this talk will give you that “must get round to” moment and furnish you with a basic overview of what it’s all about.

If I were to tell you that you could generate dynamic content including, text, images, video and audio on-the-fly without having to do any real programming, would that interest you?

If you want to aggregate content from various sources and use that content for another purpose then pipes is well worth a look. Another way to think about this was brought up on readwriteweb.com where it was concluded that with pipes, “the web essentially becomes a giant database that can be queried and remixed in any number of ways”.

What kinds of things can you do with pipes?

Well, there are a lot of things you can do with pipes, I’ve discovered several things myself that I don’t yet know about while preparing to give this talk. As this is an introduction, we’ll cover the basics and let those of you with an interest discover the rest for yourselves later.

  1. Display an RSS feed’s items on your web page without any programming using a “pipes badge”
  2. Create your own custom RSS feed pulling data from various sources, filter it and output more relevant results to display on a site/app
  3. Create custom search results by mashing up google results, yahoo results and msn results, or indeed any search engine you care to use.
  4. Pipes is not limited to text based results, it’s possible to return images, MP3 tracks (that includes podcasts, yes) and video.
  5. Possible to “clone” existing pipes and tinker with them to alter the output.
  6. You can even make a pipe from other pipes and you can take this as far as you want really.
  7. You can use pipes to power flash applications as well thanks to yahoo setting up an open crossdomain.xml file at pipes.yahooapis.com.

Who’s using pipes + for what

  1. Blondie’s website is using pipes to deal with her news RSS feed.
  2. There’s a US government dept using a pipe for aggregated news (not very interesting though!)
  3. A cycling website makes use of a pipe for their news feed.
  4. Any PHP people will have come across the CakePHP framework, they use pipes for their “super feed” here: http://cakephp.org/planet


To begin with, we’ll look at the yahoo pipes interface. It is a bit daunting when you first look at it but once you’ve dragged your first object onto stage and connected it to something else you soon get the hang of it. The list down the left hand side contains all the objects that are available to you. I’m not going to go through them all in detail, largely because I haven’t used some of them myself yet and also because of the time it would take.


I’ve prepared pipes built by myself and some other pipes created by other people. [Some of the custom pipes used have not been published for a number of reasons...sorry!]

  1. generate a basic RSS feed using the East Coast Interactive feed
  2. more advanced RSS feed – OKN-coverage [not published]
  3. display images by keyword from flickr – Photos from Flickr
  4. display picassa web albums by userID
  5. search engine mashup – search – highlander edition[not published]
  6. display last 3 podcasts from directory (loads of inputs, hence slow load time)
  7. display video from vimeo by tag
  8. multi pipe mashup with okn-coverage and highlander[not published]

In case you’re wondering how to make use of this content, yahoo have made several options available. Route 1 is to use a pipes badge which can be scaled and generated by clicking the “Get as a Badge” button. Depending on which service you want to embed your badge in, there are a few buttons that will give you specific outputs. I’m a more manual guy so I choose the “embed” button to generate me some JavaScript.

Another way to deal with using the output is to have pipes generate an RSS feed of your mashup by clicking on the “Get as RSS” button and being a “do stuff manually kind of guy”, this is the method I used live on the night of the presentation to embed some feed content as html into the OKN website by the power of CFML (you didn’t think I’d miss an opportunity to speak about the mighty ColdFusion markup language did you?) So, in proper “here’s one I made earlier” fashion all we do is use the super handy CFFEED tag to go and read the generated RSS feed and then use a loop to return the items, displaying them in a list.

<cffeed action=”read” source=”http://pipes.yahoo.com/pipes/pipe.run?_id=CKvAv4Ht3RG7trrlpQt1Yg&_render=rss” name=”OKNCoverage” />
<cfloop from=”1″ to=”#ArrayLen(OKNCoverage.item)#” index=”i”>
<cfoutput><li><a href=”#OKNCoverage.item[i].link#” title=”#OKNCoverage.item[i].title#”>#OKNCoverage.item[i].title#</a></li></cfoutput>

Job done…now we have an always up to date list of what’s being said about OKN from many locations on the internet. This can be viewed here

You’ve probably noticed there are more options including the ability to add it to your yahoo or google page, return the data in a JSON format and you can even add it directly to your RSS aggregator of choice. Additionaly you can use yahoo’s alert service to notify you by email or sms when a new item is picked up in your feed giving you the ability to be super nosey. The last item on the list gives you some form of PHP back which may be of interest to some of you. Presumably it generates the php required to output an RSS feed with php on to a page…personaly I’m happy using my 4 lines of CFML but each to their own!


Many interesting things can be done with pipes as you can see. best thing to do is to head over to the website (http://pipes.yahoo.com) and give it a go yourselves, there is a basic tutorial on the homepage and once you’re bored with that, the next step is to hit the “Browse” link and start wading through all the pipes people have built. You can dissect how they work or clone them and re-engineer them to suit your purposes.

- There is an iPhone interface for pipes at http://iphone.pipes.yahoo.com/pipes/

I’ve been speaking to fellow OKN instigator Jack Keenan about doing some kind of pipes/max/msp mashup using particular keywords found in feeds to trigger audio/video etc. When we get that done, we’ll maybe show it at a future OKN. I’d also be interested to hear any other ideas you may have so feel free to comment.

2 comments February 2nd, 2009

Redirect your old files with a 301


Assuming there is a change in structure/page names etc, an often overlooked aspect of re-developing an existing website is to add proper redirects to let users and search engines know where to find the new versions of already bookmarked/indexed pages. The chances are that when the old site has been upgraded, the content on those pages becomes out of date and incorrect fairly quickly, the old pages may also have been completely deleted or in the case of switching to a new hosting setup, have never existed. This means that the next time your users click on that bookmarked link, or when the search engines return to crawl your site again, the pages either return out of date information or even worse, inform both your users and the search engines that they don’t exist. Needless to say, this is not the result you are looking for from your new and lovingly crafted website.


Making sure your users and the search engines remain happy when they request one of your pages is actually very simple, all you need is a list of your old pages, a list of your new locations, a text editor and FTP access to your hosting. The solution is to use a “301 redirect”, this is an Apache specific method and wont work under IIS. (If you are running an IIS server, see the link at the bottom of this post for further details)


Here are the steps to take at the most basic level:

Create a new text (.txt) document and save it as .htaccess

To redirect a single file (or list of single files):

Redirect 301 /old-file.html http://www.shiny-new-website.com/new-file.html

To redirect all of the old pages to your new website:

RedirectMatch 301 (.*) http://www.shiny-new-website.com/

Upload the .htaccess to your website’s root folder (where you’re index page is) and check that when requesting the old URL’s, you are redirected to the new URL’s


Apache  - Apache mod_alias documentation

Using IIS? - IIS 6.0 Server Administration Guide

Add comment January 20th, 2009

jQuery news

We love jQuery at East Coast Interactive, simple as that really.

We use it whenever we need a bit of magic to extend websites for our clients and because of the success we’ve had with it, we look to use it to compliment all of the backend and frontend technologies we work with. First up for the treatment were our php projects, we built our applications as normal and quickly started adding bells and whistles with jQuery…it just seemed so intuitive to work with. Next up we took a look at our ColdFusion projects and although ColdFusion comes with masses of built in JS/AJAX goodness, the code it generates seemed a bit overboard for our liking…jQuery to the rescue again, nice and easy and super lightweight. So that left one area we work with short of some jQuery lovin’…the .NET side of things.

Last night an announcement popped up on the twitter radar that both Microsoft and Nokia were now using jQuery…interested, I went on to read this announcement from the jQuery blog and discovered that not only were Microsoft using jQuery, they were REALLY using jQuery and will be “looking to make jQuery part of their official development platform” and “This means that jQuery will be distributed with Visual Studio (which will include jQuery intellisense, snippets, examples, and documentation)”…fantastic! Graeme should be pleased and it means that we can use a consistant JS library accross all our development…the fact that the library is our favourite one only makes this announcement all the sweeter for us.

Add comment September 29th, 2008

Adobe CS3 Launch webcast

It’s time for the next generation of Adobe creative products to be announced, “2 years + 80 million lines of code” they say, I’ve just watched their live webcast . . some things that caught my eye . .

Flash CS3

  • Coding enhancements including a better debugger and improved editing e.g collapsing
  • Advanced QuickTime export – wahoo! At last, you can use nested and scripted elements within video output, no more arsing about trying to render from director or trying to get a screen cap movie without dropping frames..
  • Convert animation to code – handy for taking a complex path and being able to manipulate programatically.
  • Improved import with proper Illustrator handling
  • New lightweight UI components – flex quality widgets hopefully.

Photoshop CS3

  • As somebody that enjoys photography (in particular panoramas) the ‘align layers’ features is killer, I’ve been using it heavily in the cs3 beta. Chuck a bunch of photos in, 30 seconds later you have an (almost) perfectly aligned set of images just needing slight ‘crossover’ adjustments using dodge and burn tools to match exposures , and some deft eraser work to fade any glitches.
  • Faster startup
  • Video frames to layers – plus these can then be edited in a ’stack’.

Dreamweaver CS3

  • AJAX widgets built in
  • ‘Check browser compatibility’ feature – flags up common problems in certain browsers

Aftereffects CS3

  • puppet tool: chop up and animate images easily with inverse kinematics type linking of elements
  • swf integration
  • photoshop cs3 vanishing point manipulated image to animatable 3d model in aftereffects

Encore CS3

  • Output a dvd project directly to web (with all menus + video!) as a swf

The webcast was also a great example of why it’s a good idea to have a backup Plan B when presenting a live broadcast, as when their projector failed.. then they tried to do an impromptu q & a to the gathered crowd to try and fill the down time.. damn they looked awkward and ill-prepared as they struggled with the answers lol.

Shame the downtime cut back on the full program of announcements when they resumed, but details will be out soon as they all go on sale in the next few weeks.

Add comment March 27th, 2007

Useful online tools for the web developer pt1

Colour scheme creator : colortoy 2.o

Flow chart and diagram creation and sharing: Gliffy

Online drawing : sketchr (vector) and fauxto (bitmap)

Online video conversion: flvtomp3 (for flash media server people pissed at nelly moser audio), mux and the api-tastic and rapidly evolving heywatch!

1 comment March 20th, 2007


March 2019
« Feb    


Most Recent Posts