Introduction to Yahoo Pipes

What is/are Yahoo Pipes?

According to the Yahoo website, “Pipes is a powerful composition tool to aggregate, manipulate, and mashup content from around the web.”

Launched in Febuary of 2007, there was a good bit of initial excitement that surrounded the launch, and although it’s grown quite a bit in terms of functionality, for one reason or another it’s pretty much remained a “i must get round to having a look at that” item for most web people which I think is a real shame. Hopefully, for some of you, this talk will give you that “must get round to” moment and furnish you with a basic overview of what it’s all about.

If I were to tell you that you could generate dynamic content including, text, images, video and audio on-the-fly without having to do any real programming, would that interest you?

If you want to aggregate content from various sources and use that content for another purpose then pipes is well worth a look. Another way to think about this was brought up on readwriteweb.com where it was concluded that with pipes, “the web essentially becomes a giant database that can be queried and remixed in any number of ways”.

What kinds of things can you do with pipes?

Well, there are a lot of things you can do with pipes, I’ve discovered several things myself that I don’t yet know about while preparing to give this talk. As this is an introduction, we’ll cover the basics and let those of you with an interest discover the rest for yourselves later.

1. Display an RSS feed’s items on your web page without any programming using a “pipes badge”
2. Create your own custom RSS feed pulling data from various sources, filter it and output more relevant results to display on a site/app
3. Create custom search results by mashing up google results, yahoo results and msn results, or indeed any search engine you care to use.
4. Pipes is not limited to text based results, it’s possible to return images, MP3 tracks (that includes podcasts, yes) and video.
5. Possible to “clone” existing pipes and tinker with them to alter the output.
6. You can even make a pipe from other pipes and you can take this as far as you want really.
7. You can use pipes to power flash applications as well thanks to yahoo setting up an open crossdomain.xml file at pipes.yahooapis.com.

Who’s using pipes + for what

1. Blondie’s website is using pipes to deal with her news RSS feed.
2. There’s a US government dept using a pipe for aggregated news (not very interesting though!)
3. A cycling website makes use of a pipe for their news feed.
4. Any PHP people will have come across the CakePHP framework, they use pipes for their “super feed” here: http://cakephp.org/planet

Examples

To begin with, we’ll look at the yahoo pipes interface. It is a bit daunting when you first look at it but once you’ve dragged your first object onto stage and connected it to something else you soon get the hang of it. The list down the left hand side contains all the objects that are available to you. I’m not going to go through them all in detail, largely because I haven’t used some of them myself yet and also because of the time it would take.

pipes.yahoo.com

I’ve prepared pipes built by myself and some other pipes created by other people. [Some of the custom pipes used have not been published for a number of reasons...sorry!]

1. generate a basic RSS feed using the East Coast Interactive feed
2. more advanced RSS feed – OKN-coverage [not published]
3. display images by keyword from flickr – Photos from Flickr
4. display picassa web albums by userID
5. search engine mashup – search – highlander edition[not published]
6. display last 3 podcasts from directory (loads of inputs, hence slow load time)
7. display video from vimeo by tag
8. multi pipe mashup with okn-coverage and highlander[not published]

In case you’re wondering how to make use of this content, yahoo have made several options available. Route 1 is to use a pipes badge which can be scaled and generated by clicking the “Get as a Badge” button. Depending on which service you want to embed your badge in, there are a few buttons that will give you specific outputs. I’m a more manual guy so I choose the “embed” button to generate me some JavaScript.

Another way to deal with using the output is to have pipes generate an RSS feed of your mashup by clicking on the “Get as RSS” button and being a “do stuff manually kind of guy”, this is the method I used live on the night of the presentation to embed some feed content as html into the OKN website by the power of CFML (you didn’t think I’d miss an opportunity to speak about the mighty ColdFusion markup language did you?) So, in proper “here’s one I made earlier” fashion all we do is use the super handy CFFEED tag to go and read the generated RSS feed and then use a loop to return the items, displaying them in a list.

<cffeed action=”read” source=”http://pipes.yahoo.com/pipes/pipe.run?_id=CKvAv4Ht3RG7trrlpQt1Yg&_render=rss” name=”OKNCoverage” />
<ol>
<cfloop from=”1″ to=”#ArrayLen(OKNCoverage.item)#” index=”i”>
<cfoutput><li><a href=”#OKNCoverage.item[i].link#” title=”#OKNCoverage.item[i].title#”>#OKNCoverage.item[i].title#</a></li></cfoutput>
</cfloop>
</ol>

Job done…now we have an always up to date list of what’s being said about OKN from many locations on the internet. This can be viewed here

You’ve probably noticed there are more options including the ability to add it to your yahoo or google page, return the data in a JSON format and you can even add it directly to your RSS aggregator of choice. Additionaly you can use yahoo’s alert service to notify you by email or sms when a new item is picked up in your feed giving you the ability to be super nosey. The last item on the list gives you some form of PHP back which may be of interest to some of you. Presumably it generates the php required to output an RSS feed with php on to a page…personaly I’m happy using my 4 lines of CFML but each to their own!

Summary

Many interesting things can be done with pipes as you can see. best thing to do is to head over to the website (http://pipes.yahoo.com) and give it a go yourselves, there is a basic tutorial on the homepage and once you’re bored with that, the next step is to hit the “Browse” link and start wading through all the pipes people have built. You can dissect how they work or clone them and re-engineer them to suit your purposes.

- There is an iPhone interface for pipes at http://iphone.pipes.yahoo.com/pipes/

I’ve been speaking to fellow OKN instigator Jack Keenan about doing some kind of pipes/max/msp mashup using particular keywords found in feeds to trigger audio/video etc. When we get that done, we’ll maybe show it at a future OKN. I’d also be interested to hear any other ideas you may have so feel free to comment.

Assuming there is a change in structure/page names etc, an often overlooked aspect of re-developing an existing website is to add proper redirects to let users and search engines know where to find the new versions of already bookmarked/indexed pages. The chances are that when the old site has been upgraded, the content on those pages becomes out of date and incorrect fairly quickly, the old pages may also have been completely deleted or in the case of switching to a new hosting setup, have never existed. This means that the next time your users click on that bookmarked link, or when the search engines return to crawl your site again, the pages either return out of date information or even worse, inform both your users and the search engines that they don’t exist. Needless to say, this is not the result you are looking for from your new and lovingly crafted website.

Solution

Making sure your users and the search engines remain happy when they request one of your pages is actually very simple, all you need is a list of your old pages, a list of your new locations, a text editor and FTP access to your hosting. The solution is to use a “301 redirect”, this is an Apache specific method and wont work under IIS. (If you are running an IIS server, see the link at the bottom of this post for further details)

Examples

Here are the steps to take at the most basic level:

Create a new text (.txt) document and save it as .htaccess

To redirect a single file (or list of single files):

Redirect 301 /old-file.html http://www.shiny-new-website.com/new-file.html

To redirect all of the old pages to your new website:

RedirectMatch 301 (.*) http://www.shiny-new-website.com/

Upload the .htaccess to your website’s root folder (where you’re index page is) and check that when requesting the old URL’s, you are redirected to the new URL’s

Links

Apache  - Apache mod_alias documentation

Using IIS? - IIS 6.0 Server Administration Guide

www.geoffruncie.com has just been launched by East Coast Interactive.

Constructed with XHTML/CSS on the front end, this new website for successful businessman Geoff Runcie also enjoys a robust content management system allowing Geoff to keep all data relating to his career and achievements up to date with minimal fuss. An online email, calendar and document management solution was implemented meaning that Geoff now has his full portfolio of data and document assets with him wherever an internet connection is available.

This website serves as an online Curriculum Vitae and was complimented with a 4 page printed prospectus and matching business cards.

http://www.geoffruncie.com

With visitors from more than 30 countries, over 70,000 plays to date, and front page “feature” exposure on some of the biggest online Flash game websites around, the East Coast Interactive sledging game has proven to be a bit of a hit over the 2008 festive period.

Overview

Super Sledge Challenge started life as an idea near the end of November 2008 as a virtual Christmas card for East Coast Interactive to send to their clients. The idea was both simple and seasonal – To sledge down a hill as quickly as possible while dodging trees and the edges of the course while picking up presents for time bonuses. Like a lot of ideas ECI have, it quickly grew wings. While the primary function of being an interactive e-card remained very much intact, it also developed into a fully fledged online flash game with a high score table. To “go that extra mile” the ability to customize the game to add a personalised festive message was added along with a “send to a friend” feature.

Created to run on the Flash platform, the online game development technology of choice at East Coast Interactive, a number of technical options were investigated and implemented to add that extra level of playability, with a 3D library being used to give a more realistic feeling of space and motion and custom sound effects recorded to add to the atmosphere.

Exposure

After wishing our clients a Merry Christmas, we released Super Sledge Challenge as a publically available Flash online game. The Community Manager for 4mations (Channel 4, Aardman Animations and Lupus Films) was seeking out festive themed games for the 4mations website and featured it on their front  page. The next port of call was newgrounds (”the original – and still the largest – Flash portal on the web”) who accepted Super Sledge Challenge to run on their site and finally the game was submitted to Kongregate. From here the game went viral, being picked up by other gaming websites and being mentioned on the pages of Digg, the feed of Twitter and even from word of mouth, the play count has continued to rise hourly.

Building an interactive online christmas card is pretty much a given for any digital media agency these days, what East Coast Interactive set out to achieve initially was no different from any other company at that particular time of year. What ECI have shown is that with a bit of extra drive, innovation and effort, it is not only achievable to get your work in front of some of the biggest names in the industry but also to get it out to a large and varied audience globally.

Super Sledge Challenge was launched mid December 2008 and (at the time of writing) continues to be very well played.

http://www.supersledgechallenge.com

To celebrate the festive season this year we’ve created a fun e-card with a flash game. The only impact on trees is virtual (if your steering isn’t very good). Our Christmas game is a 3d sledging challenge where you need to get to the bottom of the slope as quickly as possible, by avoiding the trees and picking up presents for time bonuses on route.

To play the game go here: http://festive.eastcoastinteractive.co.uk/

You can also create your own free custom version too for your friends which will display your own Christmas greeting.

We have received a lot of applications since we’ve been on the go and we’ve formed some opinions about what we like and don’t like over this time so have decided to post this article as a guide for all future applicants…it’s not a list of rules, it’s not a definitive list by any stretch of the imagination, it’s simply a point of reference for designers/developers who want to join our team and have done their homework a bit. The biggest mistakes we see are the following:

1. The applicant has not researched us in depth. Specifically, who the point of contact is, who’s currently on the team and what it is we actually do (yes, really!)
2. Bad spelling and grammar in covering letter/email/CV
3. No online examples of work or immediately available alternatives

Now, this post doesn’t exist only to make you aware of these things…nope, we’re cooler than that. This post is here to give you a little push in the right direction. We’re not horrible people, if you’re determined to work for us, we have no problem advising you on how best to make an impression.

Our number one grumble as listed above is people not researching us properly so to immediately give us an indication of your ambition to join ECI, we reckon that anyone who’s 100% up for the challenge will have been through our website with a fine tooth comb and will have come across this very post. So to answer the questions you will have from point 1, the point of contact for all applications is Stuart Ebdy who is the managing director, on the team at the moment we have Stuart, Mike Duguid, Graeme Benzie and Mark McAulay and to find out what we do, you need to have a look at our folio of work. That’s item 1 taken care of then!

Bad spelling and grammar on an application is pretty much unacceptable in this day and age. Word processors, email clients and even the Firefox browser come with spell checking so to avoid giving us a bad impression, use it. As for grammar, why not have a couple of people you trust read over what you intend to send us? It’s amazing how many little errors a fresh pair of eyes will find and it’ll give you peace of mind that what you are sending is as good as it can be, or at least it’ll give you someone to blame

Absence of anything to show us about your visual/technical skills is a quick way to raise doubt in our minds. A suggestion we regularly give is that if you don’t have an online folio of work, or any other work you’ve done on the web, send us several links of sites/designs/apps that you like with a brief paragraph about why you like them and also any work done by other designers that you aspire to. In the absence of your own work, this will give us a good idea of what you’re all about.

In general, what we want to see is imagination, creativity, integrity and relevance to East Coast Interactive, what are you going to bring to the party, what ideas do you have, what do you think we could do differently/better? How ambitious are you? Are you in for the long haul or are you just testing the water to see if you like it? There are no right or wrong answers to any of these questions, all we ask for is honesty…be straight up with us and we’ll do the same for you.

So, hope this has been a helpful little nugget of information to find and we look forward to your well researched and correctly spelled application!

www.firstphotographics.co.uk has been upgraded to include a range of new features and e-commerce facilities.

Customers can now easily preview and purchase their images of private portraits, events and property photographs online. The site also allows customers to upload their own photos and select from a range of restoration, re-touching and stylising options to produce prints and wall art.

The bespoke shopping cart system and administrative back end was developed using PHP and MySQL. JQuery, Lightbox and Flash have been used throughout the site to add extra flare to the presentation of images.

www.firstphotographics.co.uk

To compliment the new site, a new flash based magazine reader was lovingly assembled and rolled out in tandem with the new site.

A new look venue, new look website and new look magazine viewer…those Snafu folks don’t mess about.

www.clubsnafu.com

www.snafumagazine.com

We love jQuery at East Coast Interactive, simple as that really.

We use it whenever we need a bit of magic to extend websites for our clients and because of the success we’ve had with it, we look to use it to compliment all of the backend and frontend technologies we work with. First up for the treatment were our php projects, we built our applications as normal and quickly started adding bells and whistles with jQuery…it just seemed so intuitive to work with. Next up we took a look at our ColdFusion projects and although ColdFusion comes with masses of built in JS/AJAX goodness, the code it generates seemed a bit overboard for our liking…jQuery to the rescue again, nice and easy and super lightweight. So that left one area we work with short of some jQuery lovin’…the .NET side of things.

Last night an announcement popped up on the twitter radar that both Microsoft and Nokia were now using jQuery…interested, I went on to read this announcement from the jQuery blog and discovered that not only were Microsoft using jQuery, they were REALLY using jQuery and will be “looking to make jQuery part of their official development platform” and “This means that jQuery will be distributed with Visual Studio (which will include jQuery intellisense, snippets, examples, and documentation)”…fantastic! Graeme should be pleased and it means that we can use a consistant JS library accross all our development…the fact that the library is our favourite one only makes this announcement all the sweeter for us.

www.mansellhomes.co.uk