made-to-order content with yahoo pipes · yahoo pipes is a particularly interesting utility that...

Made-to-order content with Yahoo PipesCombine and rework content feeds to suit your taste

Skill Level: Introductory

Jeff K. Wilson ([email protected])e-business architectIBM

05 Jun 2007

Explore the steps and benefits of using a content feed filtering utility like Yahoo Pipesto better capture, merge, and alter specific data from available streams. This tutorialoutlines some techniques to approach feed transformations, and includes threedemonstrations featuring key areas of the environment.

Section 1. Before you start

About this tutorial

This tutorial provides a basic understanding of Yahoo Pipes, a service that filters,transforms, and aggregates content feeds. A complete run-through of the toolsetavailable is included, as are three demonstrations that illustrate the capabilities andfeatures of the service. Finally, step-by-step instructions for one of thedemonstrations complete the tutorial, along with a recorded movie showing the stepsin detail.

Objectives

The intent of this tutorial is simply to describe the Yahoo Pipes service for managingavailable content feeds, focusing on the features, tools, and capabilities available.

Prerequisites and system requirements

Made-to-order content with Yahoo Pipes© Copyright IBM Corporation 1994, 2008. All rights reserved. of 28

mailto:[email protected]

http://www.ibm.com/legal/copytrade.shtml

No prerequisites are necessary for this tutorial. All functionality is available online.

Section 2. Introduction

Recently, I've had the opportunity to become immersed in various Web 2.0technologies, investigating the effect they might have on application development.The concept of Web 2.0 means different things to different people, but one tenet thatstands out is its impact on social computing, which takes many forms. What'sinteresting is that, as the population becomes more comfortable with onlineinteractions, you all become less reliant on user interfaces (UI) that give you thespecific information you happen to be looking for and more comfortable simplyasking for just the information itself—allowing you to display or use it however youwant.

Enter content feeds stage right: information without an interface.

This effect of these feeds on not only developers but also users is clear. Portals, orgateways to information, are more and more prevalent, and are frequentlyimplemented as personal homepages where users are given the tools to displayinformation. These tools allow the user to decide what data to display on the page.

For developers, the impact is on various tools that allow them to expose or use datain standardized, generic ways that, well, feed into these new flexible UIenvironments.

Most significantly, developers can now be set free from the limitations that forcedthem to be both highly skilled technicians and subject matter experts. Traditionally, alone programmer might require knowledge of interface technologies and also needto personally maintain the information that would be used in the application he orshe was building. Today, with structured data widely available, should I decide on awhim to build the next great personals community, virtual car dealership, orcomputer reseller site, I can do so from my kitchen table.

About Yahoo Pipes

Now, I have one small problem: if my newJeffsWayCoolEventCoordinatorPortal.com site relies on someone else's streamingdata, how do I strip what I want from what they deliver? Certainly, streams abound,and I frequently can do some limited filtering with them, but rarely—if ever—can Imanipulate the data as I typically can on a local data source.

For example, what if I want to get a feed of particular products, but only in aparticular color? Or if I need the items sorted in a specific way? Or if, heaven forbid, Ineed to merge data from multiple sources?

developerWorks® ibm.com/developerWorks

Made-to-order content with Yahoo Pipes of 28 © Copyright IBM Corporation 1994, 2008. All rights reserved.


Enter Yahoo Pipes stage left: no information, no interface, but the tools necessary totweak what goes between the two.

Yahoo Pipes is a particularly interesting utility that sits between an application and asource of data. This service allows developers to transform, fine-tune, and otherwiseexperiment with content before pulling it into an application.

It is a simple but powerful mechanism that saves developers much work when theybuild an internal parsing and editing engine within their applications.

I recorded the steps outlined in this tutorial in a Flash movie; you can find it inConclusion. You might want to check it out if you can't spend the time implementingyour own pipes.

Section 3. Yahoo Pipes: A tour of the facilities

Slick and exciting

To begin with, let me show you around the tooling.

To be perfectly honest, the developers of this environment really did an interestingjob pulling together an altogether user-friendly and appealing interface.

The ultimate concept is to pull content into the tool, pass it through a series of stepsthat perform some kind of work on it, and finally output a new stream. This processcertainly lends itself to a visually oriented tool like this, one that lays out the steps ina flow diagram. The ability to easily drag and drop components, set attributes usingforms embedded in these components, and draw connections from the outputs ofone to the input of another is extremely powerful in its simplicity.

The basic environment

The general tooling makes use of an editing grid and component palette, illustratedin Figure 1.

Figure 1. Drag and drop palette and editor

ibm.com/developerWorks developerWorks®



As you might expect, you drag a component of interest from the palette and drop itonto the editor grid. To connect the components into a flow, you simply draw a pipefrom the output of one to the input of another, as shown in Figure 2. Pretty simple.

Figure 2. Drag and drop palette and editor with defined flow

In addition, there is an information pane and a debugger/content view, both of whichcome in pretty handy.

Selecting a component in the palette displays a short description of the componentas well as a link where you can see an example of it in use. Similarly, selecting acomponent in the main editor shows a live view of the effect on the content in thedebugger. As a user selects the different components in the transformation flow, thedebugger shows the updated feed content. Figure 3 shows a subset of the original




feed based on the rule described in the filter component.

Figure 3. Information and debugger panes

Sources

You need a feed of some sort to begin. For this purpose, you can either use anactual RSS stream or point to sites like Google or Yahoo using the sourcecomponents supplied in the palette. The latter add a significant amount of power tothe regenerated content, since clearly not everything online is available throughsome identified content feed—sometimes it's nice to just look something up. Thus, ifyou wanted to ping Google Base to search for jobs of a specific type, or (as shown inFigure 4) use Yahoo Local to hunt for certain kinds of events in a given location, youwould not need to know about a specific feed that provides that information.

Figure 4. Yahoo Local source example




You can just point directly to the URL of an existing content feed, as shown in Figure5.

Figure 5. Using the Fetch component to point to a specific content stream

Once the initial source is defined, content is pulled into the tooling where othercomponents can begin to rework, analyze, and transform the data into somethingnew and more useful.

Operators

Most of the basic operations come from the Operators grouping of components.These operators provide capabilities that range from filtering and sorting to loopsand inputting regular expressions, as well as specific functions like languagetransformations and the extraction of usable location details.

Figures 6 and 7 show a filter placed after the original content is pulled into the pipe.The filter limits what is displayed, showing only items whose title contains the text"Wii."

Figure 6. Adding a Wii filter to a stream (debugger showing original RSScontent before filter)




Figure 7. Adding a Wii filter to a stream (debugger showing filtered RSScontent after filter)




As you can see in the screenshots, you can implement several other operators inyour pipe. To describe a few:

• For Each: Annotation: Provides a means to loop through each item in astream, adding a new node that contains the output of another stream.

• For Each: Replace: Provides a means to loop through items in a feedwhere the output of another feed replaces the item in the original feed.

In both of the For Each components, attributes of the original item can befed into the secondary stream. For example, a job bank feed containingopen positions might loop, passing keywords for each item into a newfeed that responds with resumes containing those keywords.

• Rename: Allows you to rename attributes.

• Split: Divides up streams based on some criteria.

• Count: Extracts the number of items returned. you can use it to set anumber field elsewhere.

• Truncate: Limits how many items are allowed to pass through.




• Content Analysis: Analyzes the content of an item, looking for significantdescriptive information.

• Sort: Sorts items.

• Regex: Processes with regular expressions.

• BabelFish: Translates into or out of various languages.

• Location Extractor: Determines an address by looking for a variety oflocation formats in a body of text.

• Unions: Combines streams. Commonly used after a split, where differingprocessing occurs based on some criteria and then the parts are rejoined.

• Unique: Limits repetitious items based on a given attribute.

User inputs

Other key components in the palette include the user input tools. For most inputs invarious components, you can manually define or build fields to take dynamic data.This dynamic data, though labeled as a user input, is not limited to users and issimply an input supplied by something defined elsewhere.

If a user implements the pipe from the Yahoo Pipes interface, then it is in fact a userinput and the interface in question dynamically builds and displays the required formcomponents on the fly. However, as you will see in the last demo in this tutorial, auser input field might be populated by some other part of the processing flow.

In any event, these inputs allow a translation flow to generate output based on someflexible piece of information and extend its flexibility. For example, you might connectto a technical journal feed using a filter to extract a subset of items based on someuser-defined criteria. In one implementation, a user might pass in "digital cameras"one day and "printers" the next—both extracting only the items that match thoseterms. In another implementation, the dynamic input might pass in from keywordsextracted from another "new product" feed, only showing the technical journal'sitems of recently released products, whatever they might be.

Figure 8 illustrates how you'd define the text you want to use as a filter on a contentstream. More specifically, the filtering text is not hard-coded in the filter component;in this example, it is supplied in the form and submitted, and that data is streamedinto the filter component's text field. In addition, as you will see in the final demo, youcan also tie some available property to the input field component. This way thedynamic data is determined by what else is being processed—not something directly




entered from a user.

Figure 8. Attaching a user input component to a filter

Tying the input term to a specific field is as easy as dragging the connector dot fromthe input dialog to the connector dot of the field in question. You can see the varioustypes of user input components in Figure 8.

Other components

In addition to the URL, String, and Date components for further processing on thosetypes, Yahoo Pipes offers another very powerful feature: the ability to layer pipesfrom the My Pipes section.

My Pipes displays all the processes you have developed that you can nest in otherprocess flows. This permits a certain level of reusability and componentization.Anything built with user inputs becomes a potential place for an attribute to be pipedin from other feeds that you use.

In Figure 9, a certain process is defined to search for antiques on Craigslist. Theprocess exposes one dynamically identified search term, labeled query. The inputfield allows a user—or, in this case, another feed—to pass specific terms into theantique process flow. You can manually set this query to a specific term like "table"or "buffet," or pipe it in from elsewhere, as in this example. Figure 9 illustratesmerging a New York Times feed on furniture and items available on Craigslist basedon keywords that overlap between the two data sources. The keywords are pulledfrom the New York Times feed using the Content Analysis component, whichpopulates an attribute labeled y:content_anaysis in the original feed. You cansee this term identified in the For Each loop as what ought to be used for the queryfield. This For Each: Annotate loop creates a new attribute calledstuff_on_craigslist on each item of the original feed and populates it with thereturn of the antique sub-process.




Figure 9. Using attributes of one feed as input for another nested feed

Conclusion

With its draggable interface, the Yahoo Pipes tool itself is very easy to use. It alsoincludes many of the basic transformation utilities required to accomplish the bulk ofwhat you might think to use this service for. In the coming sections, you will walkthrough building some demo transformations, all of which can be accessed, run, and(if you save your own copy) edited online.

Section 4. Demo 1: Looking for trucks in Austin

Demo overview

You can access this pipe yourself at the following URL:http://pipes.yahoo.com/pipes/pipe.info?_id=Tsxjv_nX2xGcxwfnp2IyXQ

Note that anyone can view and run the demonstration pipes that I link to in this



http://pipes.yahoo.com/pipes/pipe.info?_id=Tsxjv_nX2xGcxwfnp2IyXQ


tutorial. If you sign up for a free Yahoo account, you can also import, edit, and clonethese examples.

In the first demonstration, you'll learn the basic process to set up a pipe. In thisexample, you will simply create a feed that displays ads for vehicles for sale that fallinto certain categories you define. You can pull this feed into a reader orportal-based home page.

If you view the pipe run from the link above, to see how I made this pipe, click on thegraphic representing the flow diagram at the lower left of the page, as shown inFigure 10.

Figure 10. Click the flow diagram to edit a copy of the pipe

You can edit the existing flow if you save it to your own profile, but I will describehow to build the flow from scratch in the next part of this section.

Building the flow

1. To begin with, click Create pipe, or if you're looking at how this pipe wasmade in the editor, click New. This will give you a blank editor to workwith. (You might need to log in with a free Yahoo account before you cando this.)

2. Expand Sources if it hasn't already been expanded and drag the GoogleBase component onto the editor, as shown in Figure 11.Figure 11. Drag the Google Base component onto the editor




3. From the Find drop-down menu, select vehicles.

4. In the keyword field, enter truck.

5. From the within drop-down menu, select 20 miles.

6. In the location field, enter a zip code. Your component should now looklike Figure 12.Figure 12. Enter search criteria

7. Expand the Operators group and drag the Sort component under theGoogle Base component.

8. To draw a connection from Google Base to Sort, drag the dot at thebottom of Google Base to the dot at the top of Sort. Your screen shouldlook like Figure 13.Figure 13. Add a sort

9. In the sorting drop-down, select g:condition.




10. Click the plus sign next to Sort by to add another sort category, selectingg:year.

11. Make sure the orders for both are set to ascending. Your screen shouldnow look like Figure 14.Figure 14. Add sorting criteria

12. From the palette, drag a Filter component to the editor.

13. To connect the Sort to the Filter, drag the Sort output dot to the Filterinput dot.

14. Set the selections to Permit items that match all of the following.

15. Select the rule categories g:mileage and is less than, and enter100000, as in Figure 15.Figure 15. Add filtering criteria

16. From the palette, drag another filter onto the editor below the first filter.

17. Connect the two filters' output to the input, as before.

18. This time, select Permit again but choose any of the following.

19. Click the plus sign next to Rules .

20. Design both rules to read g:color and Contains but enter black for one




and red for the other. This filter and the previous one you createdtogether say that all items must be under 100,000 miles and can be eitherblack or red. The way you did it prevents the conditions from interferingwith each other. Your screen should now look like Figure 16.Figure 16. Add more filtering criteria

21. To clean things up, click Layout in the upper right of the editor, as shownin Figure 17.Figure 17. Cleaning up the layout

22. Finally, connect the output of the second Filter to the input of the PipeOutput, as shown in Figure 18.Figure 18. More cleanup




23. To test the pipe, select the Google Base component and view the outputin the Debugger pane below it. Things should look something like Figure19.Figure 19. Testing the Google Base connection

24. Now select the Sort component to see how the stream is affected in theDebugger pane. Figure 20 should look familiar to you.




Figure 20. Testing the sorting

25. Next, select the filters to see how they affect the output. (As you can seein Figure 21, I whittled my new truck selection down to just two in myarea.)Figure 21. My meager selection of two




26. Now you can make your pipe available to the world. Click Save and thenPublish, entering whatever you feel like for the options available, asshown in Figure 22. I named mine My new article test.Figure 22. Saving the pipe (don't forget to publish...)

27. To view the pipe page you just made available, click Pipe Preview,illustrated in Figure 23.




Figure 23. Pipe preview link

28. Run, My new article test, run! Run and look something like Figure 24!Figure 24. Pipe preview running

29. To use the pipe you've built, you can right-click the Get as RSS or Get asJSON links to save the link location, and use that URI in any applicationwhere you might normally use a feed. For instance, if you use Firefox likeme, you can click Get as RSS; then leave Live Bookmarks selected andclick Subscribe Now, as shown in Figure 25.Figure 25. Adding a Live Bookmark of your new pipe




30. Select the location where you want to store your pipe in your bookmarklist. Your new dynamic bookmark listing will always keep you up-to-dateon the latest vehicles available, as shown in Figure 26.Figure 26. Your new Live Bookmarks

31. Go back to your running pipe.

32. Select Subscribe and note all the locations where you might send yourpipe, as shown in Figure 27.Figure 27. Subscription availability

Figure 28 shows our pipe embedded in a personalized Google homepage.

Figure 28. Your new pipe in a personalized Google home page




Section 5. Demo 2: del.icio.us tag search

For my second demo, I created a pipe that uses a subscription that I set up ondel.icio.us to return popular sites other people tagged with the term "dessert." Mypipe queries that subscription and allows you to filter the sites by a keyword.

1. Navigate to the pipe's URL,http://pipes.yahoo.com/pipes/pipe.info?_id=knPV_OTX2xG9LijZp2IyXQ. Itshould look something like Figure 29.Figure 29. My pipe querying a del.icio.us subscription



http://pipes.yahoo.com/pipes/pipe.info?_id=knPV_OTX2xG9LijZp2IyXQ


This form is dynamically generated when you use User Inputcomponents in the process flow. To see how this is done, click the Howthis pipe was made graphic on the left.

2. In the Subscription field, enter dessert and click Run Pipe. This willreturn all sites tagged with the term dessert ranked by popularity (that is,by how many people have tagged a specific Web page with that term).

3. Now enter chocolate in the Search for field. This will return a subset ofthe dessert subscription, consisting of sites that include the termchocolate in the title or description of the feed.

4. To use this pipe on your own subscriptions, simply click Clone and editthe source URL to point to your own del.icio.us profile, then enter yoursubscription names in the form shown, as well as any filtering terms youmight like to add.

Key things to note

With user inputs, users can enter information when the pipe is run. You can tie theseinput components to any text field in other components and basically dictate that thevalue of the identified field will be set from some dynamic input. In Figure 30, youcan see two user input components highlighted on the far right and center as well asthe text fields to which they will provide the dynamic data.

Figure 30. Incorporating dynamic data using user input components




These input types are then automatically displayed to users as a form from theYahoo Pipes user interface. You might also hand-code them into the URL querystring, as illustrated in Figure 31.

Figure 31. What user inputs translate to in a browser and URL query string




Section 6. Demo 3: The Craigslist middleman feed

My third demo describes the process of layering pipes —basically creating a newSource component, similar to what you saw in Demo #1 with the Google Basecomponent. This demo uses two separate pipes, both of which you can runindependently, but here are used together.

The purpose of this pipe combination is to match things for sale on Craigslist.orgwith wanted items that people have identified. It works by first pulling a list of wanteditems from the site. Then, for each item, a second query is run to look for thatproduct for sale. Any items found for sale are added as a child node to the itemwanted record.

This demo is made from two pipes, one nested in the other. The first pipe does asimple search on Craigslist using some search term and generates an RSS feed of




things for sale. This is the nested pipe.

The second pipe pulls all wanted items from Craigslist.org. It then determines theactual desired item using a special component called the Content Analysiscomponent. This is important for extracting a concise term from a potentially lengthydescription. This "item wanted" term is stored an attribute and fed into the first "itemsfor sale" pipe as the dynamic input. This produces a list of "for sale items" based ona supplied "wanted item" and stores that list with the wanted item.

The result is a list of items wanted by Craigslist users that hold links to postings ofthat item or similar items for sale. Note that, if you run this pipe through YahooPipes, the associated "for sale" items do not display even though they are containedin the RSS feed. This is due to a limitation in the display, which prevents you fromchoosing which parts of the items in the feed to display (poke around on the Pipesforums and you will see people asking for more flexibility here). However, if you area feed developer and pulling streams into your own applications, then you cannavigate to and display whatever parts of the feed you want. Thus, this pipingexample is one that will be limited to use in only custom applications.

1. Navigate to the URLhttp://pipes.yahoo.com/pipes/pipe.info?_id=5ldWJJHY2xGrYNryqGIyXQ,and enter table into the Query field, as shown in Figure 32.Figure 32. My pipe querying Craigslist

2. The result list will display all tables for sale on Craigslist.

3. Now navigate to the URL



http://pipes.yahoo.com/pipes/pipe.info?_id=5ldWJJHY2xGrYNryqGIyXQ


http://pipes.yahoo.com/pipes/pipe.info?_id=hk7ADojO2xGdmQTulPXiAA.This will only show items wanted on Craigslist, but you can navigate toboth pipes in the editor from here to see how they were built.

Key things to note

The key demonstration in this example is the use of another pipe nested with aprocess flow. In Figure 33, you see a basic Craigslist search pipe being used to pulla feed for each item in the first stream. An attribute stored in the original item is usedto pipe into the second search pipe. The resulting secondary result list is stored asan attribute labeled supplies in this case. Now an application that makes use of thefinished stream might display the parent list of RSS items with a secondary level oflinks found in each item's supplies node. A standard RSS view will not see thissecondary list of links, but a custom application will have no trouble using this feed.

Figure 33. My pipe querying Craigslist

Section 7. Conclusion

Wrap up

As you can see from these basic examples of transforming feeds, Yahoo Pipes can



http://pipes.yahoo.com/pipes/pipe.info?_id=hk7ADojO2xGdmQTulPXiAA


be a powerful tool to manage the dynamic data so readily available today.

For further information on how to use this tooling, check out the documentationavailable in Resources. Also, to peruse pipes created by other users, selectingBrowse Pipes from the same pages used in my demos.

View movieShow me.

Download Adobe Flash Player.

See more about animated demos.

Animated demos

If this is your first encounter with a developerWorks article that includes demos, hereare a few things you might want to know:

Demos are an optional way to see the same steps described in the tutorial. To seean animated demo, click the Show me link. The demo opens in a new browserwindow. Each demo contains a navigation bar at the bottom of the screen. Use thenavigation bar to pause, exit, rewind, or fast forward portions of the demo. Thedemos are 800 x 600 pixels. If this is the maximum resolution of your screen or ifyour resolution is lower than this, you will have to scroll to see some areas of thedemo. JavaScript must be enabled in your browser and Macromedia Flash Player 6or higher must be installed.

Download Adobe Flash Player.



http://download.boulder.ibm.com/ibmdl/pub/software/dw/richmedia/xml/x-yahoopipes-tut/yahooPipeDemo800x600.swf

http://www.adobe.com/shockwave/download/download.cgi?P1_Prod_Version=ShockwaveFlash

http://www.adobe.com/shockwave/download/download.cgi?P1_Prod_Version=ShockwaveFlash


Resources

Learn

• Pipes in this tutorial: Play around with the Yahoo Pipes in this tutorial.

• Yahoo Pipes: Check out the Web site for this interactive feed aggregator andmanipulator.

• Most frequently run pipes: Take a look at the most popular pipes.

• "Yahoo! Pipes and the Web as database (Alex Iskold, Read/Write Web,February 2007): Read an introduction to the implications of this service.

• Digg discussion: Check out the talk of Yahoo Pipes right after its release.

• developerWorks XML zone: Find more XML resources.

• IBM XML certification: Find out how you can become an IBM-CertifiedDeveloper in XML and related technologies.

• XML technical library: See the developerWorks XML zone for a wide range oftechnical articles and tips, tutorials, standards, and IBM Redbooks.

• developerWorks technical events and webcasts: Stay current with technology inthese sessions.

Get products and technologies

• IBM trial software: Build your next development project with IBM trial software,available for download directly from developerWorks.

Discuss

• Message Boards for Pipes: Talk with other pipes users.

About the author

Jeff K. WilsonJeff K. Wilson is an architect and consultant for IBM. For the most part, he likescomputers. He talks a lot about them; stuff people put on them. He uses them quite abit. Typically, black ones for some reason. He never did much care for thedirty-looking, almondy-colored ones. He types on them; plugs all sorts of things intothem. He even has a bag for it in a special place in the hallway where he dumps itevery night. The drink coaster-like rings on the top of his laptop are not a sign ofdisrespect, but rather companionship.



http://pipes.yahoo.com/pipes/person.info?eyuid=JVLM8hgkoGKcdddfOVz06s8XVQ--

http://pipes.yahoo.com/pipes/

http://pipes.yahoo.com/pipes/pipes.popular

http://www.readwriteweb.com/archives/yahoo_pipes_web_database.php

http://www.digg.com/tech_news/What_is_Yahoo_Pipes

http://www.ibm.com/developerworks/xml/

http://www.ibm.com/certify/certs/xmsdreltop.shtml

http://www.ibm.com/developerworks/views/xml/library.jsp

http://www.ibm.com/developerworks/offers/techbriefings/?S_TACT=105AGX06&S_CMP=tut

http://www.ibm.com/developerworks/downloads/

http://discuss.pipes.yahoo.com/yahoo/Message_Boards_for_Pipes/


made-to-order content with yahoo pipes · yahoo pipes is a particularly interesting utility that...

Documents