A Personalized Recipe Database System with

User-Centered Adaptation and Tutoring Support

Student: Liping WangSupervisor: Prof. Qing Li

City University of Hong Kong, Hong Kong

Outline Introduction Recipe Modeling Recipe Cooking Graph Mining User Adaptation Prototype System Conclusion & Future Work

Introduction People can not live without foods Existing recipe websites provide huge amounts of

recipes throughout the world Fail to give support on analyzing and comparing

recipes What are important cooking principles & skills; what makes two dishes’

taste so different, etc. Unable to help users find similar recipes in a

comprehensive manner only keyword-based search on recipe names

Fail to adapt recipes to meet the real-world situation e.g. due to lack of ingredients or user preference

Our Contributions Propose a recipe model which encompasses static

attributes as well as dynamic behaviours e.g. cooking procedures and constraints

Present a novel perspective of evaluating the “quality” of a recipe by constructing and analysing its cooking graph capture both action flows and data/ingredient flows

Provide a promising way to address the problem of recipe adaptation heuristically with flexible and feasible solutions

Recipe on the WebIngredients:

Chicken Thighs 250 gScallions 10 g

Sesame Paste 2 tsp.Sugar 1 tsp.

Soybean Sauce 2 tsp.Sesame Oil ½ tsp.

… ...

Step Illustration

Steps1. Use chicken thighs and cut away skin and fat2. Poach the chicken. Drain and cool. 3. Mix the sesame paste, sugar, soybean sauce and sesame oil4. Cub the chicken lightly till soft and shred. Put to a plate.5. Put shredded scallion around the chicken and pour the sauce over the chicken.

… ...Users’ Rating

and Comments

CategoryRegion-->Sichuan

Cooking Method-->PoachedIngredient-->Chicken

Video Clip Final Look

Bang Bang Chicken

Recipe on the Web

Sample RecipeStep number

Recipe cooking procedure in steps

1 Dice bell peppers. Slice squash and mushrooms.

2 Cook pasta according to package directions in unsalted water.

3 Meanwhile, in a large skillet melt butter. Add bell peppers; cook and stir occasionally until barely tender, about three minutes.

4 Add squash and mushrooms; cook and stir occasionally until barely tender, about four minutes.

5 Drain pasta; toss with vegetables in skillet.

6

In the saucepot in which the spaghetti was cooked, combine ricotta, mozzarella, milk, Parmesan, Italian seasoning, salt and black pepper. Over a medium-low heat cook and stir cheese mixture just until hot, about 1 minute.

7 Add reserved pasta and vegetables; toss to coat; remove to a serving platter.

The Cooking Procedure of “Triple Cheese Pasta Primavera”

Sample Recipe

Triple Cheese Pasta Primavera

1: Dice bell peppers. Slice squash and mushrooms.2: Cook pasta according to package directions in unsalted water.3: Meanwhile, in a large skillet melt butter. Add bell peppers; cook and stir occasionally until barely tender, about three minutes. 4: ………

action 1: diceaction 2: sliceaction 3: cook…

action i: stir…

action n: remove

p Steps in the Web Page

Primitive LevelComposite Level

Divided into n ActionsRecipeCrawled from the Web Page

Recipe Level

Parsing the Cooking Procedure of “Triple Cheese Pasta Primavera”

Recipe Model A recipe R is modeled and represented by a tuple of three

elements:R = <M, RP, SP>

Where (a) M={Mi | i = 1.. m} – a set of ingredients. An ingredient Mi

is either a basic ingredient or a set of ingredients: Mi = <MID, MP>, MID—unique identity, MP—member level

properties (and functions) such as the name, quantity and image Three classes: Main, Minor and Seasoning;

(b) RP is a set of recipe-level properties (and functions) applied on R itself, such as the main cooking style, region, nutrition and images of the dish

of the recipe;

Recipe Model (c) SP = (V, E, Cons, Ingr) is a labeled directed “Cooking

Graph”, V={vi | i = 1..n} is a set of nodes.

vi—a cooking action “cooking action constraints”: Cons(vi)—constraint conditions to be satisfied when vi takes place. e.g. conditions on temperature and duration etc.

E is a set of directed edges on V—temporal execution flow of the cooking actions; named “action flows”. An edge <vi ,vj> —vj should take place after vi. “cooking transition constraints”: Cons(vi , vj) –the conditions to be satisfied for the flow to take place.

Ingr(vi) – ingredients that should be added into vi O(vi) –the output ingredients of vi

These inputs and outputs for the nodes are called “ingredient flows”.

Cooking Graph

bell peppers

squashmushrooms

dice

slice

add

pasta

meltcook

toss

Start Node

v1

v2

v3

v4

v5 v8

v9

v10

v7v6

M1

M2

M4

M3

stir add

cook

stir

v11cook

Loop

LoopFork

Join

Sequential

v12

drain

butterM5

milksalt

mozzarellaParmesan

ricottablack pepper

Italian seasoning

stiradd combinetossremove

v13v14v15v16v17M6M7

M8M9

M10M11

M12

End Node

Cons(v7,v8)

Cons(v7,v6)

Cons(v10,v12)

Cons(v13)

Cons(v10,v9)Cons(v4)

Cons(v3)

Cons(v12)

Cooking Graph

M : Ingredient Action Node

SP = (V, E, Cons, Ingr)

Action Flow: Explanation V: E: Ingr:

Ingredient FlowCons( ):

Constraint

The Cooking Graph of “Triple Cheese Pasta Primavera”

Basic Properties Definition 1. (Reachability) A cooking graph is

defined as “reachable” if each of its nodes is “reachable”; a node is “reachable” if it is on a directed path from a starting node to the end node.

Definition 2. (Consistency) A cooking graph is defined to be “consistent” if the conditions for each node/edge is consistent (i.e. there exists assignment to variables to make the conditions true).

Constraints and Rules Definition 3. (Constraint) A constraint is a predicate

followed by one or more terms, enclosed in parentheses and separated by commas; a term is either a constant, variable or function expression. Constraints specify all kinds of conditions or restrictions in

the recipe model; Three categories: intra-recipe constraints, inter-recipe

constraints and outer-recipe constraints. Incompatible(Spinach, Tofu) says spinach and tofu are

incompatible and should not be cooked together.

Constraints and Rules Definition 4. (Rule) A rule is a logical implication of

the form “If A Then B” (or, A->B), where A and B are sentences. Validate the correctness of a recipe through reasoning and

recognition process. Handle complex situations such as to make necessary

adjustment or compensation once an improper cooking action occurs.

Describe cooking skills that have been widely accepted and commonly used.

Over_Put(salt) → Add(vinegar|water) says that if too much salt has been put into a dish, then neutralize the salty taste by adding either vinegar or water.

Recipe Cooking Graph Mining Pattern — Some subgraphs occur in one or more cooking

graphs; have certain influence on the cooking effects (e.g. taste, appearance).

Find patterns for a set of recipes What’s usually done and what’s usually put in the cooking procedure

(one action, a series of actions, an ingredients, a set of ingredients, actions combined with ingredients)

Cooking graphs of different recipes may share the same pattern

Distinct subgraphs that determine the cooking effect (e.g. taste) should be identified

Sample Patterns

……

marinate

……

e.g. salt, sauce, garlic, scallion

Main Ingredient(s)

Seasoning Ingredient(s)

e.g. pork, chicken

……

coat

……

e.g. starch, water, egg

Main Ingredient(s)

Seasoning Ingredient(s)

e.g. pork, chicken

heat

Ingredient(s)

……

fry/ stir-fry/

deep-fry

remove from oil

……

oil

Passing Oil

boil

Ingredient(s)

……

simmer briefly

……

boiling/cold

water

remove

Blanching

Sample Cooking StyleCooking Style Pattern with Dominating ActionSoft Deep-frying Coating + Passing Oil + deep-fry

Dry Deep-frying Marinating + Coating + deep-fry

Cooked-frying Passing Oil/Blanching/Steaming+ stir-fry (+ Thickening)

Slip-frying (Marinating + Coating) + Passing Oil + stir-fry + Thickening

Soft Stirring Blanching/Steaming+ stir + Thickening

Braising Passing Oil/Blanching/Steaming + simmer in sauce (+ Thickening)

Simmering Blanching + simmer in water/broth

Generally describe how a recipe is cooked in a Pattern Combination or in Graph Abstraction.

User Adaptation Usually a user wants to make a dish that has the

same cooking result (e.g. taste, appearance) as the recipe exhibits.

Unfortunately, the user is very likely to get a slightly or even totally different dish as he/she modifies the cooking procedure.

Objective reasons—e.g. lack of some ingredients, Subjective reasons—e.g. wrong cooking actions by carelessness.

User Adaptation When the user makes an

adaptation, the system will check if the modified cooking graph is feasible.

If not, a set of feasible templates are provided.

The remaining subgraph is replaced by the user selected one.

Property check (Reachability, Consistency)

… … …...

… …

… …

… …

Remaining Original Subgraph

Templates

Adapted Subgraph

UserSelection

?

Originally One Recipe

…...

Adapted Subgraph

… …

User Selected Template

Substantial Ingredients & Constraints

Instantiation

Template Selection and Instantiation

Prototype SystemGlobal Systemvs. User Space

Global System

… ...

Conventional Recipes in Structure

Adopted & Adapted Recipes in User Organized Structure

User Space

Export

Import

Linda

Tom

Mary

User Area

Global Area

Export a Recipe “Steamed Chicken”

Search “Spicy Bean Curd”, “West Lake Fish”, “…”

Comment a Recipe “Carp Soup”

Add a Favorite Recipe “Stir-Fried Prawns”

Try a Pop Recipe“Eight Precious Rice”

Prepare a Party Menu

Prototype System

Summary Proposed a data model to represent a recipe Advocated cooking graph mining to find frequent

used patterns (actions, ingredients) Tried to solve recipe adaptation problem by using

patterns as templates Developed a prototype system—RecipeView

Further Work

Discover patterns of cooking graphs Refine and strengthen the algorithm of recipe

adaptation

Acknowledgement Thanks to Mr. Yu Li for his cooperation on

RecipeView System Many thanks to Prof. Xiaofeng Meng and

Prof. Guozhu Dong for their precious suggestions on our project

END

Questions & Answer