Enabling Calorie-Aware Cooking in a Smart Kitchen

ABSTRACTAs a daily activity, home cooking can be viewed as an act of care for family members. However, during cooking process, family cooks are commonly unaware of how many calories are in their prepared meals. Therefore, their family members may over consume calories, increasing their likelihood of obesity and chronic disease. This work presents a smart kitchen with Ubicomp technology to improve home cooking by providing the number of calories of ingredients that are used in prepared meals. In doing so, family cooks can more effectively control the meal calories based on family needs. Our kitchen has sensors to track the number of calories in food ingredients, and then provides real-time feedback to users on these values. A user study suggests that the kitchen can effectively help family cooks maintain the number of calories in meals within the healthy level recommended by nutritionists.

Author KeywordsUbiquitous Computing / Smart Environments, Home, Healthcare, Context-Aware Computing

ACM Classification KeywordsH5.m. Information interfaces and presentation (e.g., HCI): Miscellaneous.

INTRODUCTIONObesity and overweight are major contributors globally to serious diet-related chronic diseases, including cardiovascular disease, hypertension and stroke, diabetes and others [1]. To prevent obesity and overweight, special care should be taken to maintain the calorie content of every meal within daily requirements [2,3]. Packaged foods sold in grocery stores have nutritional information (Nutrition Facts [4]) including number of calories. However, a recent study has indicated that most people still favor home cooking or cooking meals from scratch [5]; in Europe, 52% and the US, 44% of people prefer scratch

Figure 1. Calorie-aware Kitchen with calorie tracking and digital feedbacks of nutritional information during

cooking process

c

ooking. Average family cooks may not know how many calories are in their cooked meals after raw food ingredients are mixed and cooked, or whether these meals are considered healthy and offer a good number of calories for their family members [6,7]. One of the reasons is that the average family cook cannot easily follow the steps of calculating calories during an intense cooking activity: first they have to estimate accurately the amount (weight) of each food ingredient used (such as oil, meat, vegetables and others), and then they have to look up a food calorie table to calculate the overall number of calories after cooking and mixing several ingredients in a course or a meal.

This study presents a Calorie-aware Kitchen that can provide family cooks with awareness on the number of calories in their home cooked meals, thus help family cooks make healthy meals with the appropriate amount of calories, as recommended by nutritionists.

Figure 1 depicts the Calorie-aware Kitchen. It is augmented with sensors that track the food ingredients used during cooking, and provides just-in-time digital feedback to raise healthy cooking awareness. For instance, when a user prepares a meal, the kitchen presents calorie information whenever the user performs a cooking action that changes the amount of food ingredients on the kitchen counter or the stove, such as by adding meat, pouring in oil, etc. Given the number of calories of each ingredient, an average family

1

cook can reconsider the amounts of ingredients or the composition of a course, to achieve a good balance between healthy and delicious cooking. The developed kitchen also suggests the recommended number of calories for a meal, based on the Harris-Benedict equation [8], for the entire family. Finally, an element of playfulness lies in a calorie-aware cooking game, which makes cooking more enjoyable.

The rest of this paper is organized as follows: First, the background and our prototype design of the Calorie-aware Kitchen are introduced. Second, details of how the kitchen tracks calories by tracking ingredients during cooking, and increasing awareness with a calorie-aware game that a user can interact with, are described. Then, a user study of how the system may help family cooks to be aware of calories and control them by adjusting the amounts of ingredients is presented. Conclusions are finally drawn, along with recommendations for future research.

BACKGROUNDPrior to describing our kitchen system, causes of obesity and conventional calorie control methods must be understood; the target users defined, and the results of a contextual inquiry into target users’ cooking behaviors introduced.

Obesity Concerns and Calorie ControlAccording to studies [1,9], obesity is caused by an energy imbalance, i.e. energy intake exceeds the energy expenditure. Every excess 7000 kilocalorie (kcal) may result in a weight gain of 1kg. Therefore, a major solution to weight gain is calorie control [10,11,12].

In practice [13], nutritionists have educated the public about the effects of an over-intake of calories on obesity and chronic diseases, as well as the importance of monitoring the amount of calories consumed. However, the traditional approach is usually ineffective on home cooking, as the knowledge of nutritionists cannot be easily put into cooking practice, for many reasons. (1) Nutrition education often occurs through the media, such as TV or newspapers, which do not present for the real context of consumers’ real cooking place (kitchen). This non real-time, non in-situ education is not effective and is often ignored by family cooks when they really cook in a kitchen [14,15]. (2) Most individuals do not find nutrition easy to understand, and they find the calculations too complex and time-consuming [10]. (3) Family cooks are more reluctant to change their cooking habits, given that they have cooked for a lifetime, unless they have a very strong motivation, such as personally experiencing obesity and related diseases.

A more effective solution is demanded. This study specifically targets family cooks and their kitchens, which are the main sources of meal/food production. Calorie-aware cooking by family cooks may result directly in

healthy, light-calorie eating by their family members and prevention of obesity problems.

Targeting Experienced Family CooksThis work targets experienced family cooks. An experienced family cook is defined as someone who can cook without following any recipes or by relying on weight scales to measure food ingredients. He or she regularly and frequently makes meals for their families over an extended period. Acquired extensive cooking experience also leads to a personal cooking preference for particular compositions and portions of food ingredients in dishes or meals. For instance, an experienced family cook may prefer to add more butter to make tastier fried steaks, or may opt for more oils when stir-frying vegetables. Although such experienced cooks may not be aware of the health consequences of their cooking habits, they are interested in knowing how to make their meals healthier because of their concern and care for family members. Such cooks are our targets.

Contextual InquiryBefore the smart kitchen was designed, a four-day contextual inquiry was conducted to understand the cooking behaviors of four experienced family cooks (aged 28, 30, 58 and 65) in their home kitchens as they were cooking a regular dinner for their family. During the cooking process, they were observed and videotaped; notes were taken, and questions asked about their cooking process, meal preparation, understanding to nutrition and calorie control. The findings are as follows.

Family cooks commonly added ingredients based on experience or preference (oil, butter, meats, for example). Most stated that they were unsure about whether their own cooking styles were healthy. However, they mentioned that they would alter their cooking style when their cooking styles may result in health-related concerns of family members, such as gaining weight. They would also perform their own trial-and-error runs to strike a balance between health and taste.

They expressed the desire to cook healthily for their family members, especially with respect to calorie control and nutritional balance. However, given busy schedules, they could not afford too much time or make much effort to learn and follow the complicated steps of weighing food ingredients and calculating nutritional values during actual cooking. They preferred simple-to-understand, practical guidelines that are relevant to their cooking styles.

Since cooking is a real-time activity that requires ongoing planning and thinking about the next cooking step, family cooks would like to focus solely on cooking. They do not like to be distracted and interrupted by unrelated activities, such as answering phones, taking care of crying children, and others, because distractions and

interruptions are likely to cause cooking errors. They suggested that they want only simple, highly relevant information on cooking or nutrition.

They regard a kitchen as part of a home and not a place of work. No standard procedure should tell them how to operate various tools in a kitchen to produce meals. They want to choose freely whatever methods they are comfortable with to produce healthy and delicious meals. Many also view cooking as an enjoyable activity, which brings them satisfaction and happiness, especially when they receive positive feedback about delicious meals from their family.

DESIGN CONSIDERATIONSThe contextual inquiry led to the following design considerations in designing the Calorie-aware Kitchen: (1) a kitchen should offer just-in-time calorie information on food ingredients and dishes during their regular cooking process, to reduce greatly the effort required to calculate calorie manually and help family cooks with controlling calories. (2) Information should help family cooks make their own decisions regarding the balance between health and taste, without constraining his or her natural cooking habits. When cooks must concentrate, they can choose to ignore the informational display. (3) Information should be presented simply, and the layout of the information should mirror actual usage and actions, so that family cooks can easily grasp the calorie information by taking quick glances. (4) Calorie recommendations should be provided for ease of comparison and adjustment. (5) Feedback should be offered playfully to make the complete cooking process enjoyable.

PROTOTYPE DESIGNBased on the above design issues, an initial prototype of the kitchen was proposed, and is presented in Figure 1. The kitchen is a standard one purchased from IKEA [16], comprising the following two modules; (1) a calorie tracker that tracks the calorie, composition, and position of food ingredients and dishes that are currently on the kitchen counter or stove; and (2) an awareness display that provides calorie information on the ingredients and dishes, with a calorie-aware game brings enjoyment to the process by which users control calories to meet recommendations. Users can operate the system according to the flow shown in Figure 2. The following two sections present these details.

CALORIE TRACKERWhenever a user performs a cooking action (adding or removing ingredients to or from a container) that may change the number of calories during cooking, the system

must detect the cooking action as a calorie-change event in

Figure 2. Operation of Calorie-aware Kitchen

Figure 3. Calorie tracker architecture

real-time. An example of such cooking actions is the addition of salad oil (130 kcal) to a pan or the removal of bacon (250 kcal) from a cutting board. Studies have shown that the number of calories can be derived form the sum of the calories of all the used ingredients and calories can be derived from weights of ingredients [17]. Therefore, to track calories, only the weight and the composition of food ingredients in dishes need to be determined.

The calorie tracker offers a hybrid sensing solution by combining weighing sensors and computer vision for accurate detection. Figure 3 depicts the architecture, which comprises hardware components (weighing-sensing surface and camera sensing) and software components:

Hardware Design and Implementation

Weighing-sensing surfaceBased on our observations of cooking activities, most main food preparation activities occur on the kitchen counter. They include putting ingredients on a plate, transferring foods among containers, cutting foods over a cutting board, mixing in a bowl and others. Hence, the system must accurately recognize the amounts (weights) of ingredients that are added to each container to calculate their calories. For the prototype, the design was based on the load sensing table [18] that four weighing sensors were installed in the four corners underneath the kitchen counter with an area of 55x44 cm2 (see Figure 4 left). All foods ingredients are assumed to be placed in or on kitchen containers (e.g., plates and bowls, cutting boards are also counted as containers here), rather than being placed directly on the

Calorie Tracker

The family cookthen gets calorie

awareness

Calorie Calculator

Ingredient Inference Engine

Weighing-sensing Surface

Awareness Display and Calorie-aware Game

Sensor Events

Transfer Events

Calorie-change Events

Nutritional Database

Food Labeling

Camera Sensing

Weight Change Detector

3The family cook performs a

cooking action

Our system tracks the

calorie change

kitchen surface. Hence, the smart counter can track the

Figure 4. Weighing sensors deployed under counter (left)

and stove (right)

Figure 5. (left) Overhead camera to capture images of the counter. (right) Example image taken by the camera

when a cooking action is detected.

position of the containers on the countertop with an accuracy of 1cm, and the weight of food ingredients in these containers. On the other hand, most cooking activities are performed on the stove, such as frying in a pan, so a weighing sensor must be present under the stove, too (Figure 4 right). All of the weighing sensors are attached to weight indicators with a resolution of 1g, which output readings through a serial port at a frequency of eight samples per second.

Camera SensingComputer vision is employed to improve the accuracy by filtering noise from the weighing-sensing surface. An overhead camera is deployed over the counter to capture an overview image of the counter (Figure 5). However, the other camera is not installed over the stove because foods over a burner may produce smoke which affects the image processing and damages the camera.

Software DesignThe design of our calorie tracker is based on a bottom-up event-triggered approach, presented in Figure 3. Table 1 presents three types of event from high-level to low-level. A high-level Calorie-change event describes ingredients and their calorie amount contained within a container. For instance, a Calorie-change event can be (“container1”, “salad oil”, 130kcal), in which the calorie value is calculated by the component calorie calculator, with a

nutritional database and middle-level Transfer events. A Transfer event describes ingredients and their weights contained within a container, e.g. (“container1”, “salad oil”, 50g). This is determined by the ingredient inference engine, which tracks the path of each ingredient from a starting container (as when bacon from a fridge is moved to the cutting board) to an ending container which holds the final cooked meal. The engine works with the low-level Sensor events such as (“position:(10,50)”, 50g), detected from the weight change detector. Each component is as follows.

Calorie-change EventsCalorie-Change (containeri, Ingredient, Δcalorie),indicates that Containeri (at a known position) has been added/removed Ingredient(s) of Δcalorie kilocalorieTransfer EventsIngredient-Change(containeri, Ingredient, Δweight),indicates that Containeri (at a known position) has been added/removed Ingredient(s) of Δweight grams.Sensor EventsWeight-Change(Δweight, position),indicates a weight change of Δweight grams at position (x,y) over the kitchen surface.

Table 1. Calorie-change, Transfer and Sensor events

Weight Change DetectorAt the bottom, the weight change detector aggregates weight samples from weighing-sensing surface. It reports Weight-Change events to the food ingredient inference engine when the weight changes, by detecting the stream of weight samples.

However, based on preliminary experiments, detection using only weighing sensors is not sufficiently accurate (recall of 54%, meaning 46 false positive detections or detections of noise per 100 weight changes), especially when cooking actions, such as cutting or stirring, generate lots of weight noise. Observations indicated that when these actions are performed, the cook performs similar motion of foods using hands and/or utensils. For instance, to cut bacon, the cook uses one hand to hold the bacon and the other hand to take the knife, cutting little by little. Therefore, video analysis using a color histogram comparison [19,20] is performed to filter false detections from weighing sensors. Comparing histograms is a good means to reduce sensitivity to the motion of objects that two camera images captured at different times with unchanging objects differ only slightly in their histograms, but a real change can still be detected.

Ingredient Inference EngineAfter the weight change detector has reported low-level Weight-Change events (weight and position), the system then uses an inference rule engine to infer middle-level

ingredient transfer activities. A weight matching algorithm similar to that in our earlier work [21] is adopted to track the transference of food from one container to another. Therefore, when weight change detector detects weight changes, by matching a weight decrease (such as from a food container on a counter) to a weight increase (such as in a pan on the stove), food ingredient transfer is inferred and an Ingredient-Change event is sent to the calorie calculator.

Food LabelingA Wizard of Oz method that involves one human observer’s manually inputting the name of an ingredient is currently used to identify new ingredients during cooking process, because of the difficulties of recognition using computer vision or RFID tags on raw ingredients. When the inference engine detects a new ingredient that cannot be inferred from the weight decrease of any container, the camera captures an image and show to a human observer to ask its name in the other display that the user does not see. A voice-dialog system is tested to enable family cooks to identify foods using a voice input, for application in the subsequent stage.

Calorie CalculatorAfter an Ingredient-Change event is received from the inference engine, a public nutritional database that provides the nutritional values of each ingredient is used by the calorie calculator to calculate the number of calories based on the weights and the names of the ingredients [10]. This value is reported to the awareness display to interact with the family cook.

Accuracy and Response TimeThe accuracy of the tracking of ingredients is important to provide correct calorie feedbacks to users. According to the user study of three participants who cooked a total of 15 meals, the hybrid method achieves a recall of 88%, where major noise is from the weighing sensors under the stove without camera sensing, and currently requires human observers to filter noise manually. The precision is 95%, with missed detection when the weight is too low (less than 1g) by weighing sensors or when camera sensor incorrectly filters the weight change. The accuracy of the calorie calculation is 97%: 97% of the real calorie amounts of ingredients can be determined, where the missing calories are associated with undetected food ingredients. For every cooking event (adding / removing ingredients), the average response time is 1 second, including the detection time associated with weighing, camera sensing and transfer recognition.

LimitationsSince our tracking method is based on weighing sensors, the current prototype has a limitation that it cannot recognize concurrent or interleaving events, such as taking two dishes from a counter simultaneously and then

immediately putting the ingredients into the pan on the stove.

AWARENESS DISPLAY AND CALORIE-AWARE GAMEAfter the calories in ingredients have been determined by the calorie tracker, the system provides real-time feedback to increase the user’s awareness of calories via an LCD display on the wall in front of the user.

Preliminary version – complex informationFigure 6 presents the preliminary version of our user interface. The right half is an overview of information on food ingredients in the system, with the name and weight of every ingredient in each container on the counter and stove; the left half shows not only the number of calories but also nutritional facts associated with the latest ingredient change.

Figure 6. Preliminary version of user interface

However, this design does not meet all of the design considerations. After a test in which one participant cooked only one meal, “spaghetti alla carbonara”, the user commented that the information was too complex and detailed (nutritional facts with decimal points, the weights of each ingredient) and she could not understand their meaning. Although she realized the importance of controlling calories, she did not know how much her family should take without a standard to refer to. Additionally, she felt interrupted when the information over the whole left screen changed whenever she performed an action. She could not remember how much nutrition she was already included in her dish. She also felt that repeatedly looking at the changing numbers after about half an hour of cooking was boring, so she decided to ignore the system. Therefore, she strongly suggested that the design should be revised to meet the goal of promoting healthy cooking.

Revised version – simple informationFigure 7 presents the revised user interface in the Calorie-aware Kitchen based on the design considerations and what we’ve learned from preliminary version. The interface has three parts - an overview of the calories in the currently used ingredients, recommended calorie needs and the total currently used calories, and a calorie-aware game to make calorie control enjoyable. These parts are introduced in detail below.

5

Real-time Awareness of Calorie in UseThe main part of this interface presents an overview of the number of calories in current ingredients on the stove and counter (Figure 7a), to enable family cooks to obtain information efficiently. Information on containers, including the total amount of calories and the names of the ingredients in it are displayed (Figure 7 right) based on the real position. This information is stronger only when the number of calories exceeds the basic threshold of 200kcal. Otherwise, ingredients with few calories (such as garlic, lemon) are left to allow users to focus on their cooking. All the changes on the interface are made with a short and simple sound to notify users.

Recommended Calorie NeedsIn the left part of our UI (Figure 7b), a vertical bar is used to show the recommended number of calories for the meal, which is determined using the Harris-Benedict equations (based on weight, height and age) and the individual activity level [8,22]. Therefore, before the system begins,

the above details on the user’s family members are input, and then the system calculates and presents the required number of calories for this family for reference. During the cooking process, the current total calories in use are presented, to facilitate control by the user. Additionally, when a user finishes one course and removes it from the system, the removal is recorded and the number of calories is kept in the bar so that users need not memorize anything.

Game Design and MechanismA calorie-aware game (Figure 7c) is designed to make the process enjoyable. A family member for whom the user cares most (such as son) is used as a character in the game: if the user cooks with more than the recommended number

of calories, then this character starts to gain weight little by little, gradually unbalancing the ground, eventually causing a huge stone on the other side rolling to hit the character (Figure 8). Family cooks must avoid putting their family members in danger by cooking in a healthy way. The figure of only one family member was used instead of all family members because participants suggested that the latter would be too distracting and stressful.

Figure 8. Calorie-aware game: use of excessive calories cause the family member (up) to become overweight and to

be in danger (down)

After the number of calories exceeds the recommended value, users can still bring the figure back into balance as long as he/she reduces the total number of calories below the recommended amount, to keep the beloved family member safe.

USER STUDYThis section describes the user study. The following questions guided this study.

Figure 7. User interface of Calorie-aware Kitchen, including (a) overview of calorie in the system; (b) recommended calorie needs and current used calories; (c) a calorie-aware game with a beloved family member to bring enjoyment of calorie control

(c)

(a)(b)

recommended calorie needs

current calorie in use in total

calorie of finished course

name of ingredients, total calorie in this container, and the most recent calorie change

How effective is the Calorie-aware Kitchen in improving the family cooks’ ability to control calories during cooking?

What cooking behaviors are affected by the Calorie-aware Kitchen?

An evaluation was performed to determine how the awareness of calories during cooking affects users. The activities in the cooking process are complex, so rather than focus on a specific behavior, a holistic view is taken to gather both quantitative and qualitative observations.

ParticipantsThree adult participants, P1, P2 and P3 (Table 2), were invited to participate in the user study. Their ages ranged from 24 to 58. They were all experienced cooks of more than five years who regularly cook meals for their family members. Among them, P1’s brother was classified as overweight based on BMI (Body Mass Index [23]), with special calorie control needs.

Experimental Design and ProcedureSince our prototype kitchen was constructed in laboratory, it could not be easily moved to each of the participants’ homes. Therefore, participants were invited to cook in the laboratory. A video camcorder was used to record the participants’ cooking sessions and their interactions with

Participants P1 P2 P3Age 24 58 25

Gender Female Female MaleHousehold size 4 3 4Profiles of the participant’s

family members(height/weight/

age/gender)

158/53/23/F156/55/22/F170/80/16/M173/70/23/M

175/63/58/M153/54/58/F155/44/23/F

165/62/52/M164/54/50/F172/50/25/M166/52/20/F

Number of family members

with obesity 1 0 0

Recommended calorie* (kcal) 2,981 1,926 2,723

Level of concern with calorie High Low Low

* For one meal (dinner), whole family members

Table 2. Profiles of participants and their family members

our system; their consent was obtained for subsequent analysis.

Our user study involved the following three phases: (1) pretest cooking without feedback on calories, (2) test cooking with feedback on calories, and (3) posttest interview.

To compare the effectiveness of our smart kitchen between pretest cooking and test cooking phases, each participant was asked to write a fixed dinner menu (Table 3) as if they

were to prepare a regular dinner for their family. P1 and P2 wrote a Western dinner menu, whereas P3 wrote a Chinese dinner menu. Based on their dinner menus, they were asked to prepare ingredients using our budget and bring them to our kitchen. Then, the three participants were asked to cook meals in the manner that they did at home, for a total of five cooking sessions per participant in one week. In each cooking session, each participant was asked to cook according to their designated dinner menu in our laboratory kitchen. The participant was given freedom to modify the ingredient composition of the courses (such as by changing the salad dressing, removing mushrooms from spaghetti, for example), but they were not allowed to add a new course or replace an existing course (such as by changing a salad to soup). At the end of the cooking session, participants were free to take their cooked foods home.

In the pretest cooking phase, each participant cooked two meals on two separate days without turning on calorie feedback. Before the start of the first pretest cooking session, the three participants were given time to become familiarized with various appliances and the arrangement of cooking tools in the laboratory kitchen.

In the test cooking phase, participants came to cook for another three meals on three separate days using the calorie feedback and the calorie-aware game. Before the start of the first test cooking session, the calorie feedback interfaces and the calorie game were explained to the participants. The

Participant MenuP1 。 Salad (with apple, celery, and thousand-

island dressing)。 Salmon。 Fried aubergine with onion。 Spaghetti (with bacon, mushroom, onion,

and milk)P2 。 New England clam chowder (from

Campbell’s Condensed Soup [24]).。 Bream roll with bacon with special sauce (including UHT whipped cream, onion, white wine, and lemon), rice and vegetables (cauliflower, carrot, and sweet corn)。 Salad (with lettuce and thousand-island dressing)

P3 。 Shrimp with scrambled egg。 Mapo tofu (fried tofu with meat sauce and

green onion)。 Asparagus with abalone。 Chinese Clam Soup。 Rice

Table 3. Menus designed by participants for testing

participants were also asked not to perform cooking actions outside the recognition limit of the calorie tracker, i.e. avoid performing concurrent cooking actions. Participants

7

followed this rule with reminders in the first cooking session, and then were able to remember it without trouble. Later interviews with participants revealed that although following these rules lengthened the cooking time, it did not affect cooking style.

A posttest interview was performed on the final test cooking day and after the participants finished their last cooking session. They were interviewed about their experience with the kitchen with calorie feedbacks.

MeasurementTo determine how effectively participants cooked healthy meals, this study first measured healthiness during five cooking sessions. The method counts the number of calories in a prepared meal by subtracting the weights of all food ingredients at the end of the cooking session from that at the start of the session. Then, the nutritional database was used to determine the total calories in every meal. Second, the change in the ingredients between the pretest and test cooking phases of each participant, are analyzed to understand how participants reduced calories used

Third, the cooking videos were analyzed and coded. The following data were recorded for each cooking session; (1) the frequency with which a participant glanced at the calorie display following a cooking action that resulted in a calorie change; (2) the average duration of a glance at the calorie display, and (3) the duration of a cooking session.

Finally, the posttest interview involved qualitative measurements of preference, ease of use, future use and comments.

RESULTS AND FINDINGS

Quantitative ResultsTable 4 presents the numbers of meal calories in each cooking session over five days, and Figure 9 presents differences between calories used and the recommended number of calories in each cooking session. The two main findings are as follows. All participants reduced the number of meal calories from the pretest cooking phase (without calorie feedback) to the test cooking (with calorie feedback) by an average amount of (195, 688, 887) kcal. All participants were able to cook meals with a number of calories that was within ±13% of the recommended amount.

Notably, participant P1 exhibited better calorie control before being involved in this user study, because her brother had an overweight problem. Participants P2 and P3 exhibited poorer calorie control and were cooking above the recommended calorie intake during the pretest cooking phase (38.1% for P2 and 45.6% for P3 over the recommended number of calories). Therefore, the system herein was more effective in reducing meal calories from pretest to test cooking phases for P2 (25.9%) and P3 (22.4%) than for P1 (6.4%).

Participants P1 P2 P3(1) Recommended calorie 2,981 1,926 2,723(2) Pretest

Day 1 3,070 2,677 3,951Day 2 3,058 2,641 3,976

Average 3,064 2,659 3,964Over recommendation 2.8% 38.1% 45.6%

(3) TestDay 3 2,937 1,916 3,308Day 4 2,780 2,099 3,027Day 5 2,890 1,897 2,896

Average 2,869 1,971 3,077Over recommendation -3.8% 2.3% 13.0%

(4) Reduction (PretestAVG -TestAVG) 195 688 887

Percentage 6.4% 25.9% 22.4%

Table 4. Meal calorie (in kcal) during each cooking session.

89 77-44

-201-91

751 715

-10173

-29

1228 1253

585

304173

-400-200

0200400600800

100012001400

1 2 3 4 5Cooking session

calor

ie diff

eren

ce

P1 P2 P3

Figure 9. Differences (over or below) between calories used and recommended number for each cooking session

P1 P2 P3ingredient ratio ingredient ratio ingredient ratio

oil 61.2% soup 75.5% meat-sauce 34.8%spaghetti 16.4% bacon 10.7% tofu 26.0%

sauce 6.9% butter 6.2% oil 19.3%

Table 5. Top three reduced ingredients associated with a total calorie decrease of more than 5%

Table 5 shows the percentage reductions in calories of ingredients whose amounts were changed enough to reduce calorie count by over 5%. The calories of these ingredients are higher than those of other ingredients, such that a minor reduction in amount may reduce the number of calories greatly. For instance, in P1’s meal, 61.2% of the total calorie decrease was from the oil. P1 planned to reduce the amount of oil when she found the calorie count was high, and thought it would help keep the number of calories under their required amount, while keeping the meal delicious. In P2’s meal, 75.5% of the total calorie decrease was achieved by reducing the amount of condensed soup. P2 noted that the soup had more calories than she expected,

and reducing the amount can help greatly to control calories, while keeping the meal still tasty. Finally, in P3’s meal, 34.8% of the total calorie decrease was achieved by changing the amounts of meat sauce and tofu. He responded that “Mapo Tofu” contained too many calories, so he just used smaller servings to reduce the number of calories. He also reduced the servings of all other courses. All participants tried to use an effective way to achieve balance between taste and health based on the provided information.

Table 6 shows the results of video analysis. The first measurement yields the glancing rate, which is defined as the percentage of the times that a participant glanced at the calorie display after a calorie-changing cooking action. A high percentage indicates a strong desire to obtain calorie information. Since the purpose of the kitchen was to promote calorie awareness in users, checking whether users actually checked the calorie display while cooking is important. The glancing rate ranged from 55 to 74%. For instance, P2 was very interested in knowing the number of calories in most ingredients, especially when she put new ingredients on the kitchen surface.

The second column in Table 6 lists the average glancing duration, which is defined as the average time a participant spends in glancing at the calorie display. A long average duration indicates that users must take considerable time to

Participants P1 P2 P3(1) Glancing rate 66.7% 74.0% 55.2%(2) Average glancing duration 2.75 sec 2.80 sec 1.48 sec

(3) Average duration of cooking session

45min52mins(15.6%)

55min64mins(16.4%)

62min67min(8.1%)

Table 6. Results of video analysis

comprehend the calorie information and then make an/no adjustment in the next cooking action. The average duration is 2 seconds. The analysis indicates that users spent less than 1 second for low-calorie ingredients (such as garlic with 2kcal), but more time for high-calorie ingredients (such as spaghetti and oil).

The third column in Table 6 presents the average duration of a cooking session. Cooking with the system took about 10 minutes longer than pretest cooking without the system, which was acceptable according to the feedback from the participants.

Qualitative ResultsThe findings of the posttest interviews are described below. P2 stated that “this kind of instant feedback is effective to remind me, for example, what I already know about using the condensed soup”, and P3 said, “I’m glad to get this kind of calorie information without additional effort, and I should really be aware of the need to use less of an

ingredient than the whole package.” P1 said, “I would like to cook with this kitchen in my daily life, to control easily the number of calories in what I regularly cook at home; I believe my brother will lose weight more easily with this."

In the testing cooking, after they knew the amounts of recommended calories, they began to observe the calories they used and control them (Figure 10). P1 said she was already aware of calorie control before using the system, but she was not sure about the exact amounts she used. She said that the system helped her better control the calories. For instance, before cooking, she had set the amount of spaghetti she preferred (1000kcal, or 250kcal for each person), and she used the system to add the exactly amount. P2 and P3 who originally did not care about calories, said that they were surprised when they found how many calories some ingredients contained and wanted to change immediately. For instance, when P2 found that the calories in the condensed soup exceeded half of the recommended calories, she instantly decided to halve the amount, because it was the first course and she still had three more to cook. She also mentioned that she hadn’t checked numbers of calories before, until the system actively showed the values. P3 made similar comments.

Participants had the following expectations of the future direction of this kitchen: (1) they are also interested in nutritional balance, which is difficult for them to measure

Figure 10. Different users checked the number of calories

associated with adding food ingredients

and understand. For instance, they would like to control how much protein, fat and carbohydrates they are using. (2) They would like cooking tips from professionals while they are cooking, such as about the substitutes for certain unhealthy ingredients (butter) or cooking habits (frying), which may also benefit from cook healthily. (3) Regarding the limitation of the food ingredient tracker, they suggested that although they could live with the current system, they would be more relaxed if we could eliminate the limitation.

RELATED WORKMuch research effort [25,26,27,28,29,30,31,32,33,34,35] has focused on augmenting kitchens with various digital media to create rich, interactive experiences for users who are cooking in a kitchen. Some work has focused on increasing awareness to support multi-tasking activities in the kitchen. For instance, the Counter Intelligence project

9

from MIT [25] augmented a kitchen with ambient interfaces to improve the usability of the physical environment. Their augmented kitchen assists users to determine temperatures, find things, follow recipes and time steps during meal preparation. Other work has focused on digital interactive recipes that guide users through a step-by-step cooking process. For instance, Siio et al. [26] automated the creation of web-ready multimedia recipes in a kitchen. By operating a foot switch, a user can capture images of the cooking workplace with voice memos and organize a multimedia recipe. Such digital recipes offer a more interactive experience than a paper-based recipe book. The CounterActive project [27] uses a digital functionality to teach people how to cook by projecting multimedia recipes onto a touch panel-like interactive kitchen counter. Also, the Intelligent Kitchen project [28] presents a human activity recognition system in the kitchen, using data mining to infer the next human action from past behaviors, and then suggesting what to do next using an LCD and a robot.

Rather than augmenting kitchens with a range of digital media to create interactive cooking experiences, our smart kitchen focuses on promoting healthy cooking by raising nutritional awareness during the cooking process, while leaving the decision about how to cook to the users.

Work and commercial products [21,36,37] have exploited ubiquitous computing or mobile devices to record personal food intake and calories. The Dietary-aware Dining Table [21] can track what and how much users eat on the dining table. Work from Mankoff et al. [36] tracks nutrition of foods users have taken and provides suggestions about healthier foods based on analysis of shopping receipt data. IE Institute Co. [37] developed software “Calorie Navi” running on Nintendo DS [38] that can help user record food intake of 300 types of foods and exercise level. However, they focus on tracking and recording food intake itself; therefore differ from our work, which focuses on raising nutritional awareness on preparing and cooking foods in the kitchen at home for all family members.

CONCLUSION AND FUTURE WORKThe Calorie-aware Kitchen employs digital technology to improve traditional meal preparation and cooking by raising awareness of calorie information in ingredients that go into a meal, to promote calorie control. The kitchen is augmented with sensors to detect cooking actions related to ingredients and calorie changes, and then to provide digital feedback on these calories. The user study suggests that the prototype can help users become aware of and control the calories used based on the recommended number of calories for the family. We believe just-in-time feedback during cooking effectively makes family cooks nutritionally aware. We plan to improve nutrition balance and find other opportunities to use Ubicomp technology in the kitchen to promote nutritional control. We would also like to invite

more users to participate in user studies in longer experiments, observe how they exploit this information with different menus, and thus improve our design to achieve our goal more effectively.

REFERENCES1. WHO (World Health Organization). Diet, Nutrition and the Prevention

of Chronic Diseases. In WHO Technical Report Series #916, Joint WHO/FAO Expert Consolation (2003).

2. American Diabetes Association. Evidence-Based Nutrition Principles and Recommendations for the Treatment and Prevention of Diabetes and Related Complications, In Diabetes Care (2002), Vol. 25, Supplement 1.

3. National Task Force on the Prevention and Treatment of Obesity. Long-term Pharmacotherapy in the Management of Obesity. In JAMA (1996), 276:1907-1915.

4. American Diabetes Association. Reading Food Labels. http://www.diabetes.org/nutrition-and-recipes/nutrition/foodlabel.jsp. (2007)

5. Datamonitor Co. Changing Cooking Behaviors & Attitudes: Beyond Convenience. Commercial Report (2006).

6. S. M. E. van Dillen, G. J. Hiddink, M. A. Koelen, C. de Graaf, and C. M. J. van Woerkum. Understanding Nutrition Communication between Health Professionals and Consumers: Development of a Model for Nutrition Awareness based on Qualitative Consumer Research. In American Journal of Clinical Nutrition (2003), Vol. 77, No. 4.

7. INSIGHT 20. Consumption of Food Group Servings: People’s Perception vs. Reality. In USDA Center for Nutrition Policy and Promotion (2000).

8. Harris J, and Benedict F. A Biometric Study of Basal Metabolism in Man. In Washington D.C. Carnegie Institute of Washington (1919).

9. WHO (World Health Organization). Obesity: Preventing and Managing the Global Epidemic. In WHO Technical Report Series #894, WHO Consolation (2004).

10. United States Department of Agriculture. Nutrient Data Products and Services. http://www.ars.usda.gov/Services/docs.htm?docid=13725 (2007)

11. G.A. Bray. Contemporary Diagnosis and Management of Obesity. In Handbook in Health Care (1998), 41-44.

12. INSIGHT 19. Beliefs and Attitudes of Americans toward Their Diet. In USDA Center for Nutrition Policy and Promotion (2000).

13. S. K. Kumanyika, and E. Obarzanek. Pathways to Obesity Prevention: Report of National Institutes of Health Workshop. In Obesity Research (2003), Vol. 11, 1263-1274.

14. Atkin, C. and Arkin, E.B. Issues and Initiatives in Communicating Health Information. In C. Atkin and L. Wallack (eds), Mass Communication and Public Health: Complexities and Conflicts (1990).

15. Donohew, L. Public health campaigns: Individual Message Strategies and a Model. In E.B. Ray and L. Donohew (eds), Communication and Health: Systems and Applications (1990), p36.

16. IKEA Group. http://www.ikea.com/ (2007)17. National Public Health Institute (KTL), Finland. Collected

Information on National Trends and Databases of Composite Foods and Industrial Ingredients. In EuroFIR Partners, European Food Information Resource Network (2005).

18. Schmidt, M. Strohbach, K. Van Laerhoven, A. Friday and H.-W. Gellersen. Context Acquisition based on Load Sensing. In Proc. Ubicomp 2002: the 4th international conference on Ubiquitous Computing (2002).

19. I. Koprinska, and S. Carrato. Temporal Video Segmentation: a Survey. In Signal Processing: Image Communication (2001), vol. 16, 477-500.

20. J.S. Boreczky, and L.A. Rowe. Comparison of Video Shot Boundary Detection Techniques. In Proc. SPIE: Storage and Retrieval for Still Image and Video Databases IV (1996), Vol. 2670.

21. K.-H. Chang, S.-Y. Liu, H.-H. Chu, J. Hsu, C. Chen, T.-Y. Lin, and P. Huang. Dietary-aware Dining Table: Observing Dietary Behaviors

over Tabletop Surface. In Proc. PERVASIVE 2006: the 4th international conference on Pervasive Computing (2006), 366–382.

22. Weill Medical College of Cornell University. Basal Energy Expenditure: Harris-Benedict Equation. http://www-users.med.cornell.edu/~spon/picu/calc/beecalc.htm (2007).

23. WHO (World Health Organization). Global Database on Body Mass Index. http://www.who.int/bmi/index.jsp?introPage=intro_3.html (2007).

24. Campbell Soup Company. Campbell's Condensed Soup. http://www.campbellsoup.com/condensed_soups.asp (2007).

25. L. Bonanni, C.-H. Lee, and T. Selker. Attention-based Design of Augmented Reality Interfaces. In Ext. Abstracts CHI 2005, ACM Press (2005), 1228–1231.

26. I. Siio, N. Mima, I. Frank, T. Ono, and H. Weintraub. Making Recipes in the Kitchen of the Future. In Ext. Abstracts CHI 2004, ACM Press (2004), 1554–1554.

27. W. Ju, R. Hurwitz, T. Judd, and B. Lee. Counteractive: an Interactive Cookbook for the Kitchen Counter. In Ext. Abstracts CHI 2001, ACM Press (2001), 269–270.

28. Y. Nakauchi, T. Fukuda, K. Noguchi, and T. Matsubara. Intelligent Kitchen: Cooking Support by LCD and Mobile Robot with IC-Labeled Objects. In Proc. IROS 2005: International Conference on Intelligent Robots and Systems (2005), 1911-1916.

29. Q. T. Tran, G. Calcaterra, and E. D. Mynatt. Cook’s collage: Déjà vu display for a home kitchen. In Proc. HOIT2005: Home-Oriented Informatics and Telematics (2005), 15–32.

30. J. S. Bradbury, J. S. Shell, and C. B. Knowles. Hands on Cooking: Towards an Attentive Kitchen. In Ext. Abstracts CHI 2003, ACM Press (2003), 996–997.

31. K. Tee, K. Moffatt, L. Findlater, E. MacGregor, J. McGrenere, B. Purves, and S. S. Fels. A Visual Recipe Book for Persons with Language Impairments. In Proc. CHI 2005: the 23rd conference on Human factors in computing systems, ACM Press (2005).

32. H. Ueda. Human-Robot Interaction in the Home Ubiquitous Network Environment. In Proc. HCI International 2007: the 12th international conference on Human-Computer Interaction (2007).

33. Reiko Hamada, Jun Okabe, Ichiro Ide, Shin'ichi Satoh, Shuichi Sakai, and Hidehiko Tanaka. Cooking Navi: Assistant for daily cooking in kitchen. In Proc. MM2005: the 13th ACM International Multimedia Conference, ACM Press (2005), 371-374.

34. M. Kranz, A. Schmidt, R. B. Rusu, A. Maldonado, M. Beetz, B. Hörnler, and G. Rigoll. Context-Aware Kitchen Utilities. In adjunct Proc. INSS 2007: 4th Intl. Conf. on Networked Sensing Systems (2007), 79-186.

35. H. Morishita, K. Watanabe, T. Juroiwa, and T. Mori. Development of Robotic Kitchen Counter: A Kitchen Counter Equipped with Sensors and Actuator for Action-adapted and Personally Fit Assistance. In Proc. IRS2003: Intl. Conf. on Intelligent Robots and Systems (2003).

36. J. Mankoff, G. Hsieh, H. C. Hung, S. Lee, and E. Nitao. Using Low-Cost Sensing to Support Nutritional Awareness. In Proc. Ubicomp 2002: the 4th international conference on Ubiquitous Computing (2002).

37. IE Institute Co. Calorie Navi. http://www.nintendo.co.jp/ds/software/ymdj/index.html (2007).

38. Nintendo Co. Nintendo DS. http://www.nintendo.com/channel/ds (2007).

11