[chi 2016] selph: progressive learning and support of manual photo color enhancement

SelPhProgressive Learning and Support ofManual Photo Color EnhancementYuki Koyama, Daisuke Sakamoto, Takeo Igarashi

Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016

■ Parameter tweaking

- Based on the designer’saesthetic preference

3

Photo Color Enhancement

[Adobe Photoshop]


■ Tedious

4

Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016 5

Enhancing Many Photos


■ Could be tedious

■ Need manual enhancement of each photo one by one, independently

6

Enhancing Many Photos


■ Tedious

7Y. Koyama, D. Sakamoto, T. Igarashi | SelPh: Progressive Learning and Support of Manual Photo Color Enhancement | CHI 2016

Solution?: Auto-Enhancement


Auto-Enhancement is NOT Perfect

Manual edit (ground truth)

Original Auto [Adobe Photoshop]



Different





One-by-one manual tweaking is necessary



Our Approach


Our Approach

Achieving automatic enhancement

Supporting manual enhancement


Our Approach



✗ ✓


Our Approach



New workflow concept: Self-reinforcing color enhancement

✗ ✓


Enhancement with a “Self-Reinforcing” System



Target photos

User’s edits



Target photos

User’s edits

…



Learn

Target photos

User’s edits

Preference model

…



Learn

Support

Target photos

User’s edits

Preference model

…



Learn

Support

Update

Learn

Target photos

User’s edits

Preference model

…



Learn

Support

Update

Learn

Support

Target photos

User’s edits

Preference model

…



Learn

Support

Update

Learn

Support

Update

Learn

Target photos

User’s edits

Preference model

…



Learn

Support

Update

Learn

Support

Update

Learn

Support

Target photos

User’s edits

Preference model

…



Learn

Support

Update

Learn

Support

Update

Learn

Support

Update

Learn

Target photos

User’s edits

Preference model

…



…

…

Learn

Support

Update

Learn

Support

Update

Learn

Support

Update

Learn

Support

Update

Target photos

User’s edits

Preference model

…


■ Implicit

- The user can work as usual - No explicit training phase

■ Progressive

- The more photos are enhanced, the more useful the system becomes

29

“Self-Reinforcing” System


■ Implicit


■ Progressive


30



■ Implicit


■ Progressive


31


Related Work


UIs for Design Exploration

Design Galleries [Marks+, SIGGRAPH97]

■ Design Galleries

- Sampling-based selection

■ Our approach

- Direct manipulation of each parameter using slider






■ Our approach




■ VisOpt Slider [Koyama+, UIST14]

- Visualization

- Optimization



VisOpt Slider [Koyama+, UIST14]

Adopted into our prototype system




■ [Koyama+, UIST14]

- Training Data: Crowdsourcing

- Interaction: Pre-computing

■ This work

- Training Data: Editing history

- Interaction: Progressive / Implicit








■ This work




A Prototype System


SelPh A self-reinforcing system for photo enhancement

43

A Prototype System


1. Goodness Visualization

2. Interactive Optimization

3. Auto-Enhancement

4. Variable Confidence

5. Reference Photos

44

User Support Functions

1. Visualization

2. Optimization

3. Auto-Enhancement

4. Variable Confidence

5. Reference Photos

Self-Reinforcement Algorithm


Interactive enhancement



Push the “Next” button




Update the distance metric

d( , ) = ?




Update the preference model

Update


d( , ) = ?




Show the next photo


Update


d( , ) = ?




Show the next photo


Update


d( , ) = ?

preference model は説明しません、という


Distance Metric Learning

d( , ) = ?


■ We adopt techniques by [Kapoor+, IJCV14]

- Learning a personalized distance metric from editing history

60

Distance Metric Learning

d( , ) = ?

d( , ) d( , )is as equivalent to as possibleS.t.

User Study


■ Qualitative evaluation of our approach / system

- How photographers enhance photos

- How they are satisfied

63

Purpose


■ Participants: 8 photographers

■ Pre-task: Take 100 photos

■ Main-tasks:

- Enhance the first 50 photos using { Baseline

- Enhance the other 50 photos using { SelPh | Baseline }

■ Post-task: Interviews and questionnaires

64

Study Method




■ Main-tasks:




65

Study Method




■ Main-tasks:




66

Study Method




■ Main-tasks:

- Enhance the first 50 photos using { Baseline | SelPh }



67

Study Method




■ Main-tasks:




68

Study Method

Prepared by limiting the user support functions of SelPh




■ Main-tasks:




69

Study Method

Prepared by limiting the user support functions of SelPh

Results


Post-Task Questionnaires

Q1#

Q2#

Q3#

Q4#

Q5#

Q6#

Q7#

Q8#

Q9#

Q10#

Q11#

Q1#

Q2#

Q3#

Q4#

Q5#

Q6#

Q7#

Q8#

Q9#

Q10#

Q11#

Q1#

Q2#

Q3#

Q4#

Q5#

Q6#

Q7#

Q8#

Q9#

Q10#

Q11#

Q1 Visualization of goodness on sliders was useful compared to the absence of it.

Q2 Interactive optimization of slider values was useful compared to the absence of it.

Q5 Auto-enhancement in SelPh was more useful than that in commercial software.

Q7 Reference photos in SelPh were more useful than those in Baseline.

Q8 Confidence value was useful.Strongly disagree

Strongly agree

About user support functions — Positive


Post-Task Questionnaires

Q1#

Q2#

Q3#

Q4#

Q5#

Q6#

Q7#

Q8#

Q9#

Q10#

Q11#

Q9 As the task proceeds, I felt that the system learns my preference or intent.

Q10 The learning result reflected my preference or intent.

Q11 It is preferable for the system to learn my preference or intent.

Strongly disagree

Strongly agree

About overall approach — Positive


■ About Overall Experience

- “The functions [in SelPh] […] evoke the feeling of collaborating with another me.”

- Using the baseline system, “[I felt] lonely.” In contrast, “there is interaction with the [self-reinforcing] system,” thus “executing the task [with SelPh] was fun.”

73

Comments from Interviews (1/2)


■ About Confidence Value

- “[According to the confidence value, I] decided to use or not to use the optimization and the auto-enhancement”

- “I could trust the system more [by knowing the confidence]”

- “It was an enjoyable experience to do the task [while knowing the confidence]”

- “[I felt] humanity from the confidence”

74








75








76








77



Quantitative Results

Marginally faster [p < .10] Significantly smaller [p < .05]


■ All the participants were satisfied with self-reinforcing color enhancement

■ The inclusion of the confidence value makes SelPh more trustworthy and enjoyable to use

81

Summary & Lessons Learned




82





83


Conclusion

■ Self-reinforcing system for manual photo enhancement

- Progressively and implicitly learning the user’s preference

■ A prototype system: SelPh

- Five user support functions (e.g., variable confidence)

■ Qualitative user study

- Photographers prefer the proposed workflow

Paper / Videos / Software / Source Codes

are available at

http://koyama.xyz/project/SelPh/

SelPhProgressive Learning and Support ofManual Photo Color EnhancementYuki Koyama, Daisuke Sakamoto, Takeo Igarashi

[chi 2016] selph: progressive learning and support of manual photo color enhancement

Technology