designing guidelines for visual analytics system to augment organizational analytics

Designing Visual Analytics Systemsfor Organizational Environments

Xiaoyu WangResearch Associate at UNC Charlotte

Visiting Research Scientist at PARC

A Framework and Its Guidelines

Motivation

Introduction Background Framework Case Study Evaluation Contribution Future Work

GTDVis

OpsVis

Taste

IRSV

Visual Analytics Systems

Evaluations and

Statistical Analysis

Familiarity with

Domain Users

Computation and

Automation

Knowledge Management

and Organizationa

l learning

What’s a systematic approach to design a user-centered visual analytics system in

organizational environment?

Two-stage Visual Analytics Design Framework


Domain General Analysis Process

Individual Analysis Process

Observation and Design

Observation and Analysis

Design Artifacts Specification

User-centric RefinementSystem Deployment and User

Training

Usage Collection and Customization

VisualAnalyticsSystem

Goals: Generalize domain analytical workflows to present high-level problem-solving direction

Construct a design framework to incorporate both general domain analytical process and individual analysis approaches

Bridge the gap between high-level design concepts and fine-grain implementation of such concepts

Augment organizational information analyses through modeling domain users’ reasoning approaches

Two-step Research Progression

• First step:• Summarize design knowledge

learnt from all my previous research activities

• Identify similarities and unique of each analytical domains and system design correspondingly

• Understand the analytical workflows

• Resulted in guidelines this paper


• Second step:• Top-down approach to create

design framework that encapsulate the knowledge gained

• Utilize existing systems for external evidents to verify and validate the framework

• Apply the framework to further design and research practices

• Resulted in A Two-stage Framework for Designing Visual Analytics System in Organizational Environment (to appear in IEEE VAST 2011 )

Collaborators and Settings


• Bridge Management Project• Team: The US Department of Transportation & Civil Engineering

Department• Scope: Research on techniques for innovative bridge maintenance

planning process

• Document Management Project• Team: Palo Alto Research Center & Xerox Corporation• Scope: Research on efficient visual abstraction for recalling and

managing personal document activities

• Network Operation Management Project• Team: Microsoft Research & Microsoft Cloud Service Team• Scope: Research on effective methods for monitoring and responding

to cloud service

Observation and Design stageObjectives: Characterize general domain analytical processes

Identify design artifacts for visual analytics implementation

Introduction Background Guidelines Evaluation Contribution Future Work

Visual Analytics SystemSummative Evaluation

Ob

serv

atio

n a

nd

Des

ign

Sta

ge

Analytics Requirements

User Analysis Task Analysis Context Analysis

Data Requirements

Data Analysis Process Analysis

Domain Characterization and Analysis Generalization

Domain Observation and

Analysis Formative Evaluation

Evaluation Metrics(key specifications for assessing the system)

Analysis Encapsulation and Visual Encoding

Formative Evaluation

Domain Analysis Dissemination

Interaction Specification

Visualization Specification

Alternative Visualization/Interaction Combinations


Fa

il


Objective: Domain Characterization and Analysis Generalization


Analytics Requirements

User Analysis Task Analysis Context Analysis

Data Requirements

Data Analysis Process Analysis

Formative EvaluationEvaluation Metrics

• Analyze organizations for their technical, analytical, and collaborative context where the visual analytics system will be applied to

Context Analysis

• Specify the tasks and analytical workflow in an organizational environment.

• Verify the design specifications and reduce design costs

Task Analysis

• Domain users’ information (e.g. Demographics, Personal Traits)• Distinguish users broadly by expertise, task experiences, usage

constrains

User Analysis

• Specify the expected analysis goals from the domain users for the designed visual analytics systems

• List key specifications for assessing the system (details in the paper)

Evaluation Metrics

Example---U.S. DOT: Domain Characterization


Analytical Workflow and its

Actionable Knowledge

Methodology: Survey: Nationwide state DOTs

Observations: North Carolina and Charlotte DOT

Gather inspection information

Select bridges within jurisdiction

Analyze collected data

Compare it with prior inspection cycles

Consulting with structural specialist for maintenance necessities

Collaborate with colleagues to balance budgets Prioritize maintenance plan

Prepare maintenance proposal

Submit for final approval

Follow-up work on the execution of the maintenance

Update existing database

Domain Analysis GeneralizationBridge the gap between high-level design concepts and fine-grain implementation of such concepts


Design Artifacts and SpecificationObjectives: Analysis Encapsulation and Visual Encoding

Disseminate high-level task activities into actionable knowledge

Transform actionable knowledge into visual encoding


Formative Evaluation

Domain Analysis Dissemination and Transformation

Interaction SpecificationVisualization Specification

Alternative Visualization/Interaction Combinations

Visual Analytics System

• Specifies the pragmatic view of knowledge utilization and application towards specific analytical ends

• Details the relations between domain analytical tasks and the related knowledge actions• Presents the analytics activities from domain users’ perspectives• Is widely accepted in organizational learning and practices• It IS the design artifacts that represents the fine-grain domain analysis processes

Actionable Knowledge Personalized content and information Easy ‘slice and dice’ information and direct content exploration Examine and depict information from multiple aspects Make sense of significant data patterns and trends

Create hypothesis based on analysis Identify evidence that supports both thesis and antithesis Depict information from multiple aspects Annotate evidence with clear statements Group evidence with reasoning logic

Content Filtering and Customization

Evidence Collection and Hypothesis

Generation

Common Task Activities Key Actionable Knowledge Personalized content and information Easy ‘slice and dice’ information and direct content

exploration Examine and depict information from multiple aspects Make sense of significant data patterns and trends

Create hypothesis based on analysis Identify evidence that supports both thesis and

antithesis Depict information from multiple aspects Annotate evidence with clear statements Group evidence with reasoning logic

Key Actionable Knowledge

Deliver contents in straightforward representation Enable facet filtering for information personalization Interactive content exploration and filtering (Optional) Employ sophisticated data structures

Allow evidence collection and annotation Support storytelling and enable interactive grouping of the

evidence with users’ reasoning logic (Optional) Trace interactions and system usage for future

automation

Visualization and Interaction Specifications

Summary of Designing VA for General Analysis


Domain General Analysis Process

Observation and Design




User-centric RefinementObjectives: Provide a “feedback” loop to incorporate individual analysis routines

and to customize (personalize) visual analytics system

Deploy visual analytics systems and provide trainings to domain users

Enable organizational communication and collaboration


Summative Evaluation


Domain Analysis Workflow Data Infrastructure Visualization

CombinationInteraction

Combination

Usage Collection (both individual level and organizational level)

Interaction Logging Annotation Tracking

Refine Analysis Focuses

Update Data Model

Customize Visualization Combination

Analysis Evaluation and Knowledge Validation

Usage Pattern Analysis and Customization

User-centric Refinement stage II

Pa

ss

DocumentationSupport

Installation

System Deployment and User Training

Training

General Visualization Concepts Analysis Scenarios

Usage Pattern and Customization StepObjectives: Support individual tasks routines and analysis preference

Enable individual’s to collect analytical findings and their analysis provenance

Establish organizational collaborations and facilitate collective decision-making



Domain Analysis Workflow Data Infrastructure Visualization Combination Interaction Combination


Interaction Logging


Update Data ModelCustomize Visualization

Combination

Analysis Evaluation and Knowledge ValidationSystem

Deployment and User Training

Interaction Logging and Capturing User’s Analysis Provenance

Objective: Reveal the relationship between problem solving and interactions

Represent the analysis trail for domain users

Indicate domain users’ analytics preferences


Empirical Proof*: Suggest the clear connection between interactions and the type of strategies users tend to develop

* Empirical study can be found in Dou et al. (2010) : “Comparing different levels of interaction constraints for deriving visual problem isomorphs”

How-to: Log Focus Log Elements

Tracing details of analysis sessions

Low level event (e.g. MouseClick, Key Stroke)

Replay key analysis frames Visual States (e.g. visualization parameters)

Reconstructing user's analysis process

Low-level events and Contextual information and etc.

Interaction Logging ExampleIntroduction Background Framework Case Study Evaluation Contribution Future Work

Log Element:

Visualization Parameters

Data Parameters

Frequencies of views used

Dwell time on each visualization

Analytical Sessions

Utilization of Interaction Log

Example

Visualization Combinations

System places more frequent used visualization combination for individual user based on analyzing the logged frequency and dwell time

Visual Mappings Update

System records the most frequently used visual encoding and its data operators. System makes suggestions if the pattern is repeated

Data and Statistics Model Update

System rearranges the priority of the data based on its usage frequency. System also connects the data with its related statistics for future analysis suggestions

Usage Pattern and Customization StepObjectives: Support individual tasks routines and analysis preference

Enable individuals to collect analytical findings and reveal their analysis provenance

Establish organizational collaborations and facilitate collective decision-making



Domain Analysis Workflow Data Infrastructure Visualization Combination Interaction Combination


Interaction Logging


Update Data ModelCustomize Visualization

Combination

Analysis Evaluation and Knowledge ValidationSystem

Deployment and User Training

Annotation Tracking and Content Sharing

Objective: Enable users to attach semantic meaning to analysis findings

Share analysis findings and results in an organization

Create environment to encourage collective decision-making


How-to:Sharing Mechanism

Content Shared Efficiency Effectiveness Information Sharing Flow

Sharing static annotations

Fixed Image;Textual Information;Drawing;

Easy to construct Can be add-on to existing visual analytics system

More effective in a small-to-medium-sized group

Typically one-way. Information comes from original analyst and shared with other colleagues

Exchanging dynamic annotation

Parameters that can be applied to in another instance of the visual analytics system

Need to be considered at the initial system design. Could be difficult to add onto a existing system

Support larger collaboration groups or departments

Bi-direction, both original analysts and peers can collectively modify and extend the analysis results

Annotation Example: DOT Web


Multiple Evidence

Collections

Freeform Selection and

Graph Connection

Detailed Annotation

Instant Sharing with Colleagues

Summary of Designing VA for Individual Analysis processes


Individual Analysis Process

User-centric Refinement

System Deployment and User Training

Usage Collection and Customization


Contributions• Constructed a two-stage visual analytics design

framework to incorporate both general domain analytical process and individual analysis approaches

• Generalize domain analytical workflows to present high-level problem-solving direction

• Bridge the gap between high-level design concepts and fine-grain implementation of such concepts

• Augment organizational information analyses through modeling domain users’ reasoning approaches


Impacts

Academia Presented a more theoretical approach to design visual analytics system

Encourage academic research for the foundation of visual analytics design

Educational purpose, a syllabus for graduate school teaching material

Industry Concrete and practical guidelines and considerations for designing visual analytics system

Present general ground to bridge research and industry on design and development


Future Work• Continue working interactive learning from domain users’

interaction logs• Machine learning• Reactive (emotion) visualization

• Contribute to the evaluation foundation of visual analytics• Create standard evaluation metrics• Identify key measures for assessing knowledge-gain through using

visual analytics


Questions

Charlotte Visualization Center

http://webpages.uncc.edu/~xwang25

Xiaoyu Wang

Probably on Skype Now..

[email protected]

Case: Design Artifacts and Specification


Summary of Observation and Design stage• Domain observation and Analysis

• Generalization of Domain Analysis Processes• Elements needs to be considered during observation and domain

characterization• Evaluation Metrics that are useful throughout the design as an

assessment to the function

• Design artifacts• Actionable knowledge is a fine-grain items to analytically examine

the domain’s general analytical workflow• Disseminate general task activities into design artifacts through

actionable knowledge• Design considerations that are generated based on design

artifacts.• Visual analytics design needs to follow these artifacts


Example---Xerox: Domain Characterization


Analytical Workflow

Methodology: Semi-structured interview: 30 Knowledge workers

Observations and Interactions: Groups of Xerox employees

Challenges: The date and time when events occurred

(e.g., when documents were received, read, created, or modified);

Content keywords that users associate with activities

(e.g. the title of a document or the name of a person or company);

Document types or applications are used to perform a particular type of activity

(e.g., Microsoft Excel or Apple Mail)

designing guidelines for visual analytics system to augment organizational analytics

Technology

design knowledge

design stageobservation

dice information

research practicesresulted

information personalization

group evidence

users analysis provenance

highlevel design concepts