3-D Selection of Neural Pathway Estimates Using Simple Mouse Gestures

Milestone #1 David Akers

ABSTRACT – A ~100-150 word summary of the interface you plan to prototype, the hypothesis of the value that it provides the user, and how you plan to evaluate if your hypothesis is true or false.

I am planning to develop a new gestural interface for 3-D selection of neural pathways estimated from MRI imaging of human brains. Selection of pathways is important to neuroscientists because it is a precursor to many forms of scientific hypothesis-generation and analysis. Existing interfaces are frustratingly inefficient, since they require the three-dimensional placement of regions-of-interest (ROIs) within the volumetric data space, using only a mouse and keyboard. The proposed system should address this inefficiency by providing an image space solution: converting simple, natural, 2-D mouse gestures into three-dimensional path selections. First, I plan to build a prototype version of the system, and use a heuristic evaluation to understand the design space. Having learned lessons from this heuristic evaluation, I hope to design and conduct a formal user study, comparing the new system to existing systems in current use. The details of this formal user study are not yet decided.

TASK ANALYSIS – Who are the users? What tasks will the users need to perform? What new tasks do they desire to perform? Where are the tasks performed? How are the tasks learned? What set of tools does the user have now? How often do your users perform the tasks? What happens when things go wrong? What’s the relationship between the user and his or her data? How do your users communicate with each other? With whom do they communicate?

My tool is intended for neuroscientists studying the structure of the brain’s white matter using Diffusion Tensor Imaging (DTI), a form of magnetic resonance imaging. Scientists are able to use DTI images to extract neural pathway estimates in individual subjects, using a set of techniques known as “white matter tractography”. As part of their daily routine, scientists use computer software to analyze these pathway estimates and compare them between different subjects. I will focus on improving the critical subtask of pathway selection: identifying sets of pathways for further analysis. Tractography may produce some pathways of no interest to the neuroscientist (and that may obscure the desired pathways!) Moreover, tractography produces many false positive connection results, and the neuroscientist must use domain-specific knowledge to remove these false

pathways. As described in the project proposal, pathway selection is an important precursor to much of the scientific inquiry that scientists perform. This activity may include visual comparisons between individuals, group (statistical) comparisons, or the training of probabilistic models. Neuroscientists at Stanford and elsewhere currently use software based on dynamic query interfaces to select pathways [Akers et al. 2004, Sherbondy et al. 2005, Blaas et al. 2005]. Such software tools provide ways to select pathways in two ways: (1) using slider bars to select min/max values for a variety of pathway statistical properties (path length, curvature, etc.) or (2) by using “volumes of interest” to select pathways that connect particular anatomical regions. Our own DTI-Query software has become quite popular since we released it this spring: over 50 groups around the world have downloaded the software to use in their research. The Wandell lab at Stanford has just started using DTI-Query on a regular basis. They also have their own software (MR Vista) for statistical analysis and visualization, so getting them to use DTI-Query required integrating it with MR Vista. In the Wandell lab, there are two patterns of use for the DTI-Query / MR Vista software combination:

Exploratory visualization is done on a daily basis, and is used to generate and evaluate scientific hypotheses. This kind of work is typically done independently by individual researchers, in their offices. Intriguing discoveries (for example, an unusual set of pathways found in an individual with a particular brain disorder) are marked and later shared with other members of the group. To support this form of collaboration, they save the entire state of the program (including all ROIs, slider settings, etc.), so that the visualization can be recreated at a convenient time.

Explanatory visualization is done more rarely, and is used to generate

convincing evidence for scientific conclusions (usually for publication). This work is often more collaborative, with several researchers looking over the shoulder of the software user. The goal is to produce the clearest possible communication of a particular set of scientific evidence.

Pathway selection plays an important, but time-consuming role in both of these activities. Most of the time is spent delicately positioning ROIs within the data set. This process is particularly difficult and frustrating because the ROIs must be positioned in three dimensions, and the input device (a mouse) only affords two-dimensional manipulation in a natural way. When an ROI is moved incorrectly, the user undoes the change. ROIs are sometimes used for two purposes simultaneously. By creating anatomically-meaningful ROIs (corresponding to previously-identified regions of

the brain), a neuroscientist can show only the pathways that connect these known regions. In this case, the ROIs serve both to (1) select a set of pathways and (2) evaluate a hypothesis about connectivity. ROIs can also be (roughly) transferred from one subject to another, using data registration techniques. This can be used to accelerate the search for a set of pathways in a new subject using knowledge from a previous selection performed on another subject. Whatever technique I develop will need to support this transference property.

IDEATION – Sketches and Storyboards of your ideas. This will be the main part of your grade for this assignment. We want to see lots and lots of ideas. Brainstorm! Not all your ideas have to be related to your proposed system. You may turn in actual pen-on-paper sketches, or scanned in sketches, or digital sketches done in Illustrator, Flash, etc. Sketches and storyboards should be accompanied by short blurbs to explain each idea.

See Appendix. EVIDENCE – Present some evidence that your idea is a good one. What observations did you make during your contextual inquiries that support your idea? What other systems exist that support similar tasks?

Contextual inquiries with neuroscientists in the Wandell lab have showed that substantial time and effort is spent selecting pathways for analysis. A solution that made pathway selection more efficient would improve scientists’ workflow substantially. Selection of objects in 3-D is generally considered to be a hard problem, especially when there are many objects and the objects occlude one another. However, in our case we should be able to leverage the structure of the objects (pathways) that scientists are selecting. Pathways naturally bundle together in the brain, meaning that adjacent pathways will often have similar orientations (at least at some point along their length). As suggested in the project proposal and illustrated in the “ideation” section, my strategy will be to interpret 2-D mouse gestures as 3-D selection commands, using the orientation cues to disambiguate the gestures when multiple pathway bundles fall within the “selection cone”. Disambiguation of 3D selection using orientation cues has been suggested by researchers studying interaction in immersive virtual environments (Forsberg et al. 1996, Pierce et al. 1997). In the system described by Pierce, the image-plane orientation of the user’s hand was matched against the orientation of each object within the current viewing cone, and the best match was chosen. For example, in the example below, the orientation of the fingers is compared with the “natural orientation” of the chair. It seems reasonable to believe that such a system could

be adapted to a mouse input device, since image plane input is inherently two-dimensional.

Figure 1: Pierce et al. 1997: The “head crusher”

technique for 3-D selection

Olwal et al. (2003) developed a 3-D selection system for augmented reality, called “SenseShapes.” They attach volumetric selection primitives (such as cones and spheres) to the user’s hands and head. The system dynamically collects and tracks statistics of each object in the scene with respect to each “sensing shape”. This enables them the user to ask questions like, “What are all objects that I was pointing at 2 seconds ago, that were visible?” While this approach shows promise in immersive virtual environments, it is not clear whether it extends to the traditional GUI-based interface. I do like the idea of gathering information over time to infer what a user is interested in selecting – perhaps there is a way to leverage this?

Figure 2: An illustration of Olwal et al.'s "SenseShapes" system, a 3-D selection mechanism for augmented reality environments.

FURTHER EVIDENCE – How will you collect further evidence to support your intuition that your system is a good one?

Until now, it has been difficult to find time to meet with members of the Wandell lab. They have been busy preparing a grant proposal due November 1. After this time, they should be much more available to meet and discuss the project. It will be very helpful to continue the contextual inquiry process with them. (Several of their team members have been so busy that I haven’t even been able to meet with them at all, yet.) I would also like to expand the contextual inquiry to include scientists outside of the Wandell lab. I have several strategies to find additional scientists:

(1) My research partner and I know several other students and faculty at Stanford who use the DTI-Query software on a regular basis. (2) I will try to contact a member of the Radiology Department at UCSF, who has downloaded DTI-Query to use in his research group.

EVALUATION PLAN – How will you evaluate your hypotheses? What type of user tests? How do you plan to set up your user tests? How many people will participate in your study?

My evaluation plans consist of two phases:

• An initial heuristic evaluation, to make sure I’m on the right track and address any major design issues.

• A formal user study, designed for summative evaluation.

For the heuristic evaluation, I will begin by defining several (2 or 3) likely interaction scenarios. To come up with realistic scenarios, I plan to consult several of the eventual users for my tool. This should ground the heuristic evaluation in meaningful, challenging tasks that real users encounter on a daily basis. I plan to build a medium-fidelity prototype of my system, to use in the heuristic evaluation. The goal of the heuristic evaluation will be to explore the design space. Most of the design issues I plan to explore are contained within the “ideation” section at the back of this document. The basic system is surprisingly easy to prototype, since it can be built on top of our existing ROI-based software. Certain difficult tasks (for example, estimating the shape of an ellipse from a user’s rough gesture) will be handled using Wizard-of-Oz prototyping techniques. This should

avoid too much unnecessary programming effort for features that end up getting dropped. Following the guidelines of Nielsen, I plan to use 5 participants in my heuristic evaluation. I am hoping to find “double experts” to partake in my study, but this may be difficult. I would try to avoid enlisting my eventual intended users, since Nielsen’s article made compelling arguments against this. I will wait to define the formal user study until after the completion of the heuristic evaluation. This should give me a much better idea of what I interfaces I should be comparing, and how I might go about comparing them.

REFERENCES Akers, D., Sherbondy, A., Mackenzie, R., Dougherty, R., and Wandell, B. (2004) "Exploration of the Brain's White Matter Pathways with Dynamic Queries." In Proceedings of IEEE Visualization 2004, pp. 377-384. Blaas, J., Botha, C., Peters, B. and Vos, F., (2005) “Fast and Reproducible Fibre Bundle Selection in DTI Visualization” Proceedings of IEEE Visualization 2005, pp. 59-64. Fitzmaurice, G. W., Ishii, H. and Buxton, W. (1995). Bricks: Laying the Foundations for Graspable User Interfaces, In Proceedings of CHI’95, 432-449. Forsberg, A., Herndon, K. and Zeleznik, R. (1996) “Aperture Based Selection for Immersive Virtual Environments.” ACM Symposium on User Interface Software and Technology. Olwal, A., Benko, H., and Feiner, S. “SenseShapes: Using Statistical Geometry for Object Selection in a Multimodal Augmented Reality System.” In Proceedings of The Second IEEE and ACM International Symposium on Mixed and Augmented Reality (ISMAR 2003). Tokyo, Japan. October 7–10, 2003. p. 300–301. Pierce, J. S., Forsberg, A. S., Conway, M. J., Hong, S., Zeleznik, R. C. & Mine, M. R. (1997), “Image plane interaction techniques in 3d immersive environments.” In Proceedings of the 1997 symposium on Interactive 3D graphics, ACM Press. Sherbondy, A., Akers, D., Mackenzie, R., Dougherty, R., and Wandell, (2005) B., "Exploring Connectivity of the Brain's White Matter with Dynamic Queries." In IEEE Transactions on Visualization and Computer Graphics, Volume 11, Issue 4 (July 2005), pp. 419-430.