parallelism is everywhere – so how do we make it accessible? daniel ernst the university of...
TRANSCRIPT
Parallelism is Everywhere –So How Do We Make It Accessible?
Daniel ErnstThe University of Wisconsin-Eau Claire
Parallelism is Everywhere –So How Do We Make It Accessible?
Daniel Ernst
Department of Computer ScienceThe University of Wisconsin – Eau Claire
Parallelism is Everywhere –So How Do We Make It Accessible?
Daniel ErnstThe University of Wisconsin-Eau Claire
A Little Background …
• “Grew Up” (Graduate School) in Computer Architecture– Efficient, fast, low-power processors (2 ISCA, 1 MICRO papers, +)
• “Dabbled” in supercomputing– (Becoming a bigger part of my job now…)
• Became a CS educator
• Now– Trying to make sense of how these are all coming together
Parallelism is Everywhere –So How Do We Make It Accessible?
Daniel ErnstThe University of Wisconsin-Eau Claire
Moore’s Dividend
Parallelism is Everywhere –So How Do We Make It Accessible?
Daniel ErnstThe University of Wisconsin-Eau Claire
Where Did All The Performance Go?Software Development, c. 1950-2005
Figure byJim Larus[CACM 09]
“What Andy giveth, Bill taketh away.”
Parallelism is Everywhere –So How Do We Make It Accessible?
Daniel ErnstThe University of Wisconsin-Eau Claire
Can Multi-Core Supplant Moore’s “Dividend”?
• Double number of cores instead of speed
• NO, at least not without major innovation– Sequential code base will not scale without lots of help– Lack of parallel algorithms (and/or poor knowledge of them)– Difficult (or at least different) programming challenges– Fewer abstractions currently available
(for individual programs)
Parallelism is Everywhere –So How Do We Make It Accessible?
Daniel ErnstThe University of Wisconsin-Eau Claire
Parallel Algorithms
“In the context of sequential algorithms, it is standard practice to design more complex algorithms that outperform simpler ones (for example, by implementing a balanced tree instead of a list). For non-blocking algorithms, however, implementing more complex data structures has been prohibitively difficult.”
[Herlihy, Luchangco, Moir, Scherer, PODC 2003](Discussing a concurrent red-black tree)
Parallelism is Everywhere –So How Do We Make It Accessible?
Daniel ErnstThe University of Wisconsin-Eau Claire
Sequential to Parallel – A History• Existing code is sequential
• We tried parallel compilers in the ‘80s-’90s– Can get good code for a few specific cases, but not generally
• We tried to get the hardware to extract parallelism in 90’s-00’s– Dynamic mechanisms cannot find more than 2–4x parallelism
• The issue is with how programmers fundamentally think about problems– Not language specific
• If we’re training programmers to think only in sequence, how well will they be prepared for dealing with explicit parallelism?
Parallelism is Everywhere –So How Do We Make It Accessible?
Daniel ErnstThe University of Wisconsin-Eau Claire
Fundamental Take-aways
• Explicit parallelism is important, and is getting more so– Being able to identify it will be an increasingly marketable skill
• Understanding the underlying architectures is also becoming more important– Necessary to have at least basic understanding to extract reasonable amounts of
parallelism– Hybrid machines will make this critical if you want good performance
• ESPECIALLY before the tools catch up (but still after)
Parallelism is Everywhere –So How Do We Make It Accessible?
Daniel ErnstThe University of Wisconsin-Eau Claire
Programming Futures• There is a lack of consensus on the parallel
programming model– Data parallelism (Vector)– Unstructured (thread) parallelism
• (Edward Lee – “The Problem with Threads”)• Speculative threading?
– Message passing
• Lots of ongoing research with not much resolved• Moral: Everyone in the CS field needs to be prepared
to learn about new parallel programming methods and tools.
Parallelism is Everywhere –So How Do We Make It Accessible?
Daniel ErnstThe University of Wisconsin-Eau Claire
State of the Game
Educators This Talk
Industry
This (slightly modified) graph shamelessly stolen from Jim Larus
Parallelism is Everywhere –So How Do We Make It Accessible?
Daniel ErnstThe University of Wisconsin-Eau Claire
An Opportunistic Approach
• Typically, he/she who develops functionality to use fully use the hardware capabilities “wins”– Google, facebook, intel, Microsoft, etc. all took advantage of technological
inflection points
• So, if we don’t know what to teach yet, how do we prepare our students to be leaders?
Parallelism is Everywhere –So How Do We Make It Accessible?
Daniel ErnstThe University of Wisconsin-Eau Claire
Some Context• UW – Eau Claire“Founded in 1916, UW-Eau Claire is one of the Midwest’s top public universities. UW-Eau
Claire provides students with small classes led by talented professors, cutting-edge programs, hands-on research and life-enhancing study abroad opportunities.
Motto: Excellence. Our measure, our motto, our goal.• What that actually means:
– Approx. 11k students, only 500 are grad students– Focus is undergraduate engagement
• Teaching excellence• Undergraduate research
Parallelism is Everywhere –So How Do We Make It Accessible?
Daniel ErnstThe University of Wisconsin-Eau Claire
UW – Eau Claire Computer Science• ABET Accredited comprehensive CS major• 6 faculty (+1 vacancy), 1 part-time academic staff• Approximately 40-50 graduates per year in 3 majors:
– Comprehensive CS– Software Engineering– Computer Engineering (CS degree with CE emphasis)
• We’re filthy rich (sort of)
• Constituencies / Industrial Partners / Landing Spots– Healthcare, insurance (Marshfield Clinic, NW Mutual)– Some systems development (Cray/IBM)
Parallelism is Everywhere –So How Do We Make It Accessible?
Daniel ErnstThe University of Wisconsin-Eau Claire
UW – Eau Claire Computer Science• Focus is on keeping an innovative curriculum
– A majority of our scholarship is within this scope– SIGCSE-related activity is large
• total of 31 papers and 7 panels presented in SIGCSE sponsored conferences and journals since 2002
– The department has recently been nominated three times as the University of Wisconsin Teaching Department of the Year
(2001, 2002, 2003)• Curricular Basics:
– Java early, C/C++ later, plus a full PL course– Full P&H Comp Org.– Late: strong software engineering coursework along with robust systems
classes (OS/Networks)Not for the casual student
Parallelism is Everywhere –So How Do We Make It Accessible?
Daniel ErnstThe University of Wisconsin-Eau Claire
Parallelism in the Curriculum – Current• Many (most?) schools treat parallel computing as an independent
topic, addressed in the senior year as an elective.
Parallelism is Everywhere –So How Do We Make It Accessible?
Daniel ErnstThe University of Wisconsin-Eau Claire
Curricular Goals at UW-ECStudents should gain:
– a generic understanding of computational concurrency and parallelism.• see parallelism in problems• be able to exploit it in a reasonable way
– exposure to parallelism as an orthogonal method with respect to other standard curricular items.• Don’t just teach parallelism by itself – integrate it in with other
elements as simply another tool in the programming box.• Teaching it in context can lower time overhead, as well
– confidence in using parallelism as a tool.
Parallelism is Everywhere –So How Do We Make It Accessible?
Daniel ErnstThe University of Wisconsin-Eau Claire
Principles for Implementation
• Simplicity– Keep to easy (deterministic) and natural concepts
• Ubiquity– More than a “1-shot” issue
• Accessibility– Don’t make the learning curve steeper than it has to be
Parallelism is Everywhere –So How Do We Make It Accessible?
Daniel ErnstThe University of Wisconsin-Eau Claire
Keeping Parallelism Simple• First, tweak the underlying concept of a computer
– A “collection of brains”
• The concepts behind basic parallelism are simple– Sharing work
• Different tasks• Split up data
• Especially if they’re given a more natural context
Simplicity
Parallelism is Everywhere –So How Do We Make It Accessible?
Daniel ErnstThe University of Wisconsin-Eau Claire
Natural Parallelism/Concurrency• We have very natural allies in the educational graphics/animation
toolsets– Alice, Scratch, etc.
• These tools make basic concurrency natural and largely deterministic– easier to debug!
Simplicity
Parallelism is Everywhere –So How Do We Make It Accessible?
Daniel ErnstThe University of Wisconsin-Eau Claire
Another Natural Example – Swarm Intelligence• “Creative” Monte Carlo solution to TSP
– Instantiate multiple agents to probabilistically discover the shortest trip– “Threads as Entities”
• Each agent (“ant”) operates in its own thread– ants leave pheromone trails along arcs to influence later ants
• Arc information needs to be synchronized
• Orthogonal topics:– Monte Carlo– heuristics
Simplicity
Parallelism is Everywhere –So How Do We Make It Accessible?
Daniel ErnstThe University of Wisconsin-Eau Claire
Other “Clean” Parallel ExamplesSimplicity
Parallelism is Everywhere –So How Do We Make It Accessible?
Daniel ErnstThe University of Wisconsin-Eau Claire
Parallelism Everywhere Parallelism the Norm• Unlike the past, sequential code written today is likely to
only see very small (if any) gains from hardware upgrades. – As the number of cores increases, non-parallel performance
will become relatively more unacceptable in practice.
• Exploiting parallelism needs to be a first-class citizen– (or at least second-class)– not an item studied once and then not used again.– Ubiquitous
Ubiquity
Parallelism is Everywhere –So How Do We Make It Accessible?
Daniel ErnstThe University of Wisconsin-Eau Claire
An Integrated Model at UW-EC• CS1 – Decomposition and Assignment
– Introduce methods for recognizing obvious parallelism– Practice with simple, natural, and integrated examples
• CS2 – Basic Orchestration– Continued practice with obvious parallelism– Introduce communication
• Algorithms – Parallelism Limits– Extend algorithm analysis to parallel algorithms– Introduce more complicated communication patterns
• Operating Systems – Synchronization Constructs– “The usual topics “
Ubiquity
Parallelism is Everywhere –So How Do We Make It Accessible?
Daniel ErnstThe University of Wisconsin-Eau Claire
After a Reality Check…• CS2 – Decomposition and Assignment
– Introduce methods for recognizing obvious parallelism– Practice with simple, natural, and integrated examples
• CS2/Algorithms – Basic Orchestration– Continued practice with obvious parallelism– Introduce communication
• Algorithms/OS – Parallelism Limits– Extend algorithm analysis to parallel algorithms– Introduce more complicated communication patterns
• Operating Systems – Synchronization Constructs– “The usual topics “
Ubiquity
Parallelism is Everywhere –So How Do We Make It Accessible?
Daniel ErnstThe University of Wisconsin-Eau Claire
CS2 Possibilities – Image Mosaic• “Nifty Assignment” (Pattis, SIGCSE 2005)
– create an image out of tiles ofsmaller images
– tiles are independent(unless you want it to look good!)
• Basic parallelism is easy, butgood quality needs synchronization– Payoff!
• Can vary synchronization difficulty
• Orthogonal topics:– searching/sorting/matching
Ubiquity
Parallelism is Everywhere –So How Do We Make It Accessible?
Daniel ErnstThe University of Wisconsin-Eau Claire
The Role of Architecture• Understanding how a program “lays out” on a machine.
– What is it capable of?– What are the bottlenecks?
• My take thus far:– Decreased emphasis on assembly programming– Slightly decreased emphasis on microarchitecture
• (Depending on program needs)– Increased emphasis on system-level view (interaction between processor
and memory)
Ubiquity
Parallelism is Everywhere –So How Do We Make It Accessible?
Daniel ErnstThe University of Wisconsin-Eau Claire
Keeping Parallelism Accessible• For each individual curriculum, the choice of language/platform
has a significant effect on accessibility.
• At UW-EC, we are largely a “Java shop”, with some C/C++– To keep learning curve gentle, we try to operate within that framework– Java.util.concurrent / OpenMP
• Taking the time to introduce a new platform eats far too much time and can lead to projection of learning difficulties*– (and they’re just going to change, anyways)
The concepts are more important than the platform
*YMMV
Accessibility
Parallelism is Everywhere –So How Do We Make It Accessible?
Daniel ErnstThe University of Wisconsin-Eau Claire
The Delicate Balance
While we want to have student learning to be as platform-agnostic as possible…
…they also need to learn real skills for the workforce…
…and they need to practice with the concepts somehow!
Accessibility
Parallelism is Everywhere –So How Do We Make It Accessible?
Daniel ErnstThe University of Wisconsin-Eau Claire
Current Challenges1. General Workload
– Especially given the ubiquity argument, there’s a lot of material that needs to be created/re-worked in new contexts
– How do we motivate instructors?
2. Faculty Readiness– At least anecdotally, the percentage of faculty who feel
comfortable teaching parallel computing is painfully small– How can we help with faculty accessibility?
3. Future Uncertainty– Given that we’re not quite sure what future programming
models will look like – what topics should we be teaching?
Parallelism is Everywhere –So How Do We Make It Accessible?
Daniel ErnstThe University of Wisconsin-Eau Claire
Current Challenges (More?!?)4. Availability of Tools/Language Support for Parallelism
– There are very few language/toolset combinations that collectively fulfill the major criteria for curricular languages:
• Language used in industry (or related to one)• Supports OO• Good support for “easy” parallelism• Not syntactic suicide
Parallelism is Everywhere –So How Do We Make It Accessible?
Daniel ErnstThe University of Wisconsin-Eau Claire
The Reality of Implementation
• If change is really going to be cross-curriculum, it requires that there is significant buy-in from every faculty/staff member that teaches any course that includes the new material.
• This is a serious professional development challenge
Parallelism is Everywhere –So How Do We Make It Accessible?
Daniel ErnstThe University of Wisconsin-Eau Claire
EAPF – Addressing The ChallengesEducational Alliance for a Parallel Future• Combined effort of Industry:
– Intel, nVidia, AMD, Microsoft, etc.• and Academia
– Members range from research (UIUC/Berkeley) to supercomputing experts to small college educators
• Goals:– Increase awareness– Encourage and support innovation in training and tools
• Keep an eye out for more…
Parallelism is Everywhere –So How Do We Make It Accessible?
Daniel ErnstThe University of Wisconsin-Eau Claire
Thank You!
Q & A