Download - Karl Lieberherr Northeastern University College of Computer and Information Science Boston, MA
The Scientific Community Game: Education and
Innovation Through Survival in a Virtual World of Claims
The Scientific Community Game: Education and
Innovation Through Survival in a Virtual World of Claims
Karl LieberherrNortheastern University
College of Computer and Information ScienceBoston, MA
joint work with Ahmed Abdelmeged and Bryan Chadwick
Karl LieberherrNortheastern University
College of Computer and Information ScienceBoston, MA
joint work with Ahmed Abdelmeged and Bryan Chadwick
Supported by Novartis
Why Scientific Community Game(SCG)
• … motives in academic publishing: – desire for recognition and respect from the people
one regards as peers, – desire to have impact (on conclusions being
reached, on the development of the discipline, etc.), and
– desire to participate in significant knowledge-building discourse.
• e.g., Scardamalia, M., & Bereiter, C. (1994)
Bionetics 2010 2
SCG is Bio-inspired
• Virtual world of scholars based on natural selection– propose, oppose (refute and strengthen) claims– maximize reputation, weak scholars are removed.
• Turn problem-solving software into virtual organisms that fend for themselves and survive in a virtual world inhabited by virtual organisms created by your peers.
Bionetics 2010 3
SCG is a web-based implementation of Karl Popper’s science ideas
• One of the greatest philosophers of science of the 20th century.
• Falsifiability or refutability is the logical possibility that an assertion could be shown false by a particular observation or physical experiment.
• Error elimination (refutation), performs a similar function for science that natural selection performs for biological evolution.
Bionetics 2010 4from Wikipedia
Comparison
• Karl Popper: Conjectures and Refutations, 1963
• Scientific Community Game: Claims and Refutations, 2007
Bionetics 2010 5
Recognition in SCG
• Scholars build their reputation by proposing and opposing claims, by defending their own claims and refuting or strengthening the claims of others.
• The higher their reputation, the more recognition.
Bionetics 2010 6
Impact in SCG
• Second-order environment– what one scholar does in adapting, changes the
environment so that others must readapt.
• Developing novel techniques to find superior solutions, challenges others to catch up.
Bionetics 2010 7
Knowledge-Building Discourse in SCG
• Communication or debate.• Refutation protocol defines the structure of
the debate and who wins. Claims are defined through a refutation protocol.
• Knowledge-building:– claims that have been defended predominantly
are candidates for truth– claims that have been refuted predominantly are
probably false.Bionetics 2010 8
Goals of SCG
• Put knowledge-building discourse on the web giving participants the option to gain recognition and to have impact.
• Focus the discourse through precise definition of claims with refutation protocols.
• Make knowledge building discourse fun and educational from the high school to the advanced research level.
Bionetics 2010 9
SCG = Scientific Community Game = Specker Challenge Game
What do we mean by science?
• Science consists of the formulation and testing of hypotheses based on observational evidence.
• Ours: Science consists of the formulation and testing of constructive claims based on observational evidence. Construction is computable.
Bionetics 2010 10
What do we mean by Scientific Method
• Hypothetico-deductive method: Formulate a hypothesis in a form that could conceivably be falsified by a test on observable data.
• Ours: Formulate a constructive claim in a form that could conceivably be falsified by a test using a protocol. The refutation protocol is part of the claim to make very explicit when refutation is successful.
Bionetics 2010 11
Bionetics 2010 12
Tartaglia against Fior1535
Tartaglia was famed for his algebraic solution of cubic equations which was published in Cardan's Ars Magna.
Outline
• Introduction– Popper Science, Renaissance History: Tartaglia and Fior
• Definition of SCG– Example (Highest safe rung)
• Applications: Teaching, Software Development, Research• Claims with secrets and other protocol variants• Output of SCG, Equilibrium• Advantages and Disadvantages• Conclusions
Bionetics 2010 13
Definition of SCG: Domain
• Problem: Set• Solution: Set• valid: relation(Problem, Solution)• quality: function(Problem, Solution)->[0..1]
14Bionetics 2010
Claim(Domain)
• Problems: Powerset(Domain.Problem)• q: Quality = [0,1]• r: Resource = N+ = positive integer
Alice claims to have a technique to solve problems in Problemswith at least quality q and using at most resources r.
15Bionetics 2010
makes predictionsabout the future
Implied Protocol of Claim(Domain)
• Alice claims (problems,q,r), Bob refutes• Bob provides problem prob in Claim.Problems. • Alice solves problem prob providing sol in
Domain.Solution.• check: valid(prob,sol) and quality(prob,sol)>=q and
sol.resource<=r.• sol.resource returns Alice’ resource consumption to
solve problem prob.
16Bionetics 2010
Karl Popper: Only hypotheses capable of clashing with observation reports are allowed to count as scientific.
Claim
• Problems: subset of problems• quality in [0,1]
Bionetics 2010 17
0
1
quality(how wellproblems inProblems can be solved)
Claim
Bionetics 2010 1818
0
1
qualitystrengthening
correct valuation
over strengthening
Bio-inspired computing: Virtual World of SCG-Avatar
• SCG-Avatar (Claim(Domain))– State: Reputation = positive rational number– Activity
• propose new claims• oppose claims of others
– refute claim(Problems, q, r)– strengthen claim(Problems, q’, r’), q’>q or r’<r
• Reputation gain: refute others’ claims and defend own claims (counter refutation attempts)
• Reputation loss: unsuccessful refutation of other’s claim and refutation of own claims
19Bionetics 2010
Tournament 1. round-robin2. Swiss-style3. elimination
1. single2. double
20Bionetics 2010
Summary of SCG Definitions
Domain Problem Solution valid(Problem, Solution) quality(Problem, Solution) →[0,1]
21Bionetics 2010
Claim(Domain) Problems: PowerSet(Domain.Problem) q: Quality = [0,1] r: Resource = N+
Rules of the Scientific Community: propose and oppose,be an active scholar, rules for reputation accumulation.
Tournaments
Highest Safe Rung
• You are doing stress-testing on various models of glass jars to determine the height from which they can be dropped and still not break. The setup for this experiment, on a particular type of jar, is as follows.
Bionetics 2010 22
Highest Safe Rung
Only two identical bottles to determinehighest safe rung
Alice Bob
23Bionetics 2010
You have a ladder with n rungs, and you want to find the highest rung from which you can drop a copy of the jar and not have it break. We call this the highest safe rung. You have a fixed ``budget'' of k > 0 jars.
Highest Safe Rung
Only two identical bottles to determinehighest safe rung
HSR(9,2) ≤ 4 I doubt it: refutation attempt!
Alice Bob
Alice constructsdecision tree T ofdepth 4 and gives itto Bob. He checkswhether T is valid.Bob wins if he findsa flaw.
24Bionetics 2010
3
1
0
6
1 2
4
3
5
9
97
6
87
2
4
5
8
x
y z
yes no
u
highest safe rung
Highest Safe Rung Decision TreeHSR(9,2)=5
25Bionetics 2010
Finding solution for HSR(n,2)
• Approximate min x in [0,n] (n/x) + x
• Exact – MaxRungs(x,y) =MaxRungs(x-1,y-1)+MaxRungs(x-1,y)– MaxRungs(x, 2) = x + MaxRungs(x – 1, 2)– MaxRungs(0, 2) = 1– Applied to HSR(9,2)
• MaxRungs(3,2) = 7 < 9• MaxRungs(4,2) = 11 > 9
26Bionetics 2010
Keith Levin CS 4800 Fall 2010
MaxRungs(x,y) = the largest numberof rungs we can test with y jars andx experiments.
breaks at root does not break at root
Find minimum x, s.t. MaxRungs(x,2) > n
MaxRungs
• MaxRungs(x,y) = sum [k=0 .. y] binomial(x,k)• All paths are of length x. At most k branches
may be left branches.• Note: y = x implies MaxRungs(x,y) = 2x
meaning a complete binary tree of depth x.• Example: binomial(3,2)+binomial(3,1)+
binomial(3,0) = 7
Bionetics 2010 27
Formal: HSR
• Domain: – Problem: (n,k), k <= n.– Solution: Decision tree to determine highest safe
rung.– quality(problem, solution): depth of decision tree /
number of rungs– valid(problem, solution): at most k left branches, ...
28Bionetics 2010
Formal: HSR
• Claim(Domain): – Alice claims ({(25,2)},9/25,5 seconds)
• {(25,2)}: set of problems (singleton)• 9/25: quality• 5 seconds: resource
• Refutation Protocol:– Bob refutes: only one problem: (25,2)– Alice: solves problem by providing decision tree t.– predicate: t is a valid decision tree for (25,2) of depth 9
Bionetics 2010 29
Claim involving Experiment
Claim ExperimentalTechnique(X,Y,q,r)I claim, given raw materials x in X,I can produce product y in Y of quality qand using resources at most r.
30Bionetics 2010
Outline
• Introduction– Popper Science, Renaissance History: Tartaglia and Fior
• Definition of SCG– Example (Highest safe rung)
• Applications: Teaching, Software Development, Research• Claims with secrets and other protocol variants• Output of SCG, Equilibrium• Advantages and Disadvantages• Conclusions
Bionetics 2010 31
Applications: Software Development
• Software Development• Teaching Constructive Domains
Bionetics 2010 32
Gamification of Software Development etc.
• Want reliable software to solve a computational problem? Design a game where the winning team will create the software you want.
• Want to teach a STEM domain? Design a game where the winning students demonstrate superior domain knowledge.
Bionetics 2010
Doesn’t TopCoder already do this?
STEM = Science, Technology, Engineering, and Mathematics
33
SCG and TopCoder
• SCG is an abstraction and generalization of what TopCoder does.
Bionetics 2010 34
The Traditional Approach
Solver A
Static Benchmark
Solver B
Solver C
Team A
Team B
Team C
Parameterized by the domain.
Software: Solving HSR Problem:construct decision tree of min. depth
measure how closeto minimumHSR(9,2)=4
HSR(25,2)=7
Ranking
36Bionetics 2010
The Bio-Inspired Approach
Team ASolver A
prop-opp A
Team CSolver C
prop-opp C
Team BSolver B
prop-opp B
VirtualWorld
(Game)Ranking
Parameterized by the domain.
AvatarA
AvatarC
AvatarB
DynamicBenchmark
37Bionetics 2010
A Virtual WorldAvatar’s View
Administrator
Avatar
Opponents’ communication,Feedback
Claims,Problems,Solutions
Results
• Problems: Benchmark output• Solutions: Software output• Claims: statements about algorithms
38Bionetics 2010
What Scholars think about!
• If I propose claim C, what is the probability that– C is successfully refuted– C is successfully strengthened
• If I try to refute claim C, what is the probability that I will fail.
• If I try to strengthen claim C, what is the probability that I will fail?
39Bionetics 2010
SCG = Scientific Community Game
• Make software development more scientific.• Software developers build reputation
– propose and defend claims about their software– oppose claims made by others
• refute claims• strengthen claims
• claim includes refutation protocol
Bionetics 2010 40
Who are Alice and Bob?
• They are avatars developed by real Alice and real Bob.
• Alice and Bob compete with 10 other avatars in a full-round robin tournament.
• Who is the winner: The avatar with the highest reputation, i.e., the avatar who has the strongest, not successfully opposed claims (like in a real scientific community).
Bionetics 2010 41
Why a web application with avatars? Fair Evaluation.
What is SCG(X)
Bionetics 2010 42
no automationhuman plays
full automationavatar plays
degree of automation used by scholar
our focus
some automationhuman plays
0 1
more applications:test constructive knowledge
transfer to reliable, efficient software
avatar Bob
Alice
Real Scholars and Avatars:Same rules
• Are encouraged to 1. propose claims that are not easily strengthened.2. offer claims that they can successfully support.3. strengthen others’ claims, if possible. 4. stay active and propose new strong claims or
oppose others’ claims.5. become famous!
43Bionetics 2010
What we want
• Engage software developers– let them produce software that models an
organism that fends for itself in a real virtual world while producing the software we want. Have fun. Focus them.
– let them propose claims about the software they produce. Reward them when they
• defend their claims successfully or • oppose the claims of others successfully.
Bionetics 2010 44
Clear Feedback Sense of Progress
Possibility of Success
Authenticity (Facebook)
SCG
• Gamification of software development for computational problems
• A Sociotechnical System for knowledge dissemination, innovation, and integration
45Bionetics 2010
Software Engineering Properties fostered by SCG
• Reliable (otherwise the avatar is removed from the game)
• Flexible, modular (otherwise the avatar cannot be easily updated between tournaments)
• Efficient (otherwise you cannot defend your claims and oppose the claims of others)
Bionetics 2010 47
Adaptive and Aspect-Oriented Software is relevant!
State of Avatar SCG-Avatar: Our Vision
• Companies come to SCG website and define a competition by defining a claim domain X.
• Participating teams get baby avatars generated from X that participate in daily competitions.
• Competition generates a wealth of information: educated employees, good (undefeated) software, good algorithms, good potential employees. Reward is paid to the winner.
Bionetics 2010 48
State of SCG-Avatar: Our Vision
• Not only companies but faculty members who want to give their students a rich learning experience for computational problem X.
• Or editors of special issues in journals who want to use a competition to get a real world comparison of all approaches to solve computational problem X.
Bionetics 2010 49
Avatars propose and oppose
Bionetics 2010 50
CA1
CA2
CA3
CA4
egoisticAlice egoistic
Bob
reputation 1000 reputation 10
CB1
CB2
opposes (1)
provides problem (2)
solves problem
not as well as she expected based on CA2 (3)WINS!LOSES
proposed claims
transfer 200
social welfare
Life of an avatar: (propose+ oppose+ provide* solve*)*
What is SCG(X)?
TeamsDesign Problem Solver
Develop SoftwareDeliver Avatar
Avatar Alice Avatar Bob
Administrator SCG police
I am the best No!!
Let’s play constructively
51Bionetics 2010
TeamAlice
TeamBob
competitive / collaborative
Bionetics 2010 52
Avatar Alice: claim C
Avatar Bob: opposes C, refutes: providesevidence for !C
loses reputation r wins knowledge k
wins reputation r makes public knowledge k
Outline
• Introduction– Popper Science, Renaissance History: Tartaglia and Fior
• Definition of SCG– Example (Highest safe rung)
• Applications: Teaching, Software Development, Research• Claims with secrets and other protocol variants• Output of SCG, Equilibrium• Advantages and Disadvantages• Conclusions
Bionetics 2010 53
Protocol Variants
• secrets: approximation problems• involving trusted third party
– renaissance: exchange of problems
Bionetics 2010 54
Example: Triple HSR
• Alice claims ({(25,2,0), (25,2,1), (25,2,2), (25,2,3), … ,(25,2,25)},9/25, 5 seconds)
• Refutation Protocol:– Bob refutes (25,2,17)– Alice solves problems (25,2,*) by providing
decision tree to trusted third party which reveals path p from root to 17.
– predicate: p is valid and length(p) <= 9
55Bionetics 2010
Highest Safe Rung
Protocol Variation Secrets
• problem has public and private part, private part is a secret solution
• predicate has secret as argument
56Bionetics 2010
Protocol Variation Secret Program for SCG-Avatar
• problem has public and private part, private part is a secret solution and goes to administrator
• Alice gives her algorithm to administrator who applies it to public part of problem
• predicate has secret as argument
57Bionetics 2010
Example Claims involving secrets
• My algorithm can solve more problems using resources r than your algorithm using r.
• If I create problems for you for which I have a solution, you cannot recreate or approximate the solution with quality q using resources r.
58Bionetics 2010
Output and Equilibrium
• Rich tournament history• What is an equilibrium in SCG?
Bionetics 2010 59
Soundness Theorem
• SCG is sound: The avatar with the best algorithms / knowledge wins (there is no way to cheat)– best: within the group of participating avatars– issues:
• Does an avatar win because she is good at solving? Or good at proposing, opposing and providing? Answer: proposing, opposing and providing all reduce to solving.
04/21/23 Games for SD 60
SCG Equilibrium
• reputations of scholars are stable• the ranking of the scholars is invariant from
tournament to tournament• the science does not progress; bugs are not
fixed, no new ideas are introduced• extreme example: All scholars are perfect:
they propose optimal claims C(ps,q) that can neither be strengthened nor refuted.
Bionetics 2010 61
• [Scientific Innovation in X] Avatars get skills programmed into them by clever scientists in domain X. Scientists use data mining to learn from competitions and manually improve the avatars.
• [Machine Learning Innovation in X] Avatars get skills programmed into them by an avatar caregiver programmed with learning skills and data mining skills for domain X. Avatar gets updated automatically.
Survival in SCG(X)
62Bionetics 2010
second-order environment!
Blame assignment
• Where is the proposer to blame?– Bad claim that is refuted.
– Bug in problem finding algorithm?
– Bug in problem solving algorithm?
63Bionetics 2010
How to use SCG(X)• Company AB needs new ideas about how to
solve optimization problems in domain X.• Define claims language for X
– X-problems– claims, includes protocol
• Submit claims language definition to SCG server.
64Bionetics 2010
How to use SCG(X)• Offer prize money for winner with conditions,
e.g., performance must be at least 10% higher as performance of avatar XY that AB provides.
• 10 teams from 6 countries sign up, committing to 6 competitions. Player executables become known to other players after each competition. One team from company AB.
• The SCG server sends them the basic avatar and the administrator for testing.
65Bionetics 2010
How to use SCG(X)
• Game histories known to all. Data mining!• First competition is at 23.59 on day 1.
Registration starts at 18.00 on same day. The competition lasts 2.5 hours.
• Repeat on days 7, 14, … 42.• The final winner is: Team Mumbai, winning
10000 Euro. Delivers source code and design document describing winning algorithm to AB.
66Bionetics 2010
Benefits for company AB of using SCG(X)
• Teams perform know-how retrieval and integration and maybe some research. – Participating teams try to find the best knowledge in
the area.– Claims language gives control!
• The non-refuted claims give hints about new X-specific knowledge.
• A well-tested solver for X-problems that integrates the current algorithmic knowledge in field X.
67Bionetics 2010
Outline
• Introduction– Popper Science, Renaissance History: Tartaglia and Fior
• Definition of SCG– Example (Highest safe rung)
• Applications: Teaching, Software Development, Research• Claims with secrets and other protocol variants• Output of SCG, Equilibrium• Advantages and Disadvantages• Conclusions
Bionetics 2010 68
Benefits/Disadvantages
• Benefits– competitive / collaborative– structured feedback, game history– Teaching– Research– Software Development
• Dynamic testing and evaluation
• Disadvantages– addictive
Bionetics 2010 69
Disadvantages of SCG
• The game is addictive. After Bob having spent 4 hours to fix his avatar and still losing against Alice, Bob really wants to know why!
• Overhead to learn to define and participate in competitions.
• The administrator for SCG(X) must perfectly supervise the game. Includes checking the legality of X-problems.– if admin does not, cheap play is possible– watching over the admin
70Bionetics 2010
How to compensatefor those disadvantages
• Warn the scholars.• Use a gentleman’s security policy: report
administrator problems, don’t exploit them to win.
• Occasionally have a non-counting “attack the administrator” competitions to find vulnerabilities in administrator.– both generic as well as X-specific vulnerabilities.
71Bionetics 2010
Benefits of SCG
• Social Welfare – Supported knowledge
• Claims are refuted and strengthened.• Better supported knowledge comes from better
algorithms and software.72Bionetics 2010
Advantage: Democratic
• Problem to be solved: Develop the best practical algorithms for solving computational problems in domain X.
• Issue: There are probably hundreds of papers on the topic with isolated implementations. What are the best practical algorithms?
• Our solution: Use the scientific community game SCG(X) with a suitably designed claims language to compare the software. The winning avatar has the best practical algorithms/software.
73Bionetics 2010
Experience with MAX-CSP
• MAX-CSP Problem Decompositions• T-Ball (one relation), Softball (several
relations, one implication tree), Baseball (several relations).
• ALL, SECRET
74Bionetics 2010
Stages for SECRET T-Ball
• MAXCUT – R(x,y)= x!=y– fair coin ½ – maximally biased coin ½ – semi-definite programming / eigenvalue
minimization 0.878
75Bionetics 2010
Stages for SECRET T-Ball
• One-in-three– R(x,y,z) = (x+y+z=1)– fair coin: 0.375– optimally biased coin: 0.444
76Bionetics 2010
Stages for ALL Baseball
• Propose/Oppose/Provide/Solve – based on fair coin– optimally biased coin
• correctly optimize polynomials
– correctly eliminate noise relations– correctly implement weights– …
77Bionetics 2010
References
• Karl Popper, Conjectures and Refutations, London: Routledge (1963).
• Scardamalia, M., & Bereiter, C. (1994). Computer support for knowledge-building communities. The Journal of the Learning Sciences, 3(3), 265-283.
• Renaissance: Tartaglia and Fior challenge (1535).
Bionetics 2010 78
Conclusions• To address a computational problem domain X:
– “map it to second life”: define a scientific community game for X on the web: SCG(X)
– let the game SCG(X) run a few times and choose the winner; use strongest unrefuted claims.
• Benefits– Evaluates fairly, frequently, constructively and
dynamically. Encourages retrieval of state-of-the-art know-how, integration and discovery.
– Challenges humans, drives innovation, both competitive and collaborative.
– Avatars point humans to what needs attention in problem solution / software.
Bionetics 2010 79
Conclusions
• Broad applicability, e.g.,• SCG(X) provides a learning process for any
constructive domain. • Benefits
– Social Engineering: makes it fun through game.– Fair: Only hard work makes you win.– Engage a large community on one domain X.
Bionetics 2010 80
Thank You
Bionetics 2010 81
Title
• The Scientific Community Game: Education and Innovation Through Survival in a Virtual World of Claims
82Bionetics 2010
Abstract• The Scientific Community Game (SCG) is a generic game for constructive domains
where claims are defined by interactive protocols. As a starter, this includes mathematical claims containing alternating quantifiers but also non-mathematical claims involving the interaction between two parties. Scholars in SCG propose and oppose claims. Opposition means refutation or, strengthening followed by refutation. The winning scholars are good at proposing strong, un-refutable claims and at spotting refutable claims of other scholars. Scholars collaborate through competition. Applications include teaching constructive topics such as calculus and algorithms.
• An especially interesting version of SCG is Avatar SCG where the scholars are implemented in software. Avatar SCG is a web application implementing competitions between hundreds of avatars spread over the web.
– Applications of Avatar SCG include:• distributed software development for computational problems• distributed knowledge maintenance and integration for computational problems• teaching software development skills
83Bionetics 2010
Extra
Bionetics 2010 84
• 1st International ICST Conference on Theory and Practice of Algorithms in (Computer) Systems,
• 18-20 April 2011 - Rome, Italy• experimental analysis of algorithms
85Bionetics 2010
Formal: Pair HSR
• Alice claims ({(25,2)},9/25)• {(25,2)}: set of problems• 9/25: quality• Refutation Protocol:
– Bob refutes: only one problem: (25,2)– Alice: solves problem by providing decision tree t.– predicate: t is a correct decision tree for (25,2) of
depth 9.
86Bionetics 2010
Domain
• Problem: Set• Solution: Set• valid: relation(Problem, Solution)• quality: function(Problem, Solution)->[0..1]
87Bionetics 2010
Claim(Domain)
• Problems: Powerset(Domain.Problem)• q: Quality = [0,1]• r: Resource = N+ = positive integer (optional)
Alice claims to have a technique to solve problems in Problemswith at least quality q and using at most resources r.
88Bionetics 2010
Implied Protocol of Claim(Domain)
• Alice claims (problems,q,r), Bob refutes• Bob provides problem prob in Claim.Problems. • Alice solves problem prob providing sol in
Domain.Solution.• check: valid(prob,sol) and quality(prob,sol)>=q and
sol.resource<=r.• sol.resource returns Alice’ resource consumption to
solve problem prob.
89Bionetics 2010
Benefit of a game SCG-Avatar(Claim(Domain))
• Knowledge discovery and evaluation– If a claim was attacked 1000 times and refuted
only twice, it is a reasonable candidate for truth.– Depends on strength of avatars: Can they defend
true claims? Can they refute wrong claims?
90Bionetics 2010
Game kinds
• 2-player game– used in tournament (full round robin or Swiss
style)
• n-player game
91Bionetics 2010
Data produced by a game SCG-Avatar(Claim(Domain))
• Ranking of avatars• History: data exchanged through refutation
protocols, claims proposed and strengthened• A claim strengthened must be opposed by
original proposer.• Data mining of history
92Bionetics 2010
Two modes
• Teaching– give warning if
• true claim is refuted• false claim is supported
• Research– don’t know the true and false claims
93Bionetics 2010
Reflection 2
• Claim: For all NPO optimization problem P, there exists an SCG-Avatar game G:– better than putting 20 people into a room– learn more (give them a test)– have more fun
• Refutation protocol:
94Bionetics 2010
Reflection 1
• For all constructive domains X, there exists a SCG game G=SCG(X):– if there is at least one good scholar, the
participating scholars compared to when they cooperate in a non-structured way:
• create more knowledge about X using G, given the same amount of time, because they stay focused on X.
• have more fun thanks to the competition and collaboration.
95Bionetics 2010
Reflection 3
• For all challenging constructive domain X teaching tasks, there exists an SCG family game G for X– students learn more
96Bionetics 2010
Reflection Software Development
• Software Development Process based on SCG
97Bionetics 2010
The Traditional Approach
Solver A
Static Benchmark
Solver B
Solver C
Team A
Team B
Team C
Parameterized by the domain.
Software: Solving HSR Problem:construct decision tree of min. depth
measure how closeto minimumHSR(9,2)=4
HSR(25,2)=7
Ranking
98Bionetics 2010
The Bio-Inspired Approach
Team ASolver A
prop-opp A
Team CSolver C
prop-opp C
Team BSolver B
prop-opp B
VirtualWorld
(Game)Ranking
Parameterized by the domain.
AvatarA
AvatarC
AvatarB
DynamicBenchmark
99Bionetics 2010
A Virtual WorldAvatar’s View
Administrator
Avatar
Opponents’ communication,Feedback
Claims,Problems,Solutions
Results
• Problems: Benchmark output• Solutions: Software output• Claims: statements about algorithms
100Bionetics 2010
What Scholars think about!
• If I propose claim C, what is the probability that– C is successfully refuted– C is successfully strengthened
• If I try to refute claim C, what is the probability that I will fail.
• If I try to strengthen claim C, what is the probability that I will fail?
101Bionetics 2010
Best response dynamicsNash Equilibria
102Bionetics 2010
Highest Safe Rung
• You are doing stress-testing on various models of glass jars to determine the height from which they can be dropped and still not break. The setup for this experiment, on a particular type of jar, is as follows.
Bionetics 2010 103
Highest Safe Rung
Only two identical bottles to determinehighest safe rung
Alice Bob
104Bionetics 2010
You have a ladder with n rungs, and you want to find the highest rung from which you can drop a copy of the jar and not have it break. We call this the highest safe rung. You have a fixed ``budget'' of k > 0 jars.
Highest Safe Rung
Only two identical bottles to determinehighest safe rung
HSR(9,2) ≤ 4 I doubt it: refutation attempt!
Alice Bob
Alice constructsdecision tree T ofdepth 4 and gives itto Bob. He checkswhether T is correct.Bob wins if he findsa flaw.
105Bionetics 2010
3
1
0
6
1 2
4
3
5
9
97
6
87
2
4
5
8
x
y z
yes no
u
highest safe rung
Highest Safe Rung Decision TreeHSR(9,2)=5
106Bionetics 2010
Finding solution for HSR(n,2)
• Approximate min x in [0,n] (n/x) + x
• Exact – MaxRungs(x,y) =MaxRungs(x-1,y-1)+MaxRungs(x-1,y)– MaxRungs(x, 2) = x + MaxRungs(x – 1, 2)– MaxRungs(0, 2) = 1– Applied to HSR(9,2)
• MaxRungs(3,2) = 7 < 9• MaxRungs(4,2) = 11 > 9
107Bionetics 2010
Keith Levin CS 4800 Fall 2010
MaxRungs(x,y) = the largest numberof rungs we can test with y jars andat most x experiments.
breaks at root does not break at root
2
1
0
4
3
2
4
1
3
x
y z
yes no
u
highest safe rung,leaf
Highest Safe Rung Decision TreeHSR(4,2)=3
(2 y (1 y h 0 h 1) n (4 y (3 y h 2 n h 3) n h 4))Properties of decision tree:
1. at most two yes from root to any leaf.2. longest root-leaf path has 3 edges.3. each rung 1..n appears exactly once as internal node of the tree.4. each rung 0..n appears exactly once as a leaf.
root
108Bionetics 2010
2
1
0
4
3
2
4
1
3
x
y z
yes no
u
root
109Bionetics 2010
x
y z
yes no
u
110Bionetics 2010
Swiss Tournament 1. round-robin2. Swiss-style3. elimination
1. single2. double
111Bionetics 2010
The smallplayer canwin unexpectedly.
112Bionetics 2010
Elimination Tournament
113Bionetics 2010
Claim involving Experiment
Claim ExperimentalTechnique(X,Y,q,r)I claim, given raw materials x in X,I can produce product y in Y of quality qand using resources r.
114Bionetics 2010
Why Scientific Community Game
• … motives in academic publishing: – desire for recognition and respect from the people
one regards as peers, – desire to have impact (on conclusions being
reached, on the development of the discipline, etc.), and
– desire to participate in significant knowledge-building discourse .
• e.g., Scardamalia, M., & Bereiter, C. (1994)
Bionetics 2010 115
Recognition in SCG
• Scholars build their reputation by proposing and opposing claims, by defending their own claims and refuting or strengthening the claims of others.
• The higher their reputation, the more recognition.
Bionetics 2010 116
Impact in SCG
• second-order environment– what one scholar does in adapting changes the
environment so that others must readapt.
• Developing novel techniques to find superior solutions challenges others to catch up.
Bionetics 2010 117
Knowledge-Building Discourse in SCG
• communication or debate.• Refutation protocol defines the structure of
the debate and who wins. Claims are defined through a refutation protocol.
• Knowledge-building:– claims that have been defended predominantly
are candidates for truth
Bionetics 2010 118
Bereiter & Scardamaglia
• Scardamalia, M., & Bereiter, C. (1994). Computer support for knowledge-building communities. The Journal of the Learning Sciences, 3(3), 265-283.
Bionetics 2010 119
Discourse
• We have roughly divided characteristics for knowledge-building discourse into three categories: – focus on problems and depth of understanding; – decentralized, open knowledge environments for
collective understanding; and – productive interaction within broadly conceived
knowledge-building communities.
• Scardamalia, M., & Bereiter, C. (1994)Bionetics 2010 120
SCG Equilibrium
• reputations of scholars are stable• the ranking of the scholars is invariant from
tournament to tournament• the science does not progress; bugs are not
fixed, no new ideas are introduced• example: All scholars are perfect: they
propose optimal claims C(ps,q) that can neither be strengthened nor refuted.
Bionetics 2010 121
Claims and Refutation Protocol
• Alice claims: I have a program that solves inputs in domain X with quality Q and resources R. – AliceClaim(X,Q,R)
• Bob is critical. He prepares an input in X and gives it to Alice who applies her program. Bob refutes AliceClaim(X,Q,R) iff Alice achieves < Q or uses > R. – Refutation protocol
Bionetics 2010 122
State of Avatar SCG
• Domain is hard-wired to Constraint Satisfaction Problems
• One Master student worked on making it generic but work is not complete.
Bionetics 2010 123