Gathering momentumMichael Bernstein Stanford University http://crowdresearch.stanford.edu
Today’s topics! Reminder of our new methods + goals! Task posting! Working! Monitoring! Homepage! Reputation! Open governance! This week’s milestones
2
Our method + plan
August 31: Platform is online, real work is being done on it
! Task creation, including Prototype Task foundation! Task feed and task submission! Task review
! Our first workers (skunkworks)! Our first work (skunkworks)
4
September 25: CHI paper! A full paper, 10 pages, at CHI, with you as a coauthor! A CHI paper will need:! A working platform! Fully thought-through and implemented foundations! Real usage that we can report
5
Let’s make a trigger! A trigger will be our formal criteria for beginning that
transition.! We want something that is a strong indicator we are
achieving liftoff and will succeed.
! Our trigger: 100 projects completed per week ! Fine print: from at least 15 different requesters — this can’t just be
just Rajan spending his savings to post work to the platform
6
Quarterly feedback! Every three months (or before a major deadline), we’ll
give people 100 credits to allocate across the contributors to the project.! We let you summarize your contributions on the wiki before this
credit vote happens! Give proportional credit to whoever you think is most critical to
our success! We will then run PageRank on this credit vote network to prevent
small groups from accidentally influencing the vote
7
PageRank and link rings! You cannot get credit by having your team all just vote
for you
8
New York Times
Bad #1 Bad #2
Bad #3Legit site
Lots of PageRankflows from NYTimesto legit, but badget almost none
Paid RA positions! There were 25 applications as of this morning! So great to hear of the interest!
! I am currently working with Stanford to create the positions, and will share results next week
! Goal: expand team as I bring in more research funds
9
Contributors: our new process! When you join a milestone on Trello, you are joining an
ad-hoc team for the week to pursue that milestone. Immediately go check in via the relevant Slack room.! “Hey @channel, I’m joining the task creation milestone this week”
! Then: tell the room whenever you’re starting, and whenever you’re pausing. This will prevent you from working on out-of-date items.! “Hey @channel, I’m going to work on the heuristic evaluation for the next two hours. Shout if I shouldn’t!” 10
DRIs: our new process! DRIs are responsible for…! Organizing any ad-hoc hangouts or hackathons to make sure that
everyone who is in the channel knows how to contribute.
! Posting in #announcements by Friday 5pm each week: an update on their team’s progress. Describe what each team member did.! Michael will use this for weekly slides and for recommendation letters.
11
Task posting
DRI: @aginzberg
Worker feed
DRI: @dmorina
Task monitoring
DRI: @ryosuzuki and @neilthemathguy
Homepage
DRI: @karolina
Next generation crowdsourcing
LEARN MORE
CREATE ACCOUNT
WELCOME TO THE FUTURE OF CROWDSOURCING!
Our elevator pitch: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nullam ac pellentesque nulla. Duis vel leo ut tortor feugiat dapibus quis non diam. Cras ac euismod tortor.
SAMPLE TASKS
How it works Login About
LOREM IPSUM DOLOR SIT AMET, CONSECTETUR ADIPISCING ELIT.
LOREM IPSUM DOLOR SIT AMET, CONSECTETUR ADIPISCING ELIT.
LOREM IPSUM DOLOR SIT AMET, CONSECTETUR ADIPISCING ELIT.! "
"
HERE GOES THE LOGO
SMALL LOGO
Next generation crowdsourcing
LEARN MORE
CREATE ACCOUNT
WELCOME TO THE FUTURE OF CROWDSOURCING!
Our elevator pitch: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nullam ac pellentesque nulla. Duis vel leo ut tortor feugiat dapibus quis non diam. Cras ac euismod tortor.
SAMPLE TASKS
How it works Login About
LOREM IPSUM DOLOR SIT AMET, CONSECTETUR ADIPISCING ELIT.
LOREM IPSUM DOLOR SIT AMET, CONSECTETUR ADIPISCING ELIT.
LOREM IPSUM DOLOR SIT AMET, CONSECTETUR ADIPISCING ELIT.! "
"
HERE GOES THE LOGO
SMALL LOGO
THESE ARE THE LAST 5 TASKS POSTED IN DAEMO:X
Prototype Task: Design Logo
31-07-2015
Develop sketches with 3 options for my photography business logo.
Payment: $10
Skills: Not listed
Number of tasks: 1
Time to complete: 1 min#
$
%
&
'
Prototype Task: Program HTML5 Website
31-07-2015
Next generation crowdsourcing
LEARN MORE
CREATE ACCOUNT
WELCOME TO THE FUTURE OF CROWDSOURCING!
Our elevator pitch: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nullam ac pellentesque nulla. Duis vel leo ut tortor feugiat dapibus quis non diam. Cras ac euismod tortor.
SAMPLE TASKS
How it works Login About
LOREM IPSUM DOLOR SIT AMET, CONSECTETUR ADIPISCING ELIT.
LOREM IPSUM DOLOR SIT AMET, CONSECTETUR ADIPISCING ELIT.
LOREM IPSUM DOLOR SIT AMET, CONSECTETUR ADIPISCING ELIT.! "
"
HERE GOES THE LOGO
SMALL LOGO
LOGINX
Username or email
Password
Forgot your password?
LOGIN
Next generation crowdsourcing
LEARN MORE
CREATE ACCOUNT
WELCOME TO THE FUTURE OF CROWDSOURCING!
Our elevator pitch: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nullam ac pellentesque nulla. Duis vel leo ut tortor feugiat dapibus quis non diam. Cras ac euismod tortor.
SAMPLE TASKS
How it works Login About
LOREM IPSUM DOLOR SIT AMET, CONSECTETUR ADIPISCING ELIT.
LOREM IPSUM DOLOR SIT AMET, CONSECTETUR ADIPISCING ELIT.
LOREM IPSUM DOLOR SIT AMET, CONSECTETUR ADIPISCING ELIT.! "
"
HERE GOES THE LOGO
SMALL LOGO
LOGINX
Username or email
Password
Forgot your password?
LOGIN
How it works Login About
TASKSA variety of tasks for a variety of crowdsourcers
! " # $ | & ' ( )
FOR EMPLOYERS FOR WORKERS
adipiscing elit.
non diam.
adipiscing elit.
non diam.
WHY DAEMO?
FOR EMPLOYERS FOR WORKERS
Create your account now: Create your account now:
SMALL LOGO
How it works Login About
TASKSA variety of tasks for a variety of crowdsourcers
! " # $ | & ' ( )
FOR EMPLOYERS FOR WORKERS
adipiscing elit.
non diam.
adipiscing elit.
non diam.
WHY DAEMO?
FOR EMPLOYERS FOR WORKERS
Create your account now: Create your account now:
SMALL LOGO
22
How it works Login About
TESTIMONIALS
ABOUT US
SMALL LOGO
⋆⋆⋆⋆⋆
“ Lorem ipsum dolor sit amet, consectetur adipiscing elit.
“
⋆⋆⋆⋆⋆
“ Lorem ipsum dolor sit amet, consectetur adipiscing elit.
“
⋆⋆⋆⋆⋆
“ Lorem ipsum dolor sit amet, consectetur adipiscing elit.
“
⋆⋆⋆⋆⋆
“ Lorem ipsum dolor sit amet, consectetur adipiscing elit.
“Snehal (Neil) Gaikwad, Durim Morina, Rohit Nistala, Megha Agarwal, Alison Cossette, Radhika Bhanu, Saiph Savage, Vishwajeet Narwal, Karan Rajpal, Jeff Regino, Aditi Mithal, Adam Ginzberg, Aditi Nath, Karolina Zi-ulkoski, Trygve Cossette, Dilrukshi Gamage, Angela Richmond-Fuller, Ryo Suzuki, Jeerel Herrejon, Kevin Viet Le, Claudia Flores-Saviaga, Haritha Thilakarathne, Kajal Gupta, William Dai, Ankita Sastry, Shirish Goyal, Thejan Rajapakshe, Niki Abolhassani, Angela Xie, Surabhi Ingle, Veronica Jaramillo, Martin Daniel, Walter Angel, Carlos Toxtli, Juan Pablo Flores, Asmita Gupta, Vineet Sethia, Diana Padilla, Kristy Milland, Kristiono Setyadi, Nuwan Wajirasena, Muthitha Batagoda, Rolan-
STANFORD CROWD RESEARCH COLLECTIVE
Lorem ipsum dolor sit amet, consectetur adipiscing
-
amet ornare orci, non ultrices sapien. Phasellus blandit, sem at mollis varius, est magna pretium leo, vel cursus tortor felis a purus. Cras mollis mi nec sapien rutrum dictum. Maecenas sed mattis lorem. Fusce ac gravida est, eu tincidunt orci. Sed ut diam ex. Vestibulum ante
23
Snehal (Neil) Gaikwad, Durim Morina, Rohit Nistala, Megha Agarwal, Alison Cossette, Radhika Bhanu, Saiph Savage, Vishwajeet Narwal, Karan Rajpal, Jeff Regino, Aditi Mithal, Adam Ginzberg, Aditi Nath, Karolina Zi-ulkoski, Trygve Cossette, Dilrukshi Gamage, Angela Richmond-Fuller, Ryo Suzuki, Jeerel Herrejon, Kevin Viet Le, Claudia Flores-Saviaga, Haritha Thilakarathne, Kajal Gupta, William Dai, Ankita Sastry, Shirish Goyal, Thejan Rajapakshe, Niki Abolhassani, Angela Xie, Surabhi Ingle, Veronica Jaramillo, Martin Daniel, Walter Angel, Carlos Toxtli, Juan Pablo Flores, Asmita Gupta, Vineet Sethia, Diana Padilla, Kristy Milland, Kristiono Setyadi, Nuwan Wajirasena, Muthitha Batagoda, Rolan-do Cruz, James Damon, Divya Nekkanti, Tejas Sarma, Mohammed Hashim, Gaby Gon, Soroosh Bateni, Gema Toledo, Alex Peña, Ryan Compton, Deen Aariff, Luis Daniel, Muna Ritter, Nisha K.K., Alan James Kay, Jana Uhrmeister, Srivalli Nistala, Milad Nasar Esfahani, Elsa Bakiu, Kristiono Setyadi, Christopher Diemert, Ranjay Krishna, Geza Kovaks, Rajan Vaish, Michael Bernstein
STANFORD CROWD RESEARCH COLLECTIVE
Lorem ipsum dolor sit amet, consectetur adipiscing
-
amet ornare orci, non ultrices sapien. Phasellus blandit, sem at mollis varius, est magna pretium leo, vel cursus tortor felis a purus. Cras mollis mi nec sapien rutrum dictum. Maecenas sed mattis lorem. Fusce ac gravida est, eu tincidunt orci. Sed ut diam ex. Vestibulum ante ipsum primis in faucibus orci luctus et ultrices posuere cubilia Curae; Aenean nec est metus. Donec sagittis feugiat risus eu mollis. Fusce sed mattis dolor. Aenean egestas posuere lacus sed vehicula.
Integer libero eros, molestie in ipsum vitae, pulvinar elementum nisi. Fusce eu mollis sapien.
How it works Login AboutSMALL LOGO
PAGE TITLE
Lorem ipsum Dolor sit ametconsectetur Adipiscing
PAGE TITLE
Lorem ipsum Dolor sit ametconsectetur
PAGE TITLE
Lorem ipsum Dolor sit ametconsectetur Adipiscing
PAGE TITLE
Lorem ipsum Dolor sit ametconsectetur
PAGE TITLE
Lorem ipsum Dolor sit ametconsectetur Adipiscing
Reputation
DRIs: @arichmondfuller and @dilrukshi
Milestone work1. Identified articles to review. http://crowdresearch.stanford.edu/w/index.php?
title=Relevant_Work#Reputation-related_Links_and_Articles
1. Articles read and summarised on http://crowdresearch.stanford.edu/w/index.php?title=Summer_Milestone_9_Reputation_Systems_research_and_exploration by @williamdai, @nishakk, @juechi, @alfonsoxw, @aricmondfuller,@dilrukshi, @surabhi_i + @kajagupta, @claudiasaviaga, @aditi, @rahulsheth1016, @rcompton , @sarmatejas, @rijul, @aditi.nath , @ankita.sastry, @vineet_sethia, and @trygve. @Jsilver and @asmitagupta will submit by Saturday. https://docs.google.com/spreadsheets/d/1ci5sQOMR0mJU6xACysBXRQtGf21B2VV1QMKGnrZRBsY/edit?usp=sharing
1. Reviewed ideas submitted by previous participants https://docs.google.com/document/d/1Hl9Ui3Qmebz2h7MLsPHm1b5ry1b9NUQPq1atZRLj0hU/edit
4. The work on evaluation the reputation systems on other crowdsourcing platforms was split out into a separate wiki at http://crowdresearch.stanford.edu/w/index.php?title=Summer_Milestone_9_Evaluations_of_reputation_systems_on_other_crowdsourcing_platforms# @arichmondfuller evaluated Freelancer.
We suggest that this work is rolled over to next week’s Trello Milestone 11.
4. @arichmondfuller created a Reputation System Research and Exploration wiki as a central repository for ideas and to keep track of all the work participants have previously contributed http://crowdresearch.stanford.edu/w/index.php?title=Summer_Milestone_9_Reputation_Systems_research_and_exploration @jsilver and @dilrukshi helped to edit this wiki
Milestone work continued
Open governance
DRIs: @trygve and @arichmondfuller with an assist from @rajanvaish
This week’s milestones
Research engineering! Task posting! DRI: @maniksingh92
! Worker task submission working! DRI: @dmorina
! Monitoring page working! DRI: @ryosuzuki and @neilthemathguy
29
Reputation! Two goals:! Join one or both brainstorms for the repsys that we anchor in our
first prototype: M/W 7:30pm PDT. DRI: @michaelbernstein
! Study what other platforms (AMT, Upwork, Freelancer) do for reputation. DRI: @dilrukshi and @arichmondfuller
30
Homepage! Implement the home page design!! DRI: @karolina
31
Open govenance! First leadership board meeting — establish a governing
document (“what can and can’t we do?”)
32
Upcoming milestones
This week’s milestones! Reputation: join one or both brainstorms for the
repsys that we anchor in our first prototype: M/W 7:30pm PDT
34