steps toward developing an intelligent robotics course kutztown university

Steps Toward Developing an Steps Toward Developing an Intelligent Robotics CourseIntelligent Robotics Course

Kutztown UniversityKutztown University

PACISE 2011PACISE 2011April 9, 2011April 9, 2011

Oskars J. RiekstsOskars J. Rieksts

Jeffrey W. MintonJeffrey W. Minton

2011 Kutztown University 2

Acknowledgments – FundingAcknowledgments – FundingCS Department & LAS CollegeCS Department & LAS CollegeTYCO Electronics FoundationTYCO Electronics FoundationKutztown University FoundationKutztown University FoundationKU Faculty Professional KU Faculty Professional

Development CommitteeDevelopment CommitteeKU Undergraduate Research KU Undergraduate Research

CommitteeCommitteePASSHE Faculty Professional PASSHE Faculty Professional

Development CouncilDevelopment Council


MotivationsMotivationsRodney BrooksRodney BrooksDARPA Grand ChallengeDARPA Grand ChallengeREU @ AuburnREU @ AuburnHardware-software synthesisHardware-software synthesis

““Hardened” hardwareHardened” hardwareCentral role of softwareCentral role of software


MotivationsMotivationsApplied A.I.Applied A.I.Applied cognitionApplied cognitionMobilityMobilityAssistive roboticsAssistive robotics

Robotic wheelchairsRobotic wheelchairs


OutlineOutlineEarly effortsEarly efforts

Hacking Roomba & other venturesHacking Roomba & other venturesResearch Experience for Research Experience for

UndergraduatesUndergraduatesCreate and MindstormsCreate and MindstormsCaptain KURKCaptain KURKTrinity firefighting contestTrinity firefighting contestMyro and PythonMyro and PythonReading-Berks Science FairReading-Berks Science Fair


Coming into FocusComing into FocusEmerging goalsEmerging goals

IntelligenceIntelligenceVisionVisionCommuncationCommuncationPhilosophical issuesPhilosophical issues


Point of convergencePoint of convergenceCourse in intelligent roboticsCourse in intelligent robotics

ControlControlVisionVisionCommuncationCommuncationInvestigate cognitive issuesInvestigate cognitive issues


Course design objectivesCourse design objectivesRobot isRobot is

SituatedSituatedEmbodiedEmbodied

Shared environmentShared environmentHumanHumanMachineMachine

Shared communicationShared communicationFramework for investigationFramework for investigation


Robot is situatedRobot is situated

Operates within an environmentOperates within an environmentEmbedded in the worldEmbedded in the worldChief knowledge source:Chief knowledge source:Data stream drawn from Data stream drawn from

environmentenvironment““The world is its own best The world is its own best

representation” – Brooksrepresentation” – Brooks


Robot is embodiedRobot is embodiedEntity 1Entity 1stst, agent 2nd, agent 2nd

Self reliantSelf reliantSecretary of State modelSecretary of State model

Self-seated conceptual frameworkSelf-seated conceptual frameworkWorld concepts grounded in sensor suiteWorld concepts grounded in sensor suiteBehavior set emanates from actuator Behavior set emanates from actuator

suitesuiteDecision apparatus grounded in Decision apparatus grounded in

sensor/actuator suitessensor/actuator suites


Robot is embodiedRobot is embodiedEames: Eames: Design is a plan for arranging Design is a plan for arranging

elements to accomplish a particular elements to accomplish a particular purposepurpose

Elephants don’t play chess (Brooks)Elephants don’t play chess (Brooks)““Mind” is to fit the bodyMind” is to fit the body

Design on need to basisDesign on need to basisKnowKnowThinkThinkDoDo


Shared environmentShared environmentShared conceptual Shared conceptual frameworkframework

Robot’s Robot’s Derivative of human’sDerivative of human’sSimplified in structureSimplified in structure

Shared factbaseShared factbaseRobot’s Robot’s

Subset of human’sSubset of human’sSimplified ontologySimplified ontology

Vision basedVision based


Common communication frameworkCommon communication frameworkGrounded inGrounded in

SharedSharedEnvironmentEnvironmentConceptual frameworkConceptual frameworkOntologyOntology

Vision as main sensorVision as main sensor

Intersecting language constructsIntersecting language constructs


Projected course topics/activitiesProjected course topics/activitiesSpecialized robot control softwareSpecialized robot control softwareRobot control architectureRobot control architectureBasics of image processingBasics of image processingSpecialized image processing Specialized image processing

softwaresoftwareBasics of communication theoryBasics of communication theoryKey issues of cognitive roboticsKey issues of cognitive robotics


Slow, steady progressSlow, steady progress

Browning:Browning:Reach should exceed graspReach should exceed grasp

Rapid prototypingRapid prototypingIterative developmentIterative development


Structure of the classStructure of the classFour teamsFour teamsRolesRoles

Team leaderTeam leaderDesignerDesignerCoderCoderDocument guruDocument guruHistorianHistorianTest designerTest designerTest administratorTest administratorHardware specialistHardware specialist


PACT demonstrationPACT demonstrationPennsylvania Association of Pennsylvania Association of

Council of TrusteesCouncil of TrusteesVery early in learning curveVery early in learning curveNavigation within a “corral”Navigation within a “corral”Well receivedWell received

Gateway navigationGateway navigationBull fighting robotBull fighting robot

PR for robotics @ PASSHEPR for robotics @ PASSHE


Jeff – gateway navigationJeff – gateway navigation


Investigate cognitive issuesInvestigate cognitive issuesSight and touchSight and touch

George StrattonGeorge StrattonSpatial harmony of sight & touchSpatial harmony of sight & touch

SpaceSpaceBenjamin KuipersBenjamin KuipersSemantic spatial hierarchySemantic spatial hierarchy

MetricalMetricalTopologicalTopologicalHybridHybrid


Investigate cognitive issuesInvestigate cognitive issuesLanguage and meaningLanguage and meaning

Stevan HarnadStevan HarnadSymbol grounding problemSymbol grounding problem

Language and shared spaceLanguage and shared spaceAmichai KronfeldAmichai KronfeldShared referent problemShared referent problem

2011Kutztown University

21

Investigate cognitive issuesInvestigate cognitive issuesEmbodied cognitionEmbodied cognition

Randall BeerRandall BeerSituated, embodied, dynamicalSituated, embodied, dynamical

Principle of interactivismPrinciple of interactivismMark BickhardMark BickhardEmergence of representational Emergence of representational

contentcontent


Illustration – Shared Referent ProblemIllustration – Shared Referent ProblemFrom KronfeldFrom Kronfeld

Steve & wife Bev are at a partySteve & wife Bev are at a partySteve: Bob & his wife are certainly Steve: Bob & his wife are certainly

enjoying the partyenjoying the partyBev: Sally is not his wife!Bev: Sally is not his wife!

NoteNoteSteve & Bev share a referent, SallySteve & Bev share a referent, SallyDespite misidentification no Despite misidentification no

communication problems arise!communication problems arise!


The KLO TriadThe KLO TriadFor communication involving For communication involving

referentsreferentsFor each communicant – this triadFor each communicant – this triad

Object (the referent)Object (the referent)KB – representation of objectKB – representation of objectLanguage – reference to objectLanguage – reference to object

For successful communicationFor successful communicationReferents must matchReferents must matchFor HMC (Human-machine For HMC (Human-machine

communication)communication)Human & machine KBs do Human & machine KBs do notnot match match


Establishing co-refrenceEstablishing co-refrencePer Frege’s distinctionPer Frege’s distinctionExtensional approachExtensional approach

Pointing or equivalentPointing or equivalentIntensional approachIntensional approach

Language aloneLanguage aloneHMC goalHMC goal

Minimize extensionalMinimize extensionalMaximize intensionalMaximize intensional


The Trinity RobotThe Trinity Robot


The Trinity RobotThe Trinity RobotVEX robotics kit for chasisVEX robotics kit for chasisVEX included motors for locomotionVEX included motors for locomotionSonar rangefindersSonar rangefindersWeb-camWeb-camArduino to control motors and sensorsArduino to control motors and sensorsNetbook to control Arduino and Netbook to control Arduino and

process imagesprocess images


The Trinity RobotThe Trinity RobotInaccurate motor controlInaccurate motor controlSonar signals bounce inside Sonar signals bounce inside

cornerscornersProvide inaccurate measurementsProvide inaccurate measurements

Viewing angle of web-cam too Viewing angle of web-cam too smallsmall


BrobiBrobi


Brobi – hardwareBrobi – hardwareCreate by iRobotCreate by iRobotPlatform built onto Create cargo Platform built onto Create cargo

bay to accommodate equipmentbay to accommodate equipmentWeb-cam with increased viewing Web-cam with increased viewing

angleangleIR rangefinderIR rangefinder

IR light does not bounce like soundIR light does not bounce like soundArduinoArduino


Brobi – softwareBrobi – softwareOpenCV and PythonOpenCV and PythonConsultation – John SpletzerConsultation – John Spletzer

LehighLehighLittle Ben in Urban ChallengeLittle Ben in Urban Challenge

MATLABMATLABImage processingImage processingCreate APICreate APIA programming languageA programming language


Image processingImage processing““A picture is worth a thousand A picture is worth a thousand

words”words” need to extract discrete objects need to extract discrete objects

from images to identify themfrom images to identify themK-Means clusteringK-Means clustering


Goal: identify green ballGoal: identify green ball


K-Means ClusteringK-Means ClusteringCluster sets of dataCluster sets of data

Into user-defined number of Into user-defined number of segmentssegments

Number of segments referred to as KNumber of segments referred to as KClusters defined by MEAN of all Clusters defined by MEAN of all

values in clustervalues in cluster

K-Means AlgorithmK-Means Algorithm

K-Means ExampleK-Means Example

K-Means using k = 6

K-Means using k = 5


Natural Language ProcessingNatural Language Processing

““Go to the green ball”Go to the green ball”The meanings behind words must be The meanings behind words must be

inferredinferred““Go,” conceptually can represent Go,” conceptually can represent

many thingsmany thingsTake a turn in a gameTake a turn in a gameThe Chinese strategy gameThe Chinese strategy gameTravel to a locationTravel to a location

Determine the concept being referred Determine the concept being referred toto

Conceptual parsingConceptual parsing


Conceptual ParsingConceptual Parsing

Words are mapped to conceptsWords are mapped to concepts

ConceptsConceptsRules define set of related conceptsRules define set of related conceptsOne concept may have many One concept may have many

separate rule setsseparate rule sets

Concept TreeConcept Tree


Assessment – PlatformAssessment – PlatformBest hardware platform to dateBest hardware platform to dateOpens up many avenues of course Opens up many avenues of course

developmentdevelopmentSoftware for robot control systemsSoftware for robot control systemsRobot control architecturesRobot control architecturesImage processingImage processingCommunicationCommunicationCognitive robotics issuesCognitive robotics issues


Assessment – MATLABAssessment – MATLABCan do all 3 thingsCan do all 3 things

Sensor/actuator interface with Sensor/actuator interface with Lehigh APILehigh API

Image acquisition & processingImage acquisition & processingRobot control system programmingRobot control system programming

Good IDEGood IDEGood documentationGood documentation

17 pdf files; 64 mB17 pdf files; 64 mBOnline documentationOnline documentation


Assessment – MATLABAssessment – MATLABGood IDEGood IDEStrong user communityStrong user community

BlogsBlogsDiscussion boardsDiscussion boardsGood code segmentsGood code segments

Good programming languageGood programming languageMatrix optimizedMatrix optimizedAdvance features, e.g., lambda, Advance features, e.g., lambda,

applyapply


Assessment – CreateAssessment – CreateUsefulUsefulStableStableUbiquitousUbiquitousNegativesNegatives

Battery short lastingBattery short lastingDoes not travel in straight lineDoes not travel in straight line


Assessment – AccessibilityAssessment – AccessibilityCost not prohibitiveCost not prohibitive

Create – $130Create – $130$220 with battery and charger$220 with battery and charger

Netbook ~ $330Netbook ~ $330Arduino ~ $30Arduino ~ $30MATLABMATLAB

$900 for 10 seat license$900 for 10 seat license$130 for student version with image $130 for student version with image

acquisitionacquisition


Future directionsFuture directionsSoftware repositorySoftware repositoryLearningLearning

BayesianBayesianGenetic algorithmsGenetic algorithmsNeural networks, etc.Neural networks, etc.

Control architecuresControl architecuresBehavior-basedBehavior-basedHybridHybridBlackboard?Blackboard?


Future directionsFuture directionsExperiment with other sensorsExperiment with other sensors

Touch {whiskers}Touch {whiskers}OdorOdorSound {whistle, a la Sound of Sound {whistle, a la Sound of

Music}Music}HeatHeatOdorOdorLight/brightnessLight/brightness

Wireless interfaceWireless interface““Watson, come here. I need you”Watson, come here. I need you”


Future of robotics at KUFuture of robotics at KUStrong student interestStrong student interestRejuvenated courseRejuvenated course

New syllabusNew syllabusApprovedApproved400 level – a mixed blessing400 level – a mixed blessingNot yet scheduledNot yet scheduledIn rotation?In rotation?PoliticsPolitics


Further InformationFurther InformationJeff’s graduate thesisJeff’s graduate thesis

Do You See What I'm Saying: Do You See What I'm Saying: Relating Language and Vision to Relating Language and Vision to Create Interaction Between Humans Create Interaction Between Humans and Robotsand Robots

Delves further into concepts Delves further into concepts discussed herediscussed here

https://github.com/jeffminton/Thesis


Questions?Questions?


The EndThe End


Extra slidesExtra slides


TemplateTemplateFont 44Font 44

font 40font 40Font 44Font 44



font 40font 40


AssessmentAssessmentSensors can be addedSensors can be addedOffers new areas of explorationOffers new areas of exploration

Heat sensorsHeat sensorsSound sensorsSound sensorsOdor sensorsOdor sensors


Unsuccessful segmentationUnsuccessful segmentation


Successful segmentationSuccessful segmentation


TemplateTemplateFont 44Font 44




font 40font 40

steps toward developing an intelligent robotics course kutztown university

Documents

passhe2011kutztown university

minton2011kutztown university

cognitive issuessight

point of convergencecourse

venturesresearch experience

sensor suitebehavior

steady progressbrowning

data stream