1. Collections of War Connecting the dots for GovHack 2015 Aimee Maree Forsstrom Erle Pereira 2. OpenData and OpenGov So our story begins with You the happy hacker heading out to join the OpenGov Hackathon to play with Open Data 3. OpenData and OpenGov You arrive at the hackspace and announce I have a great idea 4. OpenData and OpenGov You browse the OpenData sets and think... Oh wow there are so many I feel like I am running around in circles.. 5. OpenData and OpenGov [Streaming tears when one realises they have to understand the connections in the data to build their app] But your a hacker, so you dont give up... 6. OpenData and OpenGov Then you think. But I just wanted to be creative, I just wanted to help Now I dont know where to start? 7. So many data sets So you wanted to make an app about World War One and you took a look at the datasets? How many World War One datasets do we have? How many formats are they displayed in? Where is the unique indentifier? 8. The problem is with the technology So the problem we find is not with the lack of enthusiam from the hackers It is the reality of the current state of Open Data sets and there disparete nature So my team decided to help aliviate your frustration :D 9. What was our Project you ask? We wanted to look at sentiment analysis of the World War One Diaries and connect it to the Portraits of the Soldiers But Louise Denoon suggested how it would be great to see sentiment analysis of their Diares (personal account) aposed to Newspapper clippings or letters to Departments (Offical correspondence) 10. What was the problem Different datasets across different formats Not enough time to understand the Schema Even if we had more time the Schema would be no help as what were are trying to do is discover the connection across different data 11. The Solution? Traditional methods of Databases are not going to solve this problem We were facing the unanswered issue of Big Data So much information how does the human mind comprehend it and make meaningful connections We needed a better Database! 12. Technology saves the Day SQL just wont cut it, you need to understand your data before you can write a join You need to understand your fields? 13. Its all about the Relationships With Graphing databases like Neo4J we can coalate a variety of diverse datasets Neo4J starts to learn the Relationships between the data for us Instead of understanding our data joins we start spending the time defining the question we want answered by our app 14. SQL Query vs Relationship Query So instead of writing SELECT firstname FROM person WHERE person.nickname= 'John Brown Soldier' You would write MATCH p:person WHERE p.nickname= 'John Brown Soldier' RETURN p.firstname p is not in the table person, or even 'is' a person, it simply has the label person unlike in SQL we need to define that John is a Nickname in the table Person aka we need to know our schema 15. SQL Query vs Relationship Query So instead of writing SELECT firstname,batallion.name FROM person JOIN batallion ON person.batallionid = batallion.id WHERE person.nickname= 'John Brown' AND order.rank = 'Soldier' You would write MATCH (p:person)-[:in]-(t:batallion) WHERE p.nickname= 'John Brown' AND o.rank = 'Soldier' RETURN p.firstname, t.name 16. Schema SQL is a Schema Database and you need to understand the Schema But in Big Data we dont have the time to understand the Schema how can we ever know the full Schema? 17. SQL vs Graphing This means in the old SQL Query world we would need to know that John was a Person and even a Soldier So p is not in the table person, or even 'is' a person, it simply has the label person, we have defined the relationship (the correlation) and asking the database to graph the connection 18. Potential In the world of Big Data we can no longer rely on knowing the schema of the information that we will be performing analysis on We need graphing databases in order to start to see the correlations between disparete sets of data 19. Gov Hack 2016? What does this mean for next years Gov Hack? We will be able to import all the datasets from WorldWarOne and give JSON access to various disparete datasets and allow the hackers that extra time and head space to create the app to consume and create aka what they do best 20. Research Potential What does this mean for research? World War One researchers do not have to try to understand mulitpule dataset schemas Because the Database does it for them researchers spend their mental effort on drawing deeper connections and relationships Understanding the web of knowledge Unlocking new discoveries from old data 21. Its only a Prototype Our team consisted of two people We had big plans for a fancy frontend and more datasets But time is always a factor This is our technology showcase to display the potential of a new way to look at datasets for to enable hackers with the least barriers to data 22. Where do we go from here? We have created a Docker Image of Neo4J please visit our github https://github.com/erlepereira/govhackau2015 please contribute please fork