Twitter, Big Data, and the Search for Meaning: Methodology in Progress
Associate Professor Axel Bruns @snurb_dot_infohttp://mappingonlinepublics.net/ Queensland University of Technology
WHY TWITTER?
• Researching Twitter:– Significant world-wide social network– ~500 million accounts (but how many active?)– Varied range of uses: from phatic communication to emergency coordination– Healthy third-party ecosystem (for now)– Strong history of user innovation:
@replies, #hashtags– Flat and open network structure:
non-reciprocal following, public profiles by default– Good API for gathering (big) data for research
NEW MEDIA AND PUBLIC COMMUNICATION: MAPPING AUSTRALIAN USER-CREATED CONTENT
IN ONLINE SOCIAL NETWORKS
• Australian Research Council (ARC) Discovery Project (2010-13) – $410,000– QUT (Brisbane), Sociomantic Labs (Berlin)– First comprehensive study of Australian social media use– Computer-assisted cultural analysis: tracking, mapping, analysing blogs, Twitter, Flickr,
YouTube as ‘networked publics’– Addressing the problem of scale (‘Big Data’) and disciplinary change in media, cultural and
communication studies – natively digital methods– Studying society with the Internet (Richard Rogers)
http://mappingonlinepublics.net/
• Data Gathering– yourTwapperkeeper + in-house crawler
• Data Processing– Gawk – open source, multiplatform, programmable command-line tool for
processing CSV documents
• Textual Analysis– Leximancer – commercial, multiplatform: extracts key concepts from large
corpora of text, examines and visualises concept co-occurrence– WordStat – commercial, PC-only text analysis tool; generates concept co-
occurrence data that can be exported for visualisation
• Visualisation– Gephi – open source, multiplatform network visualisation tool
A TWITTER RESEARCH TOOLKIT
SO NOW WHAT?
#HASHTAGS AS PUBLICS
• #hashtags– ‘#’ + keyword makes tweets easily discoverable and marks themes– E.g. #ausvotes, #qldfloods, #londonriots, #royalwedding, #euro2012, …
• Publics – Attend to matters of shared concern with some level of co-awareness– Varied in intensity and temporality– Emergent, constituted via discourse & affect
• #hashtag publics– Not all hashtags constitute publics; Twitter doesn’t ‘contain’ publics– What are the patterns in the dynamics of different hashtag-based publics?– What might account for these differences?
#SPILL: 23 JUNE 2010, 6-7 P.M.
http://mappingonlinepublics.net/2010/12/30/visualising-twitter-dynamics-in-gephi-part-2/
#SPILL: 23 JUNE 2010, 7-8 P.M.
#SPILL: 23 JUNE 2010, 8-9 P.M.
BUT WHY?
• Possible research questions:– Ad hoc events and publics:
• How do online publics form and dissolve? How do they interact, what structures do they form?
• Where do they draw information from? What do they share?• Do they simply consist of the usual suspects? How insular and disconnected
are online publics?
– Hashtags in context:• How do different hashtag events compare? Are there common types of
hashtags/publics?• How ‘big’ are they? What topics attract attention on Twitter?• What community (?) structures emerge?
DEVELOPING TWITTER METRICS
• Key data points available through the Twitter API:– text: contents of the tweet itself, in 140 characters or less– to_user_id: numerical ID of the tweet recipient (for @replies)– from_user: screen name of the tweet sender– id: numerical ID of the tweet itself– from_user_id: numerical ID of the tweet sender– iso_language_code: code (e.g. en, de, fr, ...) of the sender’s default
language– source: client software used to tweet (e.g. Web, Tweetdeck, ...)– profile_image_url: URL of the tweet sender’s profile picture– geo_type: format of the sender’s geographical coordinates– geo_coordinates_0: first element of the geographical coordinates – geo_coordinates_1: second element of the geographical coordinates– created_at: tweet timestamp in human-readable format– time: tweet timestamp as a numerical Unix timestamp
DEVELOPING TWITTER METRICS
• Additional data points from tweets:– original tweets: tweets which are neither @reply nor retweet– retweets: tweets which contain RT @user… (or similar)
• unedited retweets: retweets which start with RT @user…• edited retweets: retweets do not start with RT @user…
– genuine @replies: tweets which contain @user, but are not retweets– URL sharing: tweets which contain URLs
• Potential uses:– metrics per hashtag– metrics per timeframe (day, hour, minute, second, …)– metrics per user (or group of users)– …
(Bruns & Stieglitz, forthcoming)
#QLDFLOODS @REPLIES
mainstream media
authorities
#ROYALWEDDING
#AUSPOL (FEB.-DEC. 2011)
HA
SH
TA
G M
ET
RIC
S
TOWARDS A TYPOLOGY OF TWITTER USES
• How are hashtags used (during acute events)?– Gatewatching:
• Finding and sharing information about breaking news (before the mainstream media do?)
• Ad hoc publics: many URLs, many retweets (even unedited)
– Audiencing:• Shared experience of major (foreseen) events• Imagined community of fellow participants: few URLs, limited retweeting
• What other uses are there?– Continuing discussions (#auspol, #bundesliga, …)– Memes (#ghettohurricanenames, …)– Emotive hashtags (#fail, #win, #headdesk, …)– What about keywords?
BEYOND HASHTAGS
• Publics on Twitter:– Micro: @reply and retweet conversations– Meso: follower/followee networks– Macro: hashtag ‘communities’ (Bruns & Moe, forthcoming)
Multiple overlapping publics / networks
• What drives their formation and dissipation?• How do they interact and interweave?• How are they interleaved with the wider media ecology?• Twitter doesn’t contain publics: publics transcend Twitter
UNDERSTANDING AUSTRALIAN TWITTER USE
• What is the Australian Twitter userbase?– Large-scale snowballing project– Starting from selected hashtag communities
(e.g. #ausvotes, #qldfloods, #masterchef)– Identifying participating users, testing for ‘Australianness’:
• Timezone setting, location information, profile information
– Retrieving follower/followee information for each account (very slow)
• Progress update:– ~1.06 million Australian users identified so far
~2 million Australian users in total?
THE AUSTRALIAN TWITTERSPHERE?
Follower/followee network:~120,000 Australian Twitter users(of ~950,000 known accounts by early 2012) colour = outdegree, size = indegree
THEMATIC CLUSTERS
PerthMarketing / PR
DesignWeb
Creative
FarmingAgriculture
HardlineConservatives
ConservativesJournalists
ALPProgressives
Greens
News
OpinionNews
NGOsSocial Policy
ITTech
Social MediaTechPR
Advertising
Real EstateProperty
JobsHR
Business
BusinessProperty
Parenting
Mums CraftArts
FoodWine
Beer
Adelaide
SocialICTs
CreativeDesign
FashionBeauty
UtilitiesServices
Net Culture
BooksLiteraturePublishing
Film
TheatreArts
RadioTV Music
DanceHip Hop
Triple J
TalkbackBreakfast TVCelebritiesCycling
Union
NRL
Football
CricketAFL
SwimmingV8s
Evangelicals
Teachinge-Learning
Schools
ChristiansHillsong
Teens
Jonas Bros.Beliebers
Australian Bands
@KRuddMP
@JuliaGillard
#AUSPOL
Follower/followee network:~120,000 Australian Twitter users(of ~950,000 known accounts by early 2012) colour = #auspol tweets, size = indegree
#AUSVOTES
Follower/followee network:~120,000 Australian Twitter users(of ~950,000 known accounts by early 2012) colour = #ausvotes tweets, size = indegree
#ROYALWEDDING
Follower/followee network:~120,000 Australian Twitter users(of ~950,000 known accounts by early 2012) colour = #royalwedding tweets, size = indeg.
ABC.NET.AU URLS
Follower/followee network:~120,000 Australian Twitter users(of ~950,000 known accounts by early 2012) colour = tweets with URLs, size = indegree
AUSTRALIAN TWITTER NEWS INDEX
• ATNIX:– Tracking tweets which link to 29 key Australian news / opinion sites
(even if URLs are shortened: e.g. t.co bit.ly ow.ly abc.net.au)– Regular processing and evaluation
• Potential uses:– Examination of general market share– Impact of key events and stories– Tracking of specific articles– Examination of retweet chains for new URLs – how does news
disseminate?
– Coming soon: DeTNIX (Germany), others?
TWITTER AND/IN THE MEDIA ECOLOGY
TWITTER AND/IN THE MEDIA ECOLOGY
UNDERSTANDING TWITTER PUBLICS
• #hashtags:– Useful coordinating mechanism for core discussion– Relatively easy to capture and analyse– Fails to capture non-hashtagged tweets about the topic– Good case studies, but very little comparative work to date
• National / global Twittersphere maps– Crucial contextual baseline for #hashtag case studies– Slow and laborious data gathering process, never complete– Very long-term perspective, beyond most funded projects– Indispensable for study of Twitter as a public space
‘BIG DATA’ AND THE DIGITAL HUMANITIES
• Emerging needs in Twitter research:– Unified, compatible methods and metrics for Twitter analysis
Tools and approaches shared at http://mappingonlinepublics.net/
– Powerful infrastructure for long-term, high-volume tracking of public communication on Twitter
Data access requires substantial funding stream
– Facilities for long-term data storage and preservation Key roles for National Libraries, National Archives
– Integration with related datasets (e.g. MSM content) Need to address data interoperability questions
• Twitter as a test case for digital humanities research– Widespread, open, public platform for everyday communication– Tool for observing society at scale through Internet research
‘BIG DATA’ AND STUDENT SKILLS
• Students need interdisciplinary skill sets:– Media & communication to understand the media environment– Maths and statistics to deal with ‘big data’– Computer science to develop tools to process social media data– Communication design to develop effective visualisations– Writing and communication skills to communicate the results– …
– Where do we find them?(few people have such a diverse range of skills)
– How do we support their work? (we’re only just developing our methods and tools)
– What is our strategy for dealing with precarity?(sudden API changes, changing fortunes of platforms, …)
http://mappingonlinepublics.net/@snurb_dot_info
@jeanburgess
@_StephenH
@DrTNitins
@timhighfield
@cdtavijit