part 2. 데이터수집 - · pdf file목 차 04 facebook crawling 1 22 twitter crawling 2 28...

이원하

파이썬을 이용한 빅데이터 수집 분석과 시각화

Part 2 데이터 수집

목 차

04FACEBOOK Crawling

1

22TWITTER Crawling

2

28NAVER Crawling

3

34공공데이터포털 Crawling

4

40일반 WEB Page Crawling

5


JSON (JavaScript Object Notation)

3

JSON

Object

Array

nameJohn

age30

cars[ Ford BMW Fiat ]

파이썬을이용한빅데이터수집 분석과시각화

1FACEBOOK CRAWLING1


Facebook Graph API

5

GET [version info][Node|Edge Name]

Host graphfacebookcom

ldquofieldnamerdquo field-value

hellip


Facebook Graph API

6

data [

획득한데이터

]

paging

cursors

after MTAxNTExOTQ1MjAwNzI5NDE=

before NDMyNzQyODI3OTQw

previous httpsgraphfacebookcommealbumslimit=25ampbefore=NDMyNzQyODI3OTQw

next httpsgraphfacebookcommealbumslimit=25ampafter=MTAxNTExOTQ1MjAwNzI5NDE=


Facebook Graph API

7

data [

획득한데이터

]

paging

previous httpsgraphfacebookcommefeedlimit=25ampsince=1364849754

next httpsgraphfacebookcommefeedlimit=25ampuntil=1364587774


Facebook Graph API

8

data [

획득한데이터

]

paging




Facebook Numeric ID 가져오기

9

httpwwwfacebookcomjtbcnews

httpsgraphfacebookcomv28[page_id]access+token=[App_ID]|[Secret_Key]

gtgtgt page_name = jtbcnews

gtgtgt app_id = 20097013

gtgtgt app_secret = daccef14d595060d65e66c41d

gtgtgt access_token = app_id + | + app_secret

gtgtgt

gtgtgt base = httpsgraphfacebookcomv28

gtgtgt node = + page_name

gtgtgt parameters = access_token=s access_token

gtgtgt url = base + node + parameters

gtgtgt print (url)

httpsgraphfacebookcomv28jtbcnewsaccess_token=20097013|daccef14d595060d65e66c41d

gtgtgt

ldquonamerdquo ldquoJTBC 뉴스rdquo

ldquoidrdquo ldquo240263402699918rdquo



10

gtgtgt import urllibrequest

gtgtgt import json

gtgtgt req = urllibrequestRequest(url)

gtgtgt response = urllibrequesturlopen(req)

gtgtgt data = jsonloads(responseread()decode(utf-8))

gtgtgt page_id = data[id]

gtgtgt page_name = data[name]

gtgtgt print (s Facebook Numeric ID s (page_name page_id))

JTBC 뉴스 Facebook Numeric ID 240263402699918

gtgtgt



11

import sys

import urllibrequest

import json

if __name__ == __main__

page_name = jtbcnews

app_id = 200920440387013

app_secret = daccef14d5cd41c0e95060d65e66c41d

access_token = app_id + | + app_secret

base = httpsgraphfacebookcomv28

node = + page_name

parameters = access_token=s access_token

url = base + node + parameters

req = urllibrequestRequest(url)

try

response = urllibrequesturlopen(req)

if responsegetcode() == 200

data = jsonloads(responseread()decode(utf-8))

page_id = data[id]

print (s Facebook Numeric ID s (page_name page_id))

except Exception as e

print (e)

try 블록내부에서오류가발생하면프로그램이오류

를발생하지않고 except 블록수행

Exception 을이용오류를확인

geturl() 응답한서버의 url 반환

info() 페이지의헤더값과 meta 정보

getcode() 서버의 HTTP 응답코드(200이정상)


Facebook 포스트(post-id) Crawling

12

GET [version info]post-id


Id 포스트 ID String

comments 댓글 정보 Object

created_time 포스트 초기 생성일자 Datetime

from 포스트한 사용자에 대한 프로필 정보 Profile

link 포스트에 삽입되어 있는 링크 String

message 포스트 메시지 String

name 링크의 이름 String

object_id 업로드한 사진 또는 동영상 ID String

parent_id 해당 포스트의 부모 포스트 String

picture 포스트에 포함되어 있는 사진들의 링크 String

place 포스트를 작성한 위치 정보 Place

reactions 좋아요 화나요 등에 대한 리엑션 정보 Obejct

shares 포스트를 공유한 숫자 Object

type 포스트의 객체 형식 enumlink status photo video offer

updated_time 포스트가 최종 업데이트된 시간 Datetime



13

data[

comments

data[

]

summary

orderranked

total_count12

can_commentFalse

message즉 청와대가 최 씨의 국정개입 사건을 파 악하고

도 n은폐했다는 사실이안전수석입에서나온겁니다

typelink

shares

count46

reactions

data[

]

summary

viewer_reactionNONE

total_count443

created_time2017-02-23T000000+0000

name안종범 재단임원인사에최순실개입 알고도숨겼

다

id240263402699918 _1328805163845731

linkhttpnewsjtbcjoinscomarticlearticleaspxnew s_id=

NB11427906amppDate=20170222

]

paging

nexthttpsgraphfacebookco mv28240263402699918post

sfields=

previoushttpsgraphfacebookcomv28240263402699918

postsfields=



14

def get_request_url(url)


try



print ([s] Url Request Success datetimedatetimenow())

return responseread()decode(utf-8)


print(e)

print([s] Error for URL s (datetimedatetimenow() url))

return None

responseread()decode(lsquoutf-8rsquo)로반환

오류발생시 lsquoNonersquo반환



15

def getFacebookNumericID(page_id access_token)


node = + page_id



retData = get_request_url(url)

if (retData == None)

return None

else

jsonData = jsonloads(retData)

return jsonData[id]



16

def getFacebookPost(page_id access_token from_date to_date num_statuses)


node = sposts page_id

fields = fields=idmessagelinknametypesharesreactions +

created_timecommentslimit(0)summary(true) +

limit(0)summary(true)

duration = ampsince=sampuntil=s (from_date to_date)

parameters = amplimit=sampaccess_token=s (num_statuses access_token)

url = base + node + fields + duration + parameters



return None

else

return jsonloads(retData)



17

def getPostTotalCount(post key)

try

if key in postkeys()

return post[key][summary][total_count]

else

return 0

except

return 0

def getPostItem(post key)

try


return post[key]

else

return

except

return lsquo

reactions

data[

]

summary

viewer_reactionNONE

total_count443



18

def getFacebookReaction(post_id access_token)


node = s post_id

reactions = fields=

reactionstype(LIKE)limit(0)summary(total_count)as(like)

reactionstype(LOVE)limit(0)summary(total_count)as(love)

reactionstype(WOW)limit(0)summary(total_count)as(wow)

reactionstype(HAHA)limit(0)summary(total_count)as(haha)

reactionstype(SAD)limit(0)summary(total_count)as(sad)

reactionstype(ANGRY)limit(0)summary(total_count)as(angry)

parameters = ampaccess_token=s access_token

url = base + node + reactions + parameters



return None

else




19

def getPostData(post access_token jsonResult)

post_id = getPostItem(post id)

post_message = getPostItem(post message)

post_name = getPostItem(post name)

post_link = getPostItem(post link)

post_type = getPostItem(post type)

post_num_reactions = getPostTotalCount(post reactions)

post_num_comment = getPostTotalCount(post comments)

post_num_shares = 0 if shares not in postkeys() else post[shares][count]

post_created_time = getPostItem(post created_time)

post_created_time = datetimedatetimestrptime(post_created_time Y-m-dTHMS+0000)

post_created_time = post_created_time + datetimetimedelta(hours=+9)

post_created_time = post_created_timestrftime(Y-m-d HMS)



20

reaction = getFacebookReaction(post_id access_token) if post_created_time gt 2016-02-24 000000 else

post_num_likes = getPostTotalCount(reaction like)

post_num_likes = post_num_reactions if post_created_time lt 2016-02-24 000000 else post_num_likes

post_num_loves = getPostTotalCount(reaction love)

post_num_wows = getPostTotalCount(reaction wow)

post_num_hahas = getPostTotalCount(reaction haha)

post_num_sads = getPostTotalCount(reaction sad)

post_num_angrys = getPostTotalCount(reaction angry)

jsonResultappend(post_idpost_id messagepost_message

namepost_name linkpost_link

created_timepost_created_time num_reactionspost_num_reactions

num_commentspost_num_comment num_sharespost_num_shares

num_likespost_num_likes num_lovespost_num_loves

num_wowspost_num_wows num_hahaspost_num_hahas

num_sadspost_num_sads num_angryspost_num_angrys)



21

go_next = True

page_id = getFacebookNumericID(page_name access_token)

jsonPost = getFacebookPost(page_id access_token from_date to_date num_posts)

while (go_next)

for post in jsonPost[data]

getPostData(post access_token jsonResult)

if paging in jsonPostkeys()

jsonPost = jsonloads(get_request_url(jsonPost[paging][next]))

else

go_next = False

with open(s_facebook_s_sjson (page_name from_date to_date) w encoding=utf8) as outfile

str_ = jsondumps(jsonResult

indent=4 sort_keys=True

ensure_ascii=False)

outfilewrite(str_)


1TWITTER CRAWLING

2


OAuth

23


Twit Crawling

24

def oauth2_request(consumer_key consumer_secret access_token acces

s_secret)

try

consumer = oauth2Consumer(key=consumer_key secret=consumer_

secret)

token = oauth2Token(key=access_token secret=access_secret)

client = oauth2Client(consumer token)

return client


print(e)

return None

def get_user_timeline(client screen_name count=50 include_rts=False)

base = httpsapitwittercom11

node = statusesuser_timelinejson

fields = screen_name=sampcount=sampinclude_rts=s

(screen_name count include_rts)

fields = screen_name=s (screen_name)

url = base + node + fields

response data = clientrequest(url)

try

if response[status] == 200

return jsonloads(datadecode(utf-8))


print(e)

return None


Twit Crawling

25

def getTwitterTwit(tweet jsonResult)

tweet_id = tweet[id_str]

tweet_message = if text not in tweetkeys() else tweet[text]

screen_name = if user not in tweetkeys() else tweet[user][screen_name]

tweet_link =

if tweet[entities][urls] list

for i val in enumerate(tweet[entities][urls])

tweet_link = tweet_link + tweet[entities][urls][i][url] +

else

tweet_link =

hashtags =

if tweet[entities][hashtags] list

for i val in enumerate(tweet[entities][hashtags])

hashtags = hashtags + tweet[entities][hashtags][i][text] +

else

hashtags =


Twit Crawling

26

if created_at in tweetkeys()

Twitter used UTC Format EST = UTC + 9(Korean Time) Format ex Fri Feb 10 035727 +0000 2017

tweet_published = datetimedatetimestrptime(tweet[created_at]a b d HMS +0000 Y)

tweet_published = tweet_published + datetimetimedelta(hours=+9)

tweet_published = tweet_publishedstrftime(Y-m-d HMS)

else

tweet_published =

num_favorite_count = 0 if favorite_count not in tweetkeys() else tweet[favorite_count]

num_comments = 0

num_shares = 0 if retweet_count not in tweetkeys() else tweet[retweet_count]

num_likes = num_favorite_count

num_loves = num_wows = num_hahas = num_sads = num_angrys = 0

jsonResultappend(post_idtweet_id messagetweet_message

namescreen_name linktweet_link

created_timetweet_published num_reactionsnum_favorite_count

num_commentsnum_comments num_sharesnum_shares

num_likesnum_likes num_lovesnum_loves

num_wowsnum_wows num_hahasnum_hahas

num_sadsnum_sads num_angrysnum_angrys hashtags hashtags)


Twit Crawling

27

def main()

screen_name = jtbc_news

num_posts = 50

jsonResult = []

client = oauth2_request(CONSUMER_KEY CONSUMER_SECRET ACCESS_TOKEN ACCESS_SECRET)

tweets = get_user_timeline(client screen_name)

for tweet in tweets

getTwitterTwit(tweet jsonResult)

with open(s_twitterjson (screen_name) w encoding=utf8) as outfile

str_ = jsondumps(jsonResultindent=4 sort_keys=True ensure_ascii=False)

outfilewrite(str_)


1NAVER CRAWLING

3


Naver Search(검색) API

29



reqadd_header(X-Naver-Client-Id app_id)

reqadd_header(X-Naver-Client-Secret scxKeSJYib)

try






print(e)


return None



30

def getNaverSearchResult(sNode search_text page_start display)

base = httpsopenapinavercomv1search

node = sjson sNode

parameters = query=sampstart=sampdisplay=s (urllibparsequote(search_text) page_start display)




return None

else




31

def getPostData(post jsonResult)

title = post[title]

description = post[description]

org_link = post[originallink]

link = post[link]

Tue 14 Feb 2017 184600 +0900

pDate = datetimedatetimestrptime(post[pubDate] a d b Y HMS +0900)

pDate = pDatestrftime(Y-m-d HMS)

jsonResultappend(titletitle description description

org_linkorg_link link org_link

pDatepDate)

return

display 100

items [

description ldquo기사요약본

link ldquo[네이버 URL]

originallink ldquo[기사원본 URL]

pubDate Thu 13 Jul 2017 033800 +0900

title ldquo[기사제목]

]

lastBuildDate Thu 13 Jul 2017 035047 +0900

start 1

total 454265



32

def main()

jsonResult = []

news blog cafearticle

sNode = news

search_text = 탄핵

display_count = 100

jsonSearch = getNaverSearchResult(sNode search_text 1 display_count)

while ((jsonSearch = None) and (jsonSearch[display] = 0))

for post in jsonSearch[items]

getPostData(post jsonResult)

nStart = jsonSearch[start] + jsonSearch[display]

jsonSearch = getNaverSearchResult(sNode search_text nStart display_count)

with open(s_naver_sjson (search_text sNode) w encoding=utf8) as outfile

retJson = jsondumps(jsonResult


ensure_ascii=False)

outfilewrite(retJson)


Naver 지도 API

33

def getGeoData(address)

base = httpsopenapinavercomv1mapgeocode

node =

parameters = query=s urllibparsequote(address)




return None

else



1공공 데이터 포털 CRAWLING4


전국 유료 관광지 입장객 정보

35

def getTourPointVisitor(yyyymm sido gungu nPagenum nItems)

end_point = httpopenapitourgokropenapiserviceTourismResourceStatsServicegetPchrgTrrsrtVisitorList

parameters = _type=jsonampserviceKey= + access_key

parameters += ampYM= + yyyymm

parameters += ampSIDO= + urllibparsequote(sido)

parameters += ampGUNGU= + urllibparsequote(gungu)

parameters += ampRES_NM=amppageNo= + str(nPagenum)

parameters += ampnumOfRows= + str(nItems)

url = end_point + parameters



return None

else




36

def getTourPointData(item yyyymm jsonResult)

addrCd = 0 if addrCd not in itemkeys() else item[addrCd]

gungu = if gungu not in itemkeys() else item[gungu]

sido = if sido not in itemkeys() else item[sido]

resNm = if resNm not in itemkeys() else item[resNm]

rnum = 0 if rnum not in itemkeys() else item[rnum]

ForNum = 0 if csForCnt not in itemkeys() else item[csForCnt]

NatNum = 0 if csNatCnt not in itemkeys() else item[csNatCnt]

jsonResultappend(yyyymm yyyymm addrCd addrCd

gungu gungu sido sido resNm resNm

rnum rnum ForNum ForNum NatNum NatNum)



37

nItem = 100

nStartYear = 2011

nEndYear = 2017

for year in range(nStartYear nEndYear)

for month in range(1 13)

yyyymm = 010gt2format(str(year) str(month))

nPagenum = 1

while True

jsonData = getTourPointVisitor(yyyymm sido gungu nPagenum nItems)

if (jsonData[response][header][resultMsg] == OK)

nTotal = jsonData[response][body][totalCount]

if nTotal == 0

break

for item in jsonData[response][body][items][item]

getTourPointData(item yyyymm jsonResult)

nPage = mathceil(nTotal nItem)

if (nPagenum == nPage)

break

nPagenum += 1

else

break


출입국 관광 통계 서비스

38

def getNatVisitor(yyyymm nat_cd ed_cd)

end_point = httpopenapitourgokropenapiserviceEdrcntTourismStatsServicegetEdrcntTourismStatsList



parameters += ampNAT_CD= + nat_cd

parameters += ampED_CD= + ed_cd




return None

else




39

def main()

코드생략




jsonData = getNatVisitor(yyyymm national_code ed_cd)


krName = jsonData[response][body][items][item][natKorNm]

krName = krNamereplace( )

iTotalVisit = jsonData[response][body][items][item][num]

print(s_s s (krName yyyymm iTotalVisit))

jsonResultappend(nat_name krName nat_cd national_code

yyyymm yyyymm visit_cnt iTotalVisit)

코드생략


1 일반 WEB PAGE CRAWLING5


웹 페이지 저작권


BeautifulSoup4 ndash HTML 분석(Parsing) 패키지

[파이썬 설치경로]gtpip install beautifulsoup4



gtgtgt from bs4 import BeautifulSoup

gtgtgt html = lttd class=titlegtltdiv class=tit3gtlta href=moviebimibasicnhncode=136872 title=미녀와 야수gt미녀와 야수ltagtltdivgtlttdgt

gtgtgt soup = BeatifulSoup(html htmlparser)

gtgtgt soup

lttd class=titlegtltdiv class=tit3gtlta href=moviebimibasicnhncode=136872 title=미녀와 야수gt미녀와 야수ltagtltdivgtlttdgt

gtgtgt tag = souptd

gtgtgt tag

lttd class=titlegtltdiv class=tit3gtlta href=moviebimibasicnhncode=136872 title=미녀와 야수gt미녀와야수ltagtltdivgtlttdgt

gtgtgt tag = soupdiv

gtgtgt tag

ltdiv class=tit3gtlta href=moviebimibasicnhncode=136872 title=미녀와 야수gt미녀와 야수ltagtltdivgt

gtgtgt tag = soupa

gtgtgt tag

lta href=moviebimibasicnhncode=136872 title=미녀와 야수gt미녀와 야수ltagt

gtgtgt tagname

a






gtgtgt soup


gtgtgt tag = souptd

gtgtgt tag[class]

[title]


gtgtgt tag[class]

[tit3]

gtgtgt gtgtgt tagattrs

class [tit3]






gtgtgt soup


gtgtgt tag = soupfind(td attrs=classtitle)

gtgtgt tag


gtgtgt tag = soupfind(div attrs=classtit3)

gtgtgt tag






gtgtgt html = urllibrequesturlopen(httpmovienavercommoviesdbrankrmovienhn)

gtgtgt soup = BeautifulSoup(html htmlparser)

gtgtgt print(soupprettify())

ltDOCTYPE htmlgt

lthtmlgt

ltheadgt

ltmeta content=texthtml charset=utf-8 http-equiv=Content-Typegt

ltmeta content=IE=edge http-equiv=X-UA-Compatiblegt

ltmeta content=httpimgmovienavercomtodaynavermenaverme_profilejpg property=me2imagegt

ltmeta content=네이버영화 property=me2post_taggt

ltmeta content=네이버영화 property=me2category1gt

(이하중략)

lt-- Footer --gt

ltdivgt

ltbodygt

lthtmlgt



lttd class=titlegt

ltdiv class=tit3gt

lta href=moviebimibasicnhncode=135874 title=스파이더맨 홈커밍gt스파이더맨 홈커밍ltagt

ltdivgt

lttdgt



gtgtgt tags = soupfindAll(div attrs=classtit3)

gtgtgt tags

[ltdiv class=tit3gt


ltdivgt ltdiv class=tit3gt

lta href=moviebimibasicnhncode=146480 title=덩케르크gt덩케르크ltagt


lta href=moviebimibasicnhncode=76309 title=플립gt플립ltagt

hellip (이하중략)

lta href=moviebimibasicnhncode=149048 title=100미터gt100미터ltagt

ltdivgt]

gtgtgt for tag in tags

print(taga)




print(tagatext)

스파이더맨 홈커밍

덩케르크

hellip(이하중략)

목 차

04FACEBOOK Crawling

1

22TWITTER Crawling

2

28NAVER Crawling

3

34공공데이터포털 Crawling

4

40일반 WEB Page Crawling

5



3

JSON

Object

Array

nameJohn

age30



1FACEBOOK CRAWLING1


Facebook Graph API

5




hellip


Facebook Graph API

6

data [

획득한데이터

]

paging

cursors






Facebook Graph API

7

data [

획득한데이터

]

paging




Facebook Graph API

8

data [

획득한데이터

]

paging





9







gtgtgt





gtgtgt print (url)


gtgtgt





10


gtgtgt import json








gtgtgt



11

import sys


import json



app_id = 200920440387013




node = + page_name




try




page_id = data[id]



print (e)









12




















13

data[

comments

data[

]

summary

orderranked

total_count12

can_commentFalse



typelink

shares

count46

reactions

data[

]

summary

viewer_reactionNONE

total_count443

created_time2017-02-23T000000+0000


다

id240263402699918 _1328805163845731


NB11427906amppDate=20170222

]

paging


sfields=


postsfields=



14



try






print(e)


return None





15



node = + page_id





return None

else


return jsonData[id]



16












return None

else




17


try



else

return 0

except

return 0


try


return post[key]

else

return

except

return lsquo

reactions

data[

]

summary

viewer_reactionNONE

total_count443



18



node = s post_id

reactions = fields=











return None

else




19
















20


















21

go_next = True



while (go_next)





else

go_next = False




ensure_ascii=False)

outfilewrite(str_)


1TWITTER CRAWLING

2


OAuth

23


Twit Crawling

24


s_secret)

try


secret)



return client


print(e)

return None









try




print(e)

return None


Twit Crawling

25





tweet_link =




else

tweet_link =

hashtags =




else

hashtags =


Twit Crawling

26






else

tweet_published =


num_comments = 0












Twit Crawling

27

def main()


num_posts = 50

jsonResult = []



for tweet in tweets




outfilewrite(str_)


1NAVER CRAWLING

3



29





try






print(e)


return None



30



node = sjson sNode





return None

else




31


title = post[title]



link = post[link]

Tue 14 Feb 2017 184600 +0900





pDatepDate)

return

display 100

items [




pubDate Thu 13 Jul 2017 033800 +0900


]


start 1

total 454265



32

def main()

jsonResult = []


sNode = news


display_count = 100










ensure_ascii=False)



Naver 지도 API

33



node =





return None

else






35












return None

else




36














37

nItem = 100

nStartYear = 2011

nEndYear = 2017




nPagenum = 1

while True




if nTotal == 0

break





break

nPagenum += 1

else

break



38










return None

else




39

def main()

코드생략












코드생략













gtgtgt soup


gtgtgt tag = souptd

gtgtgt tag



gtgtgt tag


gtgtgt tag = soupa

gtgtgt tag


gtgtgt tagname

a






gtgtgt soup


gtgtgt tag = souptd

gtgtgt tag[class]

[title]


gtgtgt tag[class]

[tit3]


class [tit3]






gtgtgt soup



gtgtgt tag



gtgtgt tag









ltDOCTYPE htmlgt

lthtmlgt

ltheadgt






(이하중략)

lt-- Footer --gt

ltdivgt

ltbodygt

lthtmlgt



lttd class=titlegt

ltdiv class=tit3gt


ltdivgt

lttdgt




gtgtgt tags

[ltdiv class=tit3gt








ltdivgt]


print(taga)




print(tagatext)


덩케르크




3

JSON

Object

Array

nameJohn

age30



1FACEBOOK CRAWLING1


Facebook Graph API

5




hellip


Facebook Graph API

6

data [

획득한데이터

]

paging

cursors






Facebook Graph API

7

data [

획득한데이터

]

paging




Facebook Graph API

8

data [

획득한데이터

]

paging





9







gtgtgt





gtgtgt print (url)


gtgtgt





10


gtgtgt import json








gtgtgt



11

import sys


import json



app_id = 200920440387013




node = + page_name




try




page_id = data[id]



print (e)









12




















13

data[

comments

data[

]

summary

orderranked

total_count12

can_commentFalse



typelink

shares

count46

reactions

data[

]

summary

viewer_reactionNONE

total_count443

created_time2017-02-23T000000+0000


다

id240263402699918 _1328805163845731


NB11427906amppDate=20170222

]

paging


sfields=


postsfields=



14



try






print(e)


return None





15



node = + page_id





return None

else


return jsonData[id]



16












return None

else




17


try



else

return 0

except

return 0


try


return post[key]

else

return

except

return lsquo

reactions

data[

]

summary

viewer_reactionNONE

total_count443



18



node = s post_id

reactions = fields=











return None

else




19
















20


















21

go_next = True



while (go_next)





else

go_next = False




ensure_ascii=False)

outfilewrite(str_)


1TWITTER CRAWLING

2


OAuth

23


Twit Crawling

24


s_secret)

try


secret)



return client


print(e)

return None









try




print(e)

return None


Twit Crawling

25





tweet_link =




else

tweet_link =

hashtags =




else

hashtags =


Twit Crawling

26






else

tweet_published =


num_comments = 0












Twit Crawling

27

def main()


num_posts = 50

jsonResult = []



for tweet in tweets




outfilewrite(str_)


1NAVER CRAWLING

3



29





try






print(e)


return None



30



node = sjson sNode





return None

else




31


title = post[title]



link = post[link]

Tue 14 Feb 2017 184600 +0900





pDatepDate)

return

display 100

items [




pubDate Thu 13 Jul 2017 033800 +0900


]


start 1

total 454265



32

def main()

jsonResult = []


sNode = news


display_count = 100










ensure_ascii=False)



Naver 지도 API

33



node =





return None

else






35












return None

else




36














37

nItem = 100

nStartYear = 2011

nEndYear = 2017




nPagenum = 1

while True




if nTotal == 0

break





break

nPagenum += 1

else

break



38










return None

else




39

def main()

코드생략












코드생략













gtgtgt soup


gtgtgt tag = souptd

gtgtgt tag



gtgtgt tag


gtgtgt tag = soupa

gtgtgt tag


gtgtgt tagname

a






gtgtgt soup


gtgtgt tag = souptd

gtgtgt tag[class]

[title]


gtgtgt tag[class]

[tit3]


class [tit3]






gtgtgt soup



gtgtgt tag



gtgtgt tag









ltDOCTYPE htmlgt

lthtmlgt

ltheadgt






(이하중략)

lt-- Footer --gt

ltdivgt

ltbodygt

lthtmlgt



lttd class=titlegt

ltdiv class=tit3gt


ltdivgt

lttdgt




gtgtgt tags

[ltdiv class=tit3gt








ltdivgt]


print(taga)




print(tagatext)


덩케르크



1FACEBOOK CRAWLING1


Facebook Graph API

5




hellip


Facebook Graph API

6

data [

획득한데이터

]

paging

cursors






Facebook Graph API

7

data [

획득한데이터

]

paging




Facebook Graph API

8

data [

획득한데이터

]

paging





9







gtgtgt





gtgtgt print (url)


gtgtgt





10


gtgtgt import json








gtgtgt



11

import sys


import json



app_id = 200920440387013




node = + page_name




try




page_id = data[id]



print (e)









12




















13

data[

comments

data[

]

summary

orderranked

total_count12

can_commentFalse



typelink

shares

count46

reactions

data[

]

summary

viewer_reactionNONE

total_count443

created_time2017-02-23T000000+0000


다

id240263402699918 _1328805163845731


NB11427906amppDate=20170222

]

paging


sfields=


postsfields=



14



try






print(e)


return None





15



node = + page_id





return None

else


return jsonData[id]



16












return None

else




17


try



else

return 0

except

return 0


try


return post[key]

else

return

except

return lsquo

reactions

data[

]

summary

viewer_reactionNONE

total_count443



18



node = s post_id

reactions = fields=











return None

else




19
















20


















21

go_next = True



while (go_next)





else

go_next = False




ensure_ascii=False)

outfilewrite(str_)


1TWITTER CRAWLING

2


OAuth

23


Twit Crawling

24


s_secret)

try


secret)



return client


print(e)

return None









try




print(e)

return None


Twit Crawling

25





tweet_link =




else

tweet_link =

hashtags =




else

hashtags =


Twit Crawling

26






else

tweet_published =


num_comments = 0












Twit Crawling

27

def main()


num_posts = 50

jsonResult = []



for tweet in tweets




outfilewrite(str_)


1NAVER CRAWLING

3



29





try






print(e)


return None



30



node = sjson sNode





return None

else




31


title = post[title]



link = post[link]

Tue 14 Feb 2017 184600 +0900





pDatepDate)

return

display 100

items [




pubDate Thu 13 Jul 2017 033800 +0900


]


start 1

total 454265



32

def main()

jsonResult = []


sNode = news


display_count = 100










ensure_ascii=False)



Naver 지도 API

33



node =





return None

else






35












return None

else




36














37

nItem = 100

nStartYear = 2011

nEndYear = 2017




nPagenum = 1

while True




if nTotal == 0

break





break

nPagenum += 1

else

break



38










return None

else




39

def main()

코드생략












코드생략













gtgtgt soup


gtgtgt tag = souptd

gtgtgt tag



gtgtgt tag


gtgtgt tag = soupa

gtgtgt tag


gtgtgt tagname

a






gtgtgt soup


gtgtgt tag = souptd

gtgtgt tag[class]

[title]


gtgtgt tag[class]

[tit3]


class [tit3]






gtgtgt soup



gtgtgt tag



gtgtgt tag









ltDOCTYPE htmlgt

lthtmlgt

ltheadgt






(이하중략)

lt-- Footer --gt

ltdivgt

ltbodygt

lthtmlgt



lttd class=titlegt

ltdiv class=tit3gt


ltdivgt

lttdgt




gtgtgt tags

[ltdiv class=tit3gt








ltdivgt]


print(taga)




print(tagatext)


덩케르크



Facebook Graph API

5




hellip


Facebook Graph API

6

data [

획득한데이터

]

paging

cursors






Facebook Graph API

7

data [

획득한데이터

]

paging




Facebook Graph API

8

data [

획득한데이터

]

paging





9







gtgtgt





gtgtgt print (url)


gtgtgt





10


gtgtgt import json








gtgtgt



11

import sys


import json



app_id = 200920440387013




node = + page_name




try




page_id = data[id]



print (e)









12




















13

data[

comments

data[

]

summary

orderranked

total_count12

can_commentFalse



typelink

shares

count46

reactions

data[

]

summary

viewer_reactionNONE

total_count443

created_time2017-02-23T000000+0000


다

id240263402699918 _1328805163845731


NB11427906amppDate=20170222

]

paging


sfields=


postsfields=



14



try






print(e)


return None





15



node = + page_id





return None

else


return jsonData[id]



16












return None

else




17


try



else

return 0

except

return 0


try


return post[key]

else

return

except

return lsquo

reactions

data[

]

summary

viewer_reactionNONE

total_count443



18



node = s post_id

reactions = fields=











return None

else




19
















20


















21

go_next = True



while (go_next)





else

go_next = False




ensure_ascii=False)

outfilewrite(str_)


1TWITTER CRAWLING

2


OAuth

23


Twit Crawling

24


s_secret)

try


secret)



return client


print(e)

return None









try




print(e)

return None


Twit Crawling

25





tweet_link =




else

tweet_link =

hashtags =




else

hashtags =


Twit Crawling

26






else

tweet_published =


num_comments = 0












Twit Crawling

27

def main()


num_posts = 50

jsonResult = []



for tweet in tweets




outfilewrite(str_)


1NAVER CRAWLING

3



29





try






print(e)


return None



30



node = sjson sNode





return None

else




31


title = post[title]



link = post[link]

Tue 14 Feb 2017 184600 +0900





pDatepDate)

return

display 100

items [




pubDate Thu 13 Jul 2017 033800 +0900


]


start 1

total 454265



32

def main()

jsonResult = []


sNode = news


display_count = 100










ensure_ascii=False)



Naver 지도 API

33



node =





return None

else






35












return None

else




36














37

nItem = 100

nStartYear = 2011

nEndYear = 2017




nPagenum = 1

while True




if nTotal == 0

break





break

nPagenum += 1

else

break



38










return None

else




39

def main()

코드생략












코드생략













gtgtgt soup


gtgtgt tag = souptd

gtgtgt tag



gtgtgt tag


gtgtgt tag = soupa

gtgtgt tag


gtgtgt tagname

a






gtgtgt soup


gtgtgt tag = souptd

gtgtgt tag[class]

[title]


gtgtgt tag[class]

[tit3]


class [tit3]






gtgtgt soup



gtgtgt tag



gtgtgt tag









ltDOCTYPE htmlgt

lthtmlgt

ltheadgt






(이하중략)

lt-- Footer --gt

ltdivgt

ltbodygt

lthtmlgt



lttd class=titlegt

ltdiv class=tit3gt


ltdivgt

lttdgt




gtgtgt tags

[ltdiv class=tit3gt








ltdivgt]


print(taga)




print(tagatext)


덩케르크



Facebook Graph API

6

data [

획득한데이터

]

paging

cursors






Facebook Graph API

7

data [

획득한데이터

]

paging




Facebook Graph API

8

data [

획득한데이터

]

paging





9







gtgtgt





gtgtgt print (url)


gtgtgt





10


gtgtgt import json








gtgtgt



11

import sys


import json



app_id = 200920440387013




node = + page_name




try




page_id = data[id]



print (e)









12




















13

data[

comments

data[

]

summary

orderranked

total_count12

can_commentFalse



typelink

shares

count46

reactions

data[

]

summary

viewer_reactionNONE

total_count443

created_time2017-02-23T000000+0000


다

id240263402699918 _1328805163845731


NB11427906amppDate=20170222

]

paging


sfields=


postsfields=



14



try






print(e)


return None





15



node = + page_id





return None

else


return jsonData[id]



16












return None

else




17


try



else

return 0

except

return 0


try


return post[key]

else

return

except

return lsquo

reactions

data[

]

summary

viewer_reactionNONE

total_count443



18



node = s post_id

reactions = fields=











return None

else




19
















20


















21

go_next = True



while (go_next)





else

go_next = False




ensure_ascii=False)

outfilewrite(str_)


1TWITTER CRAWLING

2


OAuth

23


Twit Crawling

24


s_secret)

try


secret)



return client


print(e)

return None









try




print(e)

return None


Twit Crawling

25





tweet_link =




else

tweet_link =

hashtags =




else

hashtags =


Twit Crawling

26






else

tweet_published =


num_comments = 0












Twit Crawling

27

def main()


num_posts = 50

jsonResult = []



for tweet in tweets




outfilewrite(str_)


1NAVER CRAWLING

3



29





try






print(e)


return None



30



node = sjson sNode





return None

else




31


title = post[title]



link = post[link]

Tue 14 Feb 2017 184600 +0900





pDatepDate)

return

display 100

items [




pubDate Thu 13 Jul 2017 033800 +0900


]


start 1

total 454265



32

def main()

jsonResult = []


sNode = news


display_count = 100










ensure_ascii=False)



Naver 지도 API

33



node =





return None

else






35












return None

else




36














37

nItem = 100

nStartYear = 2011

nEndYear = 2017




nPagenum = 1

while True




if nTotal == 0

break





break

nPagenum += 1

else

break



38










return None

else




39

def main()

코드생략












코드생략













gtgtgt soup


gtgtgt tag = souptd

gtgtgt tag



gtgtgt tag


gtgtgt tag = soupa

gtgtgt tag


gtgtgt tagname

a






gtgtgt soup


gtgtgt tag = souptd

gtgtgt tag[class]

[title]


gtgtgt tag[class]

[tit3]


class [tit3]






gtgtgt soup



gtgtgt tag



gtgtgt tag









ltDOCTYPE htmlgt

lthtmlgt

ltheadgt






(이하중략)

lt-- Footer --gt

ltdivgt

ltbodygt

lthtmlgt



lttd class=titlegt

ltdiv class=tit3gt


ltdivgt

lttdgt




gtgtgt tags

[ltdiv class=tit3gt








ltdivgt]


print(taga)




print(tagatext)


덩케르크



Facebook Graph API

7

data [

획득한데이터

]

paging




Facebook Graph API

8

data [

획득한데이터

]

paging





9







gtgtgt





gtgtgt print (url)


gtgtgt





10


gtgtgt import json








gtgtgt



11

import sys


import json



app_id = 200920440387013




node = + page_name




try




page_id = data[id]



print (e)









12




















13

data[

comments

data[

]

summary

orderranked

total_count12

can_commentFalse



typelink

shares

count46

reactions

data[

]

summary

viewer_reactionNONE

total_count443

created_time2017-02-23T000000+0000


다

id240263402699918 _1328805163845731


NB11427906amppDate=20170222

]

paging


sfields=


postsfields=



14



try






print(e)


return None





15



node = + page_id





return None

else


return jsonData[id]



16












return None

else




17


try



else

return 0

except

return 0


try


return post[key]

else

return

except

return lsquo

reactions

data[

]

summary

viewer_reactionNONE

total_count443



18



node = s post_id

reactions = fields=











return None

else




19
















20


















21

go_next = True



while (go_next)





else

go_next = False




ensure_ascii=False)

outfilewrite(str_)


1TWITTER CRAWLING

2


OAuth

23


Twit Crawling

24


s_secret)

try


secret)



return client


print(e)

return None









try




print(e)

return None


Twit Crawling

25





tweet_link =




else

tweet_link =

hashtags =




else

hashtags =


Twit Crawling

26






else

tweet_published =


num_comments = 0












Twit Crawling

27

def main()


num_posts = 50

jsonResult = []



for tweet in tweets




outfilewrite(str_)


1NAVER CRAWLING

3



29





try






print(e)


return None



30



node = sjson sNode





return None

else




31


title = post[title]



link = post[link]

Tue 14 Feb 2017 184600 +0900





pDatepDate)

return

display 100

items [




pubDate Thu 13 Jul 2017 033800 +0900


]


start 1

total 454265



32

def main()

jsonResult = []


sNode = news


display_count = 100










ensure_ascii=False)



Naver 지도 API

33



node =





return None

else






35












return None

else




36














37

nItem = 100

nStartYear = 2011

nEndYear = 2017




nPagenum = 1

while True




if nTotal == 0

break





break

nPagenum += 1

else

break



38










return None

else




39

def main()

코드생략












코드생략













gtgtgt soup


gtgtgt tag = souptd

gtgtgt tag



gtgtgt tag


gtgtgt tag = soupa

gtgtgt tag


gtgtgt tagname

a






gtgtgt soup


gtgtgt tag = souptd

gtgtgt tag[class]

[title]


gtgtgt tag[class]

[tit3]


class [tit3]






gtgtgt soup



gtgtgt tag



gtgtgt tag









ltDOCTYPE htmlgt

lthtmlgt

ltheadgt






(이하중략)

lt-- Footer --gt

ltdivgt

ltbodygt

lthtmlgt



lttd class=titlegt

ltdiv class=tit3gt


ltdivgt

lttdgt




gtgtgt tags

[ltdiv class=tit3gt








ltdivgt]


print(taga)




print(tagatext)


덩케르크



Facebook Graph API

8

data [

획득한데이터

]

paging





9







gtgtgt





gtgtgt print (url)


gtgtgt





10


gtgtgt import json








gtgtgt



11

import sys


import json



app_id = 200920440387013




node = + page_name




try




page_id = data[id]



print (e)









12




















13

data[

comments

data[

]

summary

orderranked

total_count12

can_commentFalse



typelink

shares

count46

reactions

data[

]

summary

viewer_reactionNONE

total_count443

created_time2017-02-23T000000+0000


다

id240263402699918 _1328805163845731


NB11427906amppDate=20170222

]

paging


sfields=


postsfields=



14



try






print(e)


return None





15



node = + page_id





return None

else


return jsonData[id]



16












return None

else




17


try



else

return 0

except

return 0


try


return post[key]

else

return

except

return lsquo

reactions

data[

]

summary

viewer_reactionNONE

total_count443



18



node = s post_id

reactions = fields=











return None

else




19
















20


















21

go_next = True



while (go_next)





else

go_next = False




ensure_ascii=False)

outfilewrite(str_)


1TWITTER CRAWLING

2


OAuth

23


Twit Crawling

24


s_secret)

try


secret)



return client


print(e)

return None









try




print(e)

return None


Twit Crawling

25





tweet_link =




else

tweet_link =

hashtags =




else

hashtags =


Twit Crawling

26






else

tweet_published =


num_comments = 0












Twit Crawling

27

def main()


num_posts = 50

jsonResult = []



for tweet in tweets




outfilewrite(str_)


1NAVER CRAWLING

3



29





try






print(e)


return None



30



node = sjson sNode





return None

else




31


title = post[title]



link = post[link]

Tue 14 Feb 2017 184600 +0900





pDatepDate)

return

display 100

items [




pubDate Thu 13 Jul 2017 033800 +0900


]


start 1

total 454265



32

def main()

jsonResult = []


sNode = news


display_count = 100










ensure_ascii=False)



Naver 지도 API

33



node =





return None

else






35












return None

else




36














37

nItem = 100

nStartYear = 2011

nEndYear = 2017




nPagenum = 1

while True




if nTotal == 0

break





break

nPagenum += 1

else

break



38










return None

else




39

def main()

코드생략












코드생략













gtgtgt soup


gtgtgt tag = souptd

gtgtgt tag



gtgtgt tag


gtgtgt tag = soupa

gtgtgt tag


gtgtgt tagname

a






gtgtgt soup


gtgtgt tag = souptd

gtgtgt tag[class]

[title]


gtgtgt tag[class]

[tit3]


class [tit3]






gtgtgt soup



gtgtgt tag



gtgtgt tag









ltDOCTYPE htmlgt

lthtmlgt

ltheadgt






(이하중략)

lt-- Footer --gt

ltdivgt

ltbodygt

lthtmlgt



lttd class=titlegt

ltdiv class=tit3gt


ltdivgt

lttdgt




gtgtgt tags

[ltdiv class=tit3gt








ltdivgt]


print(taga)




print(tagatext)


덩케르크




9







gtgtgt





gtgtgt print (url)


gtgtgt





10


gtgtgt import json








gtgtgt



11

import sys


import json



app_id = 200920440387013




node = + page_name




try




page_id = data[id]



print (e)









12




















13

data[

comments

data[

]

summary

orderranked

total_count12

can_commentFalse



typelink

shares

count46

reactions

data[

]

summary

viewer_reactionNONE

total_count443

created_time2017-02-23T000000+0000


다

id240263402699918 _1328805163845731


NB11427906amppDate=20170222

]

paging


sfields=


postsfields=



14



try






print(e)


return None





15



node = + page_id





return None

else


return jsonData[id]



16












return None

else




17


try



else

return 0

except

return 0


try


return post[key]

else

return

except

return lsquo

reactions

data[

]

summary

viewer_reactionNONE

total_count443



18



node = s post_id

reactions = fields=











return None

else




19
















20


















21

go_next = True



while (go_next)





else

go_next = False




ensure_ascii=False)

outfilewrite(str_)


1TWITTER CRAWLING

2


OAuth

23


Twit Crawling

24


s_secret)

try


secret)



return client


print(e)

return None









try




print(e)

return None


Twit Crawling

25





tweet_link =




else

tweet_link =

hashtags =




else

hashtags =


Twit Crawling

26






else

tweet_published =


num_comments = 0












Twit Crawling

27

def main()


num_posts = 50

jsonResult = []



for tweet in tweets




outfilewrite(str_)


1NAVER CRAWLING

3



29





try






print(e)


return None



30



node = sjson sNode





return None

else




31


title = post[title]



link = post[link]

Tue 14 Feb 2017 184600 +0900





pDatepDate)

return

display 100

items [




pubDate Thu 13 Jul 2017 033800 +0900


]


start 1

total 454265



32

def main()

jsonResult = []


sNode = news


display_count = 100










ensure_ascii=False)



Naver 지도 API

33



node =





return None

else






35












return None

else




36














37

nItem = 100

nStartYear = 2011

nEndYear = 2017




nPagenum = 1

while True




if nTotal == 0

break





break

nPagenum += 1

else

break



38










return None

else




39

def main()

코드생략












코드생략













gtgtgt soup


gtgtgt tag = souptd

gtgtgt tag



gtgtgt tag


gtgtgt tag = soupa

gtgtgt tag


gtgtgt tagname

a






gtgtgt soup


gtgtgt tag = souptd

gtgtgt tag[class]

[title]


gtgtgt tag[class]

[tit3]


class [tit3]






gtgtgt soup



gtgtgt tag



gtgtgt tag









ltDOCTYPE htmlgt

lthtmlgt

ltheadgt






(이하중략)

lt-- Footer --gt

ltdivgt

ltbodygt

lthtmlgt



lttd class=titlegt

ltdiv class=tit3gt


ltdivgt

lttdgt




gtgtgt tags

[ltdiv class=tit3gt








ltdivgt]


print(taga)




print(tagatext)


덩케르크




10


gtgtgt import json








gtgtgt



11

import sys


import json



app_id = 200920440387013




node = + page_name




try




page_id = data[id]



print (e)









12




















13

data[

comments

data[

]

summary

orderranked

total_count12

can_commentFalse



typelink

shares

count46

reactions

data[

]

summary

viewer_reactionNONE

total_count443

created_time2017-02-23T000000+0000


다

id240263402699918 _1328805163845731


NB11427906amppDate=20170222

]

paging


sfields=


postsfields=



14



try






print(e)


return None





15



node = + page_id





return None

else


return jsonData[id]



16












return None

else




17


try



else

return 0

except

return 0


try


return post[key]

else

return

except

return lsquo

reactions

data[

]

summary

viewer_reactionNONE

total_count443



18



node = s post_id

reactions = fields=











return None

else




19
















20


















21

go_next = True



while (go_next)





else

go_next = False




ensure_ascii=False)

outfilewrite(str_)


1TWITTER CRAWLING

2


OAuth

23


Twit Crawling

24


s_secret)

try


secret)



return client


print(e)

return None









try




print(e)

return None


Twit Crawling

25





tweet_link =




else

tweet_link =

hashtags =




else

hashtags =


Twit Crawling

26






else

tweet_published =


num_comments = 0












Twit Crawling

27

def main()


num_posts = 50

jsonResult = []



for tweet in tweets




outfilewrite(str_)


1NAVER CRAWLING

3



29





try






print(e)


return None



30



node = sjson sNode





return None

else




31


title = post[title]



link = post[link]

Tue 14 Feb 2017 184600 +0900





pDatepDate)

return

display 100

items [




pubDate Thu 13 Jul 2017 033800 +0900


]


start 1

total 454265



32

def main()

jsonResult = []


sNode = news


display_count = 100










ensure_ascii=False)



Naver 지도 API

33



node =





return None

else






35












return None

else




36














37

nItem = 100

nStartYear = 2011

nEndYear = 2017




nPagenum = 1

while True




if nTotal == 0

break





break

nPagenum += 1

else

break



38










return None

else




39

def main()

코드생략












코드생략













gtgtgt soup


gtgtgt tag = souptd

gtgtgt tag



gtgtgt tag


gtgtgt tag = soupa

gtgtgt tag


gtgtgt tagname

a






gtgtgt soup


gtgtgt tag = souptd

gtgtgt tag[class]

[title]


gtgtgt tag[class]

[tit3]


class [tit3]






gtgtgt soup



gtgtgt tag



gtgtgt tag









ltDOCTYPE htmlgt

lthtmlgt

ltheadgt






(이하중략)

lt-- Footer --gt

ltdivgt

ltbodygt

lthtmlgt



lttd class=titlegt

ltdiv class=tit3gt


ltdivgt

lttdgt




gtgtgt tags

[ltdiv class=tit3gt








ltdivgt]


print(taga)




print(tagatext)


덩케르크




11

import sys


import json



app_id = 200920440387013




node = + page_name




try




page_id = data[id]



print (e)









12




















13

data[

comments

data[

]

summary

orderranked

total_count12

can_commentFalse



typelink

shares

count46

reactions

data[

]

summary

viewer_reactionNONE

total_count443

created_time2017-02-23T000000+0000


다

id240263402699918 _1328805163845731


NB11427906amppDate=20170222

]

paging


sfields=


postsfields=



14



try






print(e)


return None





15



node = + page_id





return None

else


return jsonData[id]



16












return None

else




17


try



else

return 0

except

return 0


try


return post[key]

else

return

except

return lsquo

reactions

data[

]

summary

viewer_reactionNONE

total_count443



18



node = s post_id

reactions = fields=











return None

else




19
















20


















21

go_next = True



while (go_next)





else

go_next = False




ensure_ascii=False)

outfilewrite(str_)


1TWITTER CRAWLING

2


OAuth

23


Twit Crawling

24


s_secret)

try


secret)



return client


print(e)

return None









try




print(e)

return None


Twit Crawling

25





tweet_link =




else

tweet_link =

hashtags =




else

hashtags =


Twit Crawling

26






else

tweet_published =


num_comments = 0












Twit Crawling

27

def main()


num_posts = 50

jsonResult = []



for tweet in tweets




outfilewrite(str_)


1NAVER CRAWLING

3



29





try






print(e)


return None



30



node = sjson sNode





return None

else




31


title = post[title]



link = post[link]

Tue 14 Feb 2017 184600 +0900





pDatepDate)

return

display 100

items [




pubDate Thu 13 Jul 2017 033800 +0900


]


start 1

total 454265



32

def main()

jsonResult = []


sNode = news


display_count = 100










ensure_ascii=False)



Naver 지도 API

33



node =





return None

else






35












return None

else




36














37

nItem = 100

nStartYear = 2011

nEndYear = 2017




nPagenum = 1

while True




if nTotal == 0

break





break

nPagenum += 1

else

break



38










return None

else




39

def main()

코드생략












코드생략













gtgtgt soup


gtgtgt tag = souptd

gtgtgt tag



gtgtgt tag


gtgtgt tag = soupa

gtgtgt tag


gtgtgt tagname

a






gtgtgt soup


gtgtgt tag = souptd

gtgtgt tag[class]

[title]


gtgtgt tag[class]

[tit3]


class [tit3]






gtgtgt soup



gtgtgt tag



gtgtgt tag









ltDOCTYPE htmlgt

lthtmlgt

ltheadgt






(이하중략)

lt-- Footer --gt

ltdivgt

ltbodygt

lthtmlgt



lttd class=titlegt

ltdiv class=tit3gt


ltdivgt

lttdgt




gtgtgt tags

[ltdiv class=tit3gt








ltdivgt]


print(taga)




print(tagatext)


덩케르크




12




















13

data[

comments

data[

]

summary

orderranked

total_count12

can_commentFalse



typelink

shares

count46

reactions

data[

]

summary

viewer_reactionNONE

total_count443

created_time2017-02-23T000000+0000


다

id240263402699918 _1328805163845731


NB11427906amppDate=20170222

]

paging


sfields=


postsfields=



14



try






print(e)


return None





15



node = + page_id





return None

else


return jsonData[id]



16












return None

else




17


try



else

return 0

except

return 0


try


return post[key]

else

return

except

return lsquo

reactions

data[

]

summary

viewer_reactionNONE

total_count443



18



node = s post_id

reactions = fields=











return None

else




19
















20


















21

go_next = True



while (go_next)





else

go_next = False




ensure_ascii=False)

outfilewrite(str_)


1TWITTER CRAWLING

2


OAuth

23


Twit Crawling

24


s_secret)

try


secret)



return client


print(e)

return None









try




print(e)

return None


Twit Crawling

25





tweet_link =




else

tweet_link =

hashtags =




else

hashtags =


Twit Crawling

26






else

tweet_published =


num_comments = 0












Twit Crawling

27

def main()


num_posts = 50

jsonResult = []



for tweet in tweets




outfilewrite(str_)


1NAVER CRAWLING

3



29





try






print(e)


return None



30



node = sjson sNode





return None

else




31


title = post[title]



link = post[link]

Tue 14 Feb 2017 184600 +0900





pDatepDate)

return

display 100

items [




pubDate Thu 13 Jul 2017 033800 +0900


]


start 1

total 454265



32

def main()

jsonResult = []


sNode = news


display_count = 100










ensure_ascii=False)



Naver 지도 API

33



node =





return None

else






35












return None

else




36














37

nItem = 100

nStartYear = 2011

nEndYear = 2017




nPagenum = 1

while True




if nTotal == 0

break





break

nPagenum += 1

else

break



38










return None

else




39

def main()

코드생략












코드생략













gtgtgt soup


gtgtgt tag = souptd

gtgtgt tag



gtgtgt tag


gtgtgt tag = soupa

gtgtgt tag


gtgtgt tagname

a






gtgtgt soup


gtgtgt tag = souptd

gtgtgt tag[class]

[title]


gtgtgt tag[class]

[tit3]


class [tit3]






gtgtgt soup



gtgtgt tag



gtgtgt tag









ltDOCTYPE htmlgt

lthtmlgt

ltheadgt






(이하중략)

lt-- Footer --gt

ltdivgt

ltbodygt

lthtmlgt



lttd class=titlegt

ltdiv class=tit3gt


ltdivgt

lttdgt




gtgtgt tags

[ltdiv class=tit3gt








ltdivgt]


print(taga)




print(tagatext)


덩케르크




13

data[

comments

data[

]

summary

orderranked

total_count12

can_commentFalse



typelink

shares

count46

reactions

data[

]

summary

viewer_reactionNONE

total_count443

created_time2017-02-23T000000+0000


다

id240263402699918 _1328805163845731


NB11427906amppDate=20170222

]

paging


sfields=


postsfields=



14



try






print(e)


return None





15



node = + page_id





return None

else


return jsonData[id]



16












return None

else




17


try



else

return 0

except

return 0


try


return post[key]

else

return

except

return lsquo

reactions

data[

]

summary

viewer_reactionNONE

total_count443



18



node = s post_id

reactions = fields=











return None

else




19
















20


















21

go_next = True



while (go_next)





else

go_next = False




ensure_ascii=False)

outfilewrite(str_)


1TWITTER CRAWLING

2


OAuth

23


Twit Crawling

24


s_secret)

try


secret)



return client


print(e)

return None









try




print(e)

return None


Twit Crawling

25





tweet_link =




else

tweet_link =

hashtags =




else

hashtags =


Twit Crawling

26






else

tweet_published =


num_comments = 0












Twit Crawling

27

def main()


num_posts = 50

jsonResult = []



for tweet in tweets




outfilewrite(str_)


1NAVER CRAWLING

3



29





try






print(e)


return None



30



node = sjson sNode





return None

else




31


title = post[title]



link = post[link]

Tue 14 Feb 2017 184600 +0900





pDatepDate)

return

display 100

items [




pubDate Thu 13 Jul 2017 033800 +0900


]


start 1

total 454265



32

def main()

jsonResult = []


sNode = news


display_count = 100










ensure_ascii=False)



Naver 지도 API

33



node =





return None

else






35












return None

else




36














37

nItem = 100

nStartYear = 2011

nEndYear = 2017




nPagenum = 1

while True




if nTotal == 0

break





break

nPagenum += 1

else

break



38










return None

else




39

def main()

코드생략












코드생략













gtgtgt soup


gtgtgt tag = souptd

gtgtgt tag



gtgtgt tag


gtgtgt tag = soupa

gtgtgt tag


gtgtgt tagname

a






gtgtgt soup


gtgtgt tag = souptd

gtgtgt tag[class]

[title]


gtgtgt tag[class]

[tit3]


class [tit3]






gtgtgt soup



gtgtgt tag



gtgtgt tag









ltDOCTYPE htmlgt

lthtmlgt

ltheadgt






(이하중략)

lt-- Footer --gt

ltdivgt

ltbodygt

lthtmlgt



lttd class=titlegt

ltdiv class=tit3gt


ltdivgt

lttdgt




gtgtgt tags

[ltdiv class=tit3gt








ltdivgt]


print(taga)




print(tagatext)


덩케르크




14



try






print(e)


return None





15



node = + page_id





return None

else


return jsonData[id]



16












return None

else




17


try



else

return 0

except

return 0


try


return post[key]

else

return

except

return lsquo

reactions

data[

]

summary

viewer_reactionNONE

total_count443



18



node = s post_id

reactions = fields=











return None

else




19
















20


















21

go_next = True



while (go_next)





else

go_next = False




ensure_ascii=False)

outfilewrite(str_)


1TWITTER CRAWLING

2


OAuth

23


Twit Crawling

24


s_secret)

try


secret)



return client


print(e)

return None









try




print(e)

return None


Twit Crawling

25





tweet_link =




else

tweet_link =

hashtags =




else

hashtags =


Twit Crawling

26






else

tweet_published =


num_comments = 0












Twit Crawling

27

def main()


num_posts = 50

jsonResult = []



for tweet in tweets




outfilewrite(str_)


1NAVER CRAWLING

3



29





try






print(e)


return None



30



node = sjson sNode





return None

else




31


title = post[title]



link = post[link]

Tue 14 Feb 2017 184600 +0900





pDatepDate)

return

display 100

items [




pubDate Thu 13 Jul 2017 033800 +0900


]


start 1

total 454265



32

def main()

jsonResult = []


sNode = news


display_count = 100










ensure_ascii=False)



Naver 지도 API

33



node =





return None

else






35












return None

else




36














37

nItem = 100

nStartYear = 2011

nEndYear = 2017




nPagenum = 1

while True




if nTotal == 0

break





break

nPagenum += 1

else

break



38










return None

else




39

def main()

코드생략












코드생략













gtgtgt soup


gtgtgt tag = souptd

gtgtgt tag



gtgtgt tag


gtgtgt tag = soupa

gtgtgt tag


gtgtgt tagname

a






gtgtgt soup


gtgtgt tag = souptd

gtgtgt tag[class]

[title]


gtgtgt tag[class]

[tit3]


class [tit3]






gtgtgt soup



gtgtgt tag



gtgtgt tag









ltDOCTYPE htmlgt

lthtmlgt

ltheadgt






(이하중략)

lt-- Footer --gt

ltdivgt

ltbodygt

lthtmlgt



lttd class=titlegt

ltdiv class=tit3gt


ltdivgt

lttdgt




gtgtgt tags

[ltdiv class=tit3gt








ltdivgt]


print(taga)




print(tagatext)


덩케르크




15



node = + page_id





return None

else


return jsonData[id]



16












return None

else




17


try



else

return 0

except

return 0


try


return post[key]

else

return

except

return lsquo

reactions

data[

]

summary

viewer_reactionNONE

total_count443



18



node = s post_id

reactions = fields=











return None

else




19
















20


















21

go_next = True



while (go_next)





else

go_next = False




ensure_ascii=False)

outfilewrite(str_)


1TWITTER CRAWLING

2


OAuth

23


Twit Crawling

24


s_secret)

try


secret)



return client


print(e)

return None









try




print(e)

return None


Twit Crawling

25





tweet_link =




else

tweet_link =

hashtags =




else

hashtags =


Twit Crawling

26






else

tweet_published =


num_comments = 0












Twit Crawling

27

def main()


num_posts = 50

jsonResult = []



for tweet in tweets




outfilewrite(str_)


1NAVER CRAWLING

3



29





try






print(e)


return None



30



node = sjson sNode





return None

else




31


title = post[title]



link = post[link]

Tue 14 Feb 2017 184600 +0900





pDatepDate)

return

display 100

items [




pubDate Thu 13 Jul 2017 033800 +0900


]


start 1

total 454265



32

def main()

jsonResult = []


sNode = news


display_count = 100










ensure_ascii=False)



Naver 지도 API

33



node =





return None

else






35












return None

else




36














37

nItem = 100

nStartYear = 2011

nEndYear = 2017




nPagenum = 1

while True




if nTotal == 0

break





break

nPagenum += 1

else

break



38










return None

else




39

def main()

코드생략












코드생략













gtgtgt soup


gtgtgt tag = souptd

gtgtgt tag



gtgtgt tag


gtgtgt tag = soupa

gtgtgt tag


gtgtgt tagname

a






gtgtgt soup


gtgtgt tag = souptd

gtgtgt tag[class]

[title]


gtgtgt tag[class]

[tit3]


class [tit3]






gtgtgt soup



gtgtgt tag



gtgtgt tag









ltDOCTYPE htmlgt

lthtmlgt

ltheadgt






(이하중략)

lt-- Footer --gt

ltdivgt

ltbodygt

lthtmlgt



lttd class=titlegt

ltdiv class=tit3gt


ltdivgt

lttdgt




gtgtgt tags

[ltdiv class=tit3gt








ltdivgt]


print(taga)




print(tagatext)


덩케르크




16












return None

else




17


try



else

return 0

except

return 0


try


return post[key]

else

return

except

return lsquo

reactions

data[

]

summary

viewer_reactionNONE

total_count443



18



node = s post_id

reactions = fields=











return None

else




19
















20


















21

go_next = True



while (go_next)





else

go_next = False




ensure_ascii=False)

outfilewrite(str_)


1TWITTER CRAWLING

2


OAuth

23


Twit Crawling

24


s_secret)

try


secret)



return client


print(e)

return None









try




print(e)

return None


Twit Crawling

25





tweet_link =




else

tweet_link =

hashtags =




else

hashtags =


Twit Crawling

26






else

tweet_published =


num_comments = 0












Twit Crawling

27

def main()


num_posts = 50

jsonResult = []



for tweet in tweets




outfilewrite(str_)


1NAVER CRAWLING

3



29





try






print(e)


return None



30



node = sjson sNode





return None

else




31


title = post[title]



link = post[link]

Tue 14 Feb 2017 184600 +0900





pDatepDate)

return

display 100

items [




pubDate Thu 13 Jul 2017 033800 +0900


]


start 1

total 454265



32

def main()

jsonResult = []


sNode = news


display_count = 100










ensure_ascii=False)



Naver 지도 API

33



node =





return None

else






35












return None

else




36














37

nItem = 100

nStartYear = 2011

nEndYear = 2017




nPagenum = 1

while True




if nTotal == 0

break





break

nPagenum += 1

else

break



38










return None

else




39

def main()

코드생략












코드생략













gtgtgt soup


gtgtgt tag = souptd

gtgtgt tag



gtgtgt tag


gtgtgt tag = soupa

gtgtgt tag


gtgtgt tagname

a






gtgtgt soup


gtgtgt tag = souptd

gtgtgt tag[class]

[title]


gtgtgt tag[class]

[tit3]


class [tit3]






gtgtgt soup



gtgtgt tag



gtgtgt tag









ltDOCTYPE htmlgt

lthtmlgt

ltheadgt






(이하중략)

lt-- Footer --gt

ltdivgt

ltbodygt

lthtmlgt



lttd class=titlegt

ltdiv class=tit3gt


ltdivgt

lttdgt




gtgtgt tags

[ltdiv class=tit3gt








ltdivgt]


print(taga)




print(tagatext)


덩케르크




17


try



else

return 0

except

return 0


try


return post[key]

else

return

except

return lsquo

reactions

data[

]

summary

viewer_reactionNONE

total_count443



18



node = s post_id

reactions = fields=











return None

else




19
















20


















21

go_next = True



while (go_next)





else

go_next = False




ensure_ascii=False)

outfilewrite(str_)


1TWITTER CRAWLING

2


OAuth

23


Twit Crawling

24


s_secret)

try


secret)



return client


print(e)

return None









try




print(e)

return None


Twit Crawling

25





tweet_link =




else

tweet_link =

hashtags =




else

hashtags =


Twit Crawling

26






else

tweet_published =


num_comments = 0












Twit Crawling

27

def main()


num_posts = 50

jsonResult = []



for tweet in tweets




outfilewrite(str_)


1NAVER CRAWLING

3



29





try






print(e)


return None



30



node = sjson sNode





return None

else




31


title = post[title]



link = post[link]

Tue 14 Feb 2017 184600 +0900





pDatepDate)

return

display 100

items [




pubDate Thu 13 Jul 2017 033800 +0900


]


start 1

total 454265



32

def main()

jsonResult = []


sNode = news


display_count = 100










ensure_ascii=False)



Naver 지도 API

33



node =





return None

else






35












return None

else




36














37

nItem = 100

nStartYear = 2011

nEndYear = 2017




nPagenum = 1

while True




if nTotal == 0

break





break

nPagenum += 1

else

break



38










return None

else




39

def main()

코드생략












코드생략













gtgtgt soup


gtgtgt tag = souptd

gtgtgt tag



gtgtgt tag


gtgtgt tag = soupa

gtgtgt tag


gtgtgt tagname

a






gtgtgt soup


gtgtgt tag = souptd

gtgtgt tag[class]

[title]


gtgtgt tag[class]

[tit3]


class [tit3]






gtgtgt soup



gtgtgt tag



gtgtgt tag









ltDOCTYPE htmlgt

lthtmlgt

ltheadgt






(이하중략)

lt-- Footer --gt

ltdivgt

ltbodygt

lthtmlgt



lttd class=titlegt

ltdiv class=tit3gt


ltdivgt

lttdgt




gtgtgt tags

[ltdiv class=tit3gt








ltdivgt]


print(taga)




print(tagatext)


덩케르크




18



node = s post_id

reactions = fields=











return None

else




19
















20


















21

go_next = True



while (go_next)





else

go_next = False




ensure_ascii=False)

outfilewrite(str_)


1TWITTER CRAWLING

2


OAuth

23


Twit Crawling

24


s_secret)

try


secret)



return client


print(e)

return None









try




print(e)

return None


Twit Crawling

25





tweet_link =




else

tweet_link =

hashtags =




else

hashtags =


Twit Crawling

26






else

tweet_published =


num_comments = 0












Twit Crawling

27

def main()


num_posts = 50

jsonResult = []



for tweet in tweets




outfilewrite(str_)


1NAVER CRAWLING

3



29





try






print(e)


return None



30



node = sjson sNode





return None

else




31


title = post[title]



link = post[link]

Tue 14 Feb 2017 184600 +0900





pDatepDate)

return

display 100

items [




pubDate Thu 13 Jul 2017 033800 +0900


]


start 1

total 454265



32

def main()

jsonResult = []


sNode = news


display_count = 100










ensure_ascii=False)



Naver 지도 API

33



node =





return None

else






35












return None

else




36














37

nItem = 100

nStartYear = 2011

nEndYear = 2017




nPagenum = 1

while True




if nTotal == 0

break





break

nPagenum += 1

else

break



38










return None

else




39

def main()

코드생략












코드생략













gtgtgt soup


gtgtgt tag = souptd

gtgtgt tag



gtgtgt tag


gtgtgt tag = soupa

gtgtgt tag


gtgtgt tagname

a






gtgtgt soup


gtgtgt tag = souptd

gtgtgt tag[class]

[title]


gtgtgt tag[class]

[tit3]


class [tit3]






gtgtgt soup



gtgtgt tag



gtgtgt tag









ltDOCTYPE htmlgt

lthtmlgt

ltheadgt






(이하중략)

lt-- Footer --gt

ltdivgt

ltbodygt

lthtmlgt



lttd class=titlegt

ltdiv class=tit3gt


ltdivgt

lttdgt




gtgtgt tags

[ltdiv class=tit3gt








ltdivgt]


print(taga)




print(tagatext)


덩케르크




19
















20


















21

go_next = True



while (go_next)





else

go_next = False




ensure_ascii=False)

outfilewrite(str_)


1TWITTER CRAWLING

2


OAuth

23


Twit Crawling

24


s_secret)

try


secret)



return client


print(e)

return None









try




print(e)

return None


Twit Crawling

25





tweet_link =




else

tweet_link =

hashtags =




else

hashtags =


Twit Crawling

26






else

tweet_published =


num_comments = 0












Twit Crawling

27

def main()


num_posts = 50

jsonResult = []



for tweet in tweets




outfilewrite(str_)


1NAVER CRAWLING

3



29





try






print(e)


return None



30



node = sjson sNode





return None

else




31


title = post[title]



link = post[link]

Tue 14 Feb 2017 184600 +0900





pDatepDate)

return

display 100

items [




pubDate Thu 13 Jul 2017 033800 +0900


]


start 1

total 454265



32

def main()

jsonResult = []


sNode = news


display_count = 100










ensure_ascii=False)



Naver 지도 API

33



node =





return None

else






35












return None

else




36














37

nItem = 100

nStartYear = 2011

nEndYear = 2017




nPagenum = 1

while True




if nTotal == 0

break





break

nPagenum += 1

else

break



38










return None

else




39

def main()

코드생략












코드생략













gtgtgt soup


gtgtgt tag = souptd

gtgtgt tag



gtgtgt tag


gtgtgt tag = soupa

gtgtgt tag


gtgtgt tagname

a






gtgtgt soup


gtgtgt tag = souptd

gtgtgt tag[class]

[title]


gtgtgt tag[class]

[tit3]


class [tit3]






gtgtgt soup



gtgtgt tag



gtgtgt tag









ltDOCTYPE htmlgt

lthtmlgt

ltheadgt






(이하중략)

lt-- Footer --gt

ltdivgt

ltbodygt

lthtmlgt



lttd class=titlegt

ltdiv class=tit3gt


ltdivgt

lttdgt




gtgtgt tags

[ltdiv class=tit3gt








ltdivgt]


print(taga)




print(tagatext)


덩케르크




20


















21

go_next = True



while (go_next)





else

go_next = False




ensure_ascii=False)

outfilewrite(str_)


1TWITTER CRAWLING

2


OAuth

23


Twit Crawling

24


s_secret)

try


secret)



return client


print(e)

return None









try




print(e)

return None


Twit Crawling

25





tweet_link =




else

tweet_link =

hashtags =




else

hashtags =


Twit Crawling

26






else

tweet_published =


num_comments = 0












Twit Crawling

27

def main()


num_posts = 50

jsonResult = []



for tweet in tweets




outfilewrite(str_)


1NAVER CRAWLING

3



29





try






print(e)


return None



30



node = sjson sNode





return None

else




31


title = post[title]



link = post[link]

Tue 14 Feb 2017 184600 +0900





pDatepDate)

return

display 100

items [




pubDate Thu 13 Jul 2017 033800 +0900


]


start 1

total 454265



32

def main()

jsonResult = []


sNode = news


display_count = 100










ensure_ascii=False)



Naver 지도 API

33



node =





return None

else






35












return None

else




36














37

nItem = 100

nStartYear = 2011

nEndYear = 2017




nPagenum = 1

while True




if nTotal == 0

break





break

nPagenum += 1

else

break



38










return None

else




39

def main()

코드생략












코드생략













gtgtgt soup


gtgtgt tag = souptd

gtgtgt tag



gtgtgt tag


gtgtgt tag = soupa

gtgtgt tag


gtgtgt tagname

a






gtgtgt soup


gtgtgt tag = souptd

gtgtgt tag[class]

[title]


gtgtgt tag[class]

[tit3]


class [tit3]






gtgtgt soup



gtgtgt tag



gtgtgt tag









ltDOCTYPE htmlgt

lthtmlgt

ltheadgt






(이하중략)

lt-- Footer --gt

ltdivgt

ltbodygt

lthtmlgt



lttd class=titlegt

ltdiv class=tit3gt


ltdivgt

lttdgt




gtgtgt tags

[ltdiv class=tit3gt








ltdivgt]


print(taga)




print(tagatext)


덩케르크




21

go_next = True



while (go_next)





else

go_next = False




ensure_ascii=False)

outfilewrite(str_)


1TWITTER CRAWLING

2


OAuth

23


Twit Crawling

24


s_secret)

try


secret)



return client


print(e)

return None









try




print(e)

return None


Twit Crawling

25





tweet_link =




else

tweet_link =

hashtags =




else

hashtags =


Twit Crawling

26






else

tweet_published =


num_comments = 0












Twit Crawling

27

def main()


num_posts = 50

jsonResult = []



for tweet in tweets




outfilewrite(str_)


1NAVER CRAWLING

3



29





try






print(e)


return None



30



node = sjson sNode





return None

else




31


title = post[title]



link = post[link]

Tue 14 Feb 2017 184600 +0900





pDatepDate)

return

display 100

items [




pubDate Thu 13 Jul 2017 033800 +0900


]


start 1

total 454265



32

def main()

jsonResult = []


sNode = news


display_count = 100










ensure_ascii=False)



Naver 지도 API

33



node =





return None

else






35












return None

else




36














37

nItem = 100

nStartYear = 2011

nEndYear = 2017




nPagenum = 1

while True




if nTotal == 0

break





break

nPagenum += 1

else

break



38










return None

else




39

def main()

코드생략












코드생략













gtgtgt soup


gtgtgt tag = souptd

gtgtgt tag



gtgtgt tag


gtgtgt tag = soupa

gtgtgt tag


gtgtgt tagname

a






gtgtgt soup


gtgtgt tag = souptd

gtgtgt tag[class]

[title]


gtgtgt tag[class]

[tit3]


class [tit3]






gtgtgt soup



gtgtgt tag



gtgtgt tag









ltDOCTYPE htmlgt

lthtmlgt

ltheadgt






(이하중략)

lt-- Footer --gt

ltdivgt

ltbodygt

lthtmlgt



lttd class=titlegt

ltdiv class=tit3gt


ltdivgt

lttdgt




gtgtgt tags

[ltdiv class=tit3gt








ltdivgt]


print(taga)




print(tagatext)


덩케르크



1TWITTER CRAWLING

2


OAuth

23


Twit Crawling

24


s_secret)

try


secret)



return client


print(e)

return None









try




print(e)

return None


Twit Crawling

25





tweet_link =




else

tweet_link =

hashtags =




else

hashtags =


Twit Crawling

26






else

tweet_published =


num_comments = 0












Twit Crawling

27

def main()


num_posts = 50

jsonResult = []



for tweet in tweets




outfilewrite(str_)


1NAVER CRAWLING

3



29





try






print(e)


return None



30



node = sjson sNode





return None

else




31


title = post[title]



link = post[link]

Tue 14 Feb 2017 184600 +0900





pDatepDate)

return

display 100

items [




pubDate Thu 13 Jul 2017 033800 +0900


]


start 1

total 454265



32

def main()

jsonResult = []


sNode = news


display_count = 100










ensure_ascii=False)



Naver 지도 API

33



node =





return None

else






35












return None

else




36














37

nItem = 100

nStartYear = 2011

nEndYear = 2017




nPagenum = 1

while True




if nTotal == 0

break





break

nPagenum += 1

else

break



38










return None

else




39

def main()

코드생략












코드생략













gtgtgt soup


gtgtgt tag = souptd

gtgtgt tag



gtgtgt tag


gtgtgt tag = soupa

gtgtgt tag


gtgtgt tagname

a






gtgtgt soup


gtgtgt tag = souptd

gtgtgt tag[class]

[title]


gtgtgt tag[class]

[tit3]


class [tit3]






gtgtgt soup



gtgtgt tag



gtgtgt tag









ltDOCTYPE htmlgt

lthtmlgt

ltheadgt






(이하중략)

lt-- Footer --gt

ltdivgt

ltbodygt

lthtmlgt



lttd class=titlegt

ltdiv class=tit3gt


ltdivgt

lttdgt




gtgtgt tags

[ltdiv class=tit3gt








ltdivgt]


print(taga)




print(tagatext)


덩케르크



OAuth

23


Twit Crawling

24


s_secret)

try


secret)



return client


print(e)

return None









try




print(e)

return None


Twit Crawling

25





tweet_link =




else

tweet_link =

hashtags =




else

hashtags =


Twit Crawling

26






else

tweet_published =


num_comments = 0












Twit Crawling

27

def main()


num_posts = 50

jsonResult = []



for tweet in tweets




outfilewrite(str_)


1NAVER CRAWLING

3



29





try






print(e)


return None



30



node = sjson sNode





return None

else




31


title = post[title]



link = post[link]

Tue 14 Feb 2017 184600 +0900





pDatepDate)

return

display 100

items [




pubDate Thu 13 Jul 2017 033800 +0900


]


start 1

total 454265



32

def main()

jsonResult = []


sNode = news


display_count = 100










ensure_ascii=False)



Naver 지도 API

33



node =





return None

else






35












return None

else




36














37

nItem = 100

nStartYear = 2011

nEndYear = 2017




nPagenum = 1

while True




if nTotal == 0

break





break

nPagenum += 1

else

break



38










return None

else




39

def main()

코드생략












코드생략













gtgtgt soup


gtgtgt tag = souptd

gtgtgt tag



gtgtgt tag


gtgtgt tag = soupa

gtgtgt tag


gtgtgt tagname

a






gtgtgt soup


gtgtgt tag = souptd

gtgtgt tag[class]

[title]


gtgtgt tag[class]

[tit3]


class [tit3]






gtgtgt soup



gtgtgt tag



gtgtgt tag









ltDOCTYPE htmlgt

lthtmlgt

ltheadgt






(이하중략)

lt-- Footer --gt

ltdivgt

ltbodygt

lthtmlgt



lttd class=titlegt

ltdiv class=tit3gt


ltdivgt

lttdgt




gtgtgt tags

[ltdiv class=tit3gt








ltdivgt]


print(taga)




print(tagatext)


덩케르크



Twit Crawling

24


s_secret)

try


secret)



return client


print(e)

return None









try




print(e)

return None


Twit Crawling

25





tweet_link =




else

tweet_link =

hashtags =




else

hashtags =


Twit Crawling

26






else

tweet_published =


num_comments = 0












Twit Crawling

27

def main()


num_posts = 50

jsonResult = []



for tweet in tweets




outfilewrite(str_)


1NAVER CRAWLING

3



29





try






print(e)


return None



30



node = sjson sNode





return None

else




31


title = post[title]



link = post[link]

Tue 14 Feb 2017 184600 +0900





pDatepDate)

return

display 100

items [




pubDate Thu 13 Jul 2017 033800 +0900


]


start 1

total 454265



32

def main()

jsonResult = []


sNode = news


display_count = 100










ensure_ascii=False)



Naver 지도 API

33



node =





return None

else






35












return None

else




36














37

nItem = 100

nStartYear = 2011

nEndYear = 2017




nPagenum = 1

while True




if nTotal == 0

break





break

nPagenum += 1

else

break



38










return None

else




39

def main()

코드생략












코드생략













gtgtgt soup


gtgtgt tag = souptd

gtgtgt tag



gtgtgt tag


gtgtgt tag = soupa

gtgtgt tag


gtgtgt tagname

a






gtgtgt soup


gtgtgt tag = souptd

gtgtgt tag[class]

[title]


gtgtgt tag[class]

[tit3]


class [tit3]






gtgtgt soup



gtgtgt tag



gtgtgt tag









ltDOCTYPE htmlgt

lthtmlgt

ltheadgt






(이하중략)

lt-- Footer --gt

ltdivgt

ltbodygt

lthtmlgt



lttd class=titlegt

ltdiv class=tit3gt


ltdivgt

lttdgt




gtgtgt tags

[ltdiv class=tit3gt








ltdivgt]


print(taga)




print(tagatext)


덩케르크



Twit Crawling

25





tweet_link =




else

tweet_link =

hashtags =




else

hashtags =


Twit Crawling

26






else

tweet_published =


num_comments = 0












Twit Crawling

27

def main()


num_posts = 50

jsonResult = []



for tweet in tweets




outfilewrite(str_)


1NAVER CRAWLING

3



29





try






print(e)


return None



30



node = sjson sNode





return None

else




31


title = post[title]



link = post[link]

Tue 14 Feb 2017 184600 +0900





pDatepDate)

return

display 100

items [




pubDate Thu 13 Jul 2017 033800 +0900


]


start 1

total 454265



32

def main()

jsonResult = []


sNode = news


display_count = 100










ensure_ascii=False)



Naver 지도 API

33



node =





return None

else






35












return None

else




36














37

nItem = 100

nStartYear = 2011

nEndYear = 2017




nPagenum = 1

while True




if nTotal == 0

break





break

nPagenum += 1

else

break



38










return None

else




39

def main()

코드생략












코드생략













gtgtgt soup


gtgtgt tag = souptd

gtgtgt tag



gtgtgt tag


gtgtgt tag = soupa

gtgtgt tag


gtgtgt tagname

a






gtgtgt soup


gtgtgt tag = souptd

gtgtgt tag[class]

[title]


gtgtgt tag[class]

[tit3]


class [tit3]






gtgtgt soup



gtgtgt tag



gtgtgt tag









ltDOCTYPE htmlgt

lthtmlgt

ltheadgt






(이하중략)

lt-- Footer --gt

ltdivgt

ltbodygt

lthtmlgt



lttd class=titlegt

ltdiv class=tit3gt


ltdivgt

lttdgt




gtgtgt tags

[ltdiv class=tit3gt








ltdivgt]


print(taga)




print(tagatext)


덩케르크



Twit Crawling

26






else

tweet_published =


num_comments = 0












Twit Crawling

27

def main()


num_posts = 50

jsonResult = []



for tweet in tweets




outfilewrite(str_)


1NAVER CRAWLING

3



29





try






print(e)


return None



30



node = sjson sNode





return None

else




31


title = post[title]



link = post[link]

Tue 14 Feb 2017 184600 +0900





pDatepDate)

return

display 100

items [




pubDate Thu 13 Jul 2017 033800 +0900


]


start 1

total 454265



32

def main()

jsonResult = []


sNode = news


display_count = 100










ensure_ascii=False)



Naver 지도 API

33



node =





return None

else






35












return None

else




36














37

nItem = 100

nStartYear = 2011

nEndYear = 2017




nPagenum = 1

while True




if nTotal == 0

break





break

nPagenum += 1

else

break



38










return None

else




39

def main()

코드생략












코드생략













gtgtgt soup


gtgtgt tag = souptd

gtgtgt tag



gtgtgt tag


gtgtgt tag = soupa

gtgtgt tag


gtgtgt tagname

a






gtgtgt soup


gtgtgt tag = souptd

gtgtgt tag[class]

[title]


gtgtgt tag[class]

[tit3]


class [tit3]






gtgtgt soup



gtgtgt tag



gtgtgt tag









ltDOCTYPE htmlgt

lthtmlgt

ltheadgt






(이하중략)

lt-- Footer --gt

ltdivgt

ltbodygt

lthtmlgt



lttd class=titlegt

ltdiv class=tit3gt


ltdivgt

lttdgt




gtgtgt tags

[ltdiv class=tit3gt








ltdivgt]


print(taga)




print(tagatext)


덩케르크



Twit Crawling

27

def main()


num_posts = 50

jsonResult = []



for tweet in tweets




outfilewrite(str_)


1NAVER CRAWLING

3



29





try






print(e)


return None



30



node = sjson sNode





return None

else




31


title = post[title]



link = post[link]

Tue 14 Feb 2017 184600 +0900





pDatepDate)

return

display 100

items [




pubDate Thu 13 Jul 2017 033800 +0900


]


start 1

total 454265



32

def main()

jsonResult = []


sNode = news


display_count = 100










ensure_ascii=False)



Naver 지도 API

33



node =





return None

else






35












return None

else




36














37

nItem = 100

nStartYear = 2011

nEndYear = 2017




nPagenum = 1

while True




if nTotal == 0

break





break

nPagenum += 1

else

break



38










return None

else




39

def main()

코드생략












코드생략













gtgtgt soup


gtgtgt tag = souptd

gtgtgt tag



gtgtgt tag


gtgtgt tag = soupa

gtgtgt tag


gtgtgt tagname

a






gtgtgt soup


gtgtgt tag = souptd

gtgtgt tag[class]

[title]


gtgtgt tag[class]

[tit3]


class [tit3]






gtgtgt soup



gtgtgt tag



gtgtgt tag









ltDOCTYPE htmlgt

lthtmlgt

ltheadgt






(이하중략)

lt-- Footer --gt

ltdivgt

ltbodygt

lthtmlgt



lttd class=titlegt

ltdiv class=tit3gt


ltdivgt

lttdgt




gtgtgt tags

[ltdiv class=tit3gt








ltdivgt]


print(taga)




print(tagatext)


덩케르크



1NAVER CRAWLING

3



29





try






print(e)


return None



30



node = sjson sNode





return None

else




31


title = post[title]



link = post[link]

Tue 14 Feb 2017 184600 +0900





pDatepDate)

return

display 100

items [




pubDate Thu 13 Jul 2017 033800 +0900


]


start 1

total 454265



32

def main()

jsonResult = []


sNode = news


display_count = 100










ensure_ascii=False)



Naver 지도 API

33



node =





return None

else






35












return None

else




36














37

nItem = 100

nStartYear = 2011

nEndYear = 2017




nPagenum = 1

while True




if nTotal == 0

break





break

nPagenum += 1

else

break



38










return None

else




39

def main()

코드생략












코드생략













gtgtgt soup


gtgtgt tag = souptd

gtgtgt tag



gtgtgt tag


gtgtgt tag = soupa

gtgtgt tag


gtgtgt tagname

a






gtgtgt soup


gtgtgt tag = souptd

gtgtgt tag[class]

[title]


gtgtgt tag[class]

[tit3]


class [tit3]






gtgtgt soup



gtgtgt tag



gtgtgt tag









ltDOCTYPE htmlgt

lthtmlgt

ltheadgt






(이하중략)

lt-- Footer --gt

ltdivgt

ltbodygt

lthtmlgt



lttd class=titlegt

ltdiv class=tit3gt


ltdivgt

lttdgt




gtgtgt tags

[ltdiv class=tit3gt








ltdivgt]


print(taga)




print(tagatext)


덩케르크




29





try






print(e)


return None



30



node = sjson sNode





return None

else




31


title = post[title]



link = post[link]

Tue 14 Feb 2017 184600 +0900





pDatepDate)

return

display 100

items [




pubDate Thu 13 Jul 2017 033800 +0900


]


start 1

total 454265



32

def main()

jsonResult = []


sNode = news


display_count = 100










ensure_ascii=False)



Naver 지도 API

33



node =





return None

else






35












return None

else




36














37

nItem = 100

nStartYear = 2011

nEndYear = 2017




nPagenum = 1

while True




if nTotal == 0

break





break

nPagenum += 1

else

break



38










return None

else




39

def main()

코드생략












코드생략













gtgtgt soup


gtgtgt tag = souptd

gtgtgt tag



gtgtgt tag


gtgtgt tag = soupa

gtgtgt tag


gtgtgt tagname

a






gtgtgt soup


gtgtgt tag = souptd

gtgtgt tag[class]

[title]


gtgtgt tag[class]

[tit3]


class [tit3]






gtgtgt soup



gtgtgt tag



gtgtgt tag









ltDOCTYPE htmlgt

lthtmlgt

ltheadgt






(이하중략)

lt-- Footer --gt

ltdivgt

ltbodygt

lthtmlgt



lttd class=titlegt

ltdiv class=tit3gt


ltdivgt

lttdgt




gtgtgt tags

[ltdiv class=tit3gt








ltdivgt]


print(taga)




print(tagatext)


덩케르크




30



node = sjson sNode





return None

else




31


title = post[title]



link = post[link]

Tue 14 Feb 2017 184600 +0900





pDatepDate)

return

display 100

items [




pubDate Thu 13 Jul 2017 033800 +0900


]


start 1

total 454265



32

def main()

jsonResult = []


sNode = news


display_count = 100










ensure_ascii=False)



Naver 지도 API

33



node =





return None

else






35












return None

else




36














37

nItem = 100

nStartYear = 2011

nEndYear = 2017




nPagenum = 1

while True




if nTotal == 0

break





break

nPagenum += 1

else

break



38










return None

else




39

def main()

코드생략












코드생략













gtgtgt soup


gtgtgt tag = souptd

gtgtgt tag



gtgtgt tag


gtgtgt tag = soupa

gtgtgt tag


gtgtgt tagname

a






gtgtgt soup


gtgtgt tag = souptd

gtgtgt tag[class]

[title]


gtgtgt tag[class]

[tit3]


class [tit3]






gtgtgt soup



gtgtgt tag



gtgtgt tag









ltDOCTYPE htmlgt

lthtmlgt

ltheadgt






(이하중략)

lt-- Footer --gt

ltdivgt

ltbodygt

lthtmlgt



lttd class=titlegt

ltdiv class=tit3gt


ltdivgt

lttdgt




gtgtgt tags

[ltdiv class=tit3gt








ltdivgt]


print(taga)




print(tagatext)


덩케르크




31


title = post[title]



link = post[link]

Tue 14 Feb 2017 184600 +0900





pDatepDate)

return

display 100

items [




pubDate Thu 13 Jul 2017 033800 +0900


]


start 1

total 454265



32

def main()

jsonResult = []


sNode = news


display_count = 100










ensure_ascii=False)



Naver 지도 API

33



node =





return None

else






35












return None

else




36














37

nItem = 100

nStartYear = 2011

nEndYear = 2017




nPagenum = 1

while True




if nTotal == 0

break





break

nPagenum += 1

else

break



38










return None

else




39

def main()

코드생략












코드생략













gtgtgt soup


gtgtgt tag = souptd

gtgtgt tag



gtgtgt tag


gtgtgt tag = soupa

gtgtgt tag


gtgtgt tagname

a






gtgtgt soup


gtgtgt tag = souptd

gtgtgt tag[class]

[title]


gtgtgt tag[class]

[tit3]


class [tit3]






gtgtgt soup



gtgtgt tag



gtgtgt tag









ltDOCTYPE htmlgt

lthtmlgt

ltheadgt






(이하중략)

lt-- Footer --gt

ltdivgt

ltbodygt

lthtmlgt



lttd class=titlegt

ltdiv class=tit3gt


ltdivgt

lttdgt




gtgtgt tags

[ltdiv class=tit3gt








ltdivgt]


print(taga)




print(tagatext)


덩케르크




32

def main()

jsonResult = []


sNode = news


display_count = 100










ensure_ascii=False)



Naver 지도 API

33



node =





return None

else






35












return None

else




36














37

nItem = 100

nStartYear = 2011

nEndYear = 2017




nPagenum = 1

while True




if nTotal == 0

break





break

nPagenum += 1

else

break



38










return None

else




39

def main()

코드생략












코드생략













gtgtgt soup


gtgtgt tag = souptd

gtgtgt tag



gtgtgt tag


gtgtgt tag = soupa

gtgtgt tag


gtgtgt tagname

a






gtgtgt soup


gtgtgt tag = souptd

gtgtgt tag[class]

[title]


gtgtgt tag[class]

[tit3]


class [tit3]






gtgtgt soup



gtgtgt tag



gtgtgt tag









ltDOCTYPE htmlgt

lthtmlgt

ltheadgt






(이하중략)

lt-- Footer --gt

ltdivgt

ltbodygt

lthtmlgt



lttd class=titlegt

ltdiv class=tit3gt


ltdivgt

lttdgt




gtgtgt tags

[ltdiv class=tit3gt








ltdivgt]


print(taga)




print(tagatext)


덩케르크



Naver 지도 API

33



node =





return None

else






35












return None

else




36














37

nItem = 100

nStartYear = 2011

nEndYear = 2017




nPagenum = 1

while True




if nTotal == 0

break





break

nPagenum += 1

else

break



38










return None

else




39

def main()

코드생략












코드생략













gtgtgt soup


gtgtgt tag = souptd

gtgtgt tag



gtgtgt tag


gtgtgt tag = soupa

gtgtgt tag


gtgtgt tagname

a






gtgtgt soup


gtgtgt tag = souptd

gtgtgt tag[class]

[title]


gtgtgt tag[class]

[tit3]


class [tit3]






gtgtgt soup



gtgtgt tag



gtgtgt tag









ltDOCTYPE htmlgt

lthtmlgt

ltheadgt






(이하중략)

lt-- Footer --gt

ltdivgt

ltbodygt

lthtmlgt



lttd class=titlegt

ltdiv class=tit3gt


ltdivgt

lttdgt




gtgtgt tags

[ltdiv class=tit3gt








ltdivgt]


print(taga)




print(tagatext)


덩케르크






35












return None

else




36














37

nItem = 100

nStartYear = 2011

nEndYear = 2017




nPagenum = 1

while True




if nTotal == 0

break





break

nPagenum += 1

else

break



38










return None

else




39

def main()

코드생략












코드생략













gtgtgt soup


gtgtgt tag = souptd

gtgtgt tag



gtgtgt tag


gtgtgt tag = soupa

gtgtgt tag


gtgtgt tagname

a






gtgtgt soup


gtgtgt tag = souptd

gtgtgt tag[class]

[title]


gtgtgt tag[class]

[tit3]


class [tit3]






gtgtgt soup



gtgtgt tag



gtgtgt tag









ltDOCTYPE htmlgt

lthtmlgt

ltheadgt






(이하중략)

lt-- Footer --gt

ltdivgt

ltbodygt

lthtmlgt



lttd class=titlegt

ltdiv class=tit3gt


ltdivgt

lttdgt




gtgtgt tags

[ltdiv class=tit3gt








ltdivgt]


print(taga)




print(tagatext)


덩케르크




36














37

nItem = 100

nStartYear = 2011

nEndYear = 2017




nPagenum = 1

while True




if nTotal == 0

break





break

nPagenum += 1

else

break



38










return None

else




39

def main()

코드생략












코드생략













gtgtgt soup


gtgtgt tag = souptd

gtgtgt tag



gtgtgt tag


gtgtgt tag = soupa

gtgtgt tag


gtgtgt tagname

a






gtgtgt soup


gtgtgt tag = souptd

gtgtgt tag[class]

[title]


gtgtgt tag[class]

[tit3]


class [tit3]






gtgtgt soup



gtgtgt tag



gtgtgt tag









ltDOCTYPE htmlgt

lthtmlgt

ltheadgt






(이하중략)

lt-- Footer --gt

ltdivgt

ltbodygt

lthtmlgt



lttd class=titlegt

ltdiv class=tit3gt


ltdivgt

lttdgt




gtgtgt tags

[ltdiv class=tit3gt








ltdivgt]


print(taga)




print(tagatext)


덩케르크




37

nItem = 100

nStartYear = 2011

nEndYear = 2017




nPagenum = 1

while True




if nTotal == 0

break





break

nPagenum += 1

else

break



38










return None

else




39

def main()

코드생략












코드생략













gtgtgt soup


gtgtgt tag = souptd

gtgtgt tag



gtgtgt tag


gtgtgt tag = soupa

gtgtgt tag


gtgtgt tagname

a






gtgtgt soup


gtgtgt tag = souptd

gtgtgt tag[class]

[title]


gtgtgt tag[class]

[tit3]


class [tit3]






gtgtgt soup



gtgtgt tag



gtgtgt tag









ltDOCTYPE htmlgt

lthtmlgt

ltheadgt






(이하중략)

lt-- Footer --gt

ltdivgt

ltbodygt

lthtmlgt



lttd class=titlegt

ltdiv class=tit3gt


ltdivgt

lttdgt




gtgtgt tags

[ltdiv class=tit3gt








ltdivgt]


print(taga)




print(tagatext)


덩케르크




38










return None

else




39

def main()

코드생략












코드생략













gtgtgt soup


gtgtgt tag = souptd

gtgtgt tag



gtgtgt tag


gtgtgt tag = soupa

gtgtgt tag


gtgtgt tagname

a






gtgtgt soup


gtgtgt tag = souptd

gtgtgt tag[class]

[title]


gtgtgt tag[class]

[tit3]


class [tit3]






gtgtgt soup



gtgtgt tag



gtgtgt tag









ltDOCTYPE htmlgt

lthtmlgt

ltheadgt






(이하중략)

lt-- Footer --gt

ltdivgt

ltbodygt

lthtmlgt



lttd class=titlegt

ltdiv class=tit3gt


ltdivgt

lttdgt




gtgtgt tags

[ltdiv class=tit3gt








ltdivgt]


print(taga)




print(tagatext)


덩케르크




39

def main()

코드생략












코드생략













gtgtgt soup


gtgtgt tag = souptd

gtgtgt tag



gtgtgt tag


gtgtgt tag = soupa

gtgtgt tag


gtgtgt tagname

a






gtgtgt soup


gtgtgt tag = souptd

gtgtgt tag[class]

[title]


gtgtgt tag[class]

[tit3]


class [tit3]






gtgtgt soup



gtgtgt tag



gtgtgt tag









ltDOCTYPE htmlgt

lthtmlgt

ltheadgt






(이하중략)

lt-- Footer --gt

ltdivgt

ltbodygt

lthtmlgt



lttd class=titlegt

ltdiv class=tit3gt


ltdivgt

lttdgt




gtgtgt tags

[ltdiv class=tit3gt








ltdivgt]


print(taga)




print(tagatext)


덩케르크














gtgtgt soup


gtgtgt tag = souptd

gtgtgt tag



gtgtgt tag


gtgtgt tag = soupa

gtgtgt tag


gtgtgt tagname

a






gtgtgt soup


gtgtgt tag = souptd

gtgtgt tag[class]

[title]


gtgtgt tag[class]

[tit3]


class [tit3]






gtgtgt soup



gtgtgt tag



gtgtgt tag









ltDOCTYPE htmlgt

lthtmlgt

ltheadgt






(이하중략)

lt-- Footer --gt

ltdivgt

ltbodygt

lthtmlgt



lttd class=titlegt

ltdiv class=tit3gt


ltdivgt

lttdgt




gtgtgt tags

[ltdiv class=tit3gt








ltdivgt]


print(taga)




print(tagatext)


덩케르크







gtgtgt soup


gtgtgt tag = souptd

gtgtgt tag[class]

[title]


gtgtgt tag[class]

[tit3]


class [tit3]






gtgtgt soup



gtgtgt tag



gtgtgt tag









ltDOCTYPE htmlgt

lthtmlgt

ltheadgt






(이하중략)

lt-- Footer --gt

ltdivgt

ltbodygt

lthtmlgt



lttd class=titlegt

ltdiv class=tit3gt


ltdivgt

lttdgt




gtgtgt tags

[ltdiv class=tit3gt








ltdivgt]


print(taga)




print(tagatext)


덩케르크







gtgtgt soup



gtgtgt tag



gtgtgt tag









ltDOCTYPE htmlgt

lthtmlgt

ltheadgt






(이하중략)

lt-- Footer --gt

ltdivgt

ltbodygt

lthtmlgt



lttd class=titlegt

ltdiv class=tit3gt


ltdivgt

lttdgt




gtgtgt tags

[ltdiv class=tit3gt








ltdivgt]


print(taga)




print(tagatext)


덩케르크









ltDOCTYPE htmlgt

lthtmlgt

ltheadgt






(이하중략)

lt-- Footer --gt

ltdivgt

ltbodygt

lthtmlgt



lttd class=titlegt

ltdiv class=tit3gt


ltdivgt

lttdgt




gtgtgt tags

[ltdiv class=tit3gt








ltdivgt]


print(taga)




print(tagatext)


덩케르크




lttd class=titlegt

ltdiv class=tit3gt


ltdivgt

lttdgt




gtgtgt tags

[ltdiv class=tit3gt








ltdivgt]


print(taga)




print(tagatext)


덩케르크





gtgtgt tags

[ltdiv class=tit3gt








ltdivgt]


print(taga)




print(tagatext)


덩케르크


part 2. 데이터수집 - · pdf file목 차 04 facebook crawling 1 22 twitter crawling 2 28...

Documents