observational science with python and a webcam

Observational ScienceWith Python

And a Webcam

By: Eric Floehr

Observational Science With Python and a Webcam by Eric Floehr is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.

http://www.intellovations.com/ospw

http://creativecommons.org/licenses/by-sa/3.0/

So I had a webcam...

and I made a time-lapse video.

http://bit.ly/ospw-1


I wanted to do better.

Longer

Inside location

Automated

Once a minute

So this is what I did.

Second floor window

Little-used room

Pointing west

Pull pictures down with cronjob

That runs once per minute

And it looks like this.

Two years later...

I have...

● 896,309 image files● 11,809,972,880 bytes● 10.9989 gigabytes● From 2:14pm on August 29, 2010● To 4:11pm on July 25, 2012● Still going...

Tenet #1:

Don't be afraid of big data.

What can we do?

● Moar Time-lapse!● Explore phenomenon that occurs

over long periods● Unique visualizations● Have lots of fun!

First...

We need to organize the data.

Database!

How will we access the data?Let's use Django!

Why Django?

● It has a good ORM● It has a command framework● Makes extending to the web later

easy● I already am familiar with it● Django isn't just for web :-)

Quick setup

● Create your virtualenv (with site packages)● PIL is a pain to compile● Install Django● django-admin.py startproject● Edit settings.py● createdb pics● django-admin.py startapp pics● django-admin.py syncdb

Tenet #2:

Don't let small things keep you from your big picture.

What do we need to store?

● Image file location and filename● Time the image was taken● Interesting information about the

image● Is it a valid image?

class Picture(models.Model):

filepath = models.CharField(max_length=1024) filename = models.CharField(max_length=1024)

timestamp = models.DateTimeField(db_index=True) file_timestamp = models.DateTimeField(null=True) # Auto-populated, don't manually enter hour = models.SmallIntegerField(null=True, db_index=True) minute = models.SmallIntegerField(null=True, db_index=True) file_hour = models.SmallIntegerField(null=True) file_minute = models.SmallIntegerField(null=True) # End auto-populated fields

size = models.IntegerField(default=0) center_color = models.IntegerField(null=True) mean_color = models.IntegerField(null=True) median_color = models.IntegerField(null=True)

stddev_red = models.IntegerField(null=True) stddev_green = models.IntegerField(null=True) stddev_blue = models.IntegerField(null=True)

min_color = models.IntegerField(null=True) max_color = models.IntegerField(null=True) valid = models.BooleanField(default=False)

class Meta: ordering = ['timestamp']








<-- File location and name








<-- Image Timestamp








<-- Interesting image data








<-- Is it valid? How do we order?

Loading the Data

How do we populate the table?

● Iterate through directories of image files

● Parse the file name to get timestamp

● Get file timestamp to compare● Load image to collect information

about the image

Find the image filesimport os

import fnmatch

import datetime

for dirpath, dirs, files in os.walk(directory):

for f in sorted(fnmatch.filter(files, 'image-*.jpg')):

# Pull out timestamp

timestamp = f.split('.')[0].split('-')[1]

date = datetime.datetime.fromtimestamp(int(timestamp), utc)

Use PIL to get image infoimport Image, ImageStat

From util.color import rgb_to_int

try:

im = Image.open(pic.filepath)

stats = ImageStat.Stat(im)

pic.center_color = rgb_to_int(im.getpixel((320,240)))

pic.mean_color = rgb_to_int(stats.mean)

pic.median_color = rgb_to_int(stats.median)

if im.size <> (640,480):

pic.valid = False

except:

pic.valid = False

It took a an hour or two using six threads on my desktop

Tenet #3:

You have a 1990's supercomputer on your lap or

under your desk.Use it!

Let's Explore the Data

Size Indicates Complexity in jpegpics=# select timestamp, filepath, size from pics_picture where size=(select max(size) from pics_picture);

timestamp | filepath | size ------------------------+--------------------------------------+------- 2011-10-10 18:26:01-04 | /data/pics/1318/image-1318285561.jpg | 64016



High Stddev Means Color Variationpics=# select timestamp, filepath, size from pics_picture where stddev_red+stddev_green+stddev_blue = (select max(stddev_red+stddev_green+stddev_blue) from pics_picture);

Timestamp | filepath | size ------------------------+--------------------------------------+------- 2011-09-29 17:20:01-04 | /data/pics/1317/image-1317331201.jpg | 22512



About the Datasetpics=# select min(timestamp) as start, max(timestamp) as end from pics_picture;

start | end

------------------------+------------------------

2010-08-29 14:14:01-04 | 2012-07-25 16:11:01-04

(1 row)

pics=# select count(*), sum(case when valid=false then 1 else 0 end) as invalid from pics_picture;

count | invalid

--------+---------

896309 | 29555

(1 row)

pics=# select timestamp, filepath, size from pics_picture where size>0 and valid=false order by size desc limit 1;

timestamp | filepath | size

------------------------+--------------------------------------+-------

2012-04-25 08:16:01-04 | /data/pics/1335/image-1335356161.jpg | 39287

(1 row)



Yuck! This Data Sucks!

● 29,555 invalid image files● That's 3.3% of all image files● Worse yet, there isn't a file every minute● Based on start and end times, we should

have 1,002,358 images● We are missing 10.6% of all minutes

between start and end times

It's Worse...I Forgot The Bolts!







* Interestingly, I was listening to the Serious Business remix of “All Is Not Lost” by OK Go from the Humble Music Bundle as I made the previous slide.

Tenet #4:

Real world data is messy.That's ok. Just work around it.

How we can work around it?

● Create table of all minutes● Accept images “near” missing

minutes● Use empty images when

acceptable images can't be found

Let's Make Time-lapse Movies

How do we make movies?

● Collect a set of images in frame order.

● Send that list of images to a movie maker.

● Wait for movie to be made.● Watch movie!

from movies import ImageSequencefrom pics.models import NormalizedPictureimport datetimefrom pytz import timezone

ims = ImageSequence()

ettz = timezone('US/Eastern')

start = datetime.datetime(2011,4,7).replace(tzinfo=ettz)end = start + datetime.timedelta(days=1)

pics = NormalizedPicture.objects.filter(timestamp__gte=start, timestamp__lt=end)


for pic in pics: if pic.picture is not None: ims.add_picture(pic.picture) else: ims.add_image('/tmp/no_image.png')

ims.make_timelapse('/tmp/{0}'.format(start.strftime("%Y%m%d")), fps=24)

The previous bullet points in code









Collect a set of Images









Send Images to Movie Maker









Wait For Movie To Be Made

class ImageSequence(object): def __init__(self): self.images = []

def add_picture(self, picture): self.images.append(picture.filepath)

def write_to_file(self, fileobj): for image in self.images: fileobj.write(image)

def make_timelapse(self, name, fps=25): tmpfile = tempfile.NamedTemporaryFile() self.write_to_file(tmpfile)

bitrate = int(round(60 * fps * 640 * 480 / 256.0))

execall = [] execall.append(mencoder_exe) execall.extend(mencoder_options) execall.extend(["-lavcopts", "vcodec=mpeg4:vbitrate={0}:mbd=2:keyint=132:v4mv:vqmin=3:lumi_mask=0.07:dark_mask=0.2:mpeg_quant:scplx_mask=0.1:tcplx_mask=0.1:naq".format(bitrate)]) execall.append("mf://@{0}".format(tmpfile.name)) execall.extend(["-mf", "type=jpeg:fps={0}".format(fps)]) execall.extend(["-o", "{name}.mp4".format(name=name)])

return subprocess.call(execall)

Wait For Movie To Be Made

Watch Movie!


http://bit.ly/ospw-8-raw



We aren't limited to consecutive minutes...

from movies import ImageSequencefrom pics.models import NormalizedPicture


Hour = 22 # UTC Timeminute = 0

last_pic = None

pics = NormalizedPicture.objects.filter(hour=hour, minute=minute)

last_pic = None

for pic in pics: if pic.picture is not None: ims.add_picture(pic.picture) last_pic = pic.picture else: # Use yesterdays' image if last_pic is not None: ims.add_picture(last_pic) else: # Give up ims.add_image('/tmp/no_image.jpg')

ims.make_timelapse('/tmp/time_{0:02d}{1:02d}'.format(hour,minute),fps=12)

Movie of a Specific Time

from movies import ImageSequencefrom pics.models import NormalizedPicture


hour = 22 # UTC timeminute = 0

last_pic = None

pics = NormalizedPicture.objects.filter(hour=hour, minute=minute)

last_pic = None

for pic in pics: if pic.picture is not None: ims.add_picture(pic.picture) last_pic = pic.picture else: # Use yesterdays' image if last_pic is not None: ims.add_picture(last_pic) else: # Give up ims.add_image('/tmp/no_image.jpg')

ims.make_timelapse('/tmp/time_{0:02d}{1:02d}'.format(hour,minute),fps=12)

Movie of a Specific Time

Watch Movie!





Now what if we want the Sun the same height above the

horizon, so we can better see the seasonal progression of

the Sun?

We can anchor on sunset. At a given time before sunset, the Sun will be at relatively the

same height above the horizon.

Where I thank Brandon Craig Rhodes for pyephem

import sky

obs = sky.webcam()

while current < end: # Get today's sunset time, but 70 minutes before pic_time = sky.sunset(obs, current) - datetime.timedelta(minutes=70) pic_time = pic_time.replace(second=0, microsecond=0)

pic = NormalizedPicture.objects.get(timestamp=pic_time)

if pic.picture is not None: ims.add_picture(pic.picture) else: piclist = NormalizedPicture.objects.get( timestamp__gte=pic_time - datetime.timedelta(minutes=20), timestamp__lte=pic_time + datetime.timedelta(minutes=20), picture__id__isnull=False)

if len(piclist) > 0: ims.add_picture(piclist[0].picture) else: # Use yesterdays' image if last_pic is not None: ims.add_picture(last_pic) else: # Give up ims.add_image('/tmp/no_image.jpg') current = current + datetime.timedelta(days=1)

Movie at Specific Time Before Sunset

import ephemfrom pytz import timezone, utc

sun = ephem.Sun()moon = ephem.Moon()est = timezone('EST')

def observer(): location = ephem.Observer() location.lon = '-83:23:24' location.lat = '40:13:48' location.elevation = 302 return location

def sunrise(observer, date): observer.date = date.astimezone(utc) return observer.next_rising(sun).datetime().replace(tzinfo=utc).astimezone(est)

def sunset(observer, date): observer.date = date.astimezone(utc) return observer.next_setting(sun).datetime().replace(tzinfo=utc).astimezone(est)

def moonset(observer, date): observer.date = date.astimezone(utc) return observer.next_setting(moon).datetime().replace(tzinfo=utc).astimezone(est)

Our sky module that uses pyephem

Watch Movie!





August 29, 2010 6:59pm EDT


October 22, 20105:33pm EDT




Ancient Astronomical Alignmentshttp://bit.ly/ospw-14

Photo credits:http://en.wikipedia.org/wiki/File:ChichenItzaEquinox.jpghttp://www.colorado.edu/Conferences/chaco/tour/images/dagger.jpg


http://en.wikipedia.org/wiki/File:ChichenItzaEquinox.jpg

http://www.colorado.edu/Conferences/chaco/tour/images/dagger.jpg

Tenet #5:

Once you have a foundation (data or code), however messy,

you can build higher.

So let's build higher!

We Also Take Pictures At Night

● Could I have captured a UFO in an image?

● Or a fireball?● Some other phenomenon?● What track does the moon take

through the sky?● How about Venus or Jupiter?

Moon Tracks and UFOs

● We want to find interesting things● How do we easily identify “interesting things”● At night, bright things are interesting things● Brightness is a good proxy for “interesting”● It makes it easy to identify interesting things● As it is easier to spot black spots on white images,

we'll invert the images

Making Night Tracks

● Process each image● Stack images into a single “all

night” image● From one hour after sunset to one

hour before sunrise the next day● Black splotches are interesting

things

def make_nighttrack(rootdir, day): obs = sky.observer()

# One hour after sunset start_time = sky.sunset(obs, day) + datetime.timedelta(hours=1) # Until one hour before sunrise the next day end_time = sky.sunrise(obs, tomorrow) - datetime.timedelta(hours=1)

pics = NormalizedPicture.objects.filter( timestamp__gte=start_time, timestamp__lt=end_time, picture__id__isnull=False)

print "Drawing image with {0} images".format(len(pics)) im = Image.new("L", (640,480), background_color)

for picture in pics: source = Image.open(picture.picture.filepath) # Get the negative source_gray = ImageOps.grayscale(source) source_neg = ImageOps.invert(source_gray)

# Threshold white source_thresh = Image.eval(source_neg, lambda x: 255*(x>224)) # Merge in the new image im = ImageChops.multiply(im, source_thresh)

# Put a date on the image canvas = ImageDraw.Draw(im) canvas.text((10,460), day.strftime("%Y-%m-%d"))

# And save im.save('{root}/{date}_mt.png'.format(root=rootdir,date=day.strftime("%Y%m%d")))

Venus, Jupiter, and the Moon



Lighting Up the Sky



Lightning Bug? Aircraft? UFO!

http://bit.ly/ospw-18http://bit.ly/ospw-19http://bit.ly/ospw-20

Moon

Venus

?




Shows up 3 weeks later

http://bit.ly/ospw-21http://bit.ly/ospw-22http://bit.ly/ospw-23




Last Oddity: 9/2/2011



Last Oddity: 9/2/2011

Single-Frame Oddity

http://bit.ly/ospw-30http://bit.ly/ospw-31

Crescent Moon That Night

http://bit.ly/ospw-32http://bit.ly/ospw-33





Let's make some science art!

Daystrips

● So far we've been using whole images...let's change that

● Instead of using a whole image, let's just use one line

● Kind of like a scanner● Start at midnight, stacking lines

Daystrip Scanner


12:06pm

11:40pm


Daystrips

● There are 1440 minutes in a day● Images will be 1440 pixels high● We place image line at the current

minute in the day● HOUR * 60 + MINUTE

def make_daystrip(rootdir, day): end = day + datetime.timedelta(days=1)

print "Getting data for {0}".format(day) pics = NormalizedPicture.objects.filter( timestamp__gte=day, timestamp__lt=end, picture__id__isnull=False)

im = Image.new('RGB', (640,1440), background_color) for picture in pics: source = Image.open(picture.picture.filepath) # Get row 1/3 of the way down, painting on proper row of canvas timestamp = picture.timestamp.astimezone(esttz)

y = timestamp.hour * 60 + timestamp.minute

im.paste(source.crop((0,160,640,161)),(0,y))

im.save('{root}/{date}_daystrip.png'.format(root=rootdir,date=day.strftime("%Y%m%d")))

This is beginning to look familiar







Create Our New Image







Iterate Though Our Images







And Paste The Single Row







Finally, save the image

Daystrip Example – March 17, 2011


Midnight

Moon Crossing

Sunrise

More cloudy (gray)More cloudy (gray)

Less cloudy (blue)

Sun Crossing

Midnight

Sunset


Shortest and Longest Day

Dec 22, 2011


June 20, 2012




Tenet #6:Don't be afraid to be creative.

Science can be beautiful.

Everything we've made so far spans a day or less.

What we've done so far

● Viewed full images of interest● Combined full images in movies● Stacked full images to find UFOs● Took a full line from a day's worth

of images● Everything is a day or less of data

Let's use ALL the days!

What if...

Instead of an image row being a single minute...

it was a whole day...

And each pixel in the row was a minute in the day.

Therefore, each of our 896,309 webcam images would

comprise a single pixel in our über-image.

A Visual Representation

Minutes in a day (1440)

Day

s (e

ach

row

is o

ne d

ay)

Image

Mid

nigh

t

Mid

nigh

t

Noo

n

Start Day

End Day

Each pixel is one minute in the day

pics = NormalizedPicture.objects.all()

im = Image.new('RGB', (1440,num_days), background_color)Canvas = ImageDraw.Draw(im)

for picture in pics: # Get XY timestamp = picture.timestamp.astimezone(est)

y_timedelta = timestamp - start y = y_timedelta.days x = timestamp.hour * 60 + timestamp.minute

# Paint pixel if picture.picture is not None: color = int_to_rgb(picture.picture.median_color) else: color = background_color

canvas.point((x,y), fill=color)

# All done, saveim.save(filepath)

The Result



What Exactly does DST Do?



Basically It Normalizes Sunrise TimeBy Making Summer Sunrise Later



At The Expense Of Wider Sunset Time Variation, Because Of Later Summer Sunset



Python makes it easy to answer “What If?”

So...

Tenet #7:Don't be afraid to ask

“What if?”, and don't be afraid of where it takes you.

Presentation:http://bit.ly/ospw-talk-pdf

http://bit.ly/ospw-talk (raw)

Code:http://bit.ly/ospw-code

Picture Set (9.1GB):http://bit.ly/ospw-pics

Links and more (soon):http://intellovations.com/ospw

http://bit.ly/ospw-talk-pdf

http://bit.ly/ospw-talk

http://bit.ly/ospw-code

http://bit.ly/ospw-pics

http://intellovations.com/ospw

observational science with python and a webcam

Technology

true file

true minute

true stddev

true mean

true median

true max

filename time

valid image