automating the mundane - craig maloneydecafbad.net/files/automating_the_mundane.pdfautomating the...

57
Automating The Mundane Craig Maloney February 9 th , 2019 Michigan!/usr/group

Upload: others

Post on 05-Jun-2020

3 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Automating The Mundane - Craig Maloneydecafbad.net/files/automating_the_mundane.pdfAutomating The Mundane Craig Maloney February 9th, 2019 Michigan!/usr/group. ... Prevent decision

AutomatingThe

MundaneCraig Maloney

February 9th, 2019Michigan!/usr/group

Page 2: Automating The Mundane - Craig Maloneydecafbad.net/files/automating_the_mundane.pdfAutomating The Mundane Craig Maloney February 9th, 2019 Michigan!/usr/group. ... Prevent decision

What does “Automating the Mundane” mean?

Page 3: Automating The Mundane - Craig Maloneydecafbad.net/files/automating_the_mundane.pdfAutomating The Mundane Craig Maloney February 9th, 2019 Michigan!/usr/group. ... Prevent decision

Automation

Page 4: Automating The Mundane - Craig Maloneydecafbad.net/files/automating_the_mundane.pdfAutomating The Mundane Craig Maloney February 9th, 2019 Michigan!/usr/group. ... Prevent decision

https://flic.kr/p/cp5WgS

Page 5: Automating The Mundane - Craig Maloneydecafbad.net/files/automating_the_mundane.pdfAutomating The Mundane Craig Maloney February 9th, 2019 Michigan!/usr/group. ... Prevent decision

By Rhododendrites - Own work, CC BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=51832081

Page 6: Automating The Mundane - Craig Maloneydecafbad.net/files/automating_the_mundane.pdfAutomating The Mundane Craig Maloney February 9th, 2019 Michigan!/usr/group. ... Prevent decision

Mundane

Page 7: Automating The Mundane - Craig Maloneydecafbad.net/files/automating_the_mundane.pdfAutomating The Mundane Craig Maloney February 9th, 2019 Michigan!/usr/group. ... Prevent decision

PLACEHOLDER(Image of your desk here)

Page 8: Automating The Mundane - Craig Maloneydecafbad.net/files/automating_the_mundane.pdfAutomating The Mundane Craig Maloney February 9th, 2019 Michigan!/usr/group. ... Prevent decision

By Franz Stuck - Stuck, Franz von: Sisyphus., Public Domain, https://commons.wikimedia.org/w/index.php?curid=55248

Page 9: Automating The Mundane - Craig Maloneydecafbad.net/files/automating_the_mundane.pdfAutomating The Mundane Craig Maloney February 9th, 2019 Michigan!/usr/group. ... Prevent decision

By Jean Le Tavernier - [1], Public Domain, https://commons.wikimedia.org/w/index.php?curid=74516

Page 10: Automating The Mundane - Craig Maloneydecafbad.net/files/automating_the_mundane.pdfAutomating The Mundane Craig Maloney February 9th, 2019 Michigan!/usr/group. ... Prevent decision

“Automation” + “Mundane” = ?

Page 11: Automating The Mundane - Craig Maloneydecafbad.net/files/automating_the_mundane.pdfAutomating The Mundane Craig Maloney February 9th, 2019 Michigan!/usr/group. ... Prevent decision

It’s not quite...

Page 12: Automating The Mundane - Craig Maloneydecafbad.net/files/automating_the_mundane.pdfAutomating The Mundane Craig Maloney February 9th, 2019 Michigan!/usr/group. ... Prevent decision

http://www.theoldrobots.com/hero.html

Page 13: Automating The Mundane - Craig Maloneydecafbad.net/files/automating_the_mundane.pdfAutomating The Mundane Craig Maloney February 9th, 2019 Michigan!/usr/group. ... Prevent decision

https://www.flickr.com/photos/ddebold/5900035541/in/photostream/

Page 14: Automating The Mundane - Craig Maloneydecafbad.net/files/automating_the_mundane.pdfAutomating The Mundane Craig Maloney February 9th, 2019 Michigan!/usr/group. ... Prevent decision

“Automation” + “Mundane” = ?

Page 15: Automating The Mundane - Craig Maloneydecafbad.net/files/automating_the_mundane.pdfAutomating The Mundane Craig Maloney February 9th, 2019 Michigan!/usr/group. ... Prevent decision

(What we’ll explicitly ignore...)

Page 16: Automating The Mundane - Craig Maloneydecafbad.net/files/automating_the_mundane.pdfAutomating The Mundane Craig Maloney February 9th, 2019 Michigan!/usr/group. ... Prevent decision

Check in

Check in

Check in

Trigger

Trigger

Trigger

Trigger

Trigger

Feedback

Feedback

Feedback

Feedback

Feedback

Feedback

Approval

Approval

Delivery team Version control Build & unit tests

Automated acceptance tests

User acceptance tests

Release

By Grégoire Détrez, original by Jez Humble - This file was derived from: Continuous Delivery process diagram.png, CC BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=43977816

Page 17: Automating The Mundane - Craig Maloneydecafbad.net/files/automating_the_mundane.pdfAutomating The Mundane Craig Maloney February 9th, 2019 Michigan!/usr/group. ... Prevent decision

Maybe this would be a better title...

Page 18: Automating The Mundane - Craig Maloneydecafbad.net/files/automating_the_mundane.pdfAutomating The Mundane Craig Maloney February 9th, 2019 Michigan!/usr/group. ... Prevent decision

Shell-scripting the mundane

Page 19: Automating The Mundane - Craig Maloneydecafbad.net/files/automating_the_mundane.pdfAutomating The Mundane Craig Maloney February 9th, 2019 Michigan!/usr/group. ... Prevent decision

Shell-scripting the mundane

Page 20: Automating The Mundane - Craig Maloneydecafbad.net/files/automating_the_mundane.pdfAutomating The Mundane Craig Maloney February 9th, 2019 Michigan!/usr/group. ... Prevent decision

Our focus...

Page 21: Automating The Mundane - Craig Maloneydecafbad.net/files/automating_the_mundane.pdfAutomating The Mundane Craig Maloney February 9th, 2019 Michigan!/usr/group. ... Prevent decision

Doing meaningful work;Offloading mundane and repetitive work.

Page 22: Automating The Mundane - Craig Maloneydecafbad.net/files/automating_the_mundane.pdfAutomating The Mundane Craig Maloney February 9th, 2019 Michigan!/usr/group. ... Prevent decision

https://www.flickr.com/photos/peterhess/2976755407

Page 23: Automating The Mundane - Craig Maloneydecafbad.net/files/automating_the_mundane.pdfAutomating The Mundane Craig Maloney February 9th, 2019 Michigan!/usr/group. ... Prevent decision

https://www.flickr.com/photos/arsidubu/500339878/

Page 24: Automating The Mundane - Craig Maloneydecafbad.net/files/automating_the_mundane.pdfAutomating The Mundane Craig Maloney February 9th, 2019 Michigan!/usr/group. ... Prevent decision

(Brace for over-simplified andscientific-sounding reason for not doing

repetitive or mundane work on the next slide)

Page 25: Automating The Mundane - Craig Maloneydecafbad.net/files/automating_the_mundane.pdfAutomating The Mundane Craig Maloney February 9th, 2019 Michigan!/usr/group. ... Prevent decision

Conserving Calories● Every decision you make counts

against your daily energy reserve.● Every calorie you save can be used

later for deeper thoughts and energy.

● Prevent decision fatigue, automate!

Page 26: Automating The Mundane - Craig Maloneydecafbad.net/files/automating_the_mundane.pdfAutomating The Mundane Craig Maloney February 9th, 2019 Michigan!/usr/group. ... Prevent decision

How do we decide what to automate?

Page 27: Automating The Mundane - Craig Maloneydecafbad.net/files/automating_the_mundane.pdfAutomating The Mundane Craig Maloney February 9th, 2019 Michigan!/usr/group. ... Prevent decision

Craig’s Law*

If this is the first time doing something, just do it.

If this is the second time doing this, automate it.

If there’s a checklist for it, consider automating it.

* Note: Not really a law. More like “Craig’s really neat rubric”

Page 28: Automating The Mundane - Craig Maloneydecafbad.net/files/automating_the_mundane.pdfAutomating The Mundane Craig Maloney February 9th, 2019 Michigan!/usr/group. ... Prevent decision

Deciding what to automate● Is it something that can be

reproduced with little or no human input?

● Is it something we do repeatedly?● Is it something we don’t enjoy

doing?

Page 29: Automating The Mundane - Craig Maloneydecafbad.net/files/automating_the_mundane.pdfAutomating The Mundane Craig Maloney February 9th, 2019 Michigan!/usr/group. ... Prevent decision

This task does not “spark joy”.

Page 30: Automating The Mundane - Craig Maloneydecafbad.net/files/automating_the_mundane.pdfAutomating The Mundane Craig Maloney February 9th, 2019 Michigan!/usr/group. ... Prevent decision

Process of automating● Clarify the verbs

– What does this task do?

● Clarify the nouns– Who is this task for?– What do we need to make this

happen?

Page 31: Automating The Mundane - Craig Maloneydecafbad.net/files/automating_the_mundane.pdfAutomating The Mundane Craig Maloney February 9th, 2019 Michigan!/usr/group. ... Prevent decision

Describe the task;Notice the verbs you use.

Page 32: Automating The Mundane - Craig Maloneydecafbad.net/files/automating_the_mundane.pdfAutomating The Mundane Craig Maloney February 9th, 2019 Michigan!/usr/group. ... Prevent decision

“I get the number from Bob via email and turn them into a report that gets sent via email to the

customer”.

Find the verbs

Page 33: Automating The Mundane - Craig Maloneydecafbad.net/files/automating_the_mundane.pdfAutomating The Mundane Craig Maloney February 9th, 2019 Michigan!/usr/group. ... Prevent decision

Verbs● Get (numbers)● Turn (numbers into report)● Sent (report)

Page 34: Automating The Mundane - Craig Maloneydecafbad.net/files/automating_the_mundane.pdfAutomating The Mundane Craig Maloney February 9th, 2019 Michigan!/usr/group. ... Prevent decision

Nouns● Bob (has the numbers)● Numbers (for the report)● Email (receive the numbers)● Report (is created)● Email (the message)● Customer (the final recipient)

Page 35: Automating The Mundane - Craig Maloneydecafbad.net/files/automating_the_mundane.pdfAutomating The Mundane Craig Maloney February 9th, 2019 Michigan!/usr/group. ... Prevent decision

Digging deeper● Where does Bob get the numbers from?● Is Bob the only source of these numbers?● Who is the customer?● When is this needed?● What’s the text of the email?● Is this still needed?● (etc.)

Page 36: Automating The Mundane - Craig Maloneydecafbad.net/files/automating_the_mundane.pdfAutomating The Mundane Craig Maloney February 9th, 2019 Michigan!/usr/group. ... Prevent decision

More answers● Bob gets the numbers from a database● We have access to the database● Report is in Excel● Email message text is copied from a

template● Customer email is [email protected]● Email is sent first of the month by 5pm

Page 37: Automating The Mundane - Craig Maloneydecafbad.net/files/automating_the_mundane.pdfAutomating The Mundane Craig Maloney February 9th, 2019 Michigan!/usr/group. ... Prevent decision

What this suggests● Python script to grab data directly

from the database● Output using OpenPyXL● Email using email library

Page 38: Automating The Mundane - Craig Maloneydecafbad.net/files/automating_the_mundane.pdfAutomating The Mundane Craig Maloney February 9th, 2019 Michigan!/usr/group. ... Prevent decision

Always dig deeper into the problem to see what happens throughout the lifecycle of the process...

Page 39: Automating The Mundane - Craig Maloneydecafbad.net/files/automating_the_mundane.pdfAutomating The Mundane Craig Maloney February 9th, 2019 Michigan!/usr/group. ... Prevent decision

Digging even deeper● Glenda converts data to their database

– Converts file into CSV from Excel– Glenda doesn’t care about formatting, only

consistency– Glenda has been complaining to Bob for

several months about keeping the data consistent

Page 40: Automating The Mundane - Craig Maloneydecafbad.net/files/automating_the_mundane.pdfAutomating The Mundane Craig Maloney February 9th, 2019 Michigan!/usr/group. ... Prevent decision

One possible result

#!/bin/sh

psql -c “COPY products_273 TO '/tmp/products_199.csv' WITH (FORMAT CSV, HEADER);”

mutt -a /tmp /products_199.csv -s “csv file for you” -- [email protected]

Page 41: Automating The Mundane - Craig Maloneydecafbad.net/files/automating_the_mundane.pdfAutomating The Mundane Craig Maloney February 9th, 2019 Michigan!/usr/group. ... Prevent decision

In our contrived example● Bob would take the data and copy it

by hand into Excel (with formatting).● Glenda didn’t care about formatting

because it went directly into a database.

● We replaced Bob’s complex task with a simple shell script.

Page 42: Automating The Mundane - Craig Maloneydecafbad.net/files/automating_the_mundane.pdfAutomating The Mundane Craig Maloney February 9th, 2019 Michigan!/usr/group. ... Prevent decision

What makes good candidates for automation?

Page 43: Automating The Mundane - Craig Maloneydecafbad.net/files/automating_the_mundane.pdfAutomating The Mundane Craig Maloney February 9th, 2019 Michigan!/usr/group. ... Prevent decision

Good automation● Consistent sources / destinations● Clearly defined processes & goals● No need for “human” decision-

making● Tasks that can be simplified to

default behaviors of existing tools.

Page 44: Automating The Mundane - Craig Maloneydecafbad.net/files/automating_the_mundane.pdfAutomating The Mundane Craig Maloney February 9th, 2019 Michigan!/usr/group. ... Prevent decision

Difficult automation● Inconsistent sources / destinations● Ill-defined processes & goals● “I’ll know it when I see it” or “I trust my

instincts” decisions● Tools with purpose-built or no interfaces for

import / export● Obfuscation

Page 45: Automating The Mundane - Craig Maloneydecafbad.net/files/automating_the_mundane.pdfAutomating The Mundane Craig Maloney February 9th, 2019 Michigan!/usr/group. ... Prevent decision

Types of automation● Simple script

– Low / no error checking– Minimal logging– Fragile (small changes will break the

script)

Page 46: Automating The Mundane - Craig Maloneydecafbad.net/files/automating_the_mundane.pdfAutomating The Mundane Craig Maloney February 9th, 2019 Michigan!/usr/group. ... Prevent decision

Types of automation● Complex script

– More error checking / correction– Sufficient logging– Robust

Page 47: Automating The Mundane - Craig Maloneydecafbad.net/files/automating_the_mundane.pdfAutomating The Mundane Craig Maloney February 9th, 2019 Michigan!/usr/group. ... Prevent decision

https://flic.kr/p/41aAWe

Page 48: Automating The Mundane - Craig Maloneydecafbad.net/files/automating_the_mundane.pdfAutomating The Mundane Craig Maloney February 9th, 2019 Michigan!/usr/group. ... Prevent decision

Other types of automation● Aliases

– Allows for complex commands to be executed using simple commands

● “One-liner” scripts– Common command-line switches that

aren’t immediately obvious, or are often run during the day.

Page 49: Automating The Mundane - Craig Maloneydecafbad.net/files/automating_the_mundane.pdfAutomating The Mundane Craig Maloney February 9th, 2019 Michigan!/usr/group. ... Prevent decision

Other types of automation● Cron Jobs

– Scheduled tasks with simple scheduling

– (We should have a talk about this)

Page 50: Automating The Mundane - Craig Maloneydecafbad.net/files/automating_the_mundane.pdfAutomating The Mundane Craig Maloney February 9th, 2019 Michigan!/usr/group. ... Prevent decision

Other types of automation● Makefiles

– Used for build processes– Can be used to bring together multiple

files into one file

Page 51: Automating The Mundane - Craig Maloneydecafbad.net/files/automating_the_mundane.pdfAutomating The Mundane Craig Maloney February 9th, 2019 Michigan!/usr/group. ... Prevent decision

Other types of automation● Editor plugins

– Can be used for templating code– Can check code for correctness /

formatting as it is being written

Page 52: Automating The Mundane - Craig Maloneydecafbad.net/files/automating_the_mundane.pdfAutomating The Mundane Craig Maloney February 9th, 2019 Michigan!/usr/group. ... Prevent decision
Page 53: Automating The Mundane - Craig Maloneydecafbad.net/files/automating_the_mundane.pdfAutomating The Mundane Craig Maloney February 9th, 2019 Michigan!/usr/group. ... Prevent decision

Scripts that I run / wrote● MUG Announcement script● Birthday manager● todo.txt recur script● todo.txt health check● Random background changer● Various backup scripts● Social Media posting (feed2toot, diaspora_post)● Reminders for Open Metalcast● Show-notes creator for Open Metalcast● Rss2email● ssh tunneling scripts● Script to sync my laptop with my home machine

Page 54: Automating The Mundane - Craig Maloneydecafbad.net/files/automating_the_mundane.pdfAutomating The Mundane Craig Maloney February 9th, 2019 Michigan!/usr/group. ... Prevent decision

Demo

Page 55: Automating The Mundane - Craig Maloneydecafbad.net/files/automating_the_mundane.pdfAutomating The Mundane Craig Maloney February 9th, 2019 Michigan!/usr/group. ... Prevent decision

Questions?

Page 56: Automating The Mundane - Craig Maloneydecafbad.net/files/automating_the_mundane.pdfAutomating The Mundane Craig Maloney February 9th, 2019 Michigan!/usr/group. ... Prevent decision

Links● https://gitlab.com/craigmaloney/birthday-manager

● https://gitlab.com/craigmaloney/vim-config

● https://github.com/MichiganUnixUserGroup/mug_ical

● https://gitlab.com/chaica/feed2toot

● https://debakel.github.io/feedDiasp/

● https://gitlab.com/open_metalcast

● https://pypi.org/project/rss2email/

● http://jrnl.sh/

Page 57: Automating The Mundane - Craig Maloneydecafbad.net/files/automating_the_mundane.pdfAutomating The Mundane Craig Maloney February 9th, 2019 Michigan!/usr/group. ... Prevent decision

AutomatingThe

MundaneCraig Maloney

February 9th, 2019Michigan!/usr/group