automate yo'self -- seagl

Automate Yo' Self!SeaGL 2016 Seattle, yo

John SJ Anderson @genehack

if anybody has any questions or i'm going too fast, please throw up a hand and ask -- or grab me on the hallway

track

Hi, I'm John.

hi, i'm john

@genehack

i go by genehack most places on line.today i'm going to talk to you today about some tools and tricks i have for being productive while developing. but before i do that, i need to explain a bit about why i needed it. my life is busy.

Sammy

i've got a dog.

@sammygenehack

i've got a dog.

Two kids

two daughters

A Wife

and a wife that i like to hang out with

long-suffering conference widow

A Wife

photobomber is not impressed.

A Wife

long-suffering conference widow

Job

I've also got a job

Hobbies

and i have a few hobbies -- maintain some Perl modules,

Hobbies

Trying to learn Swift, check out Angular 2,

Lots of

Hobbies

reading, hiking, cooking,

I've got a lot of balls in the air

don't have time for trivial nonsense, so i use a lot of automation and other "lifehacks"

AUTOMATE YO' SELF

occasionally, you just have to automate yo' self!

Basic Principles

whenever i'm trying to do this, there are some basic principles that i try to keep in mind

Don't Make Me Think

none of this automation stuff should require me to think -- if i have to think about it, it's not really saving me any time

This is my "you made me think" face.

the whole point is not having to think.

Consistency is Good.

so, in those terms, one thing that's essential: make everything the same. have a standard directory layout under $HOME. you shouldn't have to think about what system you're on, what shell, what project. things should just be the same -- or at least *correct* -- all the time

Idempotence is better!

Idempotence: the property of certain operations in mathematics and computer science, that can be applied multiple times without changing the result beyond the initial application

App::MiseEnPlace

so, here's an example of idempotence, a utility i wrote to manage symlinks and directories in my home directory

App::MiseEnPlace

"everything in its place"

"mise en place" is a French phrase meaning "everythng in its place" -- it comes from cooking, and the principle that you should have all your ingredients prepped and ready to go before you start cooking. i wanted something to make sure i always had my standard directory layout under $HOME as well as setting up various symlinks

% cat ~/.mise --- manage: - doc - etc - private - proj/* - src/* create: directories: - bin - proj - proj/go/src/github.com/genehack - src - var links: - Desktop: var/tmp - Desktop: tmp

here's what the top level config for mise looks like -- it goes in .mise in your home directory. we have a list of

directories we want to manage (more on that in a minute) and a set of directories and symlinks to create. links are source:target

% cat proj/emacs/.mise --- create: links: - DIR: ~/.emacs.d - bin/build-most-recent-emacs: BIN - bin/e: BIN - bin/ec: BIN - bin/git-blame-from-line-num: BIN - bin/map-test-lib: BIN

this is a per-project config file. mise has a couple special keywords DIR and BIN, that refer to the directory

containing the .mise file and ~/bin, respectively the advantage of this is you don't need to have a huge gnarly $PATH with a bunch of project directories in it,

everything is just symlinked into ~/bin

% mise [LINK] created ~/proj/emacs -> ~/.emacs.d [LINK] created ~/proj/emacs/bin/build-most-recent-emacs -> ~/bin/build-most-recent-emacs [LINK] created ~/proj/emacs/bin/e -> ~/bin/e [LINK] created ~/proj/emacs/bin/ec -> ~/bin/ec [LINK] created ~/proj/emacs/bin/git-blame-from-line-num -> ~/bin/git-blame-from-line-num [LINK] created ~/proj/emacs/bin/map-test-lib -> ~/bin/map-test-lib

% mise

% rm ~/bin/e

% mise [LINK] created ~/proj/emacs/bin/e -> ~/bin/e

when we run mise for the first time, you can see it creates all those links. if we run it again, it does NOTHING.

Idempotency for the win! If you remove a link and run it again, it creates _just_ that link

App::MiseEnPlace

get it from your favorite CPAN mirror

available on CPAN, minimal dependencies, works on any perl from this decade.

smartcd

ok, so that handles getting a consistent directory structure, and linking project binaries. what if you have other per-project stuff that you want to set up? environment variables or other stuff?

smartcd

Automatically run code when entering or leaving directories or sub-directories

enter smartcd, which hooks the 'cd' command and then runs scripts when you enter or leave a directory (or even a subdirectory)

% smartcd show enter /Users/genehack/.smartcd/scripts/Users/genehack/fake-node-proj/bash_enter exists ------------------------------------------------------------------------- ######################################################################## # smartcd enter - /Users/genehack/fake-node-proj # # This is a smartcd script. Commands you type will be run when you # enter this directory. The string __PATH__ will be replaced with # the current path. Some examples are editing your $PATH or creating # a temporary alias: # # autostash PATH=__PATH__/bin:$PATH # autostash alias restart="service stop; sleep 1; service start" # # See http://smartcd.org for more ideas about what can be put here ########################################################################

autostash NODE=4.2.3 autostash PATH=$PATH:__PATH__/node_modules/.bin/ nvm use $NODE -------------------------------------------------------------------------

smartcd has 'edit' and 'show' subcmds (and a bunch of others), and 'leave' and 'enter' scripts. here's an enter script

for an arbitrary node project. the autostash keyword sets up an environment variable that will be *unset* when you leave this directory. you can also run arbitrary commands; the 'nvm' command here selects a version of node to use

% cd ~/fake-node-proc Now using node v4.2.3 (npm v2.14.7)

% echo xx$NODE xx4.2.3

% cd ..

% echo xx$NODE xx

here's what it looks like when you cd into that directory. and you can see the env var is set. if we change back out of

the directory, the environment variable is unset.

smartcd

https://github.com/cxreg/smartcd

pretty cool, available on github. works with bash and zsh. really easy to install.

i can pay this the highest possible compliment: i haven't forked it or needed to patch it in any way, i just use a checkout from the upstream repo

another option: direnv

http://direnv.net/ https://github.com/direnv/direnv

another way to do this is called direnv

written in go, hooks into shells in a similar way

big difference: config stored _inside_ project directory

$ cd ~/my_project $ echo ${FOO-nope} nope $ echo export FOO=foo > .envrc .envrc is not allowed $ direnv allow . direnv: reloading direnv: loading .envrc direnv export: +FOO $ echo ${FOO-nope} foo $ cd .. direnv: unloading direnv export: ~FOO $ echo ${FOO-nope} nope

direnv supports similar directory-specific env vars

$ cd ~/my_project $ echo ${FOO-nope} nope $ echo export FOO=foo > .envrc .envrc is not allowed $ direnv allow . direnv: reloading direnv: loading .envrc direnv export: +FOO $ echo ${FOO-nope} foo $ cd .. direnv: unloading direnv export: ~FOo $ echo ${FOO-nope} nope


$ cd ~/my_project $ echo ${FOO-nope} nope $ echo export FOO=foo > .envrc .envrc is not allowed $ direnv allow . direnv: reloading direnv: loading .envrc direnv export: +FOO $ echo ${FOO-nope} foo $ cd .. direnv: unloading direnv export: ~FOO $ echo ${FOO-nope} nope


direnv may be better for collaborative projects.

direnv may be better for collaborative projects?

get you one that can do both?

App/Env Builders

are your

friends

speaking of per-project settings, let's talk for a minute about application builders.

perlbrew plenv

rubyenv nvm

App::GitGitr

build-most-recent-emacs

here are some of the ones i've used or use now. perlbrew and plenv let you have multiple perls. nvm does the same thing for node. similar tools exist for python, ruby, etc.

then there's GitGitr, which I wrote while maintaining Git wrapper library. I needed to be able to quickly install arbitrary Git versions while responding to bug reports -- so I wrote a little tool that does that.

Similarly, I'm an Emacs user. If a new version is released, I want to upgrade, across all my systems - so I scripted that.

Consistency Corollary: Don't trust system binaries

Back at the beginning, I said "consistency is good". Now, if you're developing on MacOS and deploying to Linux (or dev-ing on Ubuntu and deploying to Debian), you're probably not going to have the same version of tool from the OS (and if you do now, it's not going to last). Even if the versions are the same, there may have been vendor patches applied.

Automate building your critical tools.

No, if you really want to be consistent, the best approach is to build the tools that are most critical for your project. ("Build" in this case may just mean automating the install; it doesn't have to mean "build from source".) Note: only do this for the *important* stuff. (include examples)

The Silver Searcher

Speaking of tools, here's a tool that has literally improved my entire development life -- the silver searcher. Anybody

here using this?

grep?

ok, so, grep -- everybody knows grep, right? let's you search for text inside files, which is something you do a lot

while developing code.

powerful, speedy, indiscriminate

so, grep is super powerful in terms of what you can search for, and it's pretty quick, but it's not very selective. you

can list all the files you want to search, or search whole dir trees, but you quickly realize this sucks, because of things like .git directories. anybody ever do a recursive grep on a big git checkout? yeah.

ack?

anybody here use ack? ack is a grep-like tool written in perl.

powerful, selective, slow

it's just as powerful as grep in terms of what you can search for, but it's recursive by default (which is what you want)

and it's smart about ignoring .git and SVN stuff. the problem is, it's pretty slow, particularly to start up (because Perl)

ag!

enter ag (which is what the binary for the silver searcher is called.

powerful, selective, FAST

ag works much like ack (not _exactly_, but close enough), but it's written in C and it's oh so fast.

The Silver Searcher

https://github.com/ggreer/the_silver_searcher

available on Github, also packaged in several Linux distros. again, no higher compliment than to say I just build

from a checkout of the upstream. have never needed to fork or patch

retraining your

forearms for fun & profit

karabiner, setting up keyboard driver to force correct shift key usage

My biggest productivity/automation tip

so, here's my single biggest tip in this whole talk. are you ready? brace yourselves.

revision control $HOME

this is it. anybody know what this is?

this is your home directory under revision control

Why bother?

no, but seriously. it's a little bit of a pain to develop the discipline but once you get used to having _everything_ under revision control, it's nice. you don't have to worry about experimenting with anything, backing stuff up, or dealing with cross-machine variation in your environments

say automation

again

originally i had series of kludgy shell scripts to manage repo updates and checkoutssuper ugly and not worth sharingand then, inspiration: Ingy döt Net talking about App::AYCABTU @ PPW2010

Ingy döt Net

this is ingy - he's a crazy awesome open source hacker guy who has done a whole bunch of stuff. probably best

known for being one of the inventors of YAML

Things I wanted to steal

the thing ingy had developed had a bunch of stuff i wanted to steal: The basic ideaThe interfaceInfo about repositories in config fileFlexible ways of selecting repos for operations – by #, by name, by tag

Things I wanted to add

Support for more than just GitLocate repositories in arbitrary locationsEasily add and remove repositoriesAbility to easily extend with more subcommandsMost importantly: better name!

App::AYCABTU?!?!‽

and finally, the name. what even is this.

GitGot

Thus was born GitGothttp://search.cpan.org/dist/App-GitGot/Installs a ‘got’ commandUses Moo and App::Cmd under the coversAdd new subcommands by writing a single class!

Whirlwind Tour

Let's take a whirlwind tour of how got works

got add

you tell got about repos using the 'add' command, from inside the git repo

% got add Name: foo URL: Path: /Users/genehack/foo Tags: bar

it'll prompt you for required info, and supply sensible defaults. note that you can also apply tags (space-delimited)

got add -D

or you can just add the '-D' switch and it'll automatically use the defaults

got clone <REPO URL>

you can also clone a remote repo, which will check it out into the working directory and add it to got, prompting

you for details

% got clone [email protected]:genehack/app-gitgot.git Name: [app-gitgot]: Path: [/Users/genehack/app-gitgot]: Tags: Cloning into '/Users/genehack/app-gitgot'...

it'll prompt you for required info, and supply sensible defaults. note that you can also apply tags (space-delimited)

got clone -D <REPO URL>

got clone also respects the '-D' switch

got fork <GITHUB URL>

finally, you can give got a github url, and it will fork that project under your github id, then check it out into the

current directory and add it to got

~/.gitgot

all the info about the repos managed by got lives in this .gitgot file in your home directory

- name: App-Amylase path: /Users/genehack/proj/App-Amylase repo: [email protected]:genehack/App-Amylase.git type: git - name: Git-Wrapper path: /Users/genehack/proj/Git-Wrapper repo: [email protected]:genehack/Git-Wrapper.git tags: git type: git - name: HiD path: /Users/genehack/proj/HiD repo: [email protected]:genehack/HiD.git type: git - name: Perl-Build path: /opt/plenv/plugins/perl-build repo: git://github.com/tokuhirom/Perl-Build.git type: git

it's just a simple YAML formatted line, totally hand-editable (although you shouldn't _need_ to do that, you can)

note that repos can be located anywhere on the disk, don't have to under a common dir or in your home or

whatever. anywhere you can write to is fair game

But now what?

ok, so you've added all your git repositories to got. what now?

got list

well, you can get a list of them

got ls

which you can shorten to this

1) App-Amylase git [email protected]:genehack/App-Amylase.git 2) Git-Wrapper git [email protected]:genehack/Git-Wrapper.git 3) HiD git [email protected]:genehack/HiD.git 4) Perl-Build git git://github.com/tokuhirom/Perl-Build.git 5) Perl-Critic git [email protected]:genehack/Perl-Critic.git 6) STAMPS git [email protected]:genehack/STAMPS.git 7) advanced-moose-class git ssh://[email protected]/train/advanced-moose-class.git 8) app-gitgitr git [email protected]:genehack/app-gitgitr.git 9) app-gitgot git [email protected]:genehack/app-gitgot.git

that'll get you this sort of listing.

got ls -q

if you don't want to see the upstream repo info, you can use the '-q' or '--quiet' switch

1) App-Amylase 2) Git-Wrapper 3) HiD 4) Perl-Build 5) Perl-Critic 6) STAMPS 7) advanced-moose-class 8) app-gitgitr 9) app-gitgot

and that'll get you this output.

note the numbers - those will give you a way to select repos for other commands

got ls [repos]

easiest way to demo that is with an example. you can restrict the listing

got ls 5

this will just list repo #5 for example

5) Perl-Critic

also, note that the list is always sorted the same way, so the numbers will be stable (unless you add new repos)

got ls 5-8

you can also give a range of repos

5) Perl-Critic 6) STAMPS 7) advanced-moose-class 8) app-gitgitr

and that'll give you that range, like you would expect

got ls HiD

you can also specify repos by name

3) HiD

got ls -t git

or by using tags. can specify multiple tags with multiple '-t' switches. they combine with 'or' semantics.

2) Git-Wrapper 8) app-gitgitr 9) app-gitgot

here are all the repos tagged with 'git' (at least in our example)

got ls 5-8 HiD 21 -t git

finally, you can combine all of these selection methods together. here we're asking for repos 5 thru 8, the repo

named HiD, repo 21, and all repos tagged with git

2) Git-Wrapper 3) HiD 5) Perl-Critic 6) STAMPS 7) advanced-moose-class 8) app-gitgitr 9) app-gitgot 21) etc

and this is what we get

note that most commands operate on all repos, and any that do, you can use these techniques to restrict the

command to a subset.

What else you got?

That's cool.

(no pun intended.)

got status

you can check the status of your repos.

*all* your repos

got st

or if you're into the whole brevity thing...

1) App-Amylase : OK 2) Git-Wrapper : OK 3) HiD : OK 4) Perl-Build : OK 5) Perl-Critic : OK 6) STAMPS : OK 7) advanced-moose-class : OK 8) app-gitgitr : OK 9) app-gitgot : OK

that'll get you output like this.

again, note the use of color to give quick visual cues

1) App-Amylase : OK 2) Git-Wrapper : OK 3) HiD : Dirty 4) Perl-Build : OK 5) Perl-Critic : OK 6) STAMPS : OK 7) advanced-moose-class : OK 8) app-gitgitr : OK 9) app-gitgot : OK

Dirty

got status will let you know if a repo has uncommitted changes.

super handy if, for example, working on one machine and are going to move to another one and want to see what

hasn't been committed.

1) App-Amylase : OK 2) Git-Wrapper : OK 3) HiD : OK 4) Perl-Build : OK 5) Perl-Critic : OK 6) STAMPS : OK Ahead by 1 7) advanced-moose-class : OK 8) app-gitgitr : OK 9) app-gitgot : OK

1) App-Amylase : OK 2) Git-Wrapper : OK 3) HiD : OK 4) Perl-Build : Dirty 5) Perl-Critic : OK 6) STAMPS : OK 7) advanced-moose-class : OK 8) app-gitgitr : OK 9) app-gitgot : OK

Dirty

1) App-Amylase : OK 2) Git-Wrapper : OK 3) HiD : OK 4) Perl-Build : OK 5) Perl-Critic : OK 6) STAMPS : OK Ahead by 1 7) advanced-moose-class : OK 8) app-gitgitr : OK 9) app-gitgot : OK

Ahead by 1

it'll also tell you if you have local commits that haven't been pushed to the remote yet

got st -q

finally, you can use the '-q' switch to hide the "uninteresting" stuff

got st -q 3) HiD : Dirty

6) STAMPS : OK Ahead by 1

Dirty

Ahead by 1

which in this case is all the repos that don't have changes and are up to date

got update

you can also run 'git pull' across all your repos

got up

which abbreviates to 'up'

and yeah, i should have called it pull but i'm a dummy and we're stuck with it now.

1) App-Amylase : Up to date 2) Git-Wrapper : Up to date 3) HiD : Up to date 4) Perl-Build : Updated Updating 7f25f89..72587c8 Fast-forward lib/Perl/Build.pm | 14 +++++++++++++- script/perl-build | 14 +++++++++++++- 2 files changed, 26 insertions(+), 2 deletions(-) 5) Perl-Critic : Up to date

Updated

it'll do pretty much what you expect (and it supports '-q' too)

got update_status

finally, there's a command that combines both those, because it's something i do pretty frequent -- update

everything, then look at the status of everything

got upst

surprise - you can abbreviate that one too

got upst -q

and it also supports the '--quiet' option to only show you the interesting stuff

How much would you pay?

so, how much would you pay?

Wait, don't answer yet

got fetch

if you're not a fan of the way 'git pull' works you can also run 'git fetch' via got.

got push

you can even do a push across all your repos at once. (personally, this strikes me as insane but somebody sent in a

patch, so...)

got gc

you can garbage collect all your repos

got this

at some point, somebody sent in a patch to add 'got this' -- which tells you if the current directory is under got

control

% cd proj/HiD % got this 4) HiD

got that <DIRECTORY>

this provoked somebody else to send in a 'got that' command, which does the same thing, but takes a path to

check

% got that ~/etc 37) etc

got chdir

finally, there are a number of commands that help you jump to the directory of a project. got chdir

got cd

also spelled 'got cd', will change your current working directory to the given repo (note that this is one of the few

got commands that requires a single repo)

got tmux

we also have tmux integration -- 'got tmux' will open a new tmux window with the working directory in your repo.

this _can_ be done with multiple repos. better, the tux window is persistent; as long as it's open 'got tmux' will just select the already open window, not open a new one

got tmux -s

you can also spawn whole new tmux sessions if you prefer those to windows -- and again, those will be re-used as

long as they're around

How much would

you pay now

?

Good news!

It's free!

nothing!

Works on any perl from the last 5 years

cpan App::GitGot

installation is simple

cpanm App::GitGot

can be even simpler

Find me at SeaGL and I'll help you install!

or you can find me on the hallway track and i'll help you get it installed on your machine

Easy to extend

package App::GitGot::Command::chdir;

# ABSTRACT: open a subshell in a selected project use 5.014;

use App::GitGot -command;

use Moo; extends 'App::GitGot::Command'; use namespace::autoclean;

sub command_names { qw/ chdir cd / }

sub _execute { my( $self, $opt, $args ) = @_;

unless ( $self->active_repos and $self->active_repos == 1 ) { say STDERR 'ERROR: You need to select a single repo'; exit(1); }

my( $repo ) = $self->active_repos;

chdir $repo->path or say STDERR "ERROR: Failed to chdir to repo ($!)" and exit(1);

exec $ENV{SHELL}; }

1;

package App::GitGot::Command::chdir; # ABSTRACT: open a subshell in a selected project

use Moo; extends 'App::GitGot::Command';





exec $ENV{SHELL}; }

package App::GitGot::Command::chdir;

# ABSTRACT: open a subshell in a selected project use 5.014;

use App::GitGot -command;

use Moo; extends 'App::GitGot::Command'; use namespace::autoclean;






exec $ENV{SHELL}; }

1;

Suggestions welcome!

areas for improvement:

support for other VCSen better config management tools any other crazy workflow improvement you can think of!

In conclusion…

notice the speedbumps

in your life

…but it is a great way to learn a new languageYou don't have to write a code…

can also just be about making the wrong way harder

Thanks SeaGL organizers

Ingy döt Net Yanick Champoux

Michael Greb Rolando Pereira

Chris Prather

photo credits:

all photos by speaker except Ingy döt Net photo - https://www.flickr.com/photos/bulknews/389986053/

and pug - https://upload.wikimedia.org/wikipedia/commons/d/d7/Sad-pug.jpg and automate yo'self - somewhere on the net

thanks to the company i work for for allowing me to (occasionally) mess with this stuff and sending me out to give these talks. holler at me on the hallway track if you're looking for some Open Source consulting or staff augmentation

Questions?

as i said, i *love* to talk to people about this productivity type stuff, so grab me on the hallway track. i'm friendly.

automate yo'self -- seagl

Internet