shooting from the hip - perl at the command line for dbas

Shooting from the hip.Perl at the command line for DBAs

Giuseppe MaxiaMySQL Community Team Lead

@datacharmer

$_

@ @

#!

:-)

This work is licensed under the Creative Commons Attribution-Share Alike 3.0 Unported License. 1Friday, 06 August 2010

about me -Giuseppe Maxia a.k.a. The Data Charmer MySQL Community Team Lead Long time hacking with MySQL features Formerly, database consultant, designer, coder. A passion for QA An even greater passion for open source ... and community Passionate blogger http://datacharmer.blogspot.com

2Friday, 06 August 2010

http://datacharmer.blogspot.com




















Updated presentation slides

you will find an up to date copy of these slides at: http://tinyurl.com/shoot-cl-2010

http://tinyurl.com/shoot-cl-full-2010


http://tinyurl.com/shoot-cl-2010




I love your feedback! Tell it to the Twitter community

#yapc_eu "your feedback here" Twitter me

@datacharmer "your feedback here" Blog about it


This is a story about languages


you got to understand, Bernie, it's not personal.

It's just that you are obsolete

We use language to talk to computers


10 print "hello"20 goto 10We use

languages to talk to computers


I am a polyglot


An imperfect polyglot, actually


I speak Italian


I speak Italian

with a clear Sardinian accent


I speak English


I speak English

with a distinct Italian accent


I speak Perl


I speak Perl

with a hint of a C accent


I speak French and Spanish


I speak French and Spanishbut since I use them very little, they have melted with Italian and

English13Friday, 06 August 2010

I speak Bash #!/bin/bash


I speak Bash #!/bin/bash

with a strong Perl accent


I speak fluent Italian

La donzelletta vien dallacampagna, in sul calar delsole, col suo fascio dell'erbae reca in mano un mazzolino di rose e viole, onde, siccome suole, ornare ella si appresta ...


However, Sardinian comes handy sometimes

Maissavidati frimisti! (*)

(*) May you never stop16Friday, 06 August 2010

I speak fluent English

When in the Course of human events, it becomes necessary for one people to dissolve the political bands which have connected them with another, and to assume among the powers of the earth, the separate and equal station to which the Laws of Nature and of Nature's God entitle them, a decent respect to the opinions of mankind requires that they should declare the causes which impel them to the separation.


http://en.wikipedia.org/wiki/Natural_law




hmm.wrong flag




But sometimes Italian is still necessary

maestro

virtuoso

al fresco

presto

spaghettibravo

tempo

dilettantefiasco

studio

stucco

barista

cappuccino

espresso

lasagnapizza

pasta

stanza

finale

lottostiletto

vendetta soprano piano


And sometimes Italian is needed

I tuoi perversidefunti!(*)

(*) Your wicked ancestors!19Friday, 06 August 2010

Although Sardinian is still effective here

Ancu ti currullidisa giustizia (*)

(*) May the rule of law chase you20Friday, 06 August 2010

I speak Perl fluently

#!/usr/bin/perluse strict;use warnings;use File::Find;

while (my $line = <stdin>){ # …


Sometimes, I need help from C

use Inline C;print "9 + 16 = ", add(9, 16), "\n";

__END____C__int add(int x, int y) { return x + y;}


Most of my work at the command line is Bash

$ for D in $HOME/*/; do du -sh $D;> done > /tmp/occupancy

#!/bin/bash


And sometimes, a pinch of Perl can help

$ perl -pe 's{(\d\d)-(\d\d)-(\d\d\d\d)}{$3-$1-$2}g' \ < rawfile.txt > betterfile.txt

#!/bin/bash


And then on top of all this, I need to speak SQL

$ mysql -e 'SELECT COUNT(*) FROM dbname.tbl_name WHERE col_name = 1 and other_col="A" '


I wish I could say that my SQL has a Perlish accent


use strict;use DBI;?


I wish I could say that my SQL has a Perlish accent


but it wouldn't be true


I could curse in Perl

$ perl -pe 's{(\d\d)-(\d\d)-(\d\d\d\d)}{$3-$1-$2}g' \ < rawfile.txt > betterfile.txt (*)

(*) already translated

But it would not get me far.


Then


Then

I will need to mix languages.


Then


This talk will show exactly that.


Then



Mixing languages.


Then



Mixing languages.Yes.


Then



Mixing languages.

$ at the command line

Yes.


What this tutorial is about Providing tools and tips for command line users; Learn a few tricks to make you achieve more and type

less.


What this tutorial is about Providing tools and tips for command line users; Learn a few tricks to make you achieve more and type

less. Making GUI enthusiasts green with envy;

bar

foo

baz


the command line (for rookies)

BASICS

31


The shell• It's a command interpreter• There are many flavors

> bash > zsh> ksh

32


MySQL command line client• called "mysql"

> Takes SQL commands> sends them to the server> gets results

• but it can do much more ...

33


other tools• Shell scripts• Aliases• Perl• awk• sort• …. and a lot of Unix little helper

34


other tools• Shell scripts• Aliases• Perl• awk• sort• …. and a lot of Unix little helper

34

which could replace everything else, of course


in case it wasn't clear (yeah, for rookies)

35



• command line actions are

35



• command line actions are> commands (the name of the application to invoke)

35



• command line actions are> commands (the name of the application to invoke)> parameters (optional arguments for the application)

35



• command line actions are> commands (the name of the application to invoke)> parameters (optional arguments for the application)> and the ENTER key

35



• command line actions are> commands (the name of the application to invoke)> parameters (optional arguments for the application)> and the ENTER key

35

Enter


The Unix philosophy (1)• One tool for every feature

> make one tool that does one thing well> for many features, make many tools> make them work together (see next slide)

36


The Unix philosophy (2)• To make several tool cooperate:

> pipe the output of one tool to the input of another one.> for more actions, pipe more tools

$ sort < filename | uniq | nl

37


Unix pipes

38


Unix pipes

38

inputapplication <


Unix pipes

38

outputapplication >

inputapplication <


Unix pipes

38

outputapplication >

inputapplication <

outputapplication >input<


Unix pipes

38

outputapplication >

inputapplication <


application | application


Unix pipes

38

outputapplication >

inputapplication <



input output


Unix pipes = parallel processing

39



39




39


output



39


input output



39


input output output



39


input output output input


Who gets what• shell• application• database server• pager

40


who gets what

41

shell prompt $ mysql

mysql client

mysql server


any hostwho gets what

42

shellmysql client

mysql server


who gets what$ mysql -u username -ppassword

43

First the applicationthen the server


who gets what$ mysql -u username -ppassword \ -e "select VERSION()"

44

goes to the server

goes to the client, then the server


who gets whatmysql> helpmysql> pager lessmysql> select 1;

45

goes to the server

goes to the client onlygoes to the client only


the command line

readline46


readline• Common component

> shell> mysql client> many CLI applications

47


Main cursor movements

48

Ctrl A Ctrl E

Start/End of line

Meta B Meta F

Back/Forward one word

previous/next command

Ctrl L clear screen and take current line on top


Changing text

49

Meta U Uppercase word

Meta L Lowercase word

Ctrl D delete word

Ctrl K delete to end of line

Ctrl - undo

for more tips:$ man readline


Some conventions in this talk

$ commandat the shell

mysql> command at the mysql prompt

50


the Bash shell

bash51

#!


components• commands• aliases• variables• functions

52


commands$ echo "something"something

53


aliases$ alias mydate='mysql -e "select NOW()"'$ mydate+---------------------+| now() |+---------------------+| 2010-04-12 07:14:19 |+---------------------+

54


aliases$ alias myself='mysql' $ myself -e "select NOW()"'+---------------------+| now() |+---------------------+| 2010-04-12 07:14:19 |+---------------------+

55


variables$ MYBATCH='mysql -N -B'$ MYQUERY='select curdate()' $ $MYBATCH -e "$MYQUERY"2010-04-12

56


nesting variables# t.sh CMD="SELECT TABLE_NAME" TBL='information_schema.tables' WHC1="TABLE_SCHEMA='mysql'" WHC2="TABLE_NAME like 'u%' " Q1="$CMD FROM $TBL" Q2="WHERE $WHC1 AND $WHC2" Q="$Q1 $Q2" echo $Q echo $Q | mysql

57


nesting variables$ sh t.shSELECT TABLE_NAME FROM information_schema.tables WHERE TABLE_SCHEMA='mysql' AND TABLE_NAME like 'u%'TABLE_NAMEuser

58


the mysql command line client

59

mysql>


SQL commands from a file$ mysql < filename.sqlmysql> source filename.sql

• executes the commands contained in filename.sql

• commands can be queries understood by the server

• and commands for the client

60


SQL commands at the command line$ application | mysql$ echo "SELECT 1" | mysql$ mysql -e "SELECT 1"

• executes the commands from standard input

• commands can be queries understood by the server

• and commands for the client

61


batch mode$ echo "select curdate()" | mysqlcurdate()2010-04-12

62


table mode$ echo "select curdate()" | mysql -t+------------+| curdate() |+------------+| 2010-04-12 |+------------+

63


table mode$ mysql -e "select curdate()"+------------+| curdate() |+------------+| 2010-04-12 |+------------+

64


Forcing batch modemysql -B -e 'select curdate()'curdate()2010-04-12

65


Forcing batch mode - no headermysql -N -B -e 'select curdate()'2010-04-12

66

IMPORTANT!


Shell + mysql$ echo "The current server's date is `mysql -N -B -e 'select curdate()'`."

$ echo "The current server's date is $(mysql -N -B -e 'select curdate()')."

67


advanced command line techniques

loops68

$_

(-:((((((

@@


loops basics$ for N in 1 2 3> do> echo $N> done123

69


loops basics$ for N in 1 2 3 ; do echo $N ; done123

70


loops basics$ for N in $(seq 1 3) ;\ do echo "this is number $N" ;\ done

this is number 1this is number 2this is number 3

71


loops from file$ head /usr/share/dict/words AaaaaalaaliiaamAaniaardvarkaardwolf

72


loops from file$ for U in \$(head /usr/share/dict/words ) ; do perl -le 'print uc shift' $U ; doneAAAAAALAALIIAAMAANI

73


loops from mysql results$ mysql -B -N -e 'show tables from world'CityCountryCountryLanguage

74


loops from mysql results$ for T in $(mysql -B -N -e 'show tables from world') ;\> do echo "CREATE TABLE world2.$T LIKE world.$T;" ; done

CREATE TABLE world2.City LIKE world.City;CREATE TABLE world2.Country LIKE world.Country;CREATE TABLE world2.CountryLanguage LIKE world.CountryLanguage;

75


piping loops back into mysql$ for T in $(mysql -B -N -e 'show tables from world') ;\> do echo "CREATE TABLE world2.$T LIKE world.$T;" ; done \> | mysql -vv

76



nested

77

$_

@ @

#!

:-)

#!#!#!#!#!

loops77Friday, 06 August 2010

doing something for every tablein every database$ mysql -e "show databases"+--------------------+| Database |+--------------------+| information_schema || mysql || sakila || world |+--------------------+

78


doing something for every tablein every database$ mysql -e "show tables from world"+-----------------+| Tables_in_world |+-----------------+| City || Country || CountryLanguage |+-----------------+

79


pseudocodefor each database do for each table do print the table name

80


actual codealias mybatch='mysql -B -N'for D in $(mybatch -e "SHOW SCHEMAS")do for T in \ $(mybatch -e "SHOW TABLES FROM $D") do echo "$D.$T" donedone

81



options files

82

$HOME


Default option files/etc/my.cnf/etc/mysql/my.cnf (Debian, Ubuntu)$BASEDIR/my.cnf$HOME/.my.cnf

83


Custom option files$ mysql --no-defaults$ mysql --defaults-file=filename$ mysql --defaults-extra-file=filename

84


Option files sections[mysqld]# option for server

[client]# option for any mysql client

[mysql]# options specific to 'mysql' CLI app

85


Option files example

[client]user=mynamepassword=mypassword

[mysql]prompt="mysql (\h) {\u} [\d] > "

86


Option files: caution# protect your option files!

$ chmod 600 $HOME/.my.cnf

$ ls -l .my.cnf-rw------- 1 [...] 10:04 .my.cnf

87


promptmysql> prompt gm (\h) {\u} [\d] > PROMPT set to 'gm (\h) {\u} [\d] > 'gm (localhost) {root} [(none)] > use testDatabase changedgm (localhost) {root} [test] >

# \h -> host# \u -> user# \d -> database

88



aliases89

#!

(-:

((((((@@


An useful alias$ alias mysql_as_root=\'mysql --defaults_file=~/.my_as_root.cnf'

mysql_as_root -e "select user()"+----------------+| user() |+----------------+| root@localhost |+----------------+

90


An useful alias#.my_as_root.cnf'[client]user=rootpassword=your_mysql_root_password

[mysql]prompt="### [\h] {\u} (\d) > "

91


Another useful aliasalias mybatch='mysql -B -N '

mybatch -e "select version()"5.1.45-log

92



functions93

#!

(-:

((((((@@( )


functions• like aliases, but with parameters

94


sample functionfunction grant_new { mysql_as_root -e \ "GRANT $1 on $2 to $3 identified by \"$4\" "}

$ grant_new SELECT "world.*" john \ john_secret

95


sample function$ mysql_as_root \ -e 'show grants for john\G'*** 1. row ***Grants for john@%: GRANT USAGE ON *.* TO 'john'@'%' IDENTIFIED BY PASSWORD '*73A88CE13374128844E3ABB500F439A9C02C53A3'*** 2. row ***Grants for john@%: GRANT SELECT ON `world`.* TO 'john'@'%'

96



getting query

details97

$_


which query is being executed?# wrong.sql# this script has one erroruse test ;drop table if exists t1 ;create table t1 (id int);insert into t1 values (a);

98


which query is being executed?$ mysql < wrong.sqlERROR 1054 (42S22) at line 4: Unknown column 'a' in 'field list'

# WHERE IS THE ERROR?

99


which query is being executed?$ mysql -v < wrong.sql--------------drop table if exists t1----------------------------create table t1 (id int)----------------------------insert into t1 values (a)--------------ERROR 1054 (42S22) at line 4: Unknown column 'a' in 'field list'

100


getting execution times$ mysql -vv -e \'select rating,count(*) from sakila.film group by rating'+--------+----------+| rating | count(*) |+--------+----------+| G | 178 || PG | 194 || PG-13 | 223 || R | 195 || NC-17 | 210 |+--------+----------+5 rows in set (0.00 sec) 101


$echo 'select rating,count(*) from sakila.film group by rating' | mysql -vv--------------select rating,count(*) from sakila.film group by rating--------------

rating count(*)G 178PG 194PG-13 223R 195NC-17 2105 rows in set

102

getting execution times (batch mode)


getting execution times (batch mode)$ echo 'select rating,count(*) from sakila.film group by rating' | mysql -vvv --------------select rating,count(*) from sakila.film group by rating--------------+--------+----------+| rating | count(*) |+--------+----------+| G | 178 || PG | 194 || PG-13 | 223 || R | 195 || NC-17 | 210 |+--------+----------+5 rows in set (0.00 sec) 103



getting distinct results

104

$_


getting a single value$ mysql -NB -e "show variables like 'port'" port 3306

# no header, batch mode# still two values

106


getting a single value$ mysql -NB \ -e "show variables like 'port'" \ | awk '{print $2}'

3306

$ mysql -NB \ -e "show variables like 'port'" \ | perl -lane 'print $F[1]'

3306 # filtering the wanted value

107


getting a single value: is replication working?

$ mysql \ -e "show slave status \G" \ | grep -i slave_io_running

Slave_IO_Running: Yes

# too many things. # We only want a 'yes' or 'no'

108


getting a single value: is replication working?

$ mysql \ -e "show slave status \G" \ | grep -i "slave_io_running" \ | perl -lane 'print $F[1]'

Yes

# yes!

109


What's perl -lane?

110


What's perl -lane?

110

perl -lane must turn perl


Perl command line must-know facts (1)perl -e 'print "hello\n"'# executes perl code

111


Perl command line must-know facts (2)perl -ne 'print if /â/'# starts a loop from standard input# and executes the perl code for each# iteration

perl -MO=Deparse -ne 'print if /â/'LINE: while (defined($_ = <ARGV>)) { print $_ if /â/;}

112


Perl command line must-know facts (3)perl -pe 's/(â\w+)/$1/'# like -ne, but prints each line

$ perl -MO=Deparse -pe 's/(â\w+)/$1/'LINE: while (defined($_ = <ARGV>)) { s/(â\w+)/$1/;}continue { print $_;}

113


Perl command line must-know facts (4)perl -le 'print "hello"'# strips newlines from input and adds # them to output

$ perl -e 'print "hello"'hello$ $ perl -le 'print "hello"'hello$ 114


Perl command line must-know facts (5)perl -lane 'print $F[0]'# splits input into @F array

$ echo "a b c" | \ perl -lane 'print $F[1]'b $ echo "a b c" | \ perl -lane 'print $F[2]'c 115


Perl and shell contention for $

116

Take the $$$ and run

Perl command line must-know facts (6)

#!/bin/bash


Beware of $VARIABLES on the command line$ perl -le "$x=1; print $x" syntax error at -e line 1, near "="Execution of -e aborted due to compilation errors.

117



117

consumed by the shell, before Perl can see it

double quotes allow the shell to interpolate variables



117







117


consumed by the shell, before Perl can see it$ perl -le '$x=1; print $x'

1





117


consumed by the shell, before Perl can see it$ perl -le '$x=1; print $x'

1


single quotes prevent variable consumption by the shell





combining column names

118


combining column names$ mysql -N -e 'desc world.City'+-------------+----------+----+-----+| ID | int(11) | NO | PRI || Name | char(35) | NO | | | CountryCode | char(3) | NO | | | District | char(20) | NO | | | Population | int(11) | NO | | +-------------+----------+----+-----+

119


combining column names# Expected result:# SELECT CONCAT_WS('|', # ID,Name,CountryCode,District, # Population) from City

120


combining column names (I)echo "select concat_ws('|', $(mysql -N -e 'desc world.City' | \ perl -lane 'push @f, $F[0];' \ -e 'END{print join ",", @f}')) from world.City" select concat_ws('|', ID,Name,CountryCode,District,Population) from world.City

121


combining column names (II)echo "select concat_ws('|', $( mysql -NB -e 'select group_concat(column_name) from information_schema.columns where table_name="City" and table_schema="world"')) from world.City" select concat_ws('|', ID,Name,CountryCode,District,Population) from world.City

122


combining column names (IIa)echo "select concat_ws('|', $( mysql -NB -e 'select group_concat(column_name) from information_schema.columns where table_name="Country" and table_schema="world"')) from world.Country"select concat_ws('|', Code,Name,Continent,Region,SurfaceArea,IndepYear,Population,LifeExpectancy,GNP,GNPOld,LocalName,GovernmentForm,HeadOfState,Capital,Code2) from world.Country

123


piping the result back to mysqlecho "select concat_ws('|', $( mysql -NB -e 'select group_concat(column_name) from information_schema.columns where table_name="Country" and table_schema="world"')) as line from world.Country" | mysql | head -n 4lineAFG|Afghanistan|Asia|Southern and Central Asia|652090.00|1919|22720000|45.9|5976.00|Afganistan/Afqanestan|Islamic Emirate|Mohammad Omar|1|AFNLD|Netherlands|Europe|Western Europe|41526.00|1581|15864000|78.3|371362.00|360478.00|Nederland|Constitutional Monarchy|Beatrix|5|NLANT|Netherlands Antilles|North America|Caribbean|800.00|217000|74.7|1941.00|Nederlandse Antillen|Nonmetropolitan Territory of The Netherlands|Beatrix|33|AN

124



dumping data

125

$_


dumping a tab delimited data set # the naive way$ mysql \ -e "SELECT * into outfile '/tmp/city_data' from world.City"

# it works only if:# - /tmp/city_data does not exist# - /tmp is world writable

126


dumping a tab delimited data set $ mysql -B -N \ -e "SELECT * from world.City" \ > /tmp/city_data

# it works always

# if you omit the "-N" it also # includes a header

127


dumping a CSV $ mysql -B \ -e "SELECT * from world.City" \ | perl -F"\t" -lane \ 'print join ",", map {s/"/""/g; /^[\d.]+$/ ? $_ : qq("$_")} @F ' \ > /tmp/city_data

128


dumping a CSV $ mysql -B \ -e "SELECT * from world.City" \ | perl -F"\t" -lane \ 'print join ",", map {s/"/""/g; /^[\d.]+$/ ? $_ : qq("$_")} @F ' \ > /tmp/city_data

128

splits the line by tabs, instead of the default whitespace



scriptit

129


use the mysql.sh script The companion code for this tutorial http://tinyurl.com/cli-tut-code-2010

It will save you much typing But use it at your own risk!

130


http://tinyurl.com/cli-tut-code-2010

http://tinyurl.com/cli-tut-code-2010

Usage• download the script• load it

131

$ source mysql.sh


mysql.sh functions• my_batch "query"• returns the result of the query. • Without header • Without box

132

$ my_batch "SELECT * FROM test.t1"1 aaa2 bbb3 ccc


mysql.sh functions• my_value "what"• returns a single value

• e.g.

133

$ my_value "version()"5.1.49

$ my_value "@@server_id"1


mysql.sh functions• my_var "what"• returns the value of a given variable

• e.g.

134

$ my_var port3306

$ my_var datadir/usr/local/mysql/data


mysql.sh functions• my_status "what"• returns the value of a given status item

• e.g.

135

$ my_status com_select2653

$ my_status connections452


mysql.sh functions• my_dump_dataset "query" [header]• returns the values of a given query separated by tabs• if a header is requested, it will be part of the result• e.g.

136

$ my_dump_dataset "select * from world.City" | head -n 51 Kabul AFG Kabol 17800002 Qandahar AFG Qandahar 2375003 Herat AFG Herat 1868004 Mazar-e-Sharif AFG Balkh 1278005 Amsterdam NLD Noord-Holland 731200


mysql.sh functions• my_csv "query" [separator]• returns the values of a given query separated by

commas, or by a given separator• e.g.

137

$ my_csv 'select * from world.City' |head -n 3"ID","Name","CountryCode","District","Population"1,"Kabul","AFG","Kabol",17800002,"Qandahar","AFG","Qandahar",237500


inside the command line client

handling

output138

$_

@ @

:-)


pagermysql> SELECT * FROM world.City;

rec …rec …rec …

4000 + recordsunreadable

139


pagermysql> pager lessPAGER set to 'less'mysql> SELECT * FROM world.City;

# [ records in a manageable window ]

mysql> nopagerPAGER set to stdout

# back to normal140


More pager tricksmysql> pager vim -# like 'less', but you can also edit# the results and eventually save # them

141


More pager tricksmysql> pager md5sumPAGER set to 'md5sum'mysql> select * from City;5d17ffa50d6da200dee621823ade2543 -4079 rows in set (0.01 sec)

# produce a CRC for the result set# instead of displaying it.# Useful for data comparison.# (Thanks to Baron Schwartz

142


Comparing data sets with md5summysql> select * from City;5d17ffa50d6da200dee621823ade2543 -

mysql> select * from City order by ID;5d17ffa50d6da200dee621823ade2543 -

mysql> select * from City order by name;5ab980f5b69a6e65d3b8cd5a95ec94e0 -

mysql> select * from City order by District;2bf37cbe91b3ffeaebc8c4b6084b8784 -

143


More pager tricksmysql> pager cat > /tmp/t1.txtmysql> select "one" union select "two";mysql> pager cat > /tmp/t2.txtmysql> select "one" union select "TWO";mysql> nopagermysql> \! vimdiff -o /tmp/t[12].txt

# see the difference between two # datasets

144


145


logging sessionsmysql> tee mylog.txt

# logs everything that you see on# the screen.# your commands, the results,# warnings, error messages, and so on.

mysql> notee

# back to normal146


Updated presentation slides

you will find an up to date copy of these slides at: http://tinyurl.com/shoot-cl-2010







read more http://datacharmer.blogspot.com




THANKS!Questiontime

Comments on Twitter: @datacharmer

?

This work is licensed under the Creative Commons Attribution-Share Alike 3.0 Unported License.


shooting from the hip - perl at the command line for dbas

Technology