clojure talk at münster jug

ClojureA small introduction

Alex Ott

http://alexott.net

Munster JUG, July 2010

Alex Ott (alexott@gmail .com) Clojure Munster JUG, July 2010 1 / 40

http://alexott.net

About me...

Professional software developer & architectWorking with different programming languages – C/C++, Lisps,Haskell, Erlang, ...About 20 years of Lisp experience, including commercial projectsDevelopment on Unix/Linux platformsAuthor of articles on programming (including Clojure), Emacs, etc.Participate in many open source projects, including Incanter, labrepl,etc.Maintainer of “Planet Clojure”


Outline

1 What is Clojure?

2 Language basics

3 Interoperability with Java

4 Concurrent programming

5 Clojure in real life

6 Sources of information

7 Examples


What is Clojure?

Lisp-like language, created by Rich Hickey andannounced in 2007thDesigned to work on existing platformsFunctional programming language, withimmutable dataSpecial features for concurrent programmingOpen sourced with liberal licenseAlready used in many projects, includingcommercial


Why new language?

Lisp, but without compatibility with previousversionsImmutable data and more support for purefunctional programming comparing with other LispsSupport for concurrent programing inside languageBetter integration with target platforms:JVM & .Net


Main features

Dynamically typed languageVery simple syntax, like in other Lisps (code as data)Support for interactive developmentDesigned in terms of abstractionsMetaprogrammingMultimethods and protocols (in version 1.2)Compiled into byte code of target platformsAccess to big number of available libraries


Differences from other Lisps

Changes in syntax, less parenthesesChanges in namingMore first-class data structures – maps, sets, vectorsImmutable, by default, dataAbility to link meta-information with variables and functions“Lazy” collectionsThere is no reader macrosCase-sensitive namesLack of tail call optimization (JVM’s limitation) – explicit loopsloop/recur

Exceptions instead of signals and restarts


Base data types

Integer numbers of any size – 14235344564564564564

Rational numbers – 26/7

Real numbers – 1.2345 and BigDecimals – 1.2345M

Strings (String class from Java) – "hello world"

Characters (Character from Java) – \a, \b, . . .Regular expressions – #"[abc]*"

Boolean – true & false

nil – the same as null in Java, not an empty list as in LispSymbols – test, var1, . . .Keywords – :test, :hello, . . .


Data structures

Separate syntax for different collections:Lists – (1 2 3 "abc")Vectors – [1 2 3 "abc"]Maps – {:key1 1234 :key2 "value"}Sets – #{:val1 "text" 1 2 10}

Sequence – abstraction to work with all collections (including classesfrom Java/.Net)Common set of functions to work with sequencesLazy operations on sequencesVectors, maps, and sets are functions – to simplify access to data inthemPersistent collectionsTransient collections – performance optimizationdefstruct – optimization of map to work with complex datastructures


Syntax

Very simple, homoiconic syntax – program is usual data structureSpecial procedure (reader) process text and produce data structuresAll objects represent themselves except symbols and listsSymbols link value with “variable”Lists represent forms, that could be:

Special form – def, let, if, loop, . . .MacrosFunction call or expression, that could be used as function(maps, keywords, returned functional values, . . . )

To get list as-is you need to “quote” it (with quote or ’)


Code example

( de fn f i b o( [ ] ( concat [ 1 1 ] ( f i b o 1 1 ) ) )( [ a b ]

( l e t [ n (+ a b ) ]( l a z y−seq ( cons n ( f i b o b n ) ) ) ) ) )

( take 100000000 ( f i b o ) )

( de fn v range2 [ n ]( l oop [ i 0

v ( t r a n s i e n t [ ] ) ]( i f (< i n )

( r e c u r ( i n c i ) ( con j ! v i ) )( p e r s i s t e n t ! v ) ) ) )


Life cycle of Clojure program

Code is read from file, or REPL, or from other codeReader transforms program into data structuresMacros are expandedResulting code is evaluated/compiledProduced bytecode is executed by JVM

File

Reader

Evaluator/Compiler

Macroexpand

Code

Code Datastructures

Datastructures

Datastructures

JVM

Developer(REPL)

bytecode

Execution


Functions

Functions are first-class objectsFunction can have variable number of argumentsFunction’s definition: defn – top-level functions, fn – anonymousfunctionsSimplified syntax for anonymous functions – #(code). For example:(map #(.toUpperCase %) ["test" "hello"])(map #(vector %1 %2) [1 2 3] [4 5 6])You can specify tests & pre/post-conditions. For example,

( de fn c on s t r a i n e d−s q r [ x ]{ : p r e [ ( pos ? x ) ] : po s t [(> % 16) , (< % 225) ] }(∗ x x ) )

Each function is compiled into separate Java classEach function implements Runnable interface


Metaprogramming & macros

Macros receives source code and returns new code, that will compiledBig part of language is written using macrosVariable number of arguments (same as functions)Code could be generated with list functions, but it’s much handy touse quasi-quote – ‘ and substitution operators – ~ and ~@The # suffix in names is used to generate unique namesmacroexpand-1 & macroexpand are used to debug macros


Macros examples (1)

The standard macros when:

( defmacro when[ t e s t & body ]( l i s t ’ i f t e s t ( cons ’ do body ) ) )

when used as:

(when ( pos ? a )( p r i n t l n " p o s i t i v e ") (/ b a ) )

will expanded into:

( i f ( pos ? a )( do

( p r i n t l n " p o s i t i v e ")(/ b a ) ) )


Macros examples (2)

The standard macros and

( defmacro and( [ ] t r u e )( [ x ] x )( [ x & next ]‘ ( l e t [ and# ~x ]

( i f and# ( and ~@next ) and#))))

will expanded into different code, depending on number of arguments:

use r> ( macroexpand ’ ( and ) ) ==> t r u euse r> ( macroexpand ’ ( and (= 1 2 ) ) ) ==> (= 1 2)use r> ( macroexpand ’ ( and (= 1 2) (= 3 3 ) ) ) ==>( l e t ∗ [ and__4457__auto__ (= 1 2 ) ]

( i f and__4457__auto__( c l o j u r e . co r e /and (= 3 3) )and__4457__auto__) )


Polymorphism & multimethods

ExtensibilityMultimethods aren’t bound exclusively to types/classesDispatching is performed using results of user-specified dispatchfunctionDispatching on several parametersDiffers from CLOS – absence of :before, :after, . . .defmulti – is analog of defgeneric in Common LispDefined as (defmulti func−name dispatch−fn) + set ofimplementations (via defmethod)You can also define hierarchical relationships with derive, and usethem in dispatch (via isa?)


Multimethods examples (1)

Simple dispatching using data type/class:

( d e fmu l t i m−example c l a s s )( defmethod m−example S t r i n g [ t h i s ]

( p r i n t l n "Th i s i s s t r i n g ’" t h i s " ’ " ) )( defmethod m−example j a v a . u t i l . C o l l e c t i o n [ t h i s ]

( p r i n t "Th i s i s c o l l e c t i o n ! " ) )

and we’ll get:

(m−example " He l l o ") ==> "This i s s t r i n g ’ He l l o ’ "(m−example [ 1 2 3 ] ) ==> "This i s c o l l e c t i o n !"(m−example ’ (1 2 3) ) ==> "This i s c o l l e c t i o n !"


Multimethods examples (2)

( d e fmu l t i encoun t e r( fn [ x y ] [ ( : S p e c i e s x ) ( : S p e c i e s y ) ] ) )

( defmethod encoun t e r [ : Bunny : L ion ] [ b l ] : run−away )( defmethod encoun t e r [ : L ion : Bunny ] [ l b ] : e a t )( defmethod encoun t e r [ : L ion : L ion ] [ l 1 l 2 ] : f i g h t )( defmethod encoun t e r [ : Bunny : Bunny ] [ b1 b2 ] : mate )( de f b1 { : Sp e c i e s : Bunny })( de f b2 { : Sp e c i e s : Bunny })( de f l 1 { : S p e c i e s : L ion })( de f l 2 { : S p e c i e s : L ion })

( encoun t e r b1 b2 ) ==> : mate( encoun t e r b1 l 1 ) ==> : run−away( encoun t e r l 1 b1 ) ==> : ea t( encoun t e r l 1 l 2 ) ==> : f i g h t


Protocols & Datatypes

Introduced in version 1.2 (will released shortly)Much faster than multimethodsAllows to write “Clojure in Clojure”Dispatching is done only on data typesSimilar to type classes in HaskellJava interface is created for each protocoldefrecord & deftype define new data typesextend-protocol & extend-type bind protocol(s) with data types(not only with defined in Clojure!)reify is used to implement protocols and interfaces for “once-usedtypes”


Protocol’s examples

( d e f p r o t o c o l H e l l o "Test o f p r o t o c o l "( h e l l o [ t h i s ] " h e l l o f u n c t i o n " ) )

( d e f r e c o r d B [ name ] He l l o( h e l l o [ t h i s ] ( s t r " He l l o " ( : name t h i s ) " ! " ) ) )

( h e l l o (B . "User " ) ) ==> "He l l o User !"

( extend−p r o t o c o l H e l l o S t r i n g( h e l l o [ t h i s ] ( s t r " He l l o " t h i s " ! " ) ) )

( h e l l o " wor ld ") ==> "He l l o wor ld !"

( extend−p r o t o c o l H e l l o j a v a . l ang . Object( h e l l o [ t h i s ] ( s t r " He l l o ’" t h i s

" ’ ! (" ( type t h i s ) " ) " ) ) )( h e l l o 1) ==> "He l l o ’ 1 ’ ! ( c l a s s j a v a . l ang . I n t e g e r )"


Miscellaneous

Metadata – #^{} in 1.0 & 1.1, or simply ^{} in 1.2Optional type specification (type hints) – #^Type or ^TypeYou can specify tests directly in function’s declarationScope managementAccess and change of metadata from codeDon’t change value equality

Namespaces:are first-class objectsare used to organize code into libraries

Data destructuring (function arguments, etc.)

( l e t [ [ a b & c : as e ] [ 1 2 3 4 ] ] [ a b c e ] )==> [1 2 (3 4) [ 1 2 3 4 ] ]

( l e t [ { : key s [ a b c ] : as m : or {b 3}} { : a 5 : c 6} ][ a b c m] ) ==> [5 3 6 { : a 5 : c 6} ]


Interoperability with Java

Two-way interoperability with target platform:Creation of instances – new or Class.Call of Java methods – ., .., dotoClass & interface generation – gen-class, gen-interface, or proxy(anonymous class)

You can execute Clojure code in Java programsSeparate functions to work with Java arrays:

make-array – array creationaget/aset – access to array’s elementsamap/areduce – iteration over array’s elements

memfn allows to use Java methods as arguments of map, filter, etc.The set! special form to set values inside classGeneration and catch of exception – throw & catch


Interoperability examples

Instance creation:

( new j a v a . u t i l . Date ) <==> ( j a v a . u t i l . Date . )

Call of Java methods and access to data:

( . s u b s t r i n g " He l l o World" 0 5) ==> "He l l o "( . " He l l o World" s u b s t r i n g 0 5) ==> "He l l o "Math/PI ==> 3.141592653589793( I n t e g e r / p a r s e I n t "42") ==> 42

.. is used to chained calls:

( . . System g e t P r o p e r t i e s ( ge t " os . name ") ) ==> "Mac OS X"System . g e t P r o p e r t i e s ( ) . ge t (" os . name")

doto – call of several methods for one object:

( doto ( j a v a . u t i l . HashMap . )( . put "a" 1) ( . put "b" 2) )


Concurrent programming

Special language features, that provides data mutation:Refs – synchronous, coordinated changeAgents – asynchronous, not coordinatedAtoms – synchronous, not coordinatedVars – local for current thread@ (deref) is used to get access to data

Parallel code executionfuturepmap, pvalues & pcallsNative threads

Synchronization with promise


State and Identity

Clojure separates concepts of state and identityState (value) isn’t changed!More information at http://clojure.org/state

Identity(ref/agent/atom

name)

Code(past)

Code(present)

Code(future)

State S1[1 2 3 4]

State S2[1 2 3 4 5 6 7]

State S3[?????]


http://clojure.org/state

Refs & STM

Coordinated change of data from several threadsSoftware Transaction Memory provides atomicity, consistency, andisolationChanges could be made only inside transactionYou can add function for data validationAbility to add watcher function

( de f c oun t e r s ( r e f {}) )( de fn get−coun t e r [ key ]

( @counte r s key 0 ) )( de fn inc rement−coun t e r [ key ]

( dosync( a l t e r c oun t e r s a s s o c key

( i n c ( @counte r s key 0 ) ) ) ) )


Agents

Asynchronous change of data – “fire and forget”Thread pools for data update functions: send – for “fast” functions,and send-off – for “heavyweight” functions (separate thread pool)send & send-off get function that will applied to agent’s currentstateValidators and watchersYou can wait until finish of all updates

( de f a c oun t e r s ( agent {}) )( de fn inc rement−acoun t e r [ key ]

( send a coun t e r s a s s o c key( i n c ( @counte r s key 0 ) ) ) )

( de fn get−acoun t e r [ key ]( @acounte r s key 0 ) )


Vars & Atoms

AtomsSynchronous data change without coordinationData are changed by applying function to current state of atomFunction could be called several timesFunction shouldn’t have side effects!

VarsProvide data change only in current threadbinding links new values with symbolsChange affects all functions, called from current threadBe careful with lazy sequences!

( de f ∗ va r ∗ 5)( de fn foo [ ] ∗ va r ∗)( foo ) ==> 5( b i n d i n g [∗ va r ∗ 10 ] ( foo ) ) ==> 10


Parallel code execution

futureexecutes given code in separate thread@ is used to get results of code execution@ blocks current thread, if results aren’t available yetresults of code execution are cached

promiseis used for synchronization between parts of programsdeliver sets value of promise@ reads value, or blocks execution if value isn’t set yet

pmap, pvalues & pcalls are used for “heavyweight” operations, thatcould be performed in parallel


Clojure in real life

Existing infrastructure:IDEsBuild toolsDebugging and related toolsLibrariesCode repositories

Clojure/core – commercial support & consultingClojure is used in commercial projects since 2008Sources of information


Infrastructure: IDEs and build tools

Supported by most of popular IDEs:Emacs + SLIME (most popular now)VimClojureNetBeansEclipseIntelliJ IDEA

Build tools:Clojure support in Maven, Ant and GradleLeiningen – written in Clojure, extensible and very simple


Infrastructure: libraries & repositories

Simple access to big number of existing Java librariesClojure-specific libraries:

Clojure-Contrib – “semi-standard” librariesCompojure, Ring, Scriptjure – Web developmentClojureQL – databasesIncanter – R-like environment and librariesother – see at http://clojars.org

Repositories:build.clojure.orgclojars.org


http://clojars.org

Commercial projects

FlightCaster – machine learning, etc.Relevance, Inc. – consultingRuna – marketing servicesSonian Networks – archiving solutionsBackType – social media analyticsDRW Trading Group – trading and financeDocuHarvest project by Snowtide Informatics Systems, Inc.ThorTech Solutions – scalable, mission-critical solutionsTheDeadline project by freiheit.com (http://www.freiheit.com/)and many more. . .


http://www.freiheit.com/

Sources of information (1)

Main sites:Site of the language – http://clojure.orgPlanet Clojure – http://planet.clojure.inthe #clojure IRC channel at freenode.netThe labrepl project (http://github.com/relevance/labrepl) –learning environmentTry-Clojure (http://www.try-clojure.org/) – you can executeClojure code via Web-browserhttp://clojuredocs.org/ – documentation and examples

German resources:Book in German will released in 2010 fall(http://www.clojure-buch.de/)clojure-de mailing list(http://groups.google.com/group/clojure-de)videos about Clojure in German (http://www.rheinjug.de/videos/gse.lectures.app/Talk.html#Clojure)


http://clojure.org

http://planet.clojure.in

http://github.com/relevance/labrepl

http://www.try-clojure.org/

http://clojuredocs.org/

http://www.clojure-buch.de/

http://groups.google.com/group/clojure-de

http://www.rheinjug.de/videos/gse.lectures.app/Talk.html#Clojure

http://www.rheinjug.de/videos/gse.lectures.app/Talk.html#Clojure

Sources of information (2)

Books:Programming Clojure – 2009th, Clojure v. 1.0Practical Clojure. The Definitive Guide – 2010th, Clojure, v. 1.2The Joy of Clojure (beta)Clojure in Action (beta)Clojure Programming on WikibooksClojure Notes on RubyLearning

Video-lectures & screencastsClojure user groups around the worldClojure study courses (on-site in USA & Europe, or online)


Examples. Hadoop

Hadoop word count example is only lines of code (usingclojure-hadoop), instead dozens for Java...All low-level details are handled by macros

( ns c l o j u r e−hadoop . examples . wordcount3( : impor t ( j a v a . u t i l S t r i n gTok e n i z e r ) ) )

( de fn my−map [ key v a l u e ](map ( fn [ token ] [ token 1 ] )

( enumerat ion−seq ( S t r i n gTok e n i z e r . v a l u e ) ) ) )

( de fn my−r educe [ key va l u e s−f n ][ [ key ( r educe + ( va l u e s−f n ) ) ] ] )


Examples. Cascalog

Domain specific language to query data in Hadoop with joins,aggregates, custom operations, subqueries, etc.Interactive work with data, using ClojureAvailable from http://github.com/nathanmarz/cascalog

Example: query for persons younger than 30:

(?<− ( s t dou t ) [ ? pe r son ? age ] ( age ? pe r son ? age )(< ? age 30) )


http://github.com/nathanmarz/cascalog

Examples. Concurrency

Ants:320 lines of code, with commentaries and documentationMassive parallel execution without explicit threadingMany independent objectsGraphical interface

Cluster analysis with K-means algorithm:Each cluster represented by agentAutomatically uses all available coreshttp://www.informatik.uni-ulm.de/ni/staff/HKestler/Reisensburg2009/talks/kraus.pdf


http://www.informatik.uni-ulm.de/ni/staff/HKestler/Reisensburg2009/talks/kraus.pdf

http://www.informatik.uni-ulm.de/ni/staff/HKestler/Reisensburg2009/talks/kraus.pdf

Thank you

Questions


clojure talk at münster jug

Technology

u alex ott alexott

netalex ott alexott

restartsalex ott alexott

com clojure mnster jug

comclojuremnster jug

data support

data ability

previousversionsimmutable