experience report and future directions

8/14/2019 Experience Report and Future Directions

http://slidepdf.com/reader/full/experience-report-and-future-directions 1/37



Agenda

• Rationale

•Brief overview

• Select features

• Future directions

• Rampant speculation and provocation

• Q&A



Clojure Objectives

• A Lisp

• Functional

•emphasis on immutability

• Supporting Concurrency

• language-level coordination of state

• Designed for the JVM

• exposes and embraces platform

• I’m a practitioner, Clojure is a tool3



Why the JVM/CLR?

• VMs, not OSes, are the target platforms of futurelanguages, providing:

• Type system

• Dynamic enforcement and safety

• Libraries

• Huge set of facilities

• Memory and other resource management

• GC is platform, not language, facility

•Bytecode + JIT compilation

4



Language as platform vs.

Language + platform• Old way - each language defines its own runtime

•GC, bytecode, type system, libraries etc

• New way (JVM, CLR)

• Common runtime independent of language

•Platforms are dictated by clients

• Huge investments in performance, scalability,security, libraries etc.

5



Clojure is a Different Lisp

• Maps, vectors, sets in addition to lists

• Immutable

• Defined in terms of abstractions

• Lazy and concrete seqs interoperate -no separate streams library

• Core library functions have no side effects

• let-bound locals are immutable

•loop/recur functional looping construct

6



Maps and Sets(def m {:a 1 :b 2 :c 3})

(m :b) -> 2 ;also (:b m)

(keys m) -> (:a :b :c)

(assoc m :d 4 :c 42) -> {:d 4, :a 1, :b 2, :c 42}

(merge-with + m {:a 2 :b 3}) -> {:a 3, :b 5, :c 3}

(union #{:a :b :c} #{:c :d :e}) -> #{:d :a :b :c :e}

(join #{{:a 1 :b 2 :c 3} {:a 1 :b 21 :c 42}}

#{{:a 1 :b 2 :e 5} {:a 1 :b 21 :d 4}})

-> #{{:d 4, :a 1, :b 21, :c 42}

{:a 1, :b 2, :c 3, :e 5}}8



# Norvig’s Spelling Corrector in Python

# http://norvig.com/spell-correct.html

def words(text): return re.findall('[a-z]+', text.lower())

def train(features):

model = collections.defaultdict(lambda: 1)for f in features:

model[f] += 1

return model

NWORDS = train(words(file('big.txt').read()))

alphabet = 'abcdefghijklmnopqrstuvwxyz'

def edits1(word):

n = len(word)

return set([word[0:i]+word[i+1:] for i in range(n)] +

[word[0:i]+word[i+1]+word[i]+word[i+2:] for i in range(n-1)] +

[word[0:i]+c+word[i+1:] for i in range(n) for c in alphabet] +

[word[0:i]+c+word[i:] for i in range(n+1) for c in alphabet])

def known_edits2(word):

return set(e2 for e1 in edits1(word) for e2 in edits1(e1) if e2 in NWORDS)

def known(words): return set(w for w in words if w in NWORDS)

def correct(word):

candidates = known([word]) or known(edits1(word)) or known_edits2(word) or [word]

return max(candidates, key=lambda w: NWORDS[w])



; Norvig’s Spelling Corrector in Clojure

; http://en.wikibooks.org/wiki/Clojure_Programming#Examples

(defn words [text] (re-seq #"[a-z]+" (.toLowerCase text)))

(defn train [features]

(reduce (fn [model f] (assoc model f (inc (get model f 1)))){} features))

(def *nwords* (train (words (slurp "big.txt"))))

(defn edits1 [word]

(let [alphabet "abcdefghijklmnopqrstuvwxyz", n (count word)]

(distinct (concat(for [i (range n)] (str (subs word 0 i) (subs word (inc i))))

(for [i (range (dec n))]

(str (subs word 0 i) (nth word (inc i)) (nth word i) (subs word (+ 2 i))))

(for [i (range n) c alphabet] (str (subs word 0 i) c (subs word (inc i))))

(for [i (range (inc n)) c alphabet] (str (subs word 0 i) c (subs word i)))))))

(defn known [words nwords] (for [w words :when (nwords w)] w))

(defn known-edits2 [word nwords]

(for [e1 (edits1 word) e2 (edits1 e1) :when (nwords e2)] e2))

(defn correct [word nwords]

(let [candidates (or (known [word] nwords) (known (edits1 word) nwords)

(known-edits2 word nwords) [word])]

(apply max-key #(get nwords % 1) candidates)))10

http://en.wikibooks.org/wiki/Clojure_Programming#





Coming to Terms with State

Value

• An immutable magnitude, quantity,

number... or immutablecomposite thereof

Identity

• A putative entity weassociate with a series of causally related values(states) over time

State

• Value of an identity at amoment in time

Time

• Relative before/afterordering of causal values

11



Concurrency Approach

• Programming with values is critical

• Persistent data structures

• Just need to manage the succession of values

(states) of an identity

• A timeline coordination problem

•Several semantics possible

• Managed references

• Variable-like cells with time coordination

semantics



Time constructs

• Need to ensureatomic statesuccession

• Need to providepoint-in-timevalue perception

• Multiple timelinespossible (anddesirable)

• Many implementationstrategies with differentcharacteristics/semantics

• CAS - uncoordinated 1:1

• Agents - uncoordinated,async. (Like actors, but local

and observable)

• STM - coordinated,arbitrary regions

14



Uniform state transition

model• (‘change-state’ reference function [args*])

•function will be passed current state of the

reference (plus any args)

• Return value of function will be the nextstate of the reference

• Snapshot of ‘current’ state always availablewith deref

•No user locking, no deadlocks



16

• 1:1 timeline/identity

• Atomic state succession

• Point-in-time valueperception

F

v2

F

v3

F

v4

vN+1

vNs

vN

AtomicReference

(swap! an-atom f args)

(f vN args) becomes vN+1

- can automate spin

CAS as Time Construct



Atoms

• Manage independent state

• State changes through swap!, using ordinaryfunction (state=>new-state)

• Change occurs synchronously on caller thread

• Models compare-and-set (CAS) spin swap

• Function may be called more than once!

• Guaranteed atomic transition

•Must avoid side-effects!



Atoms in Action

(def foo (atom {:a "fred" :b "ethel" :c 42 :d 17 :e 6}))

@foo -> {:d 17, :a "fred", :b "ethel", :c 42, :e 6}

(swap! foo assoc :a "lucy")

@foo -> {:d 17, :a "lucy", :b "ethel", :c 42, :e 6}



F

v1 v2 v3 v4

v1 v2 v3 v4

v1 v2 v3 v4

v1 v2 v3 v4

F

F

F

F

F

F

F F F

F

F

Transactions

STM as Time Construct



Refs and Transactions

•Software transactional memory system (STM)

• Refs can only be changed within a transaction

• All changes are Atomic and Isolated

• Every change to Refs made within atransaction occurs or none do

• No transaction sees the effects of anyother transaction while it is running

• Transactions are speculative

• Will be retried automatically if conflict

• Must avoid side-effects!



The Clojure STM

• Surround code with (dosync ...), state changesthrough alter/commute, using ordinary function

(state=>new-state)

• Uses Multiversion Concurrency Control (MVCC)

•All reads of Refs will see a consistent snapshot of

the 'Ref world' as of the starting point of thetransaction, + any changes it has made.

• All changes made to Refs during a transaction will

appear to occur at a single point in the timeline.



Refs in action(def foo (ref {:a "fred" :b "ethel" :c 42 :d 17 :e 6}))


(assoc @foo :a "lucy")

-> {:d 17, :a "lucy", :b "ethel", :c 42, :e 6}


(alter foo assoc :a "lucy")

-> IllegalStateException: No transaction running

(dosync (alter foo assoc :a "lucy"))

@foo -> {:d 17, :a "lucy", :b "ethel", :c 42, :e 6}



Implementation - STM

• Not a lock-free spinning optimistic design

• Uses locks, latches to avoid churn

• Deadlock detection + barging

• One timestamp CAS is only global resource

• No read tracking

• Coarse-grained orientation

• Refs + persistent data structures

• Readers don’t impede writers/readers, writers

don’t impede readers, supports commute



STM - commute

• Often a transaction will need to update a jobs-done counter or add its result to a map

• If done with alter, update is a read-modify-

write, so if multiple transactions contend, onewins, one retries

• If transactions don’t care about resulting value,and operation is commutative, can instead usecommute

• Both transactions will succeed without retry

• Always just an optimization



Experiences -

Concurrency Model• Programming with values is more robust

•Approachable - like variables, with semantics

• Uniform state transition model works

• Can easily move from atoms to agents or STM

• Leverages locality

• High performance

•Direct reads



Experiences on the JVM

• Some limitations

• no tail call optimization

•no fixnums

• HotSpot covers the last mile of compilation

• Runtime optimizing compilation

• Ephemeral garbage is very cheap

• Many great facilities

•Verifier, security, dynamic code loading



Benefits of the JVM

• Focus on language vs code generation ormundane libraries

• Sharing GC and type system with

implementation/FFI language is huge benefit

• Tools - e.g. breakpoint/step debugging, profilersetc.

• Libraries! Users can do UI, database, web, XML,graphics, etc right away

• Great MT infrastructure - java.util.concurrent

27



Moving Forward

• Performance

•Parallelism

• Integrated declarative logic

• Crosscutting concerns

• Platform

28



Chunked Sequences

• Being able to view all data structures assequences is useful

• Laziness is critical, esp. not realizing fullintermediate results

• But, per-step overhead

• Allocating/realizing lazy thunk

• Chunked sequences

• Increase the granularity, amortizing work

•Leverage structure of data

29



Transients

• Shares structure with persistent source• O(1) transition from/to persistent data

structures

• Same code structure as functional version

• Capture return value, use for next call

•Don't bash in place

• Not persistent, so you can't hang ontointerim values or alias

•Single-path use, thread isolation enforced

30



Parallelism

• Persistent data structures well suited forparallel ops, reduce, map etc

• Build on Java’s ForkJoin work-stealing

framework

• Ad hoc parallelism with ForkJoin

• Parallel + lazy

• Parallelism often means full realization of intermediate values

•pmap sliding parallel work window

31



Concurrency/Parallelism

Issues• Composability of independent decisions

• e.g. thread pool sizing

• Work unit granularity detection

• especially in libraries

•GCs and runtimes informed by immutability

• better ability to communicate aboutvolatility etc - "I'm done changing this"

• time == garbage32



Integrated declarative logic

• Declarative logic is over there (DB, rulesengine...)

• Too much captured in conditionals/guards

• Integrated, could facilitate:

• Higher-level data manipulation

• Predicate dispatch

• Reasoning about code

•Constraints

33



Combining operators

• A concern of parallelism and concurrency• read-modify-writes and accumulative ops don’t

work well

• Parallel ops and (commuting) transaction commitsbenefit from mergeable operations

• Associative, commutative, conflict resolving,

operational transforms etc

• Will (preduce f collection) or

• (commute ref f) work for some f?

34



Hack the JVM!

• Huge benefits to stratified architecture anddynamic compilation model

• interpretable bytecode, runtime profiling and

compilation, multiple GC models etc• All now open source

• JVM folks could use help with:

• Tail calls, fixnums, continuations...

• Making the runtime language-agnostic

• http://openjdk.java.net/projects/mlvm/35

http://openjdk.java.net/projects/mlvm/





Want more on Clojure?

• Clojure home (docs & more)

• http://clojure.org

•Google Group (2800+ members)

• http://groups.google.com/group/clojure

• Plugins for emacs, Netbeans, IntelliJ, Mathematica

• Books (1 out, 2 more on the way)

• 98 registered contributors

•Great community

36

http://groups.google.com/group/clojure

http://clojure.org/




http://clojure.org/

http://clojure.org/



Thanks for listening!

http://clojure.org

Questions?

http://www.clojure.org/

http://www.clojure.org/

experience report and future directions

Documents