Physics and Machine Learning

“All the tricks that physicists’ use eventually end up in machine

learning”

Energy – Physics definitionsEnergy - A measure of being able to do work. There are many forms of energy, such as heat, mechanical, electrical, radiant, chemical, and nuclear energies. Energy is measured in such units as the joule (J), erg, kilowatt-hour (kW-hr), kilocalorie (kcal), foot-pound (ft-lb.), electron-volt (ev), and British thermal unit (BTU). –NASA.gov

"It is important to realize that in physics today, we have no knowledge of what energy is. We do not have a picture that energy comes in little blobs of a definite amount." -Richard Feynman "Lectures on Physics"

Energy - Physics II

Images: http://meso.phys.northwestern.edu/research/magneticarrays.html

• Magnetic Arrays and Spin

Energy - Machine Learning

• H=-½iJwiJSiSJ Si SJWiJ

Vs.

Vs.

Vs. Vs.

Energy - Machine Learning II

• Energy is the difference in weight between all nodes that agree and all nodes that disagree.

• The more weights, the greater energy. • The “closer” the call, the lower |H|.

Energy Minima

• Retrieval States – attractors

• Mixture States – linear combinations of odd numbered attractors

• Spin Glass States – uncorrelated to attractors.

Ferromagnetics

Energy MetaphorImagine the atomic magnets as movable objects with the freedom to flip, but you control their position. Each iteration of learning is like forcing all magnets to be closer together, as such the network energy is potential energy and the flipping of spins is the expression of kinetic energy.

Temperature – Physics I

• Extending the ferromagnetic example• As temperature increases, the impact of

other atomic magnets’ spins is decreased.• At absolute zero, temperature has no

impact.• At the critical temperature(Tc), spin has no

impact.

Temperature – Ferromagnetics

• Si = +1 w/ probability g(hi); else -1• g(h) = 1/(1+exp(-2βh))• β = 1/(kBT)

• kB = Boltzman’s constant• T = temperature• Fβ(+/-hi)=1/(1+exp(-/+ 2βhi) Fβ(+/-hi)• Fβ(+/-hi) is a logistic function

Temperature – Machine Learning

• Logistic function• Noise• Used in the elimination of spurious

local minima

Mean Field Theory – Physics• The individual measurement and

summation of each member of a magnetic array is too expensive

• Physicists look to average values as an inexpensive way to extract further truth from a complex combinatronics problem.

Mean Field Theory – Physics• hi=JwiJSJ+hext

• <hi>=JwiJ<SJ>+hext

• <Si>=tan(β<hi>) = tanh(βJwiJ<SJ>+hext)• <S>=tanh(βJ(S))

<S>

TcT

1

Mean Field Theory – stochastic model

• <Si>=tanh(β/NJuζuiζu

J<SJ>)• We allow an assumption, that <Si> is

proportional to one of the stored patterns ζv

i • <Si>=m ζv

i

• <Ncorrect>=½N(1+m)

TcT

Mean Field Theory • <Ncorrect>=½N(1+m)• There is a point at which noise

overcomes the ability of a network to make an informed decision.

Conclusions• All these metaphors that pull from

physics are very tightly linked to energy.

• All metaphors concentrate on atomic events. (exception that proves the rule: mean field theory).

Extra Time? Extra Topics!

Entropy – Physics• The inevitable progression toward

chaos• The motion of energy and matter

away from an organized state.

Entropy – Machine Learning• S = -PlogP

• For S = -Plog2P (the binary case) this is the average amount of additional information required to specify one of the staties.

Quantum MechanicsFits within the context of our

expectation for where to look for Physics crossover

• Atomic – discrete and binary• Energy specificLast Class Lecture – use of Dyads.