graphical models - oxford statisticssteffen/teaching/grad/graphicalmodels.pdf · google gives 7 420...
Post on 20-Mar-2020
8 Views
Preview:
TRANSCRIPT
Genesis and historyExamples
Markov theoryComplex models
References
Graphical Models
Steffen Lauritzen, University of Oxford
Graduate Lectures Hilary Term 2011
January 27, 2011
Steffen Lauritzen, University of Oxford Graphical Models
Genesis and historyExamples
Markov theoryComplex models
References
I Precursors originate mostly from Physics (Gibbs, 1902),Genetics (Wright, 1921, 1934), and Economics (Wold, 1954);
I Early graphical models in statistics include covarianceselection models (Dempster, 1972) and log-linear models(Haberman, 1974);
I Papers setting the scene include Darroch et al. (1980),Wermuth and Lauritzen (1983), and Lauritzen and Wermuth(1989).
I Subject took off after Pearl (1988) and Lauritzen andSpiegelhalter (1988), and in particular after Whittaker (1990)and Lauritzen (1996).
I Developments now prolific and it is largely impossible to keeptrack. Google gives 7 420 000 hits.
Steffen Lauritzen, University of Oxford Graphical Models
Genesis and historyExamples
Markov theoryComplex models
References
I Precursors originate mostly from Physics (Gibbs, 1902),Genetics (Wright, 1921, 1934), and Economics (Wold, 1954);
I Early graphical models in statistics include covarianceselection models (Dempster, 1972) and log-linear models(Haberman, 1974);
I Papers setting the scene include Darroch et al. (1980),Wermuth and Lauritzen (1983), and Lauritzen and Wermuth(1989).
I Subject took off after Pearl (1988) and Lauritzen andSpiegelhalter (1988), and in particular after Whittaker (1990)and Lauritzen (1996).
I Developments now prolific and it is largely impossible to keeptrack. Google gives 7 420 000 hits.
Steffen Lauritzen, University of Oxford Graphical Models
Genesis and historyExamples
Markov theoryComplex models
References
I Precursors originate mostly from Physics (Gibbs, 1902),Genetics (Wright, 1921, 1934), and Economics (Wold, 1954);
I Early graphical models in statistics include covarianceselection models (Dempster, 1972) and log-linear models(Haberman, 1974);
I Papers setting the scene include Darroch et al. (1980),Wermuth and Lauritzen (1983), and Lauritzen and Wermuth(1989).
I Subject took off after Pearl (1988) and Lauritzen andSpiegelhalter (1988), and in particular after Whittaker (1990)and Lauritzen (1996).
I Developments now prolific and it is largely impossible to keeptrack. Google gives 7 420 000 hits.
Steffen Lauritzen, University of Oxford Graphical Models
Genesis and historyExamples
Markov theoryComplex models
References
I Precursors originate mostly from Physics (Gibbs, 1902),Genetics (Wright, 1921, 1934), and Economics (Wold, 1954);
I Early graphical models in statistics include covarianceselection models (Dempster, 1972) and log-linear models(Haberman, 1974);
I Papers setting the scene include Darroch et al. (1980),Wermuth and Lauritzen (1983), and Lauritzen and Wermuth(1989).
I Subject took off after Pearl (1988) and Lauritzen andSpiegelhalter (1988), and in particular after Whittaker (1990)and Lauritzen (1996).
I Developments now prolific and it is largely impossible to keeptrack. Google gives 7 420 000 hits.
Steffen Lauritzen, University of Oxford Graphical Models
Genesis and historyExamples
Markov theoryComplex models
References
I Precursors originate mostly from Physics (Gibbs, 1902),Genetics (Wright, 1921, 1934), and Economics (Wold, 1954);
I Early graphical models in statistics include covarianceselection models (Dempster, 1972) and log-linear models(Haberman, 1974);
I Papers setting the scene include Darroch et al. (1980),Wermuth and Lauritzen (1983), and Lauritzen and Wermuth(1989).
I Subject took off after Pearl (1988) and Lauritzen andSpiegelhalter (1988), and in particular after Whittaker (1990)and Lauritzen (1996).
I Developments now prolific and it is largely impossible to keeptrack. Google gives 7 420 000 hits.
Steffen Lauritzen, University of Oxford Graphical Models
Genesis and historyExamples
Markov theoryComplex models
References
A directed graphical model
Directed graphical model (Bayesian network) showing relationsbetween risk factors, diseases, and symptoms.
Steffen Lauritzen, University of Oxford Graphical Models
Genesis and historyExamples
Markov theoryComplex models
References
A pedigree
Graphical model for a pedigree from study of Werner’s syndrome.Each node is itself a graphical model.
Steffen Lauritzen, University of Oxford Graphical Models
Genesis and historyExamples
Markov theoryComplex models
References
A large pedigree
p1379p1380 p1381 p1382 p1383p1384p1385p1386p1387
p1129p1128 p1133p1134p1135 p1136p1137
p1536p1535
p1131p1408p1407p1406p1405p1404
p1126
p1396 p1395p1394 p1393p1392 p1391p1390p1389p1388
p1127
p1426 p1425p1424p1423
p1130
p1565p1564p1530
p1132
p1073
p1457p1456p1455 p1454
p1078
p1349
p1079
p1226p1225 p1224p1223 p1222p1221p1220
p1076
p1242 p1241p1240p1239 p1238p1237
p1561
p1236
p1575
p1235
p1499
p1234
p1480p1511 p1534
p1233
p1219
p1074
p1376p1375 p1374 p1373p1372 p1371p1370 p1369p1368
p1077
p1232 p1231p1230 p1229p1228p1227
p1072
p1438p1437p1436 p1435 p1434 p1439
p1080
p1362p1361 p1360
p1190
p1449p1450p1451 p1452p1453
p1187
p1427 p1428p1429 p1430p1431 p1432p1433
p1184 p1189p1188p1186p1185
p1075
p927p928p929
p331
p544p543p542
p328
p1585 p657p659p660
p1556p1555 p1557
p767
p1514
p1568
p765
p875p876p877
p766
p661p662
p745
p842
p1367 p1366p1365 p1364p1363
p746p747
p658
p506
p744p742p743
p512
p712p709p711
p784 p783p782 p781p780p779p778
p704
p864p843
p706
p856p855
p710
p869 p868p870
p708
p1465p1443p1442
p707
p1422
p705
p511
p760p701p702
p1576
p773p879
p772 p774p775 p776p777
p1572 p1571
p840
p703
p507
p716
p509p510p508
p505
p1170
p1573
p1167
p1169p1168 p1166p1165p1163p1162 p1161
p1554p1553
p1164
p964
p1463p1464
p963
p965
p644p645
p504
p571 p572
p1488 p1487p1486
p1173
p1569
p1174 p1171p1175p1176p1177 p1172
p1099
p1183
p1096
p1110 p1109p1108 p1107
p1547p1533
p1296 p1295p1297 p1298p1299 p1300p1301p1302 p1303
p1105
p1350 p1351p1352
p1102p1106p1104 p1103
p1097 p1098p1100p1101
p930 p931
p1591
p1603 p1602p1601 p1600p1599 p1598p1597p1596p1595 p1594 p1604
p1590 p1589
p1586
p1592
p1587
p932p933
p503
p327
p541
p1460p1377
p1581p1580
p1461 p1378
p1081p1082
p1044p1045
p540p937 p621p622
p750 p749
p625
p624 p759
p1093 p1091p1090p1089
p1322p1321p1320p1319 p1318 p1317p1316p1315 p1314p1313p1323
p1092
p623p626
p1083
p934 p935p936 p606
p538
p576
p539
p329p330
p70p51
p34
p1070
p1459 p1094
p1462
p1095
p1071
p1119p1118
p1479
p1117 p1116p1115p1113 p1112
p1359p1358p1357p1356p1355 p1354p1353
p1114
p1532p1542
p1120
p1032
p1159p1158 p1157
p1475p1503p1502 p1501p1500
p1155
p1515 p1543p1485
p1154
p1513p1469
p1153
p1545p1546
p1156p1152 p1151p1160
p1033p1034
p901
p959 p958 p957p956p955
p1069
p1584
p1068
p1468
p1067
p1328
p1063p1066
p1440 p1441
p1065p1064 p1061p1060
p1348 p1347 p1346p1345 p1344 p1343p1342 p1341p1340 p1339
p1062
p954
p1562
p1247
p1484
p1245
p1560 p1559
p1246 p1248p1249p1250 p1251p1252p1253 p1254
p1087p1084 p1085
p1255p1256p1257 p1258 p1259p1260 p1261 p1262 p1263p1264
p1086 p1088
p949 p953 p952p951
p1327 p1324p1325p1326
p1001p1002 p999
p1286
p1507 p1531
p1287p1288p1289 p1290p1291p1292p1293p1294
p998 p997
p1312
p1548p1549
p1304
p1582
p1306p1311 p1310p1309p1308 p1307 p1305
p1000
p1409p1410p1411 p1412 p1413p1414p1415 p1416 p1467
p1148
p1150p1403 p1400 p1401p1402
p1149
p950
p902
p1040 p1031p1041
p903p904p906
p1039p1038 p1037p1036p1035
p905
p263
p198 p26
p440 p441
p647 p646
p442
p231
p228
p446
p730
p513
p1211p1212
p1570p1574
p1210 p1213 p1214 p1215p1216p1217
p519
p518 p517 p516p515p514
p445
p600
p599
p748
p601
p602
p444
p1042
p1147
p1397 p1398p1399
p1141p1142
p1498p1497 p1496p1466
p1143
p1506 p1524p1523p1522
p1144
p1541p1505p1504 p1478p1474
p1145
p1567p1544
p1146
p1043
p892
p443p447
p227
p269
p960
p1218
p961p962
p268p632
p751
p1420p1419 p1418 p1417p1421
p752
p633
p634
p264
p620
p1593
p757
p756
p796 p795p794
p753
p854p848
p754
p755
p619p618
p267 p266
p582
p265
p226
p966p968
p1284p1283 p1282p1281p1280p1279 p1278p1277p1276p1275 p1274 p1273p1285
p969
p970 p971
p972
p973
p1209 p1208p1207 p1206p1205p1204p1203
p1566
p1202
p1470p1471p1472
p1200p1198
p1540 p1539p1538p1537
p1199
p1516p1517 p1518p1519
p1201
p967
p888
p229
p230
p5p6
p907 p908p909p910p911 p912
p560 p562
p570p569
p561p563
p106 p340
p181
p439
p438
p1550 p1551 p1552
p736
p731
p732 p846 p803p802 p801 p800 p799p798p797
p733p734p735p737
p536p594
p847
p652
p651
p650
p648 p649
p810 p809p808p807p806 p805p804
p653
p593
p1003
p764p763p762
p861p862
p761
p729
p595 p596p597p598
p437
p881 p880
p436
p526
p434
p1028
p1448p1447 p1446p1445p1444
p1029p1030
p476
p700p699 p698p697
p849
p696
p818p819p820 p821p822p823 p824 p825
p695
p865p872p878
p694 p693p689 p690 p692
p811p812 p813p814 p815p816p817
p691
p683 p1022p1024 p1025
p1528p1527p1526p1525 p1529
p1179
p1583 p1563
p1180p1181
p1178
p1023
p1243 p1244
p1473
p1140 p1138
p1489p1476
p1139
p1026p1027
p919p921 p920p887 p886
p435
p284
p581p580p579p578
p341 p342
p49 p183p182
p73 p346
p52 p185
p283
p259
p564
p260
p395
p399
p677
p758
p676
p397
p720 p722p721p719 p718 p717
p398
p396
p549
p257
p258
p394
p845
p672p667 p668 p671
p669 p670p866p857
p673 p674
p675
p391
p389p1111
p390p393p392
p261
p262
p27p28
p252
p474
p640 p643
p826p827p828 p829p830 p831 p832p839
p642p641 p639
p256
p410
p400
p412
p401
p414p413 p411
p402p403p404 p405 p406p407p408 p409
p387p386 p385
p489p490
p383 p384
p1059p1057 p1058
p388
p251 p254 p255
p863
p841p792 p791p790p789p788p787 p786p785
p738
p853p852 p851 p850
p739
p873 p874p867
p740
p502
p1579
p1056p1050
p1048 p1049 p1051
p1495
p1052p1053p1054
p1578 p1577
p1055
p501
p678
p793
p682
p681
p680p679
p500
p498p499
p253
p241
p475
p242
p943 p974p975p976p977 p978p979
p890
p1182p980p981p982 p983p984 p985
p889 p473
p1007p1008 p1009 p1010 p1011 p1012
p915p917 p918
p1021
p916
p461
p1013
p891
p243
p15 p16
p535
p306
p532
p713
p556p714
p715
p531p530p529p528p527
p304
p728p726
p1477p1490
p1193
p1491p1492 p1493p1494
p1192
p1191 p741p1194
p727
p555
p305
p36 p37
p148
p525
p3
p431
p885p884
p429
p883p882
p430
p638
p1046 p1047
p637 p635p636
p428
p1020 p1018p1016
p1512
p1458 p1197
p1017
p1019
p432
p433
p274p275
p84p85
p497
p858p844
p836
p496
p838
p688 p687 p686 p685p684
p491
p723
p494p495 p493 p492
p272
p419p416 p418p417
p420
p273
p8
p1014 p1015
p893
p995 p994
p1483p1482 p1481
p1196p1195
p996
p894
p895 p896
p232 p449
p603 p605
p604
p448
p382
p768 p769p770p771
p381
p380
p379
p378 p630p629p628 p627
p376
p617 p612p614
p1329 p1330p1331p1332 p1333 p1334p1335p1336p1337 p1338
p613
p871
p616
p615
p611
p377
p233
p450
p2
p270p271
p427
p860 p859
p837
p725 p724
p533 p534
p426
p554p553
p415
p278
p41
p577 p458p456 p454p455p453 p460p459 p457
p336p337
p42p56
p551p550 p552
p50p314
p315
p98p99
p900 p899 p897p898
p573
p942
p574 p575
p566
p941
p944
p939p940
p565p567 p568
p296p297
p92 p93
p343
p352p351 p350
p344
p104 p105
p76
p246 p349 p472p471p470p469
p468
p466
p609
p610
p607p608
p467 p465
p244
p425 p421
p423 p424 p422
p245p247
p17 p18
p1
p520
p656 p655
p664
p663
p665
p835p834 p833
p666
p654
p521
p631p922p923
p546p548 p547 p545 p926 p925p924
p522
p279
p79 p80
p299
p39p40
p477p478
p479
p480
p287
p312
p353
p311
p332 p333p523 p524
p310
p288
p89 p90
p43
p187 p55 p67
p174
p69
p1004
p1005
p1006
p913
p1125
p1124 p1123p1122
p1121
p914
p145 p463
p464
p462
p236 p234p235
p237
p61 p62
p11 p7
p83
p298
p33p178
p206
p250
p22p249p248
p19 p20
p286
p176 p285
p483 p482p481
p291 p307
p308
p309
p29 p97
p150
p159
p127
p177
p25
p301p300
p94p95
p316
p317 p318
p100
p488p487 p486p485
p484
p335p334
p292 p293p294p295
p68 p91
p31
p320p345 p451
p452
p10p282
p59 p60
p96
p125
p363
p141 p155p24
p23
p146
p120p110 p132p130
p4p12
p47
p240
p339p290 p289
p238p239
p13p14
p180p38p321
p48 p175
p144
p211 p375
p374
p369 p588
p1520p1521
p586
p1508p1509 p1510
p584
p1558
p945
p585
p368
p151 p152
p195
p372
p592
p591p589
p590
p373
p170p171
p168p364
p142
p172 p202
p131p163
p169
p118
p107 p122p119
p989
p1272p1271 p1270p1269 p1268p1267 p1266 p1265
p986 p987
p990
p991p992p993
p1614p1613 p1612p1611 p1610p1609p1608p1607p1606 p1605
p988
p938
p143 p537
p46p280 p281
p87 p88
p276
p277
p86 p72
p203
p116 p123p109
p138
p164
p200
p225
p192
p947 p1588
p946p948
p139p587
p193 p365
p367
p366
p153p154
p194
p189
p147
p124p219
p191
p126p128 p583
p362
p190p137
p188
p165 p157
p173
p114 p160
p149 p207 p112 p224
p319
p35
p9
p121
p204 p201
p216 p166
p133
p208
p220p217 p371
p370
p134 p135
p108
p117
p162
p111
p221
p115
p158
p205
p167
p215p213
p196
p197
p223
p214p209
p179
p113p212 p222p129 p210p161
p184
p103
p156
p218
p557p558 p559
p313
p44 p199
p53
p140
p82
p303 p302
p21 p30
p45 p136
p322p323p324
p63 p64
p347
p325p326
p101 p102
p74p338
p65p66
p348
p75p186
p359p360p361
p354p355 p356p357p358
p77p78
p81
Family relationship of 1641 members of Greenland Eskimopopulation.
Steffen Lauritzen, University of Oxford Graphical Models
Genesis and historyExamples
Markov theoryComplex models
References
Conditional IndependenceUndirected graphsDirected acyclic graphsMoralization
Random variables X and Y are conditionally independent given therandom variable Z if
L(X |Y ,Z ) = L(X |Z ).
We then write X ⊥⊥Y |Z (or X ⊥⊥P Y |Z )Intuitively:Knowing Z renders Y irrelevant for predicting X .Factorisation of densities:
X ⊥⊥Y |Z ⇐⇒ f (x , y , z)f (z) = f (x , z)f (y , z)
⇐⇒ ∃a, b : f (x , y , z) = a(x , z)b(y , z).
Steffen Lauritzen, University of Oxford Graphical Models
Genesis and historyExamples
Markov theoryComplex models
References
Conditional IndependenceUndirected graphsDirected acyclic graphsMoralization
Fundamental properties
For random variables X , Y , Z , and W it holds
(C1) If X ⊥⊥Y |Z then Y ⊥⊥X |Z ;
(C2) If X ⊥⊥Y |Z and U = g(Y ), then X ⊥⊥U |Z ;
(C3) If X ⊥⊥Y |Z and U = g(Y ), then X ⊥⊥Y | (Z ,U);
(C4) If X ⊥⊥Y |Z and X ⊥⊥W | (Y ,Z ), thenX ⊥⊥ (Y ,W ) |Z ;
If density w.r.t. product measure f (x , y , z ,w) > 0 also
(C5) If X ⊥⊥Y | (Z ,W ) and X ⊥⊥Z | (Y ,W ) thenX ⊥⊥ (Y ,Z ) |W .
Steffen Lauritzen, University of Oxford Graphical Models
Genesis and historyExamples
Markov theoryComplex models
References
Conditional IndependenceUndirected graphsDirected acyclic graphsMoralization
A distribution P is said to factorize w.r.t. and undirected graph ifits joint density f can be written as
f (x) = Z−1∏A∈A
φA(xA), (1)
where A are complete subsets of the graph.
Here x = (xv , v ∈ V ), xA = (xv , v ∈ A) so φA only depends theA-coordinates of x .
The factorization is matched by a global Markov property, ie thatA⊥⊥B | S if S separates A from B in G, written as A⊥G B | S(Hammersley and Clifford, 1971).
Steffen Lauritzen, University of Oxford Graphical Models
Genesis and historyExamples
Markov theoryComplex models
References
Conditional IndependenceUndirected graphsDirected acyclic graphsMoralization
Factorization example
3 6
1 5 7
2 4
u uu u u
u u@@@
���
@@@
@@@
@@@
���
���
The graph above corresponds to a factorization as
f (x) = ψ12(x1, x2)ψ13(x1, x3)ψ24(x2, x4)ψ25(x2, x5)
× ψ356(x3, x5, x6)ψ47(x4, x7)ψ567(x5, x6, x7).
Steffen Lauritzen, University of Oxford Graphical Models
Genesis and historyExamples
Markov theoryComplex models
References
Conditional IndependenceUndirected graphsDirected acyclic graphsMoralization
Global Markov property
3 6
1 5 7
2 4
u uu u u
u u@@@
���
@@@
@@@
@@@
���
���
To find conditional independence relations, one should look forseparating sets, such as {2, 3}, {4, 5, 6}, or {2, 5, 6}For example, it follows that 1⊥⊥ 7 | {2, 5, 6} and 2⊥⊥ 6 | {3, 4, 5}.
Steffen Lauritzen, University of Oxford Graphical Models
Genesis and historyExamples
Markov theoryComplex models
References
Conditional IndependenceUndirected graphsDirected acyclic graphsMoralization
Pairwise and local Markov properties
G = (V ,E ) simple undirected graph; A distribution P satisfies
(P) the pairwise Markov property if
α 6∼ β ⇒ α⊥⊥P β |V \ {α, β};
(L) the local Markov property if
∀α ∈ V : α⊥⊥P V \ cl(α) | bd(α);
Steffen Lauritzen, University of Oxford Graphical Models
Genesis and historyExamples
Markov theoryComplex models
References
Conditional IndependenceUndirected graphsDirected acyclic graphsMoralization
Pairwise Markov property
3 6
1 5 7
2 4
u uu u u
u u@@@
���
@@@
@@@
@@@
���
���
Any non-adjacent pair of random variables are conditionallyindependent given the remaning.For example, 1⊥⊥ 5 | {2, 3, 4, 6, 7} and 4⊥⊥ 6 | {1, 2, 3, 5, 7}.
Steffen Lauritzen, University of Oxford Graphical Models
Genesis and historyExamples
Markov theoryComplex models
References
Conditional IndependenceUndirected graphsDirected acyclic graphsMoralization
Local Markov property
3 6
1 5 7
2 4
u uu u u
u u@@@
���
@@@
@@@
@@@
���
���
Every variable is conditionally independent of the remaining, givenits neighbours.For example, 5⊥⊥{1, 4} | {2, 3, 6, 7} and 7⊥⊥{1, 2, 3} | {4, 5, 6}.
Steffen Lauritzen, University of Oxford Graphical Models
Genesis and historyExamples
Markov theoryComplex models
References
Conditional IndependenceUndirected graphsDirected acyclic graphsMoralization
Let (F) denote the property that f factorizes w.r.t. G and let (G),(L) and (P) denote the Markov properties as defined. Then itholds that
(F)⇒ (G)⇒ (L)⇒ (P).
All reverse implications are false in general.
If f (x) > 0 for all x it further holds that
(P)⇒ (F)
so then(F) ⇐⇒ (G) ⇐⇒ (L) ⇐⇒ (P)
(Lauritzen, 1996, Chap. 3).
Steffen Lauritzen, University of Oxford Graphical Models
Genesis and historyExamples
Markov theoryComplex models
References
Conditional IndependenceUndirected graphsDirected acyclic graphsMoralization
A probability distribution P over X = XV factorizes over a DAG Dif its density or probability mass function f has the form
f (x) =∏v∈V
fv (xv | xpa(v)).
A well-known example is a Markov chain:
X1 X2 X3 X4 X5
s s ss s- - - - - sXn
with Xi+1⊥⊥ (X1, . . . ,Xi−1) |Xi for i = 3, . . . , n.
Steffen Lauritzen, University of Oxford Graphical Models
Genesis and historyExamples
Markov theoryComplex models
References
Conditional IndependenceUndirected graphsDirected acyclic graphsMoralization
Example of DAG factorization
3 6
1 5 7
2 4
u uu u u
u u
-
@@@R
����
@@@R
-
@@@R
@@@R
����
����
-
The above graph corresponds to the factorization
f (x) = f (x1)f (x2 | x1)f (x3 | x1)f (x4 | x2)
× f (x5 | x2, x3)f (x6 | x3, x5)f (x7 | x4, x5, x6).
Steffen Lauritzen, University of Oxford Graphical Models
Genesis and historyExamples
Markov theoryComplex models
References
Conditional IndependenceUndirected graphsDirected acyclic graphsMoralization
Local directed Markov property
A distribution P satisfies the local Markov property (L) w.r.t. adirected acyclic graph D if
∀α ∈ V : α⊥⊥P {nd(α) \ pa(α)} | pa(α).
Here nd(α) are the non-descendants of α.
Steffen Lauritzen, University of Oxford Graphical Models
Genesis and historyExamples
Markov theoryComplex models
References
Conditional IndependenceUndirected graphsDirected acyclic graphsMoralization
Local directed Markov property
3 6
1 5 7
2 4
u uu u u
u u
-
@@@R
����
@@@R
-
@@@R
@@@R
����
����
-
For example, the local Markov property says4⊥⊥{1, 3, 5, 6} | 2,5⊥⊥{1, 4} | {2, 3}3⊥⊥{2, 4} | 1.
Steffen Lauritzen, University of Oxford Graphical Models
Genesis and historyExamples
Markov theoryComplex models
References
Conditional IndependenceUndirected graphsDirected acyclic graphsMoralization
A distribution satisfies the global Markov property w.r.t. D if
A⊥D B |S ⇒ A⊥⊥B |S .
Here ⊥D is d-separation, which is somewhat subtle.
It is always true for a DAG that
(F) ⇐⇒ (G) ⇐⇒ (L)
(Pearl, 1986; Geiger and Pearl, 1990; Lauritzen et al., 1990).
Steffen Lauritzen, University of Oxford Graphical Models
Genesis and historyExamples
Markov theoryComplex models
References
Conditional IndependenceUndirected graphsDirected acyclic graphsMoralization
Separation in DAGs
A trail τ from vertex α to vertex β in a DAG D is blocked by S ifit contains a vertex γ ∈ τ such that
I either γ ∈ S and edges of τ do not meet head-to-head at γ, or
I γ and all its descendants are not in S , and edges of τ meethead-to-head at γ.
A trail that is not blocked is active. Two subsets A and B ofvertices are d-separated by S if all trails from A to B are blockedby S . We write A⊥D B | S .
Steffen Lauritzen, University of Oxford Graphical Models
Genesis and historyExamples
Markov theoryComplex models
References
Conditional IndependenceUndirected graphsDirected acyclic graphsMoralization
Separation by example
3 6
1 5 7
2 4
u uu u u
u u
-
@@@R
����
@@@R
-
@@@R
@@@R
����
����
-
3 6
1 5 7
2 4
u uu u u
u u
-
@@@R
����
@@@R
-
@@@R
@@@R
����
����
-
For S = {5}, the trail (4, 2, 5, 3, 6) is active, whereas the trails(4, 2, 5, 6) and (4, 7, 6) are blocked.For S = {3, 5}, they are all blocked.
Steffen Lauritzen, University of Oxford Graphical Models
Genesis and historyExamples
Markov theoryComplex models
References
Conditional IndependenceUndirected graphsDirected acyclic graphsMoralization
Returning to example
3 6
1 5 7
2 4
u uu u u
u u
-
@@@R
����
@@@R
-
@@@R
@@@R
����
����
-
Hence 4⊥D 6 | 3, 5, but it is not true that 4⊥D 6 | 5 nor that4⊥D 6.
Steffen Lauritzen, University of Oxford Graphical Models
Genesis and historyExamples
Markov theoryComplex models
References
Conditional IndependenceUndirected graphsDirected acyclic graphsMoralization
The moral graph Dm of a DAG D is obtained by adding undirectededges between unmarried parents and subsequently droppingdirections, as in the example below:
3 6
1 5 7
2 4
s ss s ss s-
@@R
���
@@R
-@@R
@@R
���
���
-
3 6
1 5 7
2 4
s ss s ss s-
@@R
���
@@R
-@@R�
�@@R
���
���
-
3 6
1 5 7
2 4
s ss s ss s@@
��
@@
@@
��
��
��@@
Steffen Lauritzen, University of Oxford Graphical Models
Genesis and historyExamples
Markov theoryComplex models
References
Conditional IndependenceUndirected graphsDirected acyclic graphsMoralization
Undirected factorizations
If P factorizes w.r.t. D, it factorizes w.r.t. the moralised graph Dm.
This is seen directly from the factorization:
f (x) =∏v∈V
f (xv | xpa(v)) =∏v∈V
ψ{v}∪pa(v)(x),
since {v} ∪ pa(v) are all complete in Dm.
Hence if P satisfies any of the directed Markov properties w.r.t. D,it satisfies all Markov properties for Dm.
Steffen Lauritzen, University of Oxford Graphical Models
Genesis and historyExamples
Markov theoryComplex models
References
Conditional IndependenceUndirected graphsDirected acyclic graphsMoralization
Alternative equivalent separation
To resolve query involving three sets A, B, S :
1. Reduce to subgraph induced by ancestral set DAn(A∪B∪S) ofA ∪ B ∪ S ;
2. Moralize to form (DAn(A∪B∪S))m ;
It then holds that A⊥D B | S if and only if S separates A from Bin this undirected graph.
Proof in Lauritzen (1996) needs to allow self-intersecting paths tobe correct.
Steffen Lauritzen, University of Oxford Graphical Models
Genesis and historyExamples
Markov theoryComplex models
References
Conditional IndependenceUndirected graphsDirected acyclic graphsMoralization
Forming ancestral set
3 6
1 5
2 4
u uu u
u u
-
@@@R
����
@@@R
@@@R
����
-
The subgraph induced by all ancestors of nodes involved in thequery 4⊥D 6 | 3, 5?
Steffen Lauritzen, University of Oxford Graphical Models
Genesis and historyExamples
Markov theoryComplex models
References
Conditional IndependenceUndirected graphsDirected acyclic graphsMoralization
Adding links between unmarried parents
3 6
1 5
2 4
u uu u
u u
-
@@@R
����
@@@R
@@@R
����
-
Adding an undirected edge between 2 and 3 with common child 5in the subgraph induced by all ancestors of nodes involved in thequery 4⊥D 6 | 3, 5?
Steffen Lauritzen, University of Oxford Graphical Models
Genesis and historyExamples
Markov theoryComplex models
References
Conditional IndependenceUndirected graphsDirected acyclic graphsMoralization
Dropping directions
3 6
1 5
2 4
u uu u
u u@@@
���
@@@
@@@
���
Since {3, 5} separates 4 from 6 in this graph, we can conclude that4⊥D 6 | 3, 5
Steffen Lauritzen, University of Oxford Graphical Models
Genesis and historyExamples
Markov theoryComplex models
References
Bayesian inference using Gibbs sampling
A particular successful development is associated with BUGS,(Gilks et al., 1994) (WinBUGS, OpenBUGS).
I enables a Bayesian analyst to focus on substantive modellingwhereas the technical model specification and computationalside is taken care of automatically,
I exploiting modularity, factorization, and MCMC methodology,including the Gibbs and Metropolis–Hastings sampler.
I Conforming with Bayesian paradigm, parameters andobservations are explicitly represented in model as nodes ingraph, all being observables;
Steffen Lauritzen, University of Oxford Graphical Models
Genesis and historyExamples
Markov theoryComplex models
References
Bayesian inference using Gibbs sampling
A particular successful development is associated with BUGS,(Gilks et al., 1994) (WinBUGS, OpenBUGS).
I enables a Bayesian analyst to focus on substantive modellingwhereas the technical model specification and computationalside is taken care of automatically,
I exploiting modularity, factorization, and MCMC methodology,including the Gibbs and Metropolis–Hastings sampler.
I Conforming with Bayesian paradigm, parameters andobservations are explicitly represented in model as nodes ingraph, all being observables;
Steffen Lauritzen, University of Oxford Graphical Models
Genesis and historyExamples
Markov theoryComplex models
References
Bayesian inference using Gibbs sampling
A particular successful development is associated with BUGS,(Gilks et al., 1994) (WinBUGS, OpenBUGS).
I enables a Bayesian analyst to focus on substantive modellingwhereas the technical model specification and computationalside is taken care of automatically,
I exploiting modularity, factorization, and MCMC methodology,including the Gibbs and Metropolis–Hastings sampler.
I Conforming with Bayesian paradigm, parameters andobservations are explicitly represented in model as nodes ingraph, all being observables;
Steffen Lauritzen, University of Oxford Graphical Models
Genesis and historyExamples
Markov theoryComplex models
References
Bayesian inference using Gibbs sampling
Linear regression
for(i IN 1 : N)
sigma
taubetaalpha
mu[i]
Y[i]
Linear regression as a full Bayesian graphical model.
Steffen Lauritzen, University of Oxford Graphical Models
Genesis and historyExamples
Markov theoryComplex models
References
Bayesian inference using Gibbs sampling
Linear regression
model
{
for( i in 1 : N ) {
Y[i] ~ dnorm(mu[i],tau)
mu[i] <- alpha + beta * (x[i] - xbar)
}
tau ~ dgamma(0.001,0.001) sigma <- 1 / sqrt(tau)
alpha ~ dnorm(0.0,1.0E-6)
beta ~ dnorm(0.0,1.0E-6)
}
Steffen Lauritzen, University of Oxford Graphical Models
Genesis and historyExamples
Markov theoryComplex models
References
Bayesian inference using Gibbs sampling
Data and BUGS model for pumps
The number of failures Xi is assumed to follow a Poissondistribution with parameter θi ti , i = 1, . . . , 10where θi is the failure rate for pump i and ti is the length ofoperation time of the pump (in 1000s of hours). The data areshown below.
Pump 1 2 3 4 5 6 7 8 9 10ti 94.5 15.7 62.9 126 5.24 31.4 1.05 1.05 2.01 10.5xi 5 1 5 14 3 19 1 1 4 22
A gamma prior distribution is adopted for the failure rates:θi ∼ Γ(α, β), i = 1, . . . , 10
Steffen Lauritzen, University of Oxford Graphical Models
Genesis and historyExamples
Markov theoryComplex models
References
Bayesian inference using Gibbs sampling
Gamma model for pumpdata
for(i IN 1 : N)
betaalpha
t[i]theta[i]
lambda[i]
x[i]
Failure of 10 power plant pumps.
Steffen Lauritzen, University of Oxford Graphical Models
Genesis and historyExamples
Markov theoryComplex models
References
Bayesian inference using Gibbs sampling
BUGS program for pumps
With suitable priors the program becomes
model
{
for (i in 1 : N) {
theta[i] ~ dgamma(alpha, beta)
lambda[i] <- theta[i] * t[i]
x[i] ~ dpois(lambda[i])
}
alpha ~ dexp(1)
beta ~ dgamma(0.1, 1.0)
}
Steffen Lauritzen, University of Oxford Graphical Models
Genesis and historyExamples
Markov theoryComplex models
References
Bayesian inference using Gibbs sampling
Growth of rats
for(j IN 1 : T)for(i IN 1 : N)
sigma
tau.cx[j]
Y[i, j]
mu[i, j]
beta[i]alpha[i]
beta.taubeta.calpha0alpha.calpha.tau
Growth of 30 young rats.
Steffen Lauritzen, University of Oxford Graphical Models
Genesis and historyExamples
Markov theoryComplex models
References
Bayesian inference using Gibbs sampling
Finding full conditionals for Gibbs sampler
Inference in Bayesian complex graphical models as above uses theGibbs sampler.
For a DAG the densities of full conditional distributions are:
f (xi | xV \i ) ∝∏v∈V
f (xv | xpa(v))
∝ f (xi | xpa(i))∏
v∈ch(i)
f (xv | xpa(v))
= f (xi | xbl(i)),
x where bl(i) is the Markov blanket of node i :
bl(i) = pa(i) ∪ ch(i) ∪{∪v∈ch(i) pa(v) \ {i}
}.
Steffen Lauritzen, University of Oxford Graphical Models
Genesis and historyExamples
Markov theoryComplex models
References
Bayesian inference using Gibbs sampling
Markov blanket
3 6
1 5 7
2 4
u uu u u
u u
-
@@@R
����
@@@R
-
@@@R
@@@R
����
����
-
Markov blanket of 6 is bl(6) = {3, 5, 7, 4}.
Steffen Lauritzen, University of Oxford Graphical Models
Genesis and historyExamples
Markov theoryComplex models
References
Bayesian inference using Gibbs sampling
3 6
1 5 7
2 4
u uu u u
u u
-
@@@R
����
@@@R
-
@@@R
@@@R
����
����
-
3 6
1 5 7
2 4
u uu u u
u u@@@
���
@@@
@@@
���
���
��� @
@@
The Markov blanket is just the neighbours of in the moral graph:
bl(v) = nem(v) so bl(6) = {3, 5, 7, 4} and bl(3) = {1, 5, 6, 2}.The DAG is used for modular specification of the model, and themoral graph for local computation.
Steffen Lauritzen, University of Oxford Graphical Models
Genesis and historyExamples
Markov theoryComplex models
References
Bayesian inference using Gibbs sampling
I Is a huge conceptual extension of so-called Bayesianhierarchical models;
I distinction prior/likelihood and parameter/random variableless well defined;
I If founder nodes in network are considered fixed and unknown,no reason not to consider models in Fisherian paradigm.
Steffen Lauritzen, University of Oxford Graphical Models
Genesis and historyExamples
Markov theoryComplex models
References
Bayesian inference using Gibbs sampling
I Is a huge conceptual extension of so-called Bayesianhierarchical models;
I distinction prior/likelihood and parameter/random variableless well defined;
I If founder nodes in network are considered fixed and unknown,no reason not to consider models in Fisherian paradigm.
Steffen Lauritzen, University of Oxford Graphical Models
Genesis and historyExamples
Markov theoryComplex models
References
Bayesian inference using Gibbs sampling
I Is a huge conceptual extension of so-called Bayesianhierarchical models;
I distinction prior/likelihood and parameter/random variableless well defined;
I If founder nodes in network are considered fixed and unknown,no reason not to consider models in Fisherian paradigm.
Steffen Lauritzen, University of Oxford Graphical Models
Genesis and historyExamples
Markov theoryComplex models
References
Darroch, J. N., S. L. Lauritzen, and T. P. Speed (1980). Markovfields and log-linear interaction models for contingency tables.The Annals of Statistics 8, 522–539.
Dempster, A. P. (1972). Covariance selection. Biometrics 28,157–175.
Geiger, D. and J. Pearl (1990). On the logic of causal models. InR. D. Shachter, T. S. Levitt, L. N. Kanal, and J. F. Lemmer(Eds.), Uncertainty in Artificial Intelligence 4, pp. 3–14.Amsterdam, The Netherlands: North-Holland.
Gibbs, W. (1902). Elementary Principles of Statistical Mechanics.NewHaven, Connecticut: Yale University Press.
Gilks, W. R., A. Thomas, and D. J. Spiegelhalter (1994). BUGS: alanguage and program for complex Bayesian modelling. TheStatistician 43, 169–178.
Haberman, S. J. (1974). The Analysis of Frequency Data.Chicago, Illinois: University of Chicago Press.
Steffen Lauritzen, University of Oxford Graphical Models
Genesis and historyExamples
Markov theoryComplex models
References
Hammersley, J. M. and P. E. Clifford (1971). Markov fields onfinite graphs and lattices. Unpublished manuscript.
Lauritzen, S. L. (1996). Graphical Models. Oxford, UnitedKingdom: Clarendon Press.
Lauritzen, S. L., A. P. Dawid, B. N. Larsen, and H.-G. Leimer(1990). Independence properties of directed Markov fields.Networks 20, 491–505.
Lauritzen, S. L. and D. J. Spiegelhalter (1988). Localcomputations with probabilities on graphical structures and theirapplication to expert systems (with discussion). Journal of theRoyal Statistical Society, Series B 50, 157–224.
Lauritzen, S. L. and N. Wermuth (1989). Graphical models forassociations between variables, some of which are qualitativeand some quantitative. The Annals of Statistics 17, 31–57.
Pearl, J. (1986). A constraint–propagation approach toprobabilistic reasoning. In L. N. Kanal and J. F. Lemmer (Eds.),
Steffen Lauritzen, University of Oxford Graphical Models
Genesis and historyExamples
Markov theoryComplex models
References
Uncertainty in Artificial Intelligence, Amsterdam, TheNetherlands, pp. 357–370. North-Holland.
Pearl, J. (1988). Probabilistic Inference in Intelligent Systems. SanMateo, CA: Morgan Kaufmann Publishers.
Wermuth, N. and S. L. Lauritzen (1983). Graphical and recursivemodels for contingency tables. Biometrika 70, 537–552.
Whittaker, J. (1990). Graphical Models in Applied MultivariateStatistics. Chichester, United Kingdom: John Wiley and Sons.
Wold, H. O. A. (1954). Causality and econometrics.Econometrica 22, 162–177.
Wright, S. (1921). Correlation and causation. Journal ofAgricultural Research 20, 557–585.
Wright, S. (1934). The method of path coefficients. The Annals ofMathematical Statistics 5, 161–215.
Steffen Lauritzen, University of Oxford Graphical Models
top related