gdc 2015 - amazon developer day · gdc 2015 - amazon developer day deploying a...

GDC 2015 - Amazon Developer Day

Deploying a Low-Latency Multiplayer Game Globally

Nate WigerPrincipal , Gaming Solutions

~ @nateware

[gJ [email protected]

James GwertzmanCEG, PlayFab

~ @playfabnetwork

[gJ b!!P:/Iplayfab.com

jI.fi1f@l'amaZOniii' web services

[Game) Online Problems Continue; FeaturesDisabled as [Game Studio) Works on Fixes

Days later, [Game Studio)leavingdisconnected players stranded

"This is total bull**** 1paid for a preorder and1can 't even connect WTF [Game Studio)??"

[Game) has a great online mode - if youcan get online, that is

"We are working hard to get everyone onlineas soon as possible," [Game Studio) says.

Domestic latency in selected regions:"higher latency decreases game response time, user performanceand perceived game quality."

N. America

41.7 msEurope

11.6 ms

Trans-Pacifie

Trans-Atlantic

79.6 msEMEAto Asia

Pacifie

137.9 ms

Japan

16.8 ms

1 3.2 ms103.8 ms Latin America Asla Pacifie

97.9

http://bit.ly/superdata-1 atencySource: www.&lperDataResearch.c See also: http://bit.ly/verizon-Iatency

100+ms

100+ms

100+ms

100+ms

How To Be Awesome

1. Game servers near players

2. Everything else as HTTP APis

3. Data replication = bad

4. Local caches = good

5. Nobody really "plays everywhere"

~ AWS Gaming Solutions_ }-'Ii'" _Il'amazon" web services

••

•,,

Frankfurt~ ~

UDP

Tep /(' _ .

HTIP............,.".

Virginia

HTIP

••••••••••••••••••••••••••••••••

TCP /UDP

•••••

••••••••

· Oregon i.....................................

~ .,•••

-; Tokyo .....•.•.••......................••.

Loosely Coupled Pods..... .... ... .... .... ....... .... ......

Region

_ }-'Ii'"_Il'amazon" web services

•

•

•..••.

•

VPC Subne!

VPCSubne!

HTIP

Availability Zone B 1

...=.=..=.=..=.=..=.:.:.:.: .~ 'À'~zon

. - - _ . _ ., Ctoüdf'ront•

1•

1group .. _ ._ ._ ._ .-

----- - - --Availability Zone

(

1 1 11 1 1 11 1 1

Au! s/:ali' - ' - ' - ' - ' .-

VPC Subn e! 1 11 11 11 11 11 11 •

VPC Subne!

..... _-----_/

Cognito

VPC Subne!

•.....

... ..... ... .... .... ............ . ....

SES

~ AWS Gaming Solutions

SNS Mobile ."'-== ,------ -,Push

Game Flow

(.;;:...... .=.~ (..=.= ::::=..~.. .. . . 1 .

! 1 1 1 1"1 1 1 1

1 1 1 11 1 1 11 1 1 11 1 1 1 11 1 1 l ,

iL J :~ ~~~:~s_J 1•. Region ....

~. .'...................................................................................

CD Login via HTIP API(î) Download Game AsselsQ) Malchmaking 10 Game Server@ Connecllo Server@ Hack Apart Your Friends@ GameOver(J) Wrile via HTIP API

~ AWS Gaming Solutions_ }-'Ii'"_Il'amazon" web services

1 1

11

_.Game Server Pods +-.:. -Goe· ·

••••.•••••••••••••••••••••••••..•..•• ...... .... .... .. .. .. ..... ....... .... .... ....•... •..•••.•. ... ..... .... .... .... ... .... .... .... .. .... ..•.,- .....~ -------------, - --------, .~r> \ /' \ \.( If 1

11 111 1Il!

~~Çl 1;VPC Public Sub 11 VPC Public Subnet

Region

VPC Private Subnet

Availability Zone A

111111l ,

./-------------

VPC Private Subnet

~ AWS Gaming Solutions. ..:"..~..U.li·amazonli' web services

Make. Network. Faster.

• C3/C4 instance family (or R3 for more memory)

• ln a VPC

• HVM AMI- not PV

• Enable Enhanced Networking (SR-IOV)http://bit.ly/ec2-enhanced-net

• Linux: Tweak Networking Stackhttp://bit.ly/l inux-tuning


Enhanced Networking (SR-IOV)VMDq

w /NetQueuesDirect x/a Assignment

WjthSR~IOV

VM. VM, VM,...- .,-

VMM Switch

Before:Hypervisor VMM

NIC

Passes through thevswrtcf

After:Hardware

NIC

••Rabbit..M.9 ::':: ~J.~.§.! .!.ç bQ.~.~ ê.~.I.~.D..ç..i.~g ~GOC· ·

, " "' ./ . ------- - - - - - - , -------------, ....

... /' /' \ ".

Î il :1111

TCP' S672

•

•

•

•10.2.0.16

VPC Private Subnet•

11 :~·F~ra~b~bitmq-nOde2TCI? 14369& 25672

10.1 .0.13

rabbitmq-node1

VPC Private Subnet

•

11111111

. \ Availability Zone A J \ Availability Zone B J !\ ' / ,-------------/ .1. .-....

Region


RabbitMQ + Elastic Load Balancing

• Clustering Tutorialhttps://www.rabbitmg.com/clustering .html

• Set Queue HA policy to "ail"https://www.rabbitmg.com/ha.html

• Create Internai load balancer- Listen Port: TCP 5672

• VPC Security Group- For load balancer: TCP 5672- Between EC2 nodes: TCP 4369 & 25672

• Set up Client to Heartbeat


letc/rabbitmq/rabbitmq.config

, ] ,

] }

,,

[{rabbit, [ {tcp_listeners, [ ]},{collect_statistics_interval, lOOOO},{heartbeat,30},{cluster_partition_handling, pause_minority} ,{cluster_nodes, { [ 'rabbit@

'rabbit@di sc}} ] },

{rabbi tmq_management, [ {http_log_di r , "/tmp/ rabbi t -mgmt"},{listener, [{port, BOBO}]} ] i.{rabbi tmq_management_agent , [ {force_fine_statistics, true}

] .

••

•

•

••

••,

. ..:"..~..U_li'amazonli' web services

. _ . _. _. - . _ . -•

Redis Pub/SubRedis Pub/Sub

VPC Private Subnet•••

111111 VPC Private Subnet

i 11\ , Availability Zone A I ' Availability Zone B I ,\ . ' / ,----- / ~i

-'. ....~

_.-Goe-·............................................................................................. . ..... ....

~ ------- - - -- -- , - -- ------, .~f / \ / \ \1 ( ( '- '- '- '- '- '- '- '- '- ï ( ._.

. 11! 11 1

L._ ._. ._._._._~~t?_ . ~r.g gr_ p_ ._ ._ ._ ._ ._ ._ ._!VPC Pu ie Subnet 1 VPC Public Subnet

::======~ç=::S~·1 :1111

Redis Pub ISub

••••••••••••••

Region i••••••••••••••••••••••••••••••••


CloudFormation + Chef

AWS CloudFormation:Launch, Bootstrap

•••

... .................................................................................. -"

;"./, / - - - - - - - - - - - - - ....... \1 ·\1 r '- '- '- '- '- '- '- '- ')

•: • 1. 1 1• • • 1

· 1 1 1. Auto Scaling group .\. ._ ._ ._ ._ -- -_ ._ ._ . ..1 1

VPC Public Subnet 11 j

1 l1 ; ~:""----..Region :

•

•

•

•

1 1•

1 1•

••, ,

YI ..

VPC Private Subnet 11

, \ Availabil ity Zone A J .\. ....... _-----------_/ ./~ .'-'" . , '


Chef:Confi Install

_ }-'Ii'" _Il'amazon" web services

••Unreal Engine with CloudFormation -Goe· -aws cloudformation create-stack --stack-name

-- t empl at e- body file:// \-- regi on us-west-2 \--disable-rollback --parameters \

parameterKey=Acceptunreal Eula, parametervalue=parameterKey=Bucket,parametervalue=parameterKey=FileName,parametervalue=parameterKey=KeyName,parametervalue=parameterKey=SSHLocation,parametervalue=parameterKey=serverpassword,parametervalue=


\\

\

_ }-'Ii'"_Il'amazon" web services

•+-

• CIoudFOITTIaliOll Ma l\l x

C 1'1 ' a "'tps:l/~ .• ws.amazon.eo<nICIOudlonnatoonlhom... ~ tl l .. CD D e l'::l .. .: ---••c-·

-•• Service8... Edit v r'IlItzw.-e 0 tlateware .. Oregon ..

Actions ..

Fllter: AdNe "

StackName

Eventl ..

By Nam.: l6n8l

CnNotecl nme

2014-11-0112:03:00 urc-œœ

s.....

CREATE3XlMl'I.flE

Deacrtptlon

• 12:07:501 urc-œœ CREATE3X)MPLETC

• 12:01:52 UTC-0800 CREATE_COMPLETE

• ll:06:05 urc-œœ ÇRa:: TE

1:tQ6.il4 urc-œœ C>lfATE

• 12:06:02 UTC-œoD CREATE_COMPLETE

AWS;:CbldFClfYMtM)n::Staek

AW5::Cku:lformatlon::'WartCondlbOtl

AW5::CbJdformatJCW\::W8itCondrtlOn

AW5::CIoudFormanon::W8/tCond1bOtl

AWS:.:EC2::lnatarce

'-"W8llCondll1on

WartConddlOl"l

~ AWS Gaming Solutions ...._...~~mazoniii' web services

• •httpS:llus-west-2.consoIe .8WS.amazon.comIec2IV2Iho.••~ <Ci

••= ~C· ·

-•• Servicee... Ed.It v neteware • nateware .. ~ ..

•Connect Actlons ....

1to1of1

. ....~1b8 1 207 ' c3la... us aut-2tl Ni ' .'O

a. ..tane:. type c3.18rge

Pm.te ONS ~'o.27·13().

69.... west-

2-Wf'ipUl:~

Pnvwt. 1Ps 10.27.13Q.6V

PubIc IP 54214,"5.255

Eantic: IP

Awa" bIfty zone I.e. !.2b

s.curit)'~ lh........,.

U. 1lSerwo5ecill~

, MfJ9GU.ZFEl N. w*IlIf

.-

• reinvent201 4-unteal - ubuntu0ec2-S4-2"'-' 1$·255: - - I sh••-- -~ •

~-S4-Z14-115-Z55 :-S ps -tu lIV"'eO\UID PID PPID C STlME TTY TD4E OIJUl'V"eCIl &261 82S5 5 28 :87 ., 81 :80 :21 /srv/WV"eCI1/l t.~l'"Yef'/Engt nellt nor1.es.Il t. r-.ax.lUE4Serw,. l.Int"ealToul"'nlaeftt ~Dec

••••

•••

Frankfurt •~ ~

......... ... . ... . ... .... ... . ... . .......• •• •• •• •• •• •• •• •• •• •• •• •• •• •• •• •• •• •• •• •• •• •

•?

•

••

. .".

.:.

Virginia

•

..............••••

•?

' .••••••••••••••

. Oregon .~.•. .•...............•...• .••••••.~

~..........•~~.~.~ ..

•

••••

••

••

•

•

Where My Servers At?

.... ....... ,_ , .• •· .• ••••••••••••••••••••


•

·.....Q.~.9.!!.'}....

.... .. ...................\ •: : .. 1GOC- ·

••••••••••••••••••

.... ~ê~'~'..~~~~~~~~. ~=li=O"="o~i.' . ". . .,.. ---------------- . .~ : .i ( Cleanup ( ' _ . - . - . ._. _ . _ . _ . "\ 1ii 1 Loop 1!

1 I i i1 I l !

·Il!•

! 1 VPC Subnet 1 !! 1 l '•

i 1 1i 1 VPC Subnet 1! 1 1\. AvailabilJ!y~onlLA ' _A.ï.ai!;lm1i!Y Zone.Jl_ J)

..............................................................................................................................................................

Server RegistrationEC2 APIStart 1StopInstances

Server Registration & Scaling

• HTTPS POST /api/servers/register• Include an HMAC (RFC 2104)• Send Server Status

- Public IP- # Players- Game Modes

• Matchmaking Service- Maintains server list- Removes servers


•

•

•

53 Bucket

VPC Private 5ubnet

VPC Public 5ubnet

••••••••••••••••••••••\ \ Availability Zone A\ . ....... _-----------_/

'.' ..

•

••Game ~.~.~.Y..~.~ h~.g..~ ~GOC· ·

.... -------------...... ........ / \ ".

·1 IAmazon CloudWatch Logs .\

Region

•

•

•

53 Buckel

Amazon Redshift

AmazonKinesis

7fluenld

1

111111

VPC Private 5ubnel

VPC Public 5ubnel•

1 1•i 1•

l 1•i 1•\ \ Availability Zone A 1\ . ....... _------- ----_/

'.' ..

••Game ~.~.~.Y..~.~ h~.g..~ ~GOC· ·

.... -------------...... ........ / \ ".

·1 \

Region

•

•

•

•

•

.•... .... .... .... ... .••.•..i .~

•

•••••••••••••••••••••••••••; Frankfurt~ .............................

•

•••••

•

••••,

•,,

... ........ ..•...

Virginia••••

..........,•••••••••••••••••••••••••••••••••••••••

•

•

••

•••••

.................•

Oregon............................

••• •••• ••••• •••••••

'.

• •••••••••••••

Leaderboard i• •• •~ ~' , ,. '

,••••••••••••••••••••••

•

•

•

•

•

•

••

~ ~~

•

. .••

.:.

•

...............•·•

•

•

Mini Pods Everywhere


PlayFab: Loadout Architecture•


What is PlayFab?

PlayFab is a complete live game operations platform.Focus on the fun - we'lI do the boring bits.

+ +.. andmore

Tools for operating &•

Complete set of c1oud-based•

Eco-system of pre- integrated. .

Sorne Loadout Stats

• Launched 1/31/14

• Over 1 billion API Calls

• At Peak (Feb 2014):

- 300k DAU

- 120 fixed game servers

- 110 EC2 game servers

PlayFab Core Architecture Tenants

• No scheduled downtime.

• Automatic scaling for load.

• Backend changes should beinvisible to developers.

• Use buffering to smooth overspikes or failures.


•

•••

••••••

•

•

••....................~•••••

•

••• Amamn EC2 :

Game S flI\'et"S ••

• EU :~ .

[········ïïïïïi······,;

•: Hald_ _ •• •: G-.n.S....• •••\ EU ........................'

Amazon EC2 :• Game SetveB :• AUS :" ,- .

......................",· ,

••

••

••1........................

•

......... ......... ......

•••••: Iv'nazon EC2: Game Servers

Amazon EC2: Game seves :• •• •• SA •\ ~

•••••• •· "' .... ....................~

••........................•

II

. _ .- .~,

•1

Amazo nADS

knalcn fa(,AP11W1cI1~

t

~

.-

~(1).-e(1)

>oe::J-o(1)-

List of PlayFab Web APis

a.a ... ~..... .c..~~

o.H ... rS......~.lJMlOIti

Upœ.' ... lo.....m...UpQt.~..

~.lJM<S ......T_~c..." _

a..c.1IbgI_.e>.ITiIIIO..Seln.o...

...._."....'....d.AdcIu..~Cu' 1IlWICJ'

~tIo_'*"r

~1n.n.1'olW.

SJb'&1UMNi~

.....--...~~ .... o_.C_.sr.~

Go_~

R~B'IllUP'''~.

U~Q''''I1lII~

S-..soœ o--l.J»gI:o.l.og~lM

tlenlr

F-..cI 1g '••1

""',.....""'-...50"-""

lOS S~ • APboR.J ' f'OItOSPwri..........tol

"*8...c~Jt

MIIkl"&d>gAP"G4G.-s.wrR-;ja'l'......~.........AndJDoô.a.uc N'I.

And...d;M...,.Pu.,•••• • •• -..R.i;I_'an~"I.GaogIlIPlIIyPutdl_--

LognWllh.lrr1d'<:mo."",O

Logonwe.F_bo<*

l ogn WCha.m.e:-...LagnWCl!Gool;l~

LognWllhOSo. ,œlDlogn'lWrl~yf.ll

LagnWll'l9lll...

R~ ? PlilyFltlUMr

~!AIow~

~". Fil i • .c0-"""'" .....1:1Geh,F"IDaF.-.I'•• g~lo.

GeIJ..co.tw.edirloUoIlF_OOdA«nont......a--e::-...Acc:ou'lI

"".............U'*'kF ...

U'*'..U'*'llS.~

U~"'''''*ffe''

U~._d

Upd-.tJ_T_o.pllo~I_'.

~D_~

GeFr..-ta.a-s--da.1.~1ll

GeI.~Gl~

GoIU..o_GdJM'R-.tOrlyOu

Geu-s'........Upd..u..o..

U~..st.......

Example of Web API Cali

API Request:POST https: / /l.playfabapi.com/Client /MatchmakeContent -Type: application/json;x-Authentication: 7sc920BC255F7E60-0-0-SF4{

"Suildvers,on"; "S.Ol","Region": "uscentrat", "GameMode"; "0","lobbyld": "Lobby 32"."ënabl equeue": false

)


Response from Web Services:HTTP/l.l 200 OK(ontent-Type: application/json; charsetautf-8{

"Lobbytu": "lobby 32"."serverucstnaee" : "192.168.0.1",..Se rver-sor-t .. : 7777."Ticket": "e98yf289f248902f4904f0924f9pj"."Status"; "waiting","Que ue" : (

"user1","user2"

1)

_ }-'Ii'" _Il'amazon" web services

L...

Reportsservce

~~s~rld~: -60fCliol04D31114- .Mevent- ; ~user_1Q9ged_i n· .

·propt"rtiu- : [·~r{~· : M~l i~t · .

- t i t le-i d M

; 3.Mi p-; - 174. 127. 185. 21" - ,

1.· t;~sta.p· : MI014·11-05T23:21: 14. 267Z·,"com eet " : {

- l ibraryM : (- na..- : ·An.lytics . ~ET·.

·v~rsio"· : - 1. 0.0·

Log event:1

" --'l:-~--::=-Amazon 53

bud<e<

L...

AWS wildeard DNS(*.playfabapi.eom)

Elastic LoadBaancing

•,

Amazon Route 53(3.playfabapLcom)

" Ti tleld": " 3" ."username": "çwerrzman"."Password": " j amespassword"

POST https: / /3 .playfabapi.com/client/LoginwithPlayFab(ontent -Type: application/json;1

_.ELB Logging for Title Tracking -GOe· ·

Logs include title-specifieendpoint for analytics

API Request:

Game Clienl

Loadout API Usage Report (Oct 2014)AVERAGE AVERAGE AVERAGE TOTAL AVERAGE AVERAGE

rOTAL BACKEND RESPONSE REQUEST REQUEST RECIEVED SENT TOTALAPI NAME CAll5 TlME ' 5) TN E (1051 TlME (MS) TIME /10' 'BYTES' ' BrrES! ERRQR5

adminapi_userinfo 93.081 .685 0.03 0.04 0.04 59.83 0 452 1979

gameserver_usercustorndatareadonly 1.582,348 0.06 0.04 0.04 1.06 42 902 290

matd1makElf_userinfo 1,138.842 0.20 0.04 0.04 0.81 0 10046 319

malchmaker_ptayefjoined 1,138.247 0.01 0.04 0.04 0.79 64 2 93

rnald1maker_playerleft 1,136.797 0.01 0.04 0.04 0.80 64 2 400

matchmaker_8uthuser 736.807 0.01 0.03 0.04 0.52 0 48 107

gameserver_usercustomdata 297.735 0.19 0.04 0.04 0.20 0 4080 0

ga meserver_usercustomdatainiemal 297.469 0.08 0.04 0.04 0.20 0 237 2

_.Making a Singleton Highly Available -GOe· ·

1.Auto Scaling is set to keeptwo instances up.

2. Elastic Beanstalk HealthCheck term inates badinstance.

3. Amazon Route 53 failsover ail traffic to secondaryserver.

4.Auto Scaling detectsfailure and starts newinstance.

AtAo SCalingservice

( "_ ._ ._ ._ ._ ._ 0'.. 11 ._ -

1~ Matchmaker Mat maker i1 (pnmaryj (Se<:ondaryj.. AlAo Scaling 9 p 1_. . _. _ . _ . . _'

Amazon Route 53)

Elastic LoadBalandrg

Monitoring Service Health

•

l

•Amazon EC2~P1Nn:Ungl

•......c...'"

Bastc loadBal....,,,..

Handling Spikes Without Losing Data

Amazon Redshift

Customer's gamemarlagement dashboard

f......·

Reportsservice

Localevent r--r-------r--logs

local ReportsserviceEvents:

lagios andpurchases

ServerLogs

Logstashservice

~erSSO

RabbitMOmessage

queue

Logstash ElasticSearchInstéWlce


••Storage in OynamoOB, Amazon ROS, Amazon S~OC· ·Amazon Redshift

DynanoD8 AmazonRDS Amazon 53 Amazon Aedshlft

Player data: • Player stats • Log files • Analytics data• Accounts • Player currency • Game replays• Profile values• Inventory • Auditing• Save data


Managing Tables in DynamoDB

• Player data: accounts, profile , inventory, save data

AmazonDynMlOOB

I- I- I-I- I- I-I- I- l- a a aI- I- 1-User: Custom user data: Inventory:usertO: Hash User1O: Hash UserlD: HashTitlelO: range key TitlelD: range key h.emID: range key• DisplayName Keylvalue 1 • btlelD

linked accounts Keylvatue 2 Catalog Item• ... •

~ AWS Gaming Solutions_ }-,Ii," _Il'amazon" web services

•

•

•

Physical Deployment

Original deployment: us-west-1

Expanded to us-west-2 in July

Coming soon: cn-north-1

........... ...................•

............................" ,~ r---- .: l ': 1 1: 1 P1ayFab Web Il

: 1 Services, Al 1 J: '- - - - -' r - - - -, l '1 11 PlayFab Web Il1 Se rvices

Al, J'------........~.ç;:~!~~'... ..»

...............................t r r zz r: »: :, l ' ,• •

il :: 1 PlayFab Web 1: 1 Services• Al 1 )"----- .

••• •• •• •• •• •• •• •• •• •• •• •• •: Bei- :. ~.~ ~

Managing Cross-Region Data

,

u...rld- 1]29.

Cr ••ted • D.lter i .. . Utc~.

publi C (1." pl,ve,. : pl.yr.~t.~pl.y.r. U I" t~~

1[-.l on. Dyn lMlOOlv l . D.l t~I.Oy,,_...n"K.yJpublic Ul n t 64 ut>ertd 1 9"t ; ut: )PUblic n","9 (&111 { ~t ; ut: }IlUblic D.It.11_ c r ...t~ { ~t; su: }publ Ic OIt.I1_ l"ll091n { ~t : ut: }

,_.

o.t.stor• .Oy~.-oDB •

nNlJ 1 D.ltntore . Dyn-.oll. lNItiiStore •!tIl.pl.oe 1.~ u ,-_u -2~.

~[] ( ·u,~,t-2· . · us-..st - l- J>J

var pl.y." • plltyer .Get (12J'. fais.} :pl"." .l'5t1091" • D.lt.T l_.~ :

Upoûru rh<> p/,..,.,. rKOrd "' !~ SfO.... .JttJ <?fI ICHI '''' ""' '''' If ..."pl"." .UP4Ite( ) ;

c...... ru " pb,..,. f"Kord '" r,.. .'_If HG,.. MId ~I_ _

var ~1.,.,. . pl.,.r .(,.••t.(n~ pl.y.r ()

1

[o.t.~tgr.cont19(

wr; leStor• ••••dsror.or~r •writtltPQlOfl •

.udll~;onor<Wr •Player accounts are sharedacross ail regions:

• Cross-regional wrapper foruser.get sets order ofdatabases and regions tocheck

• user.get then checks forplayer account data

• High availability, transparentto customers and players

))

11


Amazon EC2Game Servers •

AUS :' .

Amazon EC2 :Game Servet'S :

: EU :.. ......................

: j· .• •

•••

r .• •

.... ... .............' "· .· ,••••••••

• Amazon EC2Game seoeee•

•

•••••

· ...."" ' '

........................ .· ,•••••••

.......................· .: :: :, .•••

•

. _. _,

._ .- .

•• •• Amaz on EC2 :r;==l========:::j;:============;:Garre seoees i: SA :........................

'- . _ .

1

Arnazon ROS

Amaz(l"l EC2(API twwjlngl

t

~

M'Ialcn Ao..te ~3

P_plII~at:Bp -co"...s::::CJ~«

~(1).-e(1)

>o(1)~

:J-CJ(1)-.-

•services

Nate WigerPrincipal, Gaming Solutions

~ @natewarel:Bl [email protected]

gdc 2015 - amazon developer day · gdc 2015 - amazon developer day deploying a...

Documents