1. Amazon Kinesis 2014.03.18 ALBERT 1

2. - http://librabuch.jp ALBERT http://albert2005.co.jp @iktakahiro 2 3. Kinesis 3 ALBERTM2MIoT Amazon Kinesis http://www.albert2005.co.jp/release/archives/201403/18_130056.html 4. Agenda 1. Amazon Kinesis 2. Amazon Kinesis API 3. fluentd 4. Amazon Kinesis 4 5. Amazon Kinesis 5 Amazon Kinesis http://aws.amazon.com/jp/kinesis/ Put Amazon Kinesis Get () 6. Amazon Kinesis API 6 Actions - Amazon Kinesis Service API Reference http://docs.aws.amazon.com/kinesis/latest/APIReference/API_Operations.html 1. CreateStream 2. DeleteStream 3. DescribeStream 4. GetRecords 5. GetShardIterator 6. ListStreams 7. MergeShards 8. PutRecord 9. SplitShard API9 7. Amazon Kinesis API 7 PutRecord Amazon Kinesis #!/usr/bin/env python # -*- coding: utf-8 -*- from boto import kinesis STREAM_NAME = sample-stream data = hoge partition_key = partition_1 connect = kinesis.connect_to_region(us-east-1) put_response = connect.put_recoad(STREAM_NAME, data, partition_key) 8. Amazon Kinesis API 8 GetRecord Amazon Kinesis stream = connect.describe_stream(STREAM_NAME) shards = stream[StreamDescription][StreamName] kinesis_iterator = connect.get_chard_iterator(STREAM_NAME, shards[0][ShardId], LATEST) response = connect.get_records(kinesis_iterator[ShardIterator], limit=2) # get_recordsSequenceNumber next_iterator = response[NextShardIterator] 9. fluentd 9 fluentd NginxLTVS tail-labeled-tsv tail S3Redshift Scale 10. fluentd 10 Scale out Architecture Nginx + fluentd Nginx + fluentd Nginx + fluentd Nginx + fluentdArchitectureThrouPut fluentd Amazon KinesisThroughPut Shard => NginxELB 11. fluentd 11 Multistep Processing Amazon Kinesis StreamPutKinesis Stream Put Amazon Kinesis Stream-1 Get Kinesis App Put Amazon Kinesis Stream-2 12. Amazon Kinesis 12 => KVSSequenceNumber APISDK Java Shard10 => OK Simple Icon => / => ry 13. 13