amazon redshift deep dive & update · © 2019, amazon web services, inc. or its affiliates. all...

48
© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Junpei Ozono, Solutions Architect, Amazon Web Services K.K. September 26, 2019 Amazon Redshift Deep Dive & Update

Upload: others

Post on 24-Sep-2020

18 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Amazon Redshift Deep Dive & Update · © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Agenda Amazon Redshift recap Redshift Deep Dive -SETDW-

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Junpei Ozono, Solutions Architect, Amazon Web Services K.K.September 26, 2019

Amazon Redshift Deep Dive & Update

Page 2: Amazon Redshift Deep Dive & Update · © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Agenda Amazon Redshift recap Redshift Deep Dive -SETDW-

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

AgendaAmazon Redshift recap

Redshift Deep Dive -SETDW-

���

Page 3: Amazon Redshift Deep Dive & Update · © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Agenda Amazon Redshift recap Redshift Deep Dive -SETDW-

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Amazon Redshift recap

Page 4: Amazon Redshift Deep Dive & Update · © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Agenda Amazon Redshift recap Redshift Deep Dive -SETDW-

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

Amazon Redshift : "����������

��������

BI

OLTP ERP CRM LOB �������&�����!$

������0*#$� 0*-5,3

���%

Amazon Redshift

(2����"�$�/+)1.�(���������������%

'4���������

Page 5: Amazon Redshift Deep Dive & Update · © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Agenda Amazon Redshift recap Redshift Deep Dive -SETDW-

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

Amazon Redshift�������$

JDBC/ODBC

Amazon Redshift������

Amazon S3���(��

���������&��)�"�)�

���������Redshift �� ',-��(���Redshift �� '����&�+*�)�)

Amazon Redshift SpectrumS3 ���'����&�+*�)�)(���(��)�%��&���)

�!)���#�����',-��(��(Parquet, ORC, JSON, CSV etc)

���������������� ����

Page 6: Amazon Redshift Deep Dive & Update · © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Agenda Amazon Redshift recap Redshift Deep Dive -SETDW-

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

Amazon Redshift '"Redshift ����� 1 ���� 2 �������3������.#

2���/8�106 �2

2� 200(� !$,�40$,�*1

%,�$, -+7� ��)5�&���

Page 7: Amazon Redshift Deep Dive & Update · © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Agenda Amazon Redshift recap Redshift Deep Dive -SETDW-

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

������������ (2018 10�)

��

���������

Page 8: Amazon Redshift Deep Dive & Update · © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Agenda Amazon Redshift recap Redshift Deep Dive -SETDW-

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

(1 0 (2019 7 )

0

/ 8 2 )

Page 9: Amazon Redshift Deep Dive & Update · © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Agenda Amazon Redshift recap Redshift Deep Dive -SETDW-

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Redshift Deep Dive

-SETDW-

Page 10: Amazon Redshift Deep Dive & Update · © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Agenda Amazon Redshift recap Redshift Deep Dive -SETDW-

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

������ ���� 5 ��

SETDW

RedshiftPerformance

Sortkey

Encoding

Table Maintenance

Distribution

WLM

S E T D W

Page 11: Amazon Redshift Deep Dive & Update · © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Agenda Amazon Redshift recap Redshift Deep Dive -SETDW-

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

Sortkey (�����)������������������� ����� �

CREATE TABLE deep_dive (aid INT --audience_id

,loc CHAR(3) --location,dt DATE --date

);

CREATE TABLE deep_dive (aid INT --audience_id

,loc CHAR(3) --location,dt DATE --date

) SORTKEY (dt, loc);

deep_diveaid loc dt1 SFO 2017-10-202 JFK 2017-10-203 SFO 2017-04-014 JFK 2017-05-14

deep_dive (sorted)aid loc dt3 SFO 2017-04-014 JFK 2017-05-142 JFK 2017-10-201 SFO 2017-10-20

deep_dive (sorted)aid loc dt3 SFO 2017-04-01

deep_dive (sorted)aid loc dt3 SFO 2017-04-014 JFK 2017-05-14

deep_dive (sorted)aid loc dt3 SFO 2017-04-014 JFK 2017-05-142 JFK 2017-10-20

deep_dive (sorted)aid loc dt3 SFO 2017-04-014 JFK 2017-05-142 JFK 2017-10-201 SFO 2017-10-20

S E T D W

Page 12: Amazon Redshift Deep Dive & Update · © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Agenda Amazon Redshift recap Redshift Deep Dive -SETDW-

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

������ '6�Redshift �������8A�����<�%�� (1MB) +�?;��1->�1'>��&"#$(�8A��,)4953*�7/• ��$=.�0:2 �%���B�@�� (��&!� )

MIN: 01-JUNE-2017MAX: 06-JUNE-2017

MIN: 07-JUNE-2017MAX: 12-JUNE-2017

MIN: 21-JUNE-2017MAX: 30-JUNE-2017

MIN: 01-JUNE-2017MAX: 20-JUNE-2017

MIN: 08-JUNE-2017MAX: 30-JUNE-2017

MIN: 02-JUNE-2017MAX: 25-JUNE-2017

SELECT count(*) FROM deep_dive WHERE dt = '06-09-2017';

S E T D W

Page 13: Amazon Redshift Deep Dive & Update · © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Agenda Amazon Redshift recap Redshift Deep Dive -SETDW-

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

%�)#��<�6� ���2�$1�*�� COMPOUND %�)#��<S���

XU�F����MD��TP��V�B�HC (WHERE W) #��?8�AL• 4O9��3G� ID

R=#��?8�� �IE���IE�J� 4 "/.>:�Q��1. HC�?8�X;�5�N2. "�( +0' ��K�N�?8• @��"/.�?8� �1�*, Vacuum 7�!�,�-&*�

S E T D W

Page 14: Amazon Redshift Deep Dive & Update · © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Agenda Amazon Redshift recap Redshift Deep Dive -SETDW-

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

������� ��������SVL_QUERY_SUMMARY ��� is_rrscan�� t �������������� (= ��� �������� ��������)

# select … from svl_query_summary where query = xxx order by stm, seg, ste; maxtime | avgtime | rows | bytes | label | is_diskbased | workmem | is_rrscan | is_delayed_scan | rows_pre_filter--------+---------+---------+----------+---------------------------------------------+--------------+---------+-----------+-----------------+----------------103579 | 102174 | 2338770 | 28065240 | scan tbl=587136 name=test_sort_key_order | f | 0 | t | f | 16501910 103579 | 102174 | 2338770 | 0 | project | f | 0 | f | f | 0 103579 | 102174 | 2338770 | 0 | project | f | 0 | f | f | 0 103579 | 102174 | 2 | 16 | aggr tbl=296 | f | 0 | f | f | 0 65 | 58 | 2 | 16 | scan tbl=296 name=Internal Worktable | f | 0 | f | f | 0 65 | 58 | 2 | 0 | return | f | 0 | f | f | 0 354 | 354 | 2 | 16 | scan tbl=52828 name=Internal Worktable | f | 0 | f | f | 0 354 | 354 | 1 | 16 | aggr tbl=300 | f | 0 | f | f | 0 74 | 74 | 1 | 16 | scan tbl=300 name=Internal Worktable | f | 0 | f | f | 0 74 | 74 | 1 | 0 | project | f | 0 | f | f | 0 74 | 74 | 1 | 0 | project | f | 0 | f | f | 0 74 | 74 | 1 | 13 | return | f | 0 | f | f | 0

S E T D W

Page 15: Amazon Redshift Deep Dive & Update · © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Agenda Amazon Redshift recap Redshift Deep Dive -SETDW-

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

Encoding (�('<=)

�('<=�.�����-0�"����� ��?�7��"�!;�6� ��$%��&+��5,�/7��

<=��+��# (�*�)�') �@423�����CREATE TABLE�9>�1A����:8

S E T D W

Page 16: Amazon Redshift Deep Dive & Update · © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Agenda Amazon Redshift recap Redshift Deep Dive -SETDW-

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

N���������COPY

(COMPUPDATE = PRESET ; )

• - RAW (• BOOLEAN / REAL / DOUBLE PRECISION - RAW ( )• - LZO

S E T D W

!

Page 17: Amazon Redshift Deep Dive & Update · © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Agenda Amazon Redshift recap Redshift Deep Dive -SETDW-

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

� �45�"����"�(%$&���������ANALYZE COMPRESSION ��"���� ��� �"����"�����26(%�.*��,(%����1-• ANALYZE COMPRESSION <���!)>;

�"�������/3���������!,+' & INSERT-SELECT�#0

S E T D W

Page 18: Amazon Redshift Deep Dive & Update · © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Agenda Amazon Redshift recap Redshift Deep Dive -SETDW-

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

:;�$����$��+�)�Redshift �� �)0��

1. 5%*�� COPY �!$����'.:;�8�

2. �� #����21�<���"��$�37��6/�(�4��ANALYZE COMPRESSION �!$��,& 9-�37

S E T D W

Page 19: Amazon Redshift Deep Dive & Update · © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Agenda Amazon Redshift recap Redshift Deep Dive -SETDW-

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

\^ 6'�,�6%�@�<�7Z:D�

SY�O���ZSTD� � LZO ������@���QU

• ZSTD �\^P�;IV & IO [F�LW���&�(�G���$ 4A9=�\^X?�!�/�1*. (CPUT`) �;�

• LZO �]�\^P� ZSTD �b���BKJ��M����&�(�

Ra>HC�N��+�05 (~10,000H) �"32�E8)�-#��RAW �@_��

S E T D W

Page 20: Amazon Redshift Deep Dive & Update · © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Agenda Amazon Redshift recap Redshift Deep Dive -SETDW-

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

������56����/*• ������4��" �.7��56�2���������#��)�1���+��3�(-� ,��" ��#���%0���!$����������'&���

0000

100K

0000

100K

5K

COL1 COL2

row_id

row_id

S E T D W

Page 21: Amazon Redshift Deep Dive & Update · © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Agenda Amazon Redshift recap Redshift Deep Dive -SETDW-

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

�� ���78����3-• .'�� ���78�25 (RAW) ������04

• �����������COL1 �1* $��!�+,��COL2 �"%�����#&�6/���)(���

����

���

0000

100K

5K

COL1 COL2

10K

row_id

row_id

S E T D W

Page 22: Amazon Redshift Deep Dive & Update · © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Agenda Amazon Redshift recap Redshift Deep Dive -SETDW-

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

<;4����"SVL_QUERY_SUMMARY � ��rows_pre_filter= (���"��!���:���1.) rows = ()0�5*&81.) �%7���9�(�����$#�!��'�+3,�����/�:�����62-��

# select … from svl_query_summary where query = xxx order by stm, seg, ste; maxtime | avgtime | rows | bytes | label | is_diskbased | workmem | is_rrscan | is_delayed_scan | rows_pre_filter--------+---------+---------+----------+---------------------------------------------+--------------+---------+-----------+-----------------+----------------103579 | 102174 | 2338770 | 28065240 | scan tbl=587136 name=test_sort_key_order | f | 0 | t | f | 16501910 103579 | 102174 | 2338770 | 0 | project | f | 0 | f | f | 0 103579 | 102174 | 2338770 | 0 | project | f | 0 | f | f | 0 103579 | 102174 | 2 | 16 | aggr tbl=296 | f | 0 | f | f | 0 65 | 58 | 2 | 16 | scan tbl=296 name=Internal Worktable | f | 0 | f | f | 0 65 | 58 | 2 | 0 | return | f | 0 | f | f | 0 354 | 354 | 2 | 16 | scan tbl=52828 name=Internal Worktable | f | 0 | f | f | 0 354 | 354 | 1 | 16 | aggr tbl=300 | f | 0 | f | f | 0 74 | 74 | 1 | 16 | scan tbl=300 name=Internal Worktable | f | 0 | f | f | 0 74 | 74 | 1 | 0 | project | f | 0 | f | f | 0 74 | 74 | 1 | 0 | project | f | 0 | f | f | 0 74 | 74 | 1 | 13 | return | f | 0 | f | f | 0

S E T D W

Page 23: Amazon Redshift Deep Dive & Update · © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Agenda Amazon Redshift recap Redshift Deep Dive -SETDW-

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

Table Maintenance

Vacuum

Sort

Analyze Vacuum

Delete

S E T D W

Page 24: Amazon Redshift Deep Dive & Update · © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Agenda Amazon Redshift recap Redshift Deep Dive -SETDW-

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

Analyze �%*(Analyze ��+>���!'$/52-��� 6,�����"�</1.�?&��8)

���"����7(���!�;@:= 0���>4����#�%*'$���

� ��� auto_analyze�7?�����%*'$� OFF �����93

Amazon RedshiftQuery planner

Table statistics Amazon Redshift Worker Thread Auto Analyze

Anal

yze

this

tabl

e

S E T D W

Page 25: Amazon Redshift Deep Dive & Update · © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Agenda Amazon Redshift recap Redshift Deep Dive -SETDW-

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

Vacuum Delete Vacuum Delete Delete Updatet o 2

t cmr 1

u 9

V 99 a 0

S

Col1 Col2 Col3 Col1 Col2 Col3

Vacuum Delete

S E T D W

Page 26: Amazon Redshift Deep Dive & Update · © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Agenda Amazon Redshift recap Redshift Deep Dive -SETDW-

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

Distribution ($8���#)�� #��������� 20����'����

ALL

Node 1

Slice 1

Slice 2

Node 2

Slice 3

Slice 4

KEY

Node 1

Slice 1

Slice 2

Node 2

Slice 3

Slice 4

keyA

keyB

keyC

keyD

Node 1

Slice 1

Slice 2

Node 2

Slice 3

Slice 4

EVEN

*%����4����!+/�1�"�� $820

1�"��9 -)�65 ����

$820

1���&�,3��"�� ����(-$����.7

S E T D W

Page 27: Amazon Redshift Deep Dive & Update · © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Agenda Amazon Redshift recap Redshift Deep Dive -SETDW-

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

!6�����%�"�Redshift �����"*�Redshift ����

1. ����/�!6�����(4 �������-)(CREATE TABLE #�!6�����+$��)

2. & ������.,�0'�2��$3�15���

S E T D W

Page 28: Amazon Redshift Deep Dive & Update · © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Agenda Amazon Redshift recap Redshift Deep Dive -SETDW-

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

AUTO );� �&Redshift �);� �&�1*�!�%&�4/����"$�&#�AUTO );� �&�8-

����,����� ALL);�����3��� EVEN );�(.69

'+ EVEN �69�����"� 57�2 ��(.�� ALL �:���

����30

AUTOALL

Node 1

Slice 1

Slice 2

Node 2

Slice 3

Slice 4

Node 1

Slice 1

Slice 2

Node 2

Slice 3

Slice 4

EVEN

S E T D W

Page 29: Amazon Redshift Deep Dive & Update · © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Agenda Amazon Redshift recap Redshift Deep Dive -SETDW-

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

Advisor DISTKEY Recommendation

Redshift

Redshift Advisor

N :ALTER TABLE

S E T D W

!

Page 30: Amazon Redshift Deep Dive & Update · © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Agenda Amazon Redshift recap Redshift Deep Dive -SETDW-

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

+?���'�.�-�)=*4��ALL, EVEN, KEY �8�+��

1. JOIN � �"'����63�:�� �"' ($��� �"') �ALL+?�63�5� �"' (Fact �"') �KEY+?� �EVEN+?�.@

2. BA� JOIN �63�5� �"',9��; �"'� JOIN �80��&%�DISTKEY��71

3. ���� JOIN ������� �"'�>2</�EVEN+?�71

#�(!� “JOIN”

S E T D W

Page 31: Amazon Redshift Deep Dive & Update · © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Agenda Amazon Redshift recap Redshift Deep Dive -SETDW-

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

Redshift JOIN Deep Dive��#���� ��� JOIN "%���!������� (Fact-Fact� ) � JOIN ���

Redshift�� JOIN ���3���• Nested Loop JOIN• Hash JOIN• Merge JOIN

Redshift��$� JOIN �Merge JOIN

S E T D W

Page 32: Amazon Redshift Deep Dive & Update · © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Agenda Amazon Redshift recap Redshift Deep Dive -SETDW-

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

Nested Loop JOIN• 4?��"�('�Nested Loop JOIN �-G��

• Nested Loop JOIN �E=�@9�B�>A• :5�;��F7�1'82�<�������"�6�D��• ��" !$% 0,�Warning �&+������������

----- Nested Loop Join in the query plan - review the join predicates to avoid Cartesian products -----

id

1

1

3

5

9

10

id

1

5

9

10

/*.

)*.

%��1#�����C3

S E T D W

Page 33: Amazon Redshift Deep Dive & Update · © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Agenda Amazon Redshift recap Redshift Deep Dive -SETDW-

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

Hash JOIN• :+.�8��-,���Hash JOIN �1@���

• Hash JOIN �$%(��*�*�" =3• ���'��")�$%(�<�� �-,�=3������>� !��#*��A2���?5

id

5

1

10

9

3

1

id

9

5

10

1

col1

9

5

10

1

C���'��");7(on $%()

B��&* D��&*�9,

604

/04

S E T D W

Page 34: Amazon Redshift Deep Dive & Update · © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Agenda Amazon Redshift recap Redshift Deep Dive -SETDW-

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

Merge JOIN• JOIN����"*5������ '?���*%�(&��

Merge JOIN �-@���• A�=�>�����*5�8B�� JOIN �������:!����);�6.�73

• 24 �� Concurrency ���� #��1$��<9

id

1

1

3

5

9

10

id

1

5

9

10

0,/

+,/

S E T D W

Page 35: Amazon Redshift Deep Dive & Update · © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Agenda Amazon Redshift recap Redshift Deep Dive -SETDW-

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

WLM (8�%7�-SB)S E T D W

WLM��%"5�N�$3���' 9��A;��$3���� ��A;��%"5�ICPL�125KU�=>A;G�A;) 0�!,�@R��

���������� ���(SQA)

E_D[����%"5�A;>?�J]�A;>?�\��J]������YC�<X$3��6�+�9&����

Concurrency Scaling

%"5�=>A;G�:Q�W�OBT�����>��.*%&4!9-�M�%4()��VH�%"5OB�#/7�-�Z^FL������

Page 36: Amazon Redshift Deep Dive & Update · © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Agenda Amazon Redshift recap Redshift Deep Dive -SETDW-

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

Auto WLMWLM( )

WLM (Auto WLM )

Auto WLM N

Redshift�����

Concurrency Query Slots = Auto

S E T D W!

Page 37: Amazon Redshift Deep Dive & Update · © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Agenda Amazon Redshift recap Redshift Deep Dive -SETDW-

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

Auto WLM – Query Priorities•• Highest• High• Normal ( )• Low• Lowest

• Redshift N

• QMR • CHANGE_QUERY_PRIORITY• CHANGE_SESSION_PRIORITY• CHANGE_USER_PRIORITY

! S E T D W

Page 38: Amazon Redshift Deep Dive & Update · © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Agenda Amazon Redshift recap Redshift Deep Dive -SETDW-

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

Auto WLM – Query Priorities

ETL N

Highest• N 1

Normal

• BI N

Normal

ETL / Priority : Highest

BI / Priority : Normal

S E T D W!

Page 39: Amazon Redshift Deep Dive & Update · © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Agenda Amazon Redshift recap Redshift Deep Dive -SETDW-

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

�'� ��)���(+��',(SQA)

���� 9H6F ����)3-/0�<G��

1

�'� ��)�=B�����)�E5.D�%�*���,����

2

)�����'� ��)��72A4���

3

SQA ��������BI / �� �����

�,#%� "�!

�,#%� "�!

�,#%� "�!

Amazon Redshift

��� �

9>��$�* �@C(&���?�;1�8:)

S E T D W

Page 40: Amazon Redshift Deep Dive & Update · © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Agenda Amazon Redshift recap Redshift Deep Dive -SETDW-

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

Concurrency Scaling Deep Dive

Redshift S3

1

2

N1

2

e

Concurrency Scaling

S E T D W!

Page 41: Amazon Redshift Deep Dive & Update · © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Agenda Amazon Redshift recap Redshift Deep Dive -SETDW-

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

Concurrency Scaling ����������������������������WLM ����� !�� �Concurrency Scaling mode” �� auto ���

S E T D W

Page 42: Amazon Redshift Deep Dive & Update · © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Agenda Amazon Redshift recap Redshift Deep Dive -SETDW-

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

������������������ �max_concurrency_scaling_clusters” �������������� 0-10 ������ ( ��� : 1)

S E T D W

Page 43: Amazon Redshift Deep Dive & Update · © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Agenda Amazon Redshift recap Redshift Deep Dive -SETDW-

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

Concurrency Scaling -%*(��"����� ���• �"��"�����dc2.8xlarge, ds2.8xlarge, dc2.large, or ds2.xlarge• ���'�2�32�

�� ���• Amazon Redshift local ���!��� Spectrum ���!�#��

Read-only�� (SELECT, Unload)

,&• ���!�Interleaved Sort Key�)������• ���!�����$+ Temporary Table�)������

S E T D W

Page 44: Amazon Redshift Deep Dive & Update · © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Agenda Amazon Redshift recap Redshift Deep Dive -SETDW-

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

PC$.&'�� ')('!+1%Q�max_concurrency_scaling_clusters = 5 �65

1.,!1$.&'��$"/�#-�O��;L��8�?D�PC$.&'�� ')(

2. ')(���PC$.&'�� ���$"/&0)*�TK���65�2 I<�PC$.&'�� ')(�BS�F:���?2A 5 ��7R�>G�V�N

PC$.&'���$"/=4�MW ���H�����$"/�J;�����3:@9�$.&'��UE �� �PC$.&'��//�& �������

S E T D W

Page 45: Amazon Redshift Deep Dive & Update · © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Agenda Amazon Redshift recap Redshift Deep Dive -SETDW-

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

Concurrency ScalingR=?• Concurrency Scaling �)�. +"#�� �,��*�L�� B�3;• �*�L����H�� �,�)�. +"#��TO��• 5C4�MA +"#��F9�EJ����<G

• MA +"#��80�� �,�)�. +"#���80�>D�Q��GD>���• K�� �,�80N6�N���21����S���21�����• ����*�L������ ��!"%(:7�"-�'$&�@/�PI

S E T D W

Page 46: Amazon Redshift Deep Dive & Update · © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Agenda Amazon Redshift recap Redshift Deep Dive -SETDW-

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

���

Page 47: Amazon Redshift Deep Dive & Update · © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Agenda Amazon Redshift recap Redshift Deep Dive -SETDW-

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

���

RedshiftPerformance

Sortkey

Encoding

Table Maintenance

Distribution

WLM

Page 48: Amazon Redshift Deep Dive & Update · © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Agenda Amazon Redshift recap Redshift Deep Dive -SETDW-

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Thank you!Junpei OzonoTwitter: @jostandard