Download - Scaling postgres
![Page 1: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/1.jpg)
/
Scaling Postgres
Denish PatelDatabase Architect
https://twitter.com/DenishPatel
Wednesday, September 18, 13
![Page 2: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/2.jpg)
1Wednesday, September 18, 13
![Page 3: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/3.jpg)
OmniTI
1Wednesday, September 18, 13
![Page 4: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/4.jpg)
OmniTI
1
• Helping customersnavigate explosive growthwith technology.
Wednesday, September 18, 13
![Page 5: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/5.jpg)
OmniTI
1
• Helping customersnavigate explosive growthwith technology.
100MM+ users
$1B+ gross online sales
Wednesday, September 18, 13
![Page 6: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/6.jpg)
OmniTI
1
• Helping customersnavigate explosive growthwith technology.
100MM+ users
$1B+ gross online sales
Open and closed source thought leaders, experts and authors
Wednesday, September 18, 13
![Page 7: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/7.jpg)
OmniTI
1
• Helping customersnavigate explosive growthwith technology.
100MM+ users
$1B+ gross online sales
Open and closed source thought leaders, experts and authors
Wednesday, September 18, 13
![Page 8: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/8.jpg)
2Wednesday, September 18, 13
![Page 9: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/9.jpg)
Talk Outline
2Wednesday, September 18, 13
![Page 10: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/10.jpg)
Talk Outline
• Scalability
• Database Scaling needs, costs, methods
• Scaling Postgres
• Vertically
• Horizontally
• Obstacles to Scalability
• Beyond Postgres
2Wednesday, September 18, 13
![Page 11: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/11.jpg)
3Wednesday, September 18, 13
![Page 12: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/12.jpg)
What is Scalability?
3Wednesday, September 18, 13
![Page 13: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/13.jpg)
What is Scalability?
3
A service is said to be scalable if when we increase the resources in a system, it results in
increased performance in a manner proportional to resources added.
Wednesday, September 18, 13
![Page 14: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/14.jpg)
4Wednesday, September 18, 13
![Page 15: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/15.jpg)
Why to scale databases?
4Wednesday, September 18, 13
![Page 16: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/16.jpg)
Why to scale databases?
• Support a higher volume of users
4Wednesday, September 18, 13
![Page 17: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/17.jpg)
Why to scale databases?
• Support a higher volume of users
4Wednesday, September 18, 13
![Page 18: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/18.jpg)
Why to scale databases?
• Support a higher volume of users
4Wednesday, September 18, 13
![Page 19: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/19.jpg)
Why to scale databases?
• Support a higher volume of users
4Wednesday, September 18, 13
![Page 20: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/20.jpg)
Why to scale databases?
• Provide better performance for existing users
• Support a higher volume of users
4Wednesday, September 18, 13
![Page 21: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/21.jpg)
Why to scale databases?
• Provide better performance for existing users
• Store a larger volume of data
• Support a higher volume of users
4Wednesday, September 18, 13
![Page 22: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/22.jpg)
Why to scale databases?
• Provide better performance for existing users
• Store a larger volume of data
• Improve system availability
• Support a higher volume of users
4Wednesday, September 18, 13
![Page 23: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/23.jpg)
Why to scale databases?
• Provide better performance for existing users
• Store a larger volume of data
• Improve system availability
• Geographic dispersion
• Support a higher volume of users
4Wednesday, September 18, 13
![Page 24: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/24.jpg)
5Wednesday, September 18, 13
![Page 25: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/25.jpg)
Why is database scalability so hard?
5Wednesday, September 18, 13
![Page 26: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/26.jpg)
Why is database scalability so hard?
5Wednesday, September 18, 13
![Page 27: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/27.jpg)
Why is database scalability so hard?
• Search
5Wednesday, September 18, 13
![Page 28: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/28.jpg)
Why is database scalability so hard?
• Search
• Concurrency
5Wednesday, September 18, 13
![Page 29: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/29.jpg)
Why is database scalability so hard?
• Search
• Concurrency
• Consistency
5Wednesday, September 18, 13
![Page 30: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/30.jpg)
Why is database scalability so hard?
• Search
• Concurrency
• Consistency
• Speed
5Wednesday, September 18, 13
![Page 31: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/31.jpg)
Why is database scalability so hard?
• Search
• Concurrency
• Consistency
• Speed
• Cost
5Wednesday, September 18, 13
![Page 32: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/32.jpg)
Why is database scalability so hard?
• Search
• Concurrency
• Consistency
• Speed
• Cost
• Cost of hardware
5Wednesday, September 18, 13
![Page 33: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/33.jpg)
Why is database scalability so hard?
• Search
• Concurrency
• Consistency
• Speed
• Cost
• Cost of hardware
• Cost deployment effort
5Wednesday, September 18, 13
![Page 34: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/34.jpg)
Why is database scalability so hard?
• Search
• Concurrency
• Consistency
• Speed
• Cost
• Cost of hardware
• Cost deployment effort
• Cost Ongoing maintenance
5Wednesday, September 18, 13
![Page 35: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/35.jpg)
6Wednesday, September 18, 13
![Page 36: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/36.jpg)
Scaling Needs & Methods
6Wednesday, September 18, 13
![Page 37: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/37.jpg)
Scaling Needs & Methods
6Wednesday, September 18, 13
![Page 38: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/38.jpg)
Scaling Needs & Methods
• Data growth
6Wednesday, September 18, 13
![Page 39: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/39.jpg)
Scaling Needs & Methods
• Data growth
• Read requests
6Wednesday, September 18, 13
![Page 40: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/40.jpg)
Scaling Needs & Methods
• Data growth
• Read requests
• Write requests
6Wednesday, September 18, 13
![Page 41: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/41.jpg)
Scaling Needs & Methods
• Data growth
• Read requests
• Write requests
• Vertical Scaling
• Horizontal Scaling
6Wednesday, September 18, 13
![Page 42: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/42.jpg)
Scaling Needs & Methods
• Data growth
• Read requests
• Write requests
• Vertical Scaling
• Horizontal Scaling
6Wednesday, September 18, 13
![Page 43: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/43.jpg)
Scaling Needs & Methods
• Data growth
• Read requests
• Write requests
• Vertical Scaling
• Horizontal Scaling
6Wednesday, September 18, 13
![Page 44: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/44.jpg)
7Wednesday, September 18, 13
![Page 45: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/45.jpg)
Vertical Scaling (Scale up)
7Wednesday, September 18, 13
![Page 46: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/46.jpg)
Vertical Scaling (Scale up)
7Wednesday, September 18, 13
![Page 47: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/47.jpg)
Vertical Scaling (Scale up)
• Pros
7Wednesday, September 18, 13
![Page 48: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/48.jpg)
Vertical Scaling (Scale up)
• Pros
• Simple to implement
7Wednesday, September 18, 13
![Page 49: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/49.jpg)
Vertical Scaling (Scale up)
• Pros
• Simple to implement
• Ease of maintenance
7Wednesday, September 18, 13
![Page 50: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/50.jpg)
Vertical Scaling (Scale up)
• Pros
• Simple to implement
• Ease of maintenance
• Cons
7Wednesday, September 18, 13
![Page 51: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/51.jpg)
Vertical Scaling (Scale up)
• Pros
• Simple to implement
• Ease of maintenance
• Cons
• Cost of hardware
7Wednesday, September 18, 13
![Page 52: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/52.jpg)
Vertical Scaling (Scale up)
• Pros
• Simple to implement
• Ease of maintenance
• Cons
• Cost of hardware
• Sometimes SPOFs
7Wednesday, September 18, 13
![Page 53: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/53.jpg)
8Wednesday, September 18, 13
![Page 54: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/54.jpg)
Horizontal Scaling (Scale Out)
8Wednesday, September 18, 13
![Page 55: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/55.jpg)
Horizontal Scaling (Scale Out)
8Wednesday, September 18, 13
![Page 56: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/56.jpg)
Horizontal Scaling (Scale Out)
8Wednesday, September 18, 13
![Page 57: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/57.jpg)
Horizontal Scaling (Scale Out)
• Pros
8Wednesday, September 18, 13
![Page 58: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/58.jpg)
Horizontal Scaling (Scale Out)
• Pros
• Cheaper in hardware cost
8Wednesday, September 18, 13
![Page 59: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/59.jpg)
Horizontal Scaling (Scale Out)
• Pros
• Cheaper in hardware cost
• Flexibility
8Wednesday, September 18, 13
![Page 60: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/60.jpg)
Horizontal Scaling (Scale Out)
• Pros
• Cheaper in hardware cost
• Flexibility
• Higher fault tolerance
8Wednesday, September 18, 13
![Page 61: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/61.jpg)
Horizontal Scaling (Scale Out)
• Pros
• Cheaper in hardware cost
• Flexibility
• Higher fault tolerance
• Cons
8Wednesday, September 18, 13
![Page 62: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/62.jpg)
Horizontal Scaling (Scale Out)
• Pros
• Cheaper in hardware cost
• Flexibility
• Higher fault tolerance
• Cons
• Complex to implement
8Wednesday, September 18, 13
![Page 63: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/63.jpg)
Horizontal Scaling (Scale Out)
• Pros
• Cheaper in hardware cost
• Flexibility
• Higher fault tolerance
• Cons
• Complex to implement
• Expensive to maintain
8Wednesday, September 18, 13
![Page 64: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/64.jpg)
Horizontal Scaling (Scale Out)
• Pros
• Cheaper in hardware cost
• Flexibility
• Higher fault tolerance
• Cons
• Complex to implement
• Expensive to maintain
• Bigger footprint in the Data Center
8Wednesday, September 18, 13
![Page 65: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/65.jpg)
Horizontal Scaling (Scale Out)
• Pros
• Cheaper in hardware cost
• Flexibility
• Higher fault tolerance
• Cons
• Complex to implement
• Expensive to maintain
• Bigger footprint in the Data Center
• No built in support in databases
8Wednesday, September 18, 13
![Page 66: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/66.jpg)
9Wednesday, September 18, 13
![Page 67: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/67.jpg)
Spec’ing Hardware
9Wednesday, September 18, 13
![Page 68: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/68.jpg)
Spec’ing Hardware
• CPU
9Wednesday, September 18, 13
![Page 69: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/69.jpg)
Spec’ing Hardware
• CPU
• 8+ cores
9Wednesday, September 18, 13
![Page 70: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/70.jpg)
Spec’ing Hardware
• CPU
• 8+ cores
• RAM
9Wednesday, September 18, 13
![Page 71: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/71.jpg)
Spec’ing Hardware
• CPU
• 8+ cores
• RAM
• 64GB+
9Wednesday, September 18, 13
![Page 72: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/72.jpg)
Spec’ing Hardware
• CPU
• 8+ cores
• RAM
• 64GB+
• Disks
9Wednesday, September 18, 13
![Page 73: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/73.jpg)
Spec’ing Hardware
• CPU
• 8+ cores
• RAM
• 64GB+
• Disks
• SSDs
9Wednesday, September 18, 13
![Page 74: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/74.jpg)
Spec’ing Hardware
• CPU
• 8+ cores
• RAM
• 64GB+
• Disks
• SSDs
• RAID 10
9Wednesday, September 18, 13
![Page 75: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/75.jpg)
Spec’ing Hardware
• CPU
• 8+ cores
• RAM
• 64GB+
• Disks
• SSDs
• RAID 10
• Network
9Wednesday, September 18, 13
![Page 76: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/76.jpg)
Spec’ing Hardware
• CPU
• 8+ cores
• RAM
• 64GB+
• Disks
• SSDs
• RAID 10
• Network
• min Gigbit, 10Gigbit
9Wednesday, September 18, 13
![Page 77: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/77.jpg)
10Wednesday, September 18, 13
![Page 78: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/78.jpg)
Tune Postgres/memory parameters
10Wednesday, September 18, 13
![Page 79: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/79.jpg)
Tune Postgres/memory parameters
• shared_buffers
10Wednesday, September 18, 13
![Page 80: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/80.jpg)
Tune Postgres/memory parameters
• shared_buffers
• effective_cache_size
10Wednesday, September 18, 13
![Page 81: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/81.jpg)
Tune Postgres/memory parameters
• shared_buffers
• effective_cache_size
• checkpoint_completion_target
10Wednesday, September 18, 13
![Page 82: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/82.jpg)
Tune Postgres/memory parameters
• shared_buffers
• effective_cache_size
• checkpoint_completion_target
• checkpoint_segments
10Wednesday, September 18, 13
![Page 83: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/83.jpg)
Tune Postgres/memory parameters
• shared_buffers
• effective_cache_size
• checkpoint_completion_target
• checkpoint_segments
• max_connections
10Wednesday, September 18, 13
![Page 84: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/84.jpg)
Tune Postgres/memory parameters
• shared_buffers
• effective_cache_size
• checkpoint_completion_target
• checkpoint_segments
• max_connections
• work_mem
10Wednesday, September 18, 13
![Page 85: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/85.jpg)
Tune Postgres/memory parameters
• shared_buffers
• effective_cache_size
• checkpoint_completion_target
• checkpoint_segments
• max_connections
• work_mem
• maintenance_work_mem
10Wednesday, September 18, 13
![Page 86: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/86.jpg)
Tune Postgres/memory parameters
• shared_buffers
• effective_cache_size
• checkpoint_completion_target
• checkpoint_segments
• max_connections
• work_mem
• maintenance_work_mem
http://wiki.postgresql.org/wiki/Tuning_Your_PostgreSQL_Server
10Wednesday, September 18, 13
![Page 87: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/87.jpg)
11Wednesday, September 18, 13
![Page 88: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/88.jpg)
Tune Postgres/logging parameters
11Wednesday, September 18, 13
![Page 89: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/89.jpg)
Tune Postgres/logging parameters
11Wednesday, September 18, 13
![Page 90: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/90.jpg)
Tune Postgres/logging parameters
11Wednesday, September 18, 13
![Page 91: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/91.jpg)
Tune Postgres/logging parameters
• logging_collector => 'on'
11Wednesday, September 18, 13
![Page 92: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/92.jpg)
Tune Postgres/logging parameters
• logging_collector => 'on'
• log_destination => 'stderr'
11Wednesday, September 18, 13
![Page 93: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/93.jpg)
Tune Postgres/logging parameters
• logging_collector => 'on'
• log_destination => 'stderr'
• log_filename => 'postgresql-%Y-%m-%d_%H%M%S.log'
11Wednesday, September 18, 13
![Page 94: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/94.jpg)
Tune Postgres/logging parameters
• logging_collector => 'on'
• log_destination => 'stderr'
• log_filename => 'postgresql-%Y-%m-%d_%H%M%S.log'
• log_line_prefix => '%m [%r] [%p]: [%l-1] user=%u,db=%d,e=%e '
11Wednesday, September 18, 13
![Page 95: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/95.jpg)
Tune Postgres/logging parameters
• logging_collector => 'on'
• log_destination => 'stderr'
• log_filename => 'postgresql-%Y-%m-%d_%H%M%S.log'
• log_line_prefix => '%m [%r] [%p]: [%l-1] user=%u,db=%d,e=%e '
• log_min_duration_statement => 1000ms
11Wednesday, September 18, 13
![Page 96: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/96.jpg)
Tune Postgres/logging parameters
• logging_collector => 'on'
• log_destination => 'stderr'
• log_filename => 'postgresql-%Y-%m-%d_%H%M%S.log'
• log_line_prefix => '%m [%r] [%p]: [%l-1] user=%u,db=%d,e=%e '
• log_min_duration_statement => 1000ms
• log_autovacuum_min_duration => '0'
11Wednesday, September 18, 13
![Page 97: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/97.jpg)
12Wednesday, September 18, 13
![Page 98: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/98.jpg)
Tune Postgres/logging parameters
12Wednesday, September 18, 13
![Page 99: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/99.jpg)
Tune Postgres/logging parameters
12Wednesday, September 18, 13
![Page 100: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/100.jpg)
Tune Postgres/logging parameters
12Wednesday, September 18, 13
![Page 101: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/101.jpg)
Tune Postgres/logging parameters
• log_lock_waits => 'on'
12Wednesday, September 18, 13
![Page 102: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/102.jpg)
Tune Postgres/logging parameters
• log_lock_waits => 'on'
• log_temp_files => '0'
12Wednesday, September 18, 13
![Page 103: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/103.jpg)
Tune Postgres/logging parameters
• log_lock_waits => 'on'
• log_temp_files => '0'
• log_checkpoints => 'on'
12Wednesday, September 18, 13
![Page 104: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/104.jpg)
Tune Postgres/logging parameters
• log_lock_waits => 'on'
• log_temp_files => '0'
• log_checkpoints => 'on'
• log_connections => 'on'
12Wednesday, September 18, 13
![Page 105: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/105.jpg)
Tune Postgres/logging parameters
• log_lock_waits => 'on'
• log_temp_files => '0'
• log_checkpoints => 'on'
• log_connections => 'on'
• log_disconnections => 'on'
12Wednesday, September 18, 13
![Page 106: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/106.jpg)
Tune Postgres/logging parameters
• log_lock_waits => 'on'
• log_temp_files => '0'
• log_checkpoints => 'on'
• log_connections => 'on'
• log_disconnections => 'on'
• log_min_error_statement => 'warning'
12Wednesday, September 18, 13
![Page 107: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/107.jpg)
Tune Postgres/logging parameters
• log_lock_waits => 'on'
• log_temp_files => '0'
• log_checkpoints => 'on'
• log_connections => 'on'
• log_disconnections => 'on'
• log_min_error_statement => 'warning'
• log_min_messages => 'warning'
12Wednesday, September 18, 13
![Page 108: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/108.jpg)
Tune Postgres/logging parameters
• log_lock_waits => 'on'
• log_temp_files => '0'
• log_checkpoints => 'on'
• log_connections => 'on'
• log_disconnections => 'on'
• log_min_error_statement => 'warning'
• log_min_messages => 'warning'
• log_statement => 'ddl'
12Wednesday, September 18, 13
![Page 109: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/109.jpg)
13Wednesday, September 18, 13
![Page 110: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/110.jpg)
Optimize Queries/report slow queries
13Wednesday, September 18, 13
![Page 111: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/111.jpg)
Optimize Queries/report slow queries
13Wednesday, September 18, 13
![Page 112: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/112.jpg)
Optimize Queries/report slow queries
13Wednesday, September 18, 13
![Page 113: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/113.jpg)
14Wednesday, September 18, 13
![Page 114: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/114.jpg)
Optimize Queries/Extensive monitoring
14Wednesday, September 18, 13
![Page 115: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/115.jpg)
Optimize Queries/Extensive monitoring
14Wednesday, September 18, 13
![Page 116: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/116.jpg)
15Wednesday, September 18, 13
![Page 117: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/117.jpg)
Optimize Queries /Explain Analyze
15Wednesday, September 18, 13
![Page 118: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/118.jpg)
Optimize Queries /Explain Analyze
explain (analyze,buffers) select col1,col2 from demo_ios where col2 between 0.01 and 0.02; QUERY PLAN ----------------------------------------------------------------------------------------------------------------------------------------------- Index Only Scan using idx_demo_ios on demo_ios (cost=0.00..35330.93 rows=993633 width=16) (actual time=58.100..3250.589 rows=1000392 loops=1) Index Cond: ((col2 >= 0.01::double precision) AND (col2 <= 0.02::double precision)) Heap Fetches: 0 Buffers: shared hit=923073 read=3848 Total runtime: 4297.405 ms
15Wednesday, September 18, 13
![Page 119: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/119.jpg)
Optimize Queries /Explain Analyze
explain (analyze,buffers) select col1,col2 from demo_ios where col2 between 0.01 and 0.02; QUERY PLAN ----------------------------------------------------------------------------------------------------------------------------------------------- Index Only Scan using idx_demo_ios on demo_ios (cost=0.00..35330.93 rows=993633 width=16) (actual time=58.100..3250.589 rows=1000392 loops=1) Index Cond: ((col2 >= 0.01::double precision) AND (col2 <= 0.02::double precision)) Heap Fetches: 0 Buffers: shared hit=923073 read=3848 Total runtime: 4297.405 ms
15Wednesday, September 18, 13
![Page 120: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/120.jpg)
16Wednesday, September 18, 13
![Page 121: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/121.jpg)
Optimize Queries /track functions
16Wednesday, September 18, 13
![Page 122: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/122.jpg)
Optimize Queries /track functions
16Wednesday, September 18, 13
![Page 123: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/123.jpg)
Optimize Queries /track functions
• track_functions = pl # none, pl, all
16Wednesday, September 18, 13
![Page 124: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/124.jpg)
Optimize Queries /track functions
• track_functions = pl # none, pl, all
• reload online
16Wednesday, September 18, 13
![Page 125: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/125.jpg)
Optimize Queries /track functions
• track_functions = pl # none, pl, all
• reload online
• select * from pg_stat_user_functions;
16Wednesday, September 18, 13
![Page 126: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/126.jpg)
Optimize Queries /track functions
• track_functions = pl # none, pl, all
• reload online
• select * from pg_stat_user_functions;
16Wednesday, September 18, 13
![Page 127: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/127.jpg)
Optimize Queries /track functions
• track_functions = pl # none, pl, all
• reload online
• select * from pg_stat_user_functions;
funcid | schemaname | funcname | calls | total_time | self_time
16Wednesday, September 18, 13
![Page 128: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/128.jpg)
Optimize Queries /track functions
• track_functions = pl # none, pl, all
• reload online
• select * from pg_stat_user_functions;
funcid | schemaname | funcname | calls | total_time | self_time
16Wednesday, September 18, 13
![Page 129: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/129.jpg)
17Wednesday, September 18, 13
![Page 130: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/130.jpg)
Partitioning
17Wednesday, September 18, 13
![Page 131: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/131.jpg)
Partitioning
• As table size grows, queries eventually slows down, even with indexing
17Wednesday, September 18, 13
![Page 132: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/132.jpg)
Partitioning
• As table size grows, queries eventually slows down, even with indexing
• Allows data added, removed and queried fast
17Wednesday, September 18, 13
![Page 133: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/133.jpg)
Partitioning
• As table size grows, queries eventually slows down, even with indexing
• Allows data added, removed and queried fast
• Partitioning pruning queries
17Wednesday, September 18, 13
![Page 134: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/134.jpg)
Partitioning
• As table size grows, queries eventually slows down, even with indexing
• Allows data added, removed and queried fast
• Partitioning pruning queries
• Manage partitions
17Wednesday, September 18, 13
![Page 135: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/135.jpg)
18Wednesday, September 18, 13
![Page 136: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/136.jpg)
Partitioning /Postgres
18Wednesday, September 18, 13
![Page 137: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/137.jpg)
Partitioning /Postgres
18Wednesday, September 18, 13
![Page 138: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/138.jpg)
Partitioning /Postgres
• Postgres partitioning
18Wednesday, September 18, 13
![Page 139: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/139.jpg)
Partitioning /Postgres
• Postgres partitioning
• Trigger based
18Wednesday, September 18, 13
![Page 140: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/140.jpg)
Partitioning /Postgres
• Postgres partitioning
• Trigger based
• Rule based
18Wednesday, September 18, 13
![Page 141: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/141.jpg)
Partitioning /Postgres
• Postgres partitioning
• Trigger based
• Rule based
• Lack of built-in Postgres partition management
18Wednesday, September 18, 13
![Page 142: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/142.jpg)
Partitioning /Postgres
• Postgres partitioning
• Trigger based
• Rule based
• Lack of built-in Postgres partition management
• Postgres partition management extension: pg_partman
18Wednesday, September 18, 13
![Page 143: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/143.jpg)
Partitioning /Postgres
• Postgres partitioning
• Trigger based
• Rule based
• Lack of built-in Postgres partition management
• Postgres partition management extension: pg_partman
• http://pgxn.org/dist/pg_partman/doc/pg_partman.html
18Wednesday, September 18, 13
![Page 144: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/144.jpg)
Partitioning /Postgres
• Postgres partitioning
• Trigger based
• Rule based
• Lack of built-in Postgres partition management
• Postgres partition management extension: pg_partman
• http://pgxn.org/dist/pg_partman/doc/pg_partman.html
• Tomorrow’s session at 11:30AM (When Postgres can’t....)
18Wednesday, September 18, 13
![Page 145: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/145.jpg)
19Wednesday, September 18, 13
![Page 146: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/146.jpg)
Partitioning /functional
19Wednesday, September 18, 13
![Page 147: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/147.jpg)
Partitioning /functional
Configuration Data Transaction data Session data
Configuration Tools Reporting Tools Monitoring Tools
Web Applications Other Applications
19Wednesday, September 18, 13
![Page 148: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/148.jpg)
Partitioning /functional
Configuration Data Transaction data Session data
Configuration Tools Reporting Tools Monitoring Tools
Web Applications Other Applications
• Partition data based on functionality
19Wednesday, September 18, 13
![Page 149: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/149.jpg)
Partitioning /functional
Configuration Data Transaction data Session data
Configuration Tools Reporting Tools Monitoring Tools
Web Applications Other Applications
• Partition data based on functionality
• Separate Postgres clusters
19Wednesday, September 18, 13
![Page 150: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/150.jpg)
Partitioning /functional
Configuration Data Transaction data Session data
Configuration Tools Reporting Tools Monitoring Tools
Web Applications Other Applications
• Partition data based on functionality
• Separate Postgres clusters
• Start with Separate schemas
19Wednesday, September 18, 13
![Page 151: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/151.jpg)
Partitioning /functional
Configuration Data Transaction data Session data
Configuration Tools Reporting Tools Monitoring Tools
Web Applications Other Applications
• Partition data based on functionality
• Separate Postgres clusters
• Start with Separate schemas
• No relationship between data
19Wednesday, September 18, 13
![Page 152: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/152.jpg)
Partitioning /functional
Configuration Data Transaction data Session data
Configuration Tools Reporting Tools Monitoring Tools
Web Applications Other Applications
• Partition data based on functionality
• Separate Postgres clusters
• Start with Separate schemas
• No relationship between data
• Help to spread the load across server
19Wednesday, September 18, 13
![Page 153: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/153.jpg)
Partitioning /functional
Configuration Data Transaction data Session data
Configuration Tools Reporting Tools Monitoring Tools
Web Applications Other Applications
• Partition data based on functionality
• Separate Postgres clusters
• Start with Separate schemas
• No relationship between data
• Help to spread the load across server
• Less complex compare to sharding!
19Wednesday, September 18, 13
![Page 154: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/154.jpg)
20Wednesday, September 18, 13
![Page 155: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/155.jpg)
pgbouncer
20Wednesday, September 18, 13
![Page 156: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/156.jpg)
pgbouncer
20Wednesday, September 18, 13
![Page 157: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/157.jpg)
pgbouncer
20Wednesday, September 18, 13
![Page 158: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/158.jpg)
pgbouncer
20Wednesday, September 18, 13
![Page 159: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/159.jpg)
pgbouncer
• A lightweight connection pooler
20Wednesday, September 18, 13
![Page 160: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/160.jpg)
pgbouncer
• A lightweight connection pooler
• Helps to reduce # of newly created connections on DB server
20Wednesday, September 18, 13
![Page 161: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/161.jpg)
pgbouncer
• A lightweight connection pooler
• Helps to reduce # of newly created connections on DB server
• Abstracts DBs from App
20Wednesday, September 18, 13
![Page 162: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/162.jpg)
pgbouncer
• A lightweight connection pooler
• Helps to reduce # of newly created connections on DB server
• Abstracts DBs from App
• Helps to instrument smooth and easy failover
20Wednesday, September 18, 13
![Page 163: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/163.jpg)
pgbouncer
• A lightweight connection pooler
• Helps to reduce # of newly created connections on DB server
• Abstracts DBs from App
• Helps to instrument smooth and easy failover
• Connection pooling Options
20Wednesday, September 18, 13
![Page 164: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/164.jpg)
pgbouncer
• A lightweight connection pooler
• Helps to reduce # of newly created connections on DB server
• Abstracts DBs from App
• Helps to instrument smooth and easy failover
• Connection pooling Options
• Session, Transaction, Statement pooling options
20Wednesday, September 18, 13
![Page 165: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/165.jpg)
pgbouncer
• A lightweight connection pooler
• Helps to reduce # of newly created connections on DB server
• Abstracts DBs from App
• Helps to instrument smooth and easy failover
• Connection pooling Options
• Session, Transaction, Statement pooling options
• Beware! Transaction pooling doesn’t support prepared transactions
20Wednesday, September 18, 13
![Page 166: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/166.jpg)
21Wednesday, September 18, 13
![Page 167: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/167.jpg)
Caching
21Wednesday, September 18, 13
![Page 168: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/168.jpg)
Caching
21Wednesday, September 18, 13
![Page 169: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/169.jpg)
Caching
• Memcached
21Wednesday, September 18, 13
![Page 170: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/170.jpg)
Caching
• Memcached
• Open source, High-performance distributed memory object caching system
21Wednesday, September 18, 13
![Page 171: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/171.jpg)
Caching
• Memcached
• Open source, High-performance distributed memory object caching system
• Speeds up dynamic web applications by alleviating database load.
21Wednesday, September 18, 13
![Page 172: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/172.jpg)
Caching
• Memcached
• Open source, High-performance distributed memory object caching system
• Speeds up dynamic web applications by alleviating database load.
• An in-memory key-value store for small chunks of arbitrary data
21Wednesday, September 18, 13
![Page 173: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/173.jpg)
Caching
• Memcached
• Open source, High-performance distributed memory object caching system
• Speeds up dynamic web applications by alleviating database load.
• An in-memory key-value store for small chunks of arbitrary data
• Redis
21Wednesday, September 18, 13
![Page 174: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/174.jpg)
Caching
• Memcached
• Open source, High-performance distributed memory object caching system
• Speeds up dynamic web applications by alleviating database load.
• An in-memory key-value store for small chunks of arbitrary data
• Redis
• Open source, advanced key-value store.
21Wednesday, September 18, 13
![Page 175: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/175.jpg)
Caching
• Memcached
• Open source, High-performance distributed memory object caching system
• Speeds up dynamic web applications by alleviating database load.
• An in-memory key-value store for small chunks of arbitrary data
• Redis
• Open source, advanced key-value store.
• Works with an in-memory & persistent dataset
21Wednesday, September 18, 13
![Page 176: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/176.jpg)
22Wednesday, September 18, 13
![Page 177: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/177.jpg)
Replication /built-in
22Wednesday, September 18, 13
![Page 178: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/178.jpg)
Replication /built-in
• Cluster Level Replication (Binary)
22Wednesday, September 18, 13
![Page 179: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/179.jpg)
Replication /built-in
• Cluster Level Replication (Binary)
• Streaming Replication
22Wednesday, September 18, 13
![Page 180: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/180.jpg)
Replication /built-in
• Cluster Level Replication (Binary)
• Streaming Replication
• WAL-only replication
22Wednesday, September 18, 13
![Page 181: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/181.jpg)
Replication /built-in
• Cluster Level Replication (Binary)
• Streaming Replication
• WAL-only replication
• Hybrid replication
22Wednesday, September 18, 13
![Page 182: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/182.jpg)
Replication /built-in
Master
• Cluster Level Replication (Binary)
• Streaming Replication
• WAL-only replication
• Hybrid replication
22Wednesday, September 18, 13
![Page 183: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/183.jpg)
Replication /built-in
Master
Failover • Cluster Level Replication (Binary)
• Streaming Replication
• WAL-only replication
• Hybrid replication
22Wednesday, September 18, 13
![Page 184: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/184.jpg)
Replication /built-in
Master
Failover
Read Salve 1
• Cluster Level Replication (Binary)
• Streaming Replication
• WAL-only replication
• Hybrid replication
22Wednesday, September 18, 13
![Page 185: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/185.jpg)
Replication /built-in
Master
Failover
Read Salve 1
• Cluster Level Replication (Binary)
• Streaming Replication
• WAL-only replication
• Hybrid replication
22Wednesday, September 18, 13
![Page 186: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/186.jpg)
Replication /built-in
Master
Failover
Read Salve 1
• Cluster Level Replication (Binary)
• Streaming Replication
• WAL-only replication
• Hybrid replication
22Wednesday, September 18, 13
![Page 187: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/187.jpg)
Replication /built-in
Master
Failover
Read Salve 1
Streaming Replication
• Cluster Level Replication (Binary)
• Streaming Replication
• WAL-only replication
• Hybrid replication
22Wednesday, September 18, 13
![Page 188: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/188.jpg)
Replication /built-in
Master
Failover
Read Salve 1
PITR!
Streaming Replication
• Cluster Level Replication (Binary)
• Streaming Replication
• WAL-only replication
• Hybrid replication
22Wednesday, September 18, 13
![Page 189: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/189.jpg)
Replication /built-in
Master
Failover
Read Salve 1
PITR!
PITR!
Streaming Replication
• Cluster Level Replication (Binary)
• Streaming Replication
• WAL-only replication
• Hybrid replication
22Wednesday, September 18, 13
![Page 190: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/190.jpg)
Replication /built-in
Master
Failover
Read Salve 1
PITR!
PITR!
PITR!Streaming Replication
• Cluster Level Replication (Binary)
• Streaming Replication
• WAL-only replication
• Hybrid replication
22Wednesday, September 18, 13
![Page 191: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/191.jpg)
Replication /built-in
Master
Failover
Read Salve 1
PITR!
PITR!
PITR!Streaming Replication
• Cluster Level Replication (Binary)
• Streaming Replication
• WAL-only replication
• Hybrid replication
22Wednesday, September 18, 13
![Page 192: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/192.jpg)
Replication /built-in
• Pros:
Master
Failover
Read Salve 1
PITR!
PITR!
PITR!Streaming Replication
• Cluster Level Replication (Binary)
• Streaming Replication
• WAL-only replication
• Hybrid replication
22Wednesday, September 18, 13
![Page 193: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/193.jpg)
Replication /built-in
• Pros:
• Built-in
Master
Failover
Read Salve 1
PITR!
PITR!
PITR!Streaming Replication
• Cluster Level Replication (Binary)
• Streaming Replication
• WAL-only replication
• Hybrid replication
22Wednesday, September 18, 13
![Page 194: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/194.jpg)
Replication /built-in
• Pros:
• Built-in
• Allows to open replicated database in read-only mode
Master
Failover
Read Salve 1
PITR!
PITR!
PITR!Streaming Replication
• Cluster Level Replication (Binary)
• Streaming Replication
• WAL-only replication
• Hybrid replication
22Wednesday, September 18, 13
![Page 195: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/195.jpg)
Replication /built-in
• Pros:
• Built-in
• Allows to open replicated database in read-only mode
• Cons:
Master
Failover
Read Salve 1
PITR!
PITR!
PITR!Streaming Replication
• Cluster Level Replication (Binary)
• Streaming Replication
• WAL-only replication
• Hybrid replication
22Wednesday, September 18, 13
![Page 196: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/196.jpg)
Replication /built-in
• Pros:
• Built-in
• Allows to open replicated database in read-only mode
• Cons:
• All or none
Master
Failover
Read Salve 1
PITR!
PITR!
PITR!Streaming Replication
• Cluster Level Replication (Binary)
• Streaming Replication
• WAL-only replication
• Hybrid replication
22Wednesday, September 18, 13
![Page 197: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/197.jpg)
Replication /built-in
• Pros:
• Built-in
• Allows to open replicated database in read-only mode
• Cons:
• All or none
• Doesn’t allow write on replicated database
Master
Failover
Read Salve 1
PITR!
PITR!
PITR!Streaming Replication
• Cluster Level Replication (Binary)
• Streaming Replication
• WAL-only replication
• Hybrid replication
22Wednesday, September 18, 13
![Page 198: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/198.jpg)
Replication /built-in
• Pros:
• Built-in
• Allows to open replicated database in read-only mode
• Cons:
• All or none
• Doesn’t allow write on replicated database
• Doesn’t work across major version
Master
Failover
Read Salve 1
PITR!
PITR!
PITR!Streaming Replication
• Cluster Level Replication (Binary)
• Streaming Replication
• WAL-only replication
• Hybrid replication
22Wednesday, September 18, 13
![Page 199: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/199.jpg)
Replication /built-in
• Pros:
• Built-in
• Allows to open replicated database in read-only mode
• Cons:
• All or none
• Doesn’t allow write on replicated database
• Doesn’t work across major version
• Postgres 9.2 (primary) does NOT replicate to 9.3 (secondary)
Master
Failover
Read Salve 1
PITR!
PITR!
PITR!Streaming Replication
• Cluster Level Replication (Binary)
• Streaming Replication
• WAL-only replication
• Hybrid replication
22Wednesday, September 18, 13
![Page 200: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/200.jpg)
Replication /built-in
• Pros:
• Built-in
• Allows to open replicated database in read-only mode
• Cons:
• All or none
• Doesn’t allow write on replicated database
• Doesn’t work across major version
• Postgres 9.2 (primary) does NOT replicate to 9.3 (secondary)
• Postgres 9.2.1 (primary) can replicate to 9.2.4 (secondary)
Master
Failover
Read Salve 1
PITR!
PITR!
PITR!Streaming Replication
• Cluster Level Replication (Binary)
• Streaming Replication
• WAL-only replication
• Hybrid replication
22Wednesday, September 18, 13
![Page 201: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/201.jpg)
Replication /built-in
• Pros:
• Built-in
• Allows to open replicated database in read-only mode
• Cons:
• All or none
• Doesn’t allow write on replicated database
• Doesn’t work across major version
• Postgres 9.2 (primary) does NOT replicate to 9.3 (secondary)
• Postgres 9.2.1 (primary) can replicate to 9.2.4 (secondary)
Master
Failover
Read Salve 1
PITR!
PITR!
PITR!Streaming Replication
https://wiki.postgresql.org/wiki/Binary_Replication_Tools
• Cluster Level Replication (Binary)
• Streaming Replication
• WAL-only replication
• Hybrid replication
22Wednesday, September 18, 13
![Page 202: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/202.jpg)
23Wednesday, September 18, 13
![Page 203: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/203.jpg)
Replication /built-in
23Wednesday, September 18, 13
![Page 204: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/204.jpg)
Replication /built-in
Read Salve 1
23Wednesday, September 18, 13
![Page 205: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/205.jpg)
Replication /built-in
Read Salve 1
DW System
23Wednesday, September 18, 13
![Page 206: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/206.jpg)
Replication /built-in
Read Salve 1
DW System
23Wednesday, September 18, 13
![Page 207: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/207.jpg)
Replication /built-in
Read Salve 1
DW System
postgres_fdw
23Wednesday, September 18, 13
![Page 208: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/208.jpg)
Replication /built-in
Read Salve 1
DW System
postgres_fdw
23Wednesday, September 18, 13
![Page 209: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/209.jpg)
Replication /built-in
Read Salve 1
DW System
postgres_fdw
23Wednesday, September 18, 13
![Page 210: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/210.jpg)
Replication /built-in
• postgres_fdwRead
Salve 1
DW System
postgres_fdw
23Wednesday, September 18, 13
![Page 211: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/211.jpg)
Replication /built-in
• postgres_fdw
• Postgres 9.3 feature
Read Salve 1
DW System
postgres_fdw
23Wednesday, September 18, 13
![Page 212: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/212.jpg)
Replication /built-in
• postgres_fdw
• Postgres 9.3 feature
• Allows to access data stored in external PostgreSQL servers
Read Salve 1
DW System
postgres_fdw
23Wednesday, September 18, 13
![Page 213: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/213.jpg)
Replication /built-in
• postgres_fdw
• Postgres 9.3 feature
• Allows to access data stored in external PostgreSQL servers
• cross version queries
Read Salve 1
DW System
postgres_fdw
23Wednesday, September 18, 13
![Page 214: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/214.jpg)
Replication /built-in
• postgres_fdw
• Postgres 9.3 feature
• Allows to access data stored in external PostgreSQL servers
• cross version queries
• Postgres 9.3 could query Postgres 9.1
Read Salve 1
DW System
postgres_fdw
23Wednesday, September 18, 13
![Page 215: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/215.jpg)
Replication /built-in
• postgres_fdw
• Postgres 9.3 feature
• Allows to access data stored in external PostgreSQL servers
• cross version queries
• Postgres 9.3 could query Postgres 9.1
• Application
Read Salve 1
DW System
postgres_fdw
23Wednesday, September 18, 13
![Page 216: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/216.jpg)
Replication /built-in
• postgres_fdw
• Postgres 9.3 feature
• Allows to access data stored in external PostgreSQL servers
• cross version queries
• Postgres 9.3 could query Postgres 9.1
• Application
• Run query remotely on slave db
Read Salve 1
DW System
postgres_fdw
23Wednesday, September 18, 13
![Page 217: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/217.jpg)
Replication /built-in
• postgres_fdw
• Postgres 9.3 feature
• Allows to access data stored in external PostgreSQL servers
• cross version queries
• Postgres 9.3 could query Postgres 9.1
• Application
• Run query remotely on slave db
• Data warehouse data refreshes
Read Salve 1
DW System
postgres_fdw
23Wednesday, September 18, 13
![Page 218: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/218.jpg)
24Wednesday, September 18, 13
![Page 219: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/219.jpg)
Replication /third-party tools
24Wednesday, September 18, 13
![Page 220: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/220.jpg)
Replication /third-party tools
24Wednesday, September 18, 13
![Page 221: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/221.jpg)
Replication /third-party tools
24Wednesday, September 18, 13
![Page 222: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/222.jpg)
Replication /third-party tools
• Table level Replication Tools (Trigger based)
24Wednesday, September 18, 13
![Page 223: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/223.jpg)
Replication /third-party tools
• Table level Replication Tools (Trigger based)
• Slony
24Wednesday, September 18, 13
![Page 224: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/224.jpg)
Replication /third-party tools
• Table level Replication Tools (Trigger based)
• Slony
• Bucardo
24Wednesday, September 18, 13
![Page 225: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/225.jpg)
Replication /third-party tools
• Table level Replication Tools (Trigger based)
• Slony
• Bucardo
• Mimeo: http://pgxn.org/dist/mimeo/
24Wednesday, September 18, 13
![Page 226: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/226.jpg)
Replication /third-party tools
• Table level Replication Tools (Trigger based)
• Slony
• Bucardo
• Mimeo: http://pgxn.org/dist/mimeo/
• Pros:
24Wednesday, September 18, 13
![Page 227: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/227.jpg)
Replication /third-party tools
• Table level Replication Tools (Trigger based)
• Slony
• Bucardo
• Mimeo: http://pgxn.org/dist/mimeo/
• Pros:
• Allows to open replicated database in read-write mode
24Wednesday, September 18, 13
![Page 228: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/228.jpg)
Replication /third-party tools
• Table level Replication Tools (Trigger based)
• Slony
• Bucardo
• Mimeo: http://pgxn.org/dist/mimeo/
• Pros:
• Allows to open replicated database in read-write mode
• Allows table/database level replication
24Wednesday, September 18, 13
![Page 229: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/229.jpg)
Replication /third-party tools
• Table level Replication Tools (Trigger based)
• Slony
• Bucardo
• Mimeo: http://pgxn.org/dist/mimeo/
• Pros:
• Allows to open replicated database in read-write mode
• Allows table/database level replication
• Allows rolling upgrade
24Wednesday, September 18, 13
![Page 230: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/230.jpg)
Replication /third-party tools
• Table level Replication Tools (Trigger based)
• Slony
• Bucardo
• Mimeo: http://pgxn.org/dist/mimeo/
• Pros:
• Allows to open replicated database in read-write mode
• Allows table/database level replication
• Allows rolling upgrade
• cross version replication is allowed
24Wednesday, September 18, 13
![Page 231: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/231.jpg)
Replication /third-party tools
• Table level Replication Tools (Trigger based)
• Slony
• Bucardo
• Mimeo: http://pgxn.org/dist/mimeo/
• Pros:
• Allows to open replicated database in read-write mode
• Allows table/database level replication
• Allows rolling upgrade
• cross version replication is allowed
• Multi-master replication
24Wednesday, September 18, 13
![Page 232: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/232.jpg)
Replication /third-party tools
• Table level Replication Tools (Trigger based)
• Slony
• Bucardo
• Mimeo: http://pgxn.org/dist/mimeo/
• Pros:
• Allows to open replicated database in read-write mode
• Allows table/database level replication
• Allows rolling upgrade
• cross version replication is allowed
• Multi-master replication
• Cons:
24Wednesday, September 18, 13
![Page 233: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/233.jpg)
Replication /third-party tools
• Table level Replication Tools (Trigger based)
• Slony
• Bucardo
• Mimeo: http://pgxn.org/dist/mimeo/
• Pros:
• Allows to open replicated database in read-write mode
• Allows table/database level replication
• Allows rolling upgrade
• cross version replication is allowed
• Multi-master replication
• Cons:
• Complicated to setup
24Wednesday, September 18, 13
![Page 234: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/234.jpg)
Replication /third-party tools
• Table level Replication Tools (Trigger based)
• Slony
• Bucardo
• Mimeo: http://pgxn.org/dist/mimeo/
• Pros:
• Allows to open replicated database in read-write mode
• Allows table/database level replication
• Allows rolling upgrade
• cross version replication is allowed
• Multi-master replication
• Cons:
• Complicated to setup
• Unknown territory
24Wednesday, September 18, 13
![Page 235: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/235.jpg)
25Wednesday, September 18, 13
![Page 236: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/236.jpg)
Sharding
25Wednesday, September 18, 13
![Page 237: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/237.jpg)
Sharding
25Wednesday, September 18, 13
![Page 238: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/238.jpg)
Sharding
25Wednesday, September 18, 13
![Page 239: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/239.jpg)
Sharding
• Sharding is the process of splitting up your data so it resides in different tables or often different physical databases.
25Wednesday, September 18, 13
![Page 240: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/240.jpg)
Sharding
• Sharding is the process of splitting up your data so it resides in different tables or often different physical databases.
• Application aware sharding
25Wednesday, September 18, 13
![Page 241: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/241.jpg)
Sharding
• Sharding is the process of splitting up your data so it resides in different tables or often different physical databases.
• Application aware sharding
• Application transparent sharding
25Wednesday, September 18, 13
![Page 242: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/242.jpg)
26Wednesday, September 18, 13
![Page 243: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/243.jpg)
Application aware sharding
26Wednesday, September 18, 13
![Page 244: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/244.jpg)
Application aware sharding
26Wednesday, September 18, 13
![Page 245: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/245.jpg)
Application aware sharding
26Wednesday, September 18, 13
![Page 246: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/246.jpg)
Application aware sharding
• http://instagram-engineering.tumblr.com/post/10853187575/sharding-ids-at-instagram
26Wednesday, September 18, 13
![Page 247: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/247.jpg)
Application aware sharding
• http://instagram-engineering.tumblr.com/post/10853187575/sharding-ids-at-instagram
• Postgres allows “logical” shards through Schema
26Wednesday, September 18, 13
![Page 248: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/248.jpg)
Application aware sharding
• http://instagram-engineering.tumblr.com/post/10853187575/sharding-ids-at-instagram
• Postgres allows “logical” shards through Schema
• Easy to move to “physical” shard later
26Wednesday, September 18, 13
![Page 249: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/249.jpg)
27Wednesday, September 18, 13
![Page 250: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/250.jpg)
Application transparent sharding
27Wednesday, September 18, 13
![Page 251: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/251.jpg)
Application transparent sharding
27Wednesday, September 18, 13
![Page 252: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/252.jpg)
28Wednesday, September 18, 13
![Page 253: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/253.jpg)
Sharding Challenges
28Wednesday, September 18, 13
![Page 254: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/254.jpg)
Sharding Challenges
• Reliability
28Wednesday, September 18, 13
![Page 255: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/255.jpg)
Sharding Challenges
• Reliability
• Distributed queries
28Wednesday, September 18, 13
![Page 256: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/256.jpg)
Sharding Challenges
• Reliability
• Distributed queries
• Cross-shard join
28Wednesday, September 18, 13
![Page 257: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/257.jpg)
Sharding Challenges
• Reliability
• Distributed queries
• Cross-shard join
• Auto-increment key management
28Wednesday, September 18, 13
![Page 258: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/258.jpg)
Sharding Challenges
• Reliability
• Distributed queries
• Cross-shard join
• Auto-increment key management
• Choosing shard key
28Wednesday, September 18, 13
![Page 259: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/259.jpg)
Sharding Challenges
• Reliability
• Distributed queries
• Cross-shard join
• Auto-increment key management
• Choosing shard key
• Shard schemes
28Wednesday, September 18, 13
![Page 260: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/260.jpg)
29Wednesday, September 18, 13
![Page 261: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/261.jpg)
Obstacles for Scaling Postgres
29Wednesday, September 18, 13
![Page 262: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/262.jpg)
Obstacles for Scaling Postgres
29Wednesday, September 18, 13
![Page 263: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/263.jpg)
Obstacles for Scaling Postgres
• Postgres table bloat
29Wednesday, September 18, 13
![Page 264: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/264.jpg)
Obstacles for Scaling Postgres
• Postgres table bloat
• FKs relationships
29Wednesday, September 18, 13
![Page 265: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/265.jpg)
Obstacles for Scaling Postgres
• Postgres table bloat
• FKs relationships
• Insufficient logging
29Wednesday, September 18, 13
![Page 266: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/266.jpg)
Obstacles for Scaling Postgres
• Postgres table bloat
• FKs relationships
• Insufficient logging
• Insufficient Caching
29Wednesday, September 18, 13
![Page 267: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/267.jpg)
Obstacles for Scaling Postgres
• Postgres table bloat
• FKs relationships
• Insufficient logging
• Insufficient Caching
• Insufficient Monitoring and Metrics
29Wednesday, September 18, 13
![Page 268: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/268.jpg)
Obstacles for Scaling Postgres
• Postgres table bloat
• FKs relationships
• Insufficient logging
• Insufficient Caching
• Insufficient Monitoring and Metrics
• ORMs
29Wednesday, September 18, 13
![Page 269: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/269.jpg)
Obstacles for Scaling Postgres
• Postgres table bloat
• FKs relationships
• Insufficient logging
• Insufficient Caching
• Insufficient Monitoring and Metrics
• ORMs
• Single Point of Failure
29Wednesday, September 18, 13
![Page 270: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/270.jpg)
Obstacles for Scaling Postgres
• Postgres table bloat
• FKs relationships
• Insufficient logging
• Insufficient Caching
• Insufficient Monitoring and Metrics
• ORMs
• Single Point of Failure
• Lack of communications between teams
29Wednesday, September 18, 13
![Page 271: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/271.jpg)
30Wednesday, September 18, 13
![Page 272: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/272.jpg)
Beyond Postgres
30Wednesday, September 18, 13
![Page 273: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/273.jpg)
Beyond Postgres
30Wednesday, September 18, 13
![Page 274: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/274.jpg)
Beyond Postgres
• Avoid serialization in application code
30Wednesday, September 18, 13
![Page 275: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/275.jpg)
Beyond Postgres
• Avoid serialization in application code
• Feature Flags
30Wednesday, September 18, 13
![Page 276: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/276.jpg)
Beyond Postgres
• Avoid serialization in application code
• Feature Flags
• Browse only mode (Read only mode)
30Wednesday, September 18, 13
![Page 277: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/277.jpg)
Beyond Postgres
• Avoid serialization in application code
• Feature Flags
• Browse only mode (Read only mode)
• Don’t use database for Queuing
30Wednesday, September 18, 13
![Page 278: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/278.jpg)
Beyond Postgres
• Avoid serialization in application code
• Feature Flags
• Browse only mode (Read only mode)
• Don’t use database for Queuing
• RabbitMQ
30Wednesday, September 18, 13
![Page 279: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/279.jpg)
Beyond Postgres
• Avoid serialization in application code
• Feature Flags
• Browse only mode (Read only mode)
• Don’t use database for Queuing
• RabbitMQ
• Reconsider options for Full Text Search
30Wednesday, September 18, 13
![Page 280: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/280.jpg)
Beyond Postgres
• Avoid serialization in application code
• Feature Flags
• Browse only mode (Read only mode)
• Don’t use database for Queuing
• RabbitMQ
• Reconsider options for Full Text Search
• tsearch provided by Postgres
30Wednesday, September 18, 13
![Page 281: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/281.jpg)
Beyond Postgres
• Avoid serialization in application code
• Feature Flags
• Browse only mode (Read only mode)
• Don’t use database for Queuing
• RabbitMQ
• Reconsider options for Full Text Search
• tsearch provided by Postgres
• Solr, Lucene
30Wednesday, September 18, 13
![Page 282: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/282.jpg)
31Wednesday, September 18, 13
![Page 283: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/283.jpg)
Further reading . . .
31Wednesday, September 18, 13
![Page 284: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/284.jpg)
Further reading . . .
• Scalable Internet Architectures - Theo Schlossnagle
• Web Operations: Keeping the Data On Time - John Allspaw , Jesse Robbins
• PostgreSQL 9.0 High Performance - Greg Smith
31Wednesday, September 18, 13
![Page 285: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/285.jpg)
32Wednesday, September 18, 13
![Page 286: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/286.jpg)
References
32Wednesday, September 18, 13
![Page 287: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/287.jpg)
References
32Wednesday, September 18, 13
![Page 288: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/288.jpg)
References
• http://www.postgresql.org/docs/
• http://www.circonus.com
• http://pgbouncer.projects.pgfoundry.org/doc/usage.html
• http://pgxn.org/dist/pg_partman/doc/pg_partman.html
• http://dalibo.github.io/pgbadger/
• https://github.com/omniti-labs/omnipitr
• http://instagram-engineering.tumblr.com/post/10853187575/sharding-ids-at-instagram
• http://memcached.org
• http://redis.io
• http://postgres-xc.sourceforge.net/docs/1_1/
32Wednesday, September 18, 13
![Page 289: Scaling postgres](https://reader034.vdocuments.net/reader034/viewer/2022042521/54b73e764a7959be4c8b485d/html5/thumbnails/289.jpg)
Thanks
• PostgresOpen Conference Committee
• OmniTI
• You!!
33Wednesday, September 18, 13