database driven 3d content management systems

7/30/2019 Database Driven 3D Content Management Systems

1/45

Database Driven3D Content Management

Systems

By Tim Child

3DMashUp


2/45

Outline

Biography

3D Content Applications

3D Content Categories

3D Geometry

3D Samples

3D UDTs & Functions System Architecture

3D Queries

Demo

GPGPU Acceleration

Web 2.0 Integration

Performance Challenges Further Development

Summary

Q & A Session


3/45

Biography

Tim Child

35 years experience of software development

Formerly

VP Oracle Corporation VP BEA Systems Inc.

VP Informix

Leader at Autodesk, Navteq, Intuit,

30 + years experience in 3D, CAD, GIS and DBMS

Built >10 Spatial DBMS Applications


4/45

3D CMS Issues

Ad Hoc

File/SCM based

Large number for file formats Files dispersed across the users network

Hard to Sync content between files

Poor Web 2.0 Integration Need of Real-Time Display


5/45

3D Content Applications

Augmented/Virtual Reality

Medical

10x growth in 5yr - Baptist Hospital

AEC / BIM

Arup predicts 70% growth in BIM data/yr

GIS / Earth Sciences /Environmental Sciences

CAD / CAM

Games / 3D Video


6/45

3D Content

Catagories

Vector Graphics Geometry

Mathematical Representations

Topological Rules

Many varieties of Objects

Many File Formats

Raster Graphics Samples

Measurements

Location X,Y,Z

Time t

Channels

R,G,B

Intensity levels

Discreet/Quantized

Noisy Need Calibration

Scales N * N * N


7/45

Representing

3D Geometry

Point

Line

Triangle

Polygon

Mesh


8/45

Representing 3D Samples

Pixel/Voxel

Quantized Intensity (x,y,z,t)

Point Cloud

Voxel Array


9/45

PostgreSQL

Advanced Capabilities Functions

Methods executing with in the server

Consider them a Delegate Methods Take the method to the data

Languages PGSQL

C (UDF)

Java Others,

Types Complex types and Arrays

Used Defined Types (UDT)

Examples include

XML, JSON, PostGIS

Indices BTree 1 Dimensional

RTree Multi Dimensional

Inverted Index Text searches

GIST Roll your own

User Defined Aggregate (UDA) Supports Sum, Min, Max, Average,


10/45

System Overview

Authentication

Service

HTTP

Server DBMS

Server

Web Pages

LDAPActive

Directory

DCC

Tools

(COLLADA)

SQL Table

Storage

WebDavFile

Storage

Web Browser

View

(WebGL)

Plug-

In

Existing

CMS/SCM

Web / File

Crawler

File

StorageFile

Storage

3D

MashUp

Service

3D

Analytics

Service


11/45

System Architecture


12/45

3D UDTs & Functions

Current 3D Schema

75+ Functions

15+ UDTs

20 + Tables


13/45

Recap

Native SQL representation of 3D objects

Supports

Vector Graphics

Raster Graphics 3D Content Aware

Vector math

4 x4 Homogenous coordinate transformations

Polygon to Polygon Intersections

Image processing


14/45

IMPORTANT

3D QUERIES

DBMS Goal:

Provide sufficient data to rapidly render the scene with acceptable fidelity


15/45

Camera View

Simplify Camera

Field of View to

a Cone


16/45

Spatial Cone Query


17/45

Spatial Indexing

Spatial Indexing allows scalable queries from > 100M Objects


18/45

View Coherent

Queries


19/45

Mesh Size

Stanford Bunny

70K Triangles

39K Vertices

XYZ Dragon

7.2M Triangles

3.6M Vertices

No point in using 7M triangles to render 64 x 64 pixels


20/45

Mesh Spatial

ProcessingEach mesh subdivided into1024 x 1024 x 1024 cells Each triangle centroid is

sorted on a Z Coordinate

0-

10

23

Morton Z Coordinates

Each 10 bit coordinate

Is bit interleaved

(Z9,Y9,X9, Z0,Y0,X0)


21/45

Spatially Clustered

Mesh Storage

PostgreSQL

Server

8 KB DBMS Pages

Each page Contains

50 100Mesh Triangles

Spatially Clustered

Spatially clustered meshes optimizes DBMS access!

Queries


22/45

LOD Processing

On loading a Mesh

Compute and Store

Reduced LOD

Representations


23/45

Triangle Decimation

Edge

Collapsing

Collapsing Cost

based on

Area and

Angle Between

Cost based algorithm


24/45

LOD Queries

Document

Table

Mesh

Table

Triangle

Cluster

LOD 0

Triangle

Cluster

LOD N

LOD Determined by Query

Based on Object Size

and Query Distance

Query pathfor

Detailed Objects

Query pathfor

Coarse Objects


25/45

PostgreSQL

Rules and Triggers

Select, Update, Delete Rules

Do ALSO Command, Command, Extends SQL operations

Do INSTEAD Command, Command, Replaces SQL operations

Insert, Update, Delete Trigger Events

Before Event Command

Or

After Event Command

SQL Views and Rules System hides implementation complexity


26/45

Recap

Camera SQL Queries Selects only whats visible

View coherent queries Optimizes scene data retrieval for camera motions

2 Level Spatial Indexing RTree Supports large number of (>100M) objects

Z-Ordering improves finer access for large meshes

Spatially Clustered Storage Optimizes partial mesh access

LOD processing Optimizes speed versus size for large meshes

PostgreSQL

UDTs and UDF encapsulate functionality in server

Rules System hides implementation complexity


27/45

Demo


28/45

GPGPUACCELERATION


29/45

GPGPU

Goals

OpenCL a PostgreSQL Procedural Language

OpenCL Kernels execute as SQL UDFs

Type Mapping, support common data types Vectors, Arrays, Images

Sorting

Used GPGPU based sorts in SQL queries


30/45

WEB 2.0

INTEGRATION


31/45

Web 2.0 Goals

Provide a Rich Web 2.0 UI

Integration with Web 2.0 Apps

Programmability

Tracking & Analytics


32/45

Rich UI

HTML 5

2D Canvas

WebGLDrag & Drop

.

Others?Silverlight for IE9


33/45

Web 2.0 Integration

Linking

Embedding

Social Bookmarking

Tagging

Drag & Drop

Annotations

Mash-Ups


34/45

Programmable

REST APIs Supports Browsers & other Web 2.0 Apps

SQL Queries Supports LAMP, RoR, POJ, ASP.Net,

XML Input / Output support Document Exchange and Import / Export

JSON Output provides Easy integration with JavaScript


35/45

Tracking & Analytics

Whos looking at what?

HTTP

Server

DBMSLogging

Service

Analytics

Service

Web Browsers

Content

QueriesHTTP

Request /

Response Log

Queries

Analytics

Report

Queries

View Web

Analytics

Reports


36/45

3D MashUp

Service

URILook-Up URIRequest

BrowserContent

Filter

Transform

ClipMerge

URI

Request

Web Site

A

Web Site

B

MashUp

Meta Data

HTTP

Request /

Response

HTTPRequest /

Response

HTTP

Request /Response

3D MashUp Engine

Acts as Proxy between Browser and Web Sites

Filtering, Transforming, Clipping and Merging

Content Responses


37/45

CONTENT

MANAGEMENT


38/45

CMS Goals

Track Revisions & Changes

Compare documents

Publish Documents

Annotation

Metadata Queries


39/45

Content Versioning

Rev 0 Rev 1 Rev 2 Rev 3

Rev 1.0

Rev 4

Rev 1.1

All Database Rows include Version ID

Versioning at the File, Section, and Record Levels


40/45

Contents Diff

SHA1 SHA1

SHA1 SHA1

SHA1 SHA1

SHA1 SHA1

Objects in the database for the 3D content

are signed Cryptographically

Revision N Revision M

Diff

Diff

Diff

Diff


41/45

PERFORMANCE


42/45

Performance Challenges

Visual Scenes

20K 5M Polygons

Near Real-Time Queries Queries 15m 30ms end user response

How much can I retrieve in 16.7 ms?

DB Size < RAM DB Size >= Ram Size && 1 T B


43/45

Further Development

Areas

Performance Investigations

What are the important queries?

Where are the bottlenecks?

Picking Operations Using a Cylinder to pick from a view

Editing Operations

Updating Geometry

OpenCl as Procedural Language

Allows cleaner OpenCl integration


44/45

Summary

Why a 3D in a Database? Shared

Multi-user

Scalable

Reliable

Extensible

Secure and well define Security Model

Rich Tools Set

Why choose PostgreSQL for 3D Advanced capabilities

Open and Opensource

Versatile Simple to use

Robust Developer Community

Leverages existing skill sets


45/45

Q&A