compositie filtering of application datasets
TRANSCRIPT
Composite Filtering of Datasets- - -
Simplify Data Filtering Functions & Increase Performance
Coffee ShopsServes Cold Brew?
Yes NoServes what food?
Pastry Sandwiches Fruit
Reviews
0 5
A Simple App for filtering coffee shops on a map
With 0 filters applied, the complete set of coffee shops appear
Data points in “map-space”
Coffee ShopsServes Cold Brew?
Yes NoServes what food?
Pastry Sandwiches Fruit
Reviews
0 5
A Simple App for filtering coffee shops on a map
With 0 filters applied, the complete set of coffee shops appear
Data points in “data-space”
Coffee ShopsServes Cold Brew?
Yes NoServes what food?
Pastry Sandwiches Fruit
Reviews
0 5
A Simple App for filtering coffee shops on a map
When the Cold Brew filter applied, we get a “filtered set” of data points.
How do we translate a user-interaction (e.g., click) into a filtered-set?
Coffee ShopsServes Cold Brew?
Yes NoServes what food?
Pastry Sandwiches Fruit
Reviews
0 5
A Simple App for filtering coffee shops on a map
How does an application translate a user-interaction (e.g., click) into a filtered-set?
Coffee ShopsServes Cold Brew?
Yes NoServes what food?
Pastry Sandwiches Fruit
Reviews
0 5
“DB-centric” filtering approach
1
Adding a single filter1) User click’s on “cold brew” filter
Coffee ShopsServes Cold Brew?
Yes NoServes what food?
Pastry Sandwiches Fruit
Reviews
0 5
“DB-centric” filtering approach
1
{“cold brew”: true }
2
Adding a single filter1) User click’s on “cold brew” filter2) Cold Brew filter criteria sent to database
Coffee ShopsServes Cold Brew?
Yes NoServes what food?
Pastry Sandwiches Fruit
Reviews
0 5
“DB-centric” filtering approach
1
{“cold brew”: true }
2
3
Adding a single filter1) User click’s on “cold brew” filter2) Cold Brew filter criteria sent to database3) Filtered set of coffee shops returned
Coffee ShopsServes Cold Brew?
Yes NoServes what food?
Pastry Sandwiches Fruit
Reviews
0 5
1
{ “cold brew”: true “food” : sandwiches }
2
3
Adding an additional filter 1) User click’s on “food” filter2) Cold Brew and Food filter-criteria sent to database.3) Filtered set of coffee shops returned
“DB-centric” filtering approach
Coffee ShopsServes Cold Brew?
Yes NoServes what food?
Pastry Sandwiches Fruit
Reviews
0 51
{ “cold brew”: true “food” : sandwiches “reviews”: [3,5] }
2
3
Adding a 3rd filter 1) User click’s on “food” filter2) Cold Brew, Food, Review Range filter-criteria sent to database3) Filtered set of coffee shops returned
“Stateless” filtering approach
Coffee ShopsServes Cold Brew?
Yes NoServes what food?
Pastry Sandwiches Fruit
Reviews
0 51
{ “cold brew”: true “food” : sandwiches “reviews”: [3,5] }
2
3
Inefficiency identified This approach does not leverage the DB processing performed on previous
user interactions
“Stateless” filtering approach
Details1) Each user interaction adds single criteria, but ALL criteria sent and processed
after each interaction
2) The DB filter operation gets more expensive as the number of potentially submitted filters increases
3) “Unselecting” a filter requires another database operation, even though the application had the resultant dataset prior to toggling this filter on
Inefficiency identified This approach does not leverage the DB processing performed on
previous user clicks
“Stateless” filtering approach
“Stateless” filtering approach
Additional downsides of this DB-centric approach:1) A single “catch-all” DB function gets increasingly complicated as number and
types of possible filters grow
2) A single DB function has to handle “null” filter-critera - it needs to be ready to accept values for all filter types - but all might not be sent
Example: Cold-brew filter sent, but food and reviews filter values are undefined; a single DB function needs to be ready to process food and review even though they aren’t in the incoming request
Alternative: “Stateful” filtering approach
1) Store the result of each individual filter set on browser2) Move set intersections from DB to the browser
{ “coffeeShops”:{ totalSet: [1,2,3,…,n] }
}
Angular service object (Keep track of the ID set associated with each individual filter)
On App load: retrieve the total set, store it as ID array
total-set
“Stateful” filtering
Coffee ShopsServes Cold Brew?
Yes NoServes what food?
Pastry Sandwiches Fruit
Reviews
0 5
{ “coffeeShops”:{ totalSet: [1,2,3,…,n], finalSet:[1,2,3,…,n] }
}
When no filters: total set = final set
total-set
Angular service object (Keep track of the ID set associated with each individual filter)
“Stateful” filtering
Coffee ShopsServes Cold Brew?
Yes NoServes what food?
Pastry Sandwiches Fruit
Reviews
0 5
{ “coffeeShops”:{ totalSet: [1,2,3,…,n], filteredSets: { coldBrew:{resultSet: [2,4, …, n] } }
}
Serves Cold Brew?
Yes NoServes what food?
Pastry Sandwiches Fruit
Reviews
0 5
Coffee Shops
Add a filter
coldbrew-set
Angular service object (Keep track of the ID set associated with each individual filter)
“Stateful” filtering
{ “coffeeShops”:{ totalSet: [1,2,3,…,n], filteredSets: { coldBrew:{resultSet: [2,4, …, n] } finalSet:[2,4,…,n] }
}
single filter result-set = final set
coldbrew-set
Angular service object (Keep track of the ID set associated with each individual filter)
“Stateful” filtering
Serves Cold Brew?
Yes NoServes what food?
Pastry Sandwiches Fruit
Reviews
0 5
Coffee Shops
{ “coffeeShops”:{ totalSet: [1,2,3,…,n], filteredSets: { coldBrew:{ resultSet: [2,4, …, n], food: { resultSet: [4,5, … n] } }
}
Add additional filter
coldbrew-set
food-set
Serves Cold Brew?
Yes NoServes what food?
Pastry Sandwiches Fruit
Reviews
0 5
Coffee Shops
Angular service object (Keep track of the ID set associated with each individual filter)
This set stored during previous user interaction
“Stateful” filtering
{ “coffeeShops”:{ totalSet: [1,2,3,…,n], filteredSets: { coldBrew:{ resultSet: [2,4, …, n], food: { resultSet: [4,5, … n] }, finalSet: [4,7 … n] }
}
Angular service object
coldbrew-set food-set
INTERSECT ID ARRAYS IN THE ANGULAR SERVICE!
Intersection of the two arrays = final set
“Stateful” filtering
Serves Cold Brew?
Yes NoServes what food?
Pastry Sandwiches Fruit
Reviews
0 5
Coffee Shops
{ “coffeeShops”:{ totalSet: [1,2,3,…,n], filteredSets: { coldBrew:{ resultSet: [2,4, …, n], food: { resultSet: [4,5, … n] }, finalSet: [2,4 … n] }
}
Angular service object
coldbrew-set food-set
Removing a filter does not require another round trip to DB
Remove a filter
“Stateful” filtering
Serves Cold Brew?
Yes NoServes what food?
Pastry Sandwiches Fruit
Reviews
0 5
Coffee Shops
Coffee ShopsAttributes:
Serves Cold Brew?Yes No
Serves what food?Pastry Sandwiches Fruit
Reviews
0 5
“Stateful” filtering
Angular program flow for “stateful filtering”.
1. User clicks on Cold Brew “yes”, fires an ng-click with info about the filter.
2. Controller tells service what type of filter this is (cold brew) and the value (yes)
3. Service checks a state object to see if this particular filter already has a value and a result set associated with it (e.g., previous UX had cold brew : no filter); if so it clears that value
4. Service determines which API endpoint to request a the filtered set from, and passes on the relevant request data { coldbrew: true} onto the endpoint
5. Endpoint returns a integer array of IDs meeting the filter-criteria to the service
6. The service completes the intersection of result sets from all currently toggled on filters
7. The service broadcasts the final intersected ID set to an listeners (e.g. a map controller the uses the ID array to show or hide map points).
“Stateful” filtering
1) Distributes processing across user interactions2) Eliminates DB trip for “filter-removal” event3) More performant - each DB operation is less complex; Intersections on the
browser are very quick (e.g, intersecting 4 10000 element integer arrays took 0.033s)
4) Simplifies DB function development5) Allows for a more customized set of DB filters.
Example Application instance “AAA”: food filter operates like an AND (pastries AND sandwiches) Application instance “BBB”: the filter operates like and OR (pastries OR sandwiches)You can easily have separate simple API endpoints/DB functions that handle each of the above cases for this type of filter instead of multiple large, complicated DB functions that handle all possible filter combinations in a simple request.
Benefits