improving performance and flexibility of content listings using criteria api
TRANSCRIPT
![Page 1: Improving Performance and Flexibility of Content Listings Using Criteria API](https://reader031.vdocuments.net/reader031/viewer/2022021813/5885f2c21a28ab864f8b61f5/html5/thumbnails/1.jpg)
Improving Performance and Flexibility of Content Listings Using Criteria API Nils Breunese
![Page 2: Improving Performance and Flexibility of Content Listings Using Criteria API](https://reader031.vdocuments.net/reader031/viewer/2022021813/5885f2c21a28ab864f8b61f5/html5/thumbnails/2.jpg)
Public Broadcaster since 1926 The Netherlands
![Page 3: Improving Performance and Flexibility of Content Listings Using Criteria API](https://reader031.vdocuments.net/reader031/viewer/2022021813/5885f2c21a28ab864f8b61f5/html5/thumbnails/3.jpg)
Online since 1994 Open-source CMS released in 1997
![Page 4: Improving Performance and Flexibility of Content Listings Using Criteria API](https://reader031.vdocuments.net/reader031/viewer/2022021813/5885f2c21a28ab864f8b61f5/html5/thumbnails/4.jpg)
Using Magnolia since 2010 Still migrating websites
![Page 5: Improving Performance and Flexibility of Content Listings Using Criteria API](https://reader031.vdocuments.net/reader031/viewer/2022021813/5885f2c21a28ab864f8b61f5/html5/thumbnails/5.jpg)
Tens of thousands of pages Multiple sites like that
![Page 6: Improving Performance and Flexibility of Content Listings Using Criteria API](https://reader031.vdocuments.net/reader031/viewer/2022021813/5885f2c21a28ab864f8b61f5/html5/thumbnails/6.jpg)
Overview pages Lots of them
![Page 7: Improving Performance and Flexibility of Content Listings Using Criteria API](https://reader031.vdocuments.net/reader031/viewer/2022021813/5885f2c21a28ab864f8b61f5/html5/thumbnails/7.jpg)
Thanks for the warning… Even 10 seconds would be way too long
WARN info.magnolia.module.cache.filter.CacheFilter -- The following URL took longer than 10 seconds (63969 ms) to render. This might cause timeout exceptions on other requests to the same URI.
![Page 8: Improving Performance and Flexibility of Content Listings Using Criteria API](https://reader031.vdocuments.net/reader031/viewer/2022021813/5885f2c21a28ab864f8b61f5/html5/thumbnails/8.jpg)
Overview models Standard Templating Kit
![Page 9: Improving Performance and Flexibility of Content Listings Using Criteria API](https://reader031.vdocuments.net/reader031/viewer/2022021813/5885f2c21a28ab864f8b61f5/html5/thumbnails/9.jpg)
Tracking back from the template newsOverview.ftl
(...) [#assign pager = model.pager] [#assign newsList = cmsfn.asContentMapList(pager.pageItems)!] (...)
![Page 10: Improving Performance and Flexibility of Content Listings Using Criteria API](https://reader031.vdocuments.net/reader031/viewer/2022021813/5885f2c21a28ab864f8b61f5/html5/thumbnails/10.jpg)
Constructing the pager AbstractItemListModel
public STKPager getPager() throws RepositoryException { (...) return new STKPager(currentPageLink, getItems(), content); }
![Page 11: Improving Performance and Flexibility of Content Listings Using Criteria API](https://reader031.vdocuments.net/reader031/viewer/2022021813/5885f2c21a28ab864f8b61f5/html5/thumbnails/11.jpg)
Four step pipeline AbstractItemListModel
public Collection<Node> getItems() throws RepositoryException { List<Node> itemsList = search(); this.filter(itemsList); this.sort(itemsList); itemsList = this.shrink(itemsList); return itemsList;}
1
23
4
![Page 12: Improving Performance and Flexibility of Content Listings Using Criteria API](https://reader031.vdocuments.net/reader031/viewer/2022021813/5885f2c21a28ab864f8b61f5/html5/thumbnails/12.jpg)
Step 1a: Constructing the query TemplateCategoryUtil
public static List<Node> getContentListByTemplateNames(...) { (...) StringBuffer sql = new StringBuffer( "select * from nt:base where jcr:path like '" + path + "/%'"); (...add 'mgnl:template=' clauses...) (...add 'ORDER BY' clauses...) return getWrappedNodesFromQuery(sql.toString(), repository, maxResultSize); } maxResultSize == Integer.MAX_VALUE
![Page 13: Improving Performance and Flexibility of Content Listings Using Criteria API](https://reader031.vdocuments.net/reader031/viewer/2022021813/5885f2c21a28ab864f8b61f5/html5/thumbnails/13.jpg)
Step 1b: Executing the query TemplateCategoryUtil
public static List<Node> getContentListByTemplateNames(...) { (...) NodeIterator items = QueryUtil.search( repository, sql.toString(), Query.SQL, NodeTypes.Content.NAME); }
![Page 14: Improving Performance and Flexibility of Content Listings Using Criteria API](https://reader031.vdocuments.net/reader031/viewer/2022021813/5885f2c21a28ab864f8b61f5/html5/thumbnails/14.jpg)
Step 2: Filtering the item list STKDateContentUtil
public static void filterDateContentList(...) { CollectionUtils.filter(itemsList, new Predicate() { @Override public boolean evaluate(Object object) { (...) return date.after(minDate) && date.before(maxDate); } });}
![Page 15: Improving Performance and Flexibility of Content Listings Using Criteria API](https://reader031.vdocuments.net/reader031/viewer/2022021813/5885f2c21a28ab864f8b61f5/html5/thumbnails/15.jpg)
Step 3: Time to sort STKDateContentUtil
public static void sortDateContentList(...) { Collections.sort(itemsList, new Comparator<Node>() { @Override public int compare(Node c1, Node c2) { (...) if (StringUtils.equals(sortDirection, ASCENDING)) { return date2.compareTo(date1); } return date1.compareTo(date2); } });}
![Page 16: Improving Performance and Flexibility of Content Listings Using Criteria API](https://reader031.vdocuments.net/reader031/viewer/2022021813/5885f2c21a28ab864f8b61f5/html5/thumbnails/16.jpg)
Step 4: Shrinking the list STKTemplatingFunctions
public List<Node> cutList(List<Node> itemsList, final int maxResults) { if (itemsList.size() > maxResults) { return itemsList.subList(0, maxResults); } return itemsList;}
NewsOverviewModel passes Integer.MAX_VALUE, so shrink does effectively nothing in this case
![Page 17: Improving Performance and Flexibility of Content Listings Using Criteria API](https://reader031.vdocuments.net/reader031/viewer/2022021813/5885f2c21a28ab864f8b61f5/html5/thumbnails/17.jpg)
Step 5: Get the items from the pager STKPager
public Collection getPageItems() { Collection subList = items; int offset = getOffset(); if (count > 0) { int limit = maxResultsPerPage + offset; if (items.size() < limit) { limit = count; } subList = ((List) items).subList(offset, limit); } return subList;}
maxResultsPerPage is typically something like 20
![Page 18: Improving Performance and Flexibility of Content Listings Using Criteria API](https://reader031.vdocuments.net/reader031/viewer/2022021813/5885f2c21a28ab864f8b61f5/html5/thumbnails/18.jpg)
When this becomes a problem We have multiple sites like this
select * from nt:base where jcr:path like '/siteX/news/%' AND
mgnl:template = 'standard-templating-kit:pages/stkNews'
20000 pages under website:/siteX/news
Four step pipeline returns STKPager with 20000 items (page nodes)
[#assign model.pager]
[#assign newsList = cmsfn.asContentMapList(pager.pageItems)!]
STKPager returns list with 20 page nodes
19980 Node objects created, but not rendered
![Page 19: Improving Performance and Flexibility of Content Listings Using Criteria API](https://reader031.vdocuments.net/reader031/viewer/2022021813/5885f2c21a28ab864f8b61f5/html5/thumbnails/19.jpg)
A query could do all steps at once JCR queries are pretty flexible
![Page 20: Improving Performance and Flexibility of Content Listings Using Criteria API](https://reader031.vdocuments.net/reader031/viewer/2022021813/5885f2c21a28ab864f8b61f5/html5/thumbnails/20.jpg)
Everything in a single JCR query Only 20 nodes returned
SELECT * FROM nt:base WHERE jcr:path LIKE '/siteX/news/%' AND
mgnl:template = 'standard-templating-kit:pages/stkNews'
AND jcr:created < cast('2016-06-07T00:00:00.000Z' AS DATE)
ORDER BY date ASCENDING
LIMIT 20 OFFSET 20
Search
Filter
Sort
Paging
![Page 21: Improving Performance and Flexibility of Content Listings Using Criteria API](https://reader031.vdocuments.net/reader031/viewer/2022021813/5885f2c21a28ab864f8b61f5/html5/thumbnails/21.jpg)
Criteria API For those familiar with Hibernate/JPA
Criteria criteria = JCRCriteriaFactory.createCriteria() .setBasePath("/siteX/news") .add(Restrictions.eq( "@mgnl:template", "standard-templating-kit:pages/stkNews")) .add(Restrictions.betweenDates("@jcr:created", minDate, maxDate)) .addOrder(Order.asc("date")) .setPaging(20, 1); ResultIterator<...> items = criteria.execute(session).getItems();
SortPaging
Filter
Search
![Page 22: Improving Performance and Flexibility of Content Listings Using Criteria API](https://reader031.vdocuments.net/reader031/viewer/2022021813/5885f2c21a28ab864f8b61f5/html5/thumbnails/22.jpg)
Criteria API for Magnolia CMS Magnolia module by Openmind
![Page 24: Improving Performance and Flexibility of Content Listings Using Criteria API](https://reader031.vdocuments.net/reader031/viewer/2022021813/5885f2c21a28ab864f8b61f5/html5/thumbnails/24.jpg)
Custom pager Only a single page worth of items
public class VtkPager<T> extends STKPager { private final List<? extends T> items; private final int pageSize; private final int count; (...) @Override public List<? extends T> getPageItems() { return items; } }
![Page 25: Improving Performance and Flexibility of Content Listings Using Criteria API](https://reader031.vdocuments.net/reader031/viewer/2022021813/5885f2c21a28ab864f8b61f5/html5/thumbnails/25.jpg)
Use it in your model classes VtkContentListModel (vpro)
public abstract class VtkContentListModel ... { protected final VtkPager<ContentMap> pager; @Override public String execute() { pager = createPager(); return super.execute(); } protected abstract VtkPager<T> createPager(); (...) }
![Page 26: Improving Performance and Flexibility of Content Listings Using Criteria API](https://reader031.vdocuments.net/reader031/viewer/2022021813/5885f2c21a28ab864f8b61f5/html5/thumbnails/26.jpg)
Concrete Example VtkNewsOverviewModel (vpro)
@Overrideprotected VtkPager<Node> createPager() { (...) AdvancedResult result = JCRCriteriaFactory.createCriteria() .setBasePath(path) .add(Restrictions.in("@mgnl:template", templates)) .add(Restrictions.betweenDates("@jcr:created", minDate, maxDate)) .addOrder(Order.asc("date")) .setPaging(itemsPerPage, pageNumberStartingFromOne) .execute(session);
List<Node> items = new ArrayList<>(); for (AdvancedResultItem item : result.getItems()) { items.add(item.getJCRNode()); } int count = result.getTotalSize(); return new VtkPager<>(link, items, content, itemsPerPage, count); }
![Page 27: Improving Performance and Flexibility of Content Listings Using Criteria API](https://reader031.vdocuments.net/reader031/viewer/2022021813/5885f2c21a28ab864f8b61f5/html5/thumbnails/27.jpg)
Still this. Was it all for nothing? :o(
WARN info.magnolia.module.cache.filter.CacheFilter -- The following URL took longer than 10 seconds (63969 ms) to render. This might cause timeout exceptions on other requests to the same URI.
![Page 28: Improving Performance and Flexibility of Content Listings Using Criteria API](https://reader031.vdocuments.net/reader031/viewer/2022021813/5885f2c21a28ab864f8b61f5/html5/thumbnails/28.jpg)
Example VtkNewsOverviewModel (vpro)
@Overrideprotected VtkPager<Node> createPager() { (...) AdvancedResult result = JCRCriteriaFactory.createCriteria() .setBasePath(path) .add(Restrictions.in("@mgnl:template", templates)) .add(Restrictions.betweenDates("@jcr:created", minDate, maxDate)) .addOrder(Order.asc("date")) .setPaging(itemsPerPage, pageNumberStartingFromOne) .execute(session);
List<Node> items = new ArrayList<>(); for (AdvancedResultItem item : result.getItems()) { items.add(item.getJCRNode()); } int count = result.getTotalSize(); return new VtkPager<>(link, items, content, itemsPerPage, count); }
This call takes 10-60+ seconds!
![Page 29: Improving Performance and Flexibility of Content Listings Using Criteria API](https://reader031.vdocuments.net/reader031/viewer/2022021813/5885f2c21a28ab864f8b61f5/html5/thumbnails/29.jpg)
AdvancedResultImpl (jcr-criteria)
@Overridepublic int getTotalSize() { if (totalResults == null) { int queryTotalSize = -1; try { // jcrQueryResult instanceof JackrabbitQueryResult) { Method m = jcrQueryResult.getClass().getMethod("getTotalSize"); queryTotalSize = (int) m.invoke(jcrQueryResult); } catch (InvocationTargetException | IllegalAccessException e) { LOG.error(e.getMessage(), e); } catch (NoSuchMethodException e) { } if (queryTotalSize == -1 && (itemsPerPage == 0 || applyLocalPaging)) { try { totalResults = (int) jcrQueryResult.getNodes().getSize(); } catch (RepositoryException e) { // ignore, the standard total size will be returned } } if (queryTotalSize == -1) { totalResults = queryCounter.getAsInt(); } else { totalResults = queryTotalSize; } } return totalResults; }
We end up here
![Page 30: Improving Performance and Flexibility of Content Listings Using Criteria API](https://reader031.vdocuments.net/reader031/viewer/2022021813/5885f2c21a28ab864f8b61f5/html5/thumbnails/30.jpg)
jackrabbit-core 2.8.0
protected void getResults(long size) throws RepositoryException { (...) result = executeQuery(maxResultSize); // Lucene query (...) // Doesn’t use result.getSize(), call collectScoreNodes(...) }
private void collectScoreNodes(...) { while (collector.size() < maxResults) { ScoreNode[] sn = hits.nextScoreNodes(); (...) // check access if (isAccessGranted(sn)) { collector.add(sn); } else { invalid++; } }} QueryResultImpl
![Page 31: Improving Performance and Flexibility of Content Listings Using Criteria API](https://reader031.vdocuments.net/reader031/viewer/2022021813/5885f2c21a28ab864f8b61f5/html5/thumbnails/31.jpg)
It used to be fast! https://issues.apache.org/jira/browse/JCR-3858
![Page 32: Improving Performance and Flexibility of Content Listings Using Criteria API](https://reader031.vdocuments.net/reader031/viewer/2022021813/5885f2c21a28ab864f8b61f5/html5/thumbnails/32.jpg)
jackrabbit-core 2.10.0+
protected void getResults(long size) throws RepositoryException { (...) if (sizeEstimate) { numResults = result.getSize(); // Use count from Lucene } else { // do things the Jackrabbit 2.8.0 way (...) } (...) }
QueryResultImpl
![Page 33: Improving Performance and Flexibility of Content Listings Using Criteria API](https://reader031.vdocuments.net/reader031/viewer/2022021813/5885f2c21a28ab864f8b61f5/html5/thumbnails/33.jpg)
Enable Jackrabbit’s 'sizeEstimate' Jackrabbit 2.10+
<SearchIndex class="org.apache.jackrabbit.core.query.lucene.SearchIndex"> (...) <param name="sizeEstimate" value="true"/></SearchIndex>
![Page 34: Improving Performance and Flexibility of Content Listings Using Criteria API](https://reader031.vdocuments.net/reader031/viewer/2022021813/5885f2c21a28ab864f8b61f5/html5/thumbnails/34.jpg)
Rendering times down to 1-2 seconds Bingo
![Page 35: Improving Performance and Flexibility of Content Listings Using Criteria API](https://reader031.vdocuments.net/reader031/viewer/2022021813/5885f2c21a28ab864f8b61f5/html5/thumbnails/35.jpg)
Time for questions
Anyone?