Eudoxus.grTechnology & Architecture

 Fotis Stamatelopoulos (@fstama)

  

JHUG Meeting @CollabAthensMay 28, 2011

What is Eudoxus - Disclaimer• A distributed system that supports and streamlines

processes and operations related to the textbooks distribution for the higher education institutions of Greece

 • Stakeholder: Hellenic Ministry of Education

http://minedu.gov.gr/ • Developed, operated and supported by GRNET

http://grnet.gr/  • Users: Students, Secretariats, Publishers, Distribution

Points (e.g. Bookstores), related Ministry depts, Courier service, Helpdesk- More than 270.000 individual users

• Multiple APIs and integration with other ISshttp://eudoxus.gr

Presentation Focus

• Design and software architectureo by design based on 100% FOSS infrastructureo full JEE stack, rich web clients (GWT/Java)o scalability, performance and bottlenecks

 • Building within a very tight schedule and ever changing

requirementso agile methodologyo human environmento “collateral damage”

Design Goals - Challenges

• Tight schedule, fixed milestones - deadlineso Incrementally release modules in production

• Not finalized requirementso Be flexible, handle changing requirements

• High availabilityo Redundant architectureo Live application updates – no downtime

Design Goals – Challenges (cont'd)

• Impossible to predict usage load– design for scalability, – elastically use infrastructure to serve growing usage, – prepare for the worst– efficient use of hardware infrastructure

• Safeguard the transactional nature of related processes

• Synchronize data and connect to other systems

Technology decisions

• Web-based user interfaces

• Support various APIs• RESTFul APIs (JSON / XML payload)• use SOAP-based web services for integration in

specific cases (e.g. courier service)

• Java-based core (business logic tier)

• Javascript-based / GWT-based (Java again!) rich GUIs

Technology decisions (cont'd)

• Back-end storage: RDBMS vs noSQL decisiono noSQL options offer better scalability than typical RDBMso needed at least a minimal transactional core - eventual

consistency is not acceptableo dropped the initial hybrid approach - full RDBMS design

• Use caching as much as possible

• Authentication / Credentials• Stateless mechanism (killed JAAS)• Support both–Shibboleth-based authentication for students– form based login (other user classes)

High level architecture

Infrastructure Software• Google Web Toolkit (GWT) for the rich GUI clients (browser)

• Apache, nginx front-ends• Varnish web cache

• JBoss Application Servero stateless EJBs implementing core business logico JPA ORM (hibernate substrate)o JMS Queues and Message Driven Beanso JMX MBeans for scheduled tasks and maintanance

• PostgreSQL RDBMS

• Solr (Lucene server) indexing / search server

Technical Architecture

search.,eudoxus.gr

Losing the HTTP Session

• No HTTP session => workers don't need to replicate – synchronize

• Application state is kept in the client• Authentication generates a short-lived auth token

for the specific user & session• Token is kept in a cookie and send back with every

request (over SSL), be it a REST or GWT-remoting call– In GSS used the token to encrypt the header

• Completely stateless middle tier

The worker pool

The worker pool (current)

The worker

• Stateless EJBs implement business functions• POJOs (JPA) for the data model• Mbeans for management tasks (no UI – access via

the jmx-console)• Persistent JMS queues (hornetQ) & MDBs for

syncing with other systems, serializing reports, handling file indexing (Solr)

• REST API implementaiton will move to JAX-RS (Jersey)

The data store

• Single RDBMS (postgresql) instance with fallback• “Plan B” alternatives– Cold replicas for reports– Sharding / hot replication (one writer / multiple readers)– Moving parts of the data to NoSQL or even use Solr as

a NoSQL data store• Currently we handle the load without sweating

• Varnish caches specific HTML output and REST responses for URLs that do not change (Eudoxus controls this via http headers)

• 2nd level caching - EHCache (per worker)• We will be introducing distributed read-write

second level caching (working with Infinispan right now)

More on caching

Meeting Tight Schedules

• Typical setupo VCS (mercurial), wiki for requirements, issue trackero dedicated release engineer, dedicated DBAo local development -> test server -> production servero FOSS dev tools (Eclipse, ant, etc)

• Agile approach• Documentation was the first victim of the pressing

schedule :( “updated unit tests” was the second :o• A few statistics on the team

o 5 months from scratch to productiono ~10 developerso ~10 testers (functional testing)o ~20 helpdesk members for end-user supporto Full GRNET NOC support

http://eudoxus.gr

linkedin.com/in/fstamatelopoulos

@fstama