project meniscus
TRANSCRIPT
Project Meniscus
Simplex Internet / Research Institute / Jeong, Ki-Chul
2013. 4. 29
Meniscus?
• LAAS(Logging-as-a-Service) Backend API
• Python based system
• Event - collection, transit, processing
• Large-scale
• Cloud Logging
APIS
Tenant API
• Tenant = {Host, Event Producer, Profile}
• A Tenant = A Consumer
• Event Producer = APP or Service
• Profile = [Event Producer, …]
• Host = Node , link to a profile(1:1)
Tenant API
• Requests to coordinator worker
• Tenant, Host, Event Producer, Profile
=> CRUD Available
• has a TOKEN
=> validate, reset a token
Correlator API
• Requests
To a Correlator worker server
From a Consumer server
via http(s)
with a Message token(Tenant token)
• Stores a collection named “logs” in Mongo
Coordinator API
• Pairing
- registers a worker
• Configuration
– requests a worker’s configuration
• Status
– updates a workers status
Status API
• Worker Status API (PUT, Manually)
- System Load Average
- Disk Usage
• Coordinator Status API
- grid status
- individual worker status
Broadcaster API
• What’s Brodcaster API?
I don’t know well!
• Communicates with coordinator worker
server.
• New Feature!
GETTING STARTED
Server deployment example
Host1
• Coordinator
• MongoDB
Host2
• Correlator
Host3
• Storage
HOST1
• Creating a development environment
[LINK]
• Install MongoDB
[LINK]
uWSGI.ini
meniscus/etc/meniscus/meniscus.conf
Run uwsgi daemon!!
# uwsgi uwsgi.ini
HOST2
• Creating a development environment
[LINK]
• Run uwsgi daemon!!
Default Persona : pairing
CURL Request
• Attempts pairing with coordinator
• Becomes correlation worker after automatically restart
HOST3
• Creating a development environment
[LINK]
• Run uwsgi daemon!!
Default Persona : pairing
CURL Request
• Attempts pairing with coordinator
• Becomes store worker after automatically restart
COMPLETED FOR SETTING UP
Logging a data
• Already MUST register a tenant
• Request to correlator server(host2)
Flow
Correlator
(host2)
Storage
(host3)
MongoDB
(host1)
Conclusion
• Well-made backend APIs for logging
(Good architecture)
• Distributed workers processing
• Diverse Persona(worker)
• In early development
• Nobody in its community
(Stack Overflow, Google Groups)
Not yet
• Data Store – Only MongoDB
Hadoop / HDFS / Hbase
Elastic Search
• Perfect documentation
need more examples
need exact examples
The thing we can now do with meniscus is
JUST to create log data into mongodb via
restful api and to fetch log data from
mongodb.
THANKS