meteor @ sharethis
Post on 09-Aug-2015
163 Views
Preview:
TRANSCRIPT
SHARETHISInsights
Ishika Paul
Principle Engineer
The Team
Jack Zhang Neil Lobo Jeff Blanding Ishika Paul
SHARING TOOLS AT SCALE
2.4 MM SITES AND APPS
120 SOCIAL CHANNELS
1. comScore Media Matrix Report * Includes PC, Tablet, and Mobile sites.
210 MM US USERS1
95% REACH*
EVENT INSIGHTS
BRAND ACTIVATION INSIGHTS30 DAY ACTIVITY
TRENDSCHANNEL BREAKOUT
GENDER BREAKOUT AGE BREAKOUT
Challenge● Shut down 3 hadoop
clusters (~100 machines)● integrate 4 existing
products ● Support over 30 insights
reports● Release to clients by
June 1 (dev started in early March)
Challenge
● Build a new team
● Analyze 350TB of Data
● increase response time
by 100x
Challenge Accepted
● Relatively Clean Code
● Reuseable code
● Reactive and
Responsive UI
● Unit Testing
Reactiveness with Meteor
● Built in integration with Mongodb
● Publish - Subscribe for immediate
reactiveness
● No polling for up-to-date data
● Use of websockets is opaque to developers
Response Architecture Demo
Response Architecture
ServerMeteor.publish("job-status", function (jobId) { return JobCollection.find(jobId);});
ClientTracker.autorun(function () { var jobId =
Session.get("jobId"); Meteor.subscribe("job-status",
jobId); var item =
Downloads.findOne({"jobId":jobId}); if (!item) { return; } var job =
jobService.getJob(jobId); if (job && job.key) { downloadReport(job,
item.campaignGroup); }
}
Jasmine Integration
Meteor: Jasmine integration
What we loved
● writing test are easy!
● Peace of mind
Not so much
● mocking meteor mongo
● no code coverage
● have to remember to add
tests
Free Stuff!
What is a Insights Report?
Each type a client calls a report:● Create a valid job for that particular type of Insights
● Return cached report data if job already ran
● Create new job, send back jobId to client.
Spawn off Async job to:o Call Rest-Insights
o save BQ data into S3 and key into Mongo
o Update Job state (Success | Failed)
Building New Reports is Easy CampaignInsights Prototypes Example
CampaignInsights = function() {};
CampaignInsights.prototype = new baseInsights(new CampaignJobService());CampaignInsights.prototype._decorateData = function (results) {};CampaignInsights.prototype._callInsights = function (job, report, callback) {};
CampaignJobService = function() {};CampaignJobService.prototype = Object.create(JobService.prototype);
CampaignJobService.prototype._requiredFields = [campaign required fields];
CampaignJobService.prototype._generatedFields = [generated fields that will be produced];CampaignJobService.prototype.getFilter = function getFilter(aJob) {};CampaignJobService.prototype._enrichJob = function (job) {};CampaignJobService.prototype._generateS3FileName = function (job, partialName){};
Building New Reports is Easy CampaignJobService Prototype Example
Prototype Benefits
● No code duplication
● Faster development turn around time
● Common functionality guaranteed to work!
Response Architecture
● BigQuery supports SQL!● Common Inputs● Report Definition Loading into Mongo:
insertReportWithDefaults({ reportName: "Income Segments of Ad Clickers", insightsType: INSIGHT_TYPE.CAMPAIGN, funcName: 'getInsightsByCampaign', fileName: 'income_segments_clickers', schema: 'segment_name,dlx,count', tooltip: "Number of ad clickers by household income group." });
Demo
We are hiring :)www.sharethis.com/rocketship
▪ Software Engineers ▪ Site Reliability Engineer / DevOps ▪ Data Scientists▪ Big Data▪ Full Stack▪ Machine Learning / Optimizations
QA
top related