beyond logging: using mongodb to power a private social network (oh, and log millions of e-mails...
DESCRIPTION
Beyond Logging: Using MongoDB to Power a Private Social Network (Oh, and log millions of e-mails too.)TRANSCRIPT
![Page 1: Beyond Logging: Using MongoDB to Power a Private Social Network (Oh, and log millions of e-mails too.)](https://reader035.vdocuments.net/reader035/viewer/2022062513/555831f1d8b42acb078b45e6/html5/thumbnails/1.jpg)
Building a Private Social Network with MongoDBJustin Jenkins@LearnMongo
![Page 2: Beyond Logging: Using MongoDB to Power a Private Social Network (Oh, and log millions of e-mails too.)](https://reader035.vdocuments.net/reader035/viewer/2022062513/555831f1d8b42acb078b45e6/html5/thumbnails/2.jpg)
Learn Mongo (.com that is)
MongoDB resources aimed at not scaring off the beginner.
Twitter: @LearnMongo
![Page 3: Beyond Logging: Using MongoDB to Power a Private Social Network (Oh, and log millions of e-mails too.)](https://reader035.vdocuments.net/reader035/viewer/2022062513/555831f1d8b42acb078b45e6/html5/thumbnails/3.jpg)
Saddleback Church
21,000+ per weekend @ 10 locations. 4,000 online.
4,300 “small groups”20k – 32k people
~1 million user records
3.2 million e-mails a week
Service trips to ~180 countries
Food bank serving Orange County
![Page 4: Beyond Logging: Using MongoDB to Power a Private Social Network (Oh, and log millions of e-mails too.)](https://reader035.vdocuments.net/reader035/viewer/2022062513/555831f1d8b42acb078b45e6/html5/thumbnails/4.jpg)
Features, easy of use …
![Page 5: Beyond Logging: Using MongoDB to Power a Private Social Network (Oh, and log millions of e-mails too.)](https://reader035.vdocuments.net/reader035/viewer/2022062513/555831f1d8b42acb078b45e6/html5/thumbnails/5.jpg)
Project Goals
• Provide a private way for groups to interact and share online.
• Tie into other social networks but keep group data private.
• Provide groups with video & audio resources for growth.
• Create a simple & fast user experience.
![Page 6: Beyond Logging: Using MongoDB to Power a Private Social Network (Oh, and log millions of e-mails too.)](https://reader035.vdocuments.net/reader035/viewer/2022062513/555831f1d8b42acb078b45e6/html5/thumbnails/6.jpg)
MongoDB SQL Server
• Flexible schema• Fairly easy to scale• Optimized for read / inserts
Uh Ohs• New, little past use• Can be harder to query• Different data format
• Solid, durable• Years of use• Very easy to query
Uh Ohs• Not optimized for the
web• Can be complex to scale• Stringent schema
![Page 7: Beyond Logging: Using MongoDB to Power a Private Social Network (Oh, and log millions of e-mails too.)](https://reader035.vdocuments.net/reader035/viewer/2022062513/555831f1d8b42acb078b45e6/html5/thumbnails/7.jpg)
Playing nice with SQL Server
![Page 8: Beyond Logging: Using MongoDB to Power a Private Social Network (Oh, and log millions of e-mails too.)](https://reader035.vdocuments.net/reader035/viewer/2022062513/555831f1d8b42acb078b45e6/html5/thumbnails/8.jpg)
Playing nice with SQL Server
• Keep users accounts in SQL• Use the same user id’s in both SQL and
MongoDB.• Store app specific user data in MongoDB.
• Keep group data and hierarchy in SQL• Extract group data from SQL when any user of
the group logs in, compare.• Push any changes back to SQL in the
background.
![Page 9: Beyond Logging: Using MongoDB to Power a Private Social Network (Oh, and log millions of e-mails too.)](https://reader035.vdocuments.net/reader035/viewer/2022062513/555831f1d8b42acb078b45e6/html5/thumbnails/9.jpg)
Problem Solved?
![Page 10: Beyond Logging: Using MongoDB to Power a Private Social Network (Oh, and log millions of e-mails too.)](https://reader035.vdocuments.net/reader035/viewer/2022062513/555831f1d8b42acb078b45e6/html5/thumbnails/10.jpg)
The Problems MongoDB Solved
• Flexible schema design • Each feed item can have totally different
attributes.• New features can be added quickly.
• Reduces impact on production SQL Server• Focus server resources on internal staff.
• BSON to JSON• Ideal for a highly AJAX / JavaScript site.
![Page 11: Beyond Logging: Using MongoDB to Power a Private Social Network (Oh, and log millions of e-mails too.)](https://reader035.vdocuments.net/reader035/viewer/2022062513/555831f1d8b42acb078b45e6/html5/thumbnails/11.jpg)
The Feed
![Page 12: Beyond Logging: Using MongoDB to Power a Private Social Network (Oh, and log millions of e-mails too.)](https://reader035.vdocuments.net/reader035/viewer/2022062513/555831f1d8b42acb078b45e6/html5/thumbnails/12.jpg)
![Page 13: Beyond Logging: Using MongoDB to Power a Private Social Network (Oh, and log millions of e-mails too.)](https://reader035.vdocuments.net/reader035/viewer/2022062513/555831f1d8b42acb078b45e6/html5/thumbnails/13.jpg)
The Feed
• The main feature of the site is “the feed” where the group can interact and share discussions, prayers, videos, etc.
• We need something very flexible …• Discussions• Videos• Events• Meetings• Prayers• Web Links• Pictures
![Page 14: Beyond Logging: Using MongoDB to Power a Private Social Network (Oh, and log millions of e-mails too.)](https://reader035.vdocuments.net/reader035/viewer/2022062513/555831f1d8b42acb078b45e6/html5/thumbnails/14.jpg)
Schema Design
• Flexible Schema• Less clutter.• Cleaner, leaner storage.• Simpler queries.• JOINs largely unneeded.
• BSON Format• Converts into JSON with very little effort to make
JavaScript / AJAX centric apps happy.• Move from the client to the database, back to the
client easily.
![Page 15: Beyond Logging: Using MongoDB to Power a Private Social Network (Oh, and log millions of e-mails too.)](https://reader035.vdocuments.net/reader035/viewer/2022062513/555831f1d8b42acb078b45e6/html5/thumbnails/15.jpg)
{ "_id" : "4d19fb939ac0900274000005", "_t" : "Discussion", "created" : "Tue, 28 Dec 2010 07:00:35 GMT", "GroupID" : 129242, "eIndividualID" : "22872F9", "firstName" : "Bill", "lastName" : "Finch", "discussionBody" : "I've been not feeling well lately, I hope it's nothing major.", "comments": [ { "_id" : "4d19fe329ac090027400000a", "eIndividualID" : "FFE4251", "created" : "2010-12-28T15:11:46.6760000Z", "firstName" : "Robin", "lastName" : "Tally", "commentText" : "I hope that everything is OK, whatever it is!" }, { "_id" : "4d1df3b29ac0900d64000003", ... } ]}
![Page 16: Beyond Logging: Using MongoDB to Power a Private Social Network (Oh, and log millions of e-mails too.)](https://reader035.vdocuments.net/reader035/viewer/2022062513/555831f1d8b42acb078b45e6/html5/thumbnails/16.jpg)
Models (the pretty kind)
![Page 17: Beyond Logging: Using MongoDB to Power a Private Social Network (Oh, and log millions of e-mails too.)](https://reader035.vdocuments.net/reader035/viewer/2022062513/555831f1d8b42acb078b45e6/html5/thumbnails/17.jpg)
using System;using System.Collections.Generic;using System.Linq;using System.Web;using Newtonsoft.Json;using MySmallGroup.Helpers.JsonConverters;
namespace MySmallGroup.Models{ public abstract class Feed : IFeed, IUserCreatable { [JsonConverter(typeof(MongoOidConverter))] public MongoDB.Oid Id { get; set; }
public DateTime created { get; set; } public int GroupID { get; set; } public string eIndividualID { get; set; } public string firstName { get; set; } public string lastName { get; set; } public abstract string shortDescription { get; } public int commentCount { get; set; } public IList<comment> comments { get; set; } public abstract void ReceivedJsonEncode(); public Feed Get() { return this; } }
public interface IFeed : IUserCreatable { MongoDB.Oid Id { get; set; } int GroupID { get; set; } string eIndividualID { get; set; } string firstName { get; set; } string lastName { get; set; } string shortDescription { get; } int commentCount { get; set; } IList<comment> comments { get; set; } void ReceivedJsonEncode(); Feed Get(); }}
![Page 18: Beyond Logging: Using MongoDB to Power a Private Social Network (Oh, and log millions of e-mails too.)](https://reader035.vdocuments.net/reader035/viewer/2022062513/555831f1d8b42acb078b45e6/html5/thumbnails/18.jpg)
using System;using System.Collections.Generic;using System.Linq;using System.Web;using MySmallGroup.Helpers.ExtensionMethods;
namespace MySmallGroup.Models{ public class Discussion : Feed { public string discussionBody { get; set;} public Discussion() { this.commentCount = new int(); this.comments = new List<comment>(); this.created = DateTime.Now; }
public override void ReceivedJsonEncode() { discussionBody = discussionBody.ReceivedJsonEncode(); }
public override string shortDescription { get { return discussionBody; } } }}
![Page 19: Beyond Logging: Using MongoDB to Power a Private Social Network (Oh, and log millions of e-mails too.)](https://reader035.vdocuments.net/reader035/viewer/2022062513/555831f1d8b42acb078b45e6/html5/thumbnails/19.jpg)
Live Chat & Video
![Page 20: Beyond Logging: Using MongoDB to Power a Private Social Network (Oh, and log millions of e-mails too.)](https://reader035.vdocuments.net/reader035/viewer/2022062513/555831f1d8b42acb078b45e6/html5/thumbnails/20.jpg)
![Page 21: Beyond Logging: Using MongoDB to Power a Private Social Network (Oh, and log millions of e-mails too.)](https://reader035.vdocuments.net/reader035/viewer/2022062513/555831f1d8b42acb078b45e6/html5/thumbnails/21.jpg)
Live Chat & Live Video{ "_id": "4cd261549ac0900c24000006", "senderEIndividualID": "1B0D6E8", "senderFirstName": "Justin", "senderLastName": "Jenkins", "sendDate": "Thu, 04 Nov 2010 00:31:32 GMT", "groupID": 119420, "message": "way faster then dev tho"}{ "_id": "4cd261a69ac0900c2400000a", "senderEIndividualID": "B656641", "senderFirstName": "Steve", "senderLastName": "Chen", "sendDate": "Thu, 04 Nov 2010 00:32:54 GMT", "groupID": 119420, "message": "oh"} { "_id": "4cd2621b9ac0900c2400001f", "senderEIndividualID": "1B0D6E8", "senderFirstName": "Justin", "senderLastName": "Jenkins", "sendDate": "Thu, 04 Nov 2010 00:34:51 GMT", "groupID": 119420, "message": "I think it's the facebook pic reload voodoo Jeff made"}
![Page 22: Beyond Logging: Using MongoDB to Power a Private Social Network (Oh, and log millions of e-mails too.)](https://reader035.vdocuments.net/reader035/viewer/2022062513/555831f1d8b42acb078b45e6/html5/thumbnails/22.jpg)
User Settings
![Page 23: Beyond Logging: Using MongoDB to Power a Private Social Network (Oh, and log millions of e-mails too.)](https://reader035.vdocuments.net/reader035/viewer/2022062513/555831f1d8b42acb078b45e6/html5/thumbnails/23.jpg)
User Settings{ "eIndividualID": "A0BE87A", "_id": "4cd332389ac0900c2400001c", "ChatTimestamps": false, "GroupSettings": [ { "GroupId": 119420, "FeedNotificationEmail": false, "CommentNotificationEmail": false }, { "GroupId": 128834, "FeedNotificationEmail": true, "CommentNotificationEmail": false }, { "GroupId": 125680, "FeedNotificationEmail": true, "CommentNotificationEmail": true } ]}
![Page 24: Beyond Logging: Using MongoDB to Power a Private Social Network (Oh, and log millions of e-mails too.)](https://reader035.vdocuments.net/reader035/viewer/2022062513/555831f1d8b42acb078b45e6/html5/thumbnails/24.jpg)
Email Tracking …
![Page 25: Beyond Logging: Using MongoDB to Power a Private Social Network (Oh, and log millions of e-mails too.)](https://reader035.vdocuments.net/reader035/viewer/2022062513/555831f1d8b42acb078b45e6/html5/thumbnails/25.jpg)
E-mail Tracking (the boring stuff.)
Emails Sent• 350,000 normal e-mails sent a week.• 460,000 daily subscription e-mails sent a day.
Tracking Needs• Track each e-mail open, per individual.• Date• User Agent• IP
• If a user opens an e-mail more than once, don’t double count but still track.
• Deduce how quickly users open an e-mail and how many don’t open the e-mail at all.
![Page 26: Beyond Logging: Using MongoDB to Power a Private Social Network (Oh, and log millions of e-mails too.)](https://reader035.vdocuments.net/reader035/viewer/2022062513/555831f1d8b42acb078b45e6/html5/thumbnails/26.jpg)
Email Tracking{ "EmailLogID": 14, "EmailLogDocumentID": "4d12a55e00ae2611f8000002", "EIndividualID": “ZZZZZZZ", "Email": "[email protected]", "SendDate": "Wed, 22 Dec 2010 17:26:54 GMT", "_id": "4d12a56a18bee80a7c00243d", "Opens": [ { "OpenDate": "2010-12-23T01:59:45", "UserAgent": "Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_1 like Mac OS X; en-us) ...", "IP": "72.111.250.111" }, { "OpenDate": "2010-12-23T16:22:19", "UserAgent": "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_5; en-us) ...", "IP": "70.111.66.11" } ]}
![Page 27: Beyond Logging: Using MongoDB to Power a Private Social Network (Oh, and log millions of e-mails too.)](https://reader035.vdocuments.net/reader035/viewer/2022062513/555831f1d8b42acb078b45e6/html5/thumbnails/27.jpg)
> db.Recipient.find({ "EmailLogID" : 32, "Opens" : { "$exists" : true } }).count();
141762