amazon-fresh-project-report (1)

AMAZON FARMERS FRESH

1 | P a g eCMPE 273 Group Project

Spring 2016

CMPE 273 – Enterprise Distributed Systems

FINAL GROUP REPORT

On

Amazon Market Fresh

Instructor:

Dr. Simon Shim

Submission Date: Submitted By:

May 1, 2016 Dhiraj Gurnani Satyateja Pothuru Ashwini Chellagurki Vimal Muraleedharan Nair Vaishampayan Reddy Pathuri



INDEX

1. Introduction: state your goals, purpose of system...................................32. System Design: Describe your chosen system design...............................33. Object Management Policy.......................................................................8

3.1 Planning and Requirement Analysis....................................................83.2 Consideration of the Functional Requirements..................................83.3 Development......................................................................................93.4 Testing the System...............................................................................9

4. Handling of Heavy Weight Resources........................................................94.1 Usage of Connection Pooling..............................................................94.2 Usage of Rabbit MQ............................................................................104.3 Usage of Redis Caching.......................................................................11

5. Policy used to decide when to write to data.............................................126. Implementation Details: Performance/Mocha Tests................................127. Use Cases/Work Flow – Screenshots…......................................................168. Dynamic Pricing Algorithm…......................................................................43



1. Introduction :

The goal of our system is to develop an application which is similar to AmazonFresh Farmers Market to demostrate the use of REST based Web Services in our Middleware technology , the use of RabbitMQ as a messaging platform for communication between front-end channels with backend systems, and implementing the connection pooling concept to effectively manage our connections and how it effects the overall performance of the system and finally to showcase as to where and which pages to use the SQL and NOSQL database in our system. Our Application offers wide range of grocery items for sale, as well as a subset of items for purchase to the customers.Items that are ordered are available for home delivery on the same day or the next day, depending on the time of the order and the availability of trucks which are also managed by our application.Here the local farmers can sell their fresh produce such as lettuce, tomato, potato, oranges, carrots, etc to customers by getting them approved by the admin. The farmers put their descriptions of products and introduction to their farms that includes history, farming methods, video tour, owner profile etc. Customers can browse many local farmers and items, and make purchases.They can check the order progress in the web and also check the real time locatin of the delivery truck through google maps.

2. System Design:

Block Diagram for Amazon Fresh



Schema Diagram:

Mongo DB:



Sessions (Implemented using Passport.JS and stored in Mongo)

System Design(Continued….)Amazon Fresh Client represents the front-end where customers can select various agricultural items from different forms. Functionality such as available fresh produce items, delivery method and description of farmers and forming methods can be accessed from this page. User interface is customized for the role of the logged in user. If the logged in user is a customer and if the request is made to buy a product then server enques this in a request queue with a specific correlation_id. This request will be forther forwarded to service defined for this type of request. It is here RabbitMQ plays



an important role of decoupling the different components to address scalability. After the request has been serviced, the response will be enqueued in the RabbitMQ response queue for the predefined correlation_id. Finally, it will be sent back to client for display. All the session level and configuration information is stored in the Redis cache. Once the user is authenticated, for the subsequent requests the information stored in the Redis cache will be retrived for the validation.

In our AmazonFresh Farmers Market application, we are making use of bootstrap, HTML5 and angularjs for the client side code and Node js for the server side code. The application provides the users with a AmazonFresh Farmers Market like interface with the basic functionalities like allowing the farmer to sell his fresh produce by creating his account and by updating his products which are in prior approved by the admin and are purchased by the customers who can view all the products, rate them and review them and are also provided with information about a registered delivery truck and can live track it while it is out for delivery.The customers are also provided with a bill at the end of each order placed.The products are also dynamically priced with their price varying on the days when they have high demand etc. The customer should be logged in while doing a checkout of the products , if not he can signup and carry out the request. The farmer,customer,admin product, trucks and billing information is stored in Mysql and is retrieved whenever required whereas the farmer introduction video and all images are stored in MongoDb for easy management.We have made use of connection pooling in the database to effectively manage our connections and to not create a overhead on the system as we will be testing our application on huge data. We have made use of RabbitMq as a messaging platform for high availability and to increase the performance.

Test Scenarios:

We have also used testing frameworks to test our application namely Jmeter for testing the performance of the system when subjected to varying load of the clients and the other namely mocha testing framework.

Github:We have used GitHub which is a web-based Git repository to host our Application.

URL: https://github.com/vaisham92/AmazonFreshMock



3.Object Management Policy

3.1 Planning and Requirement Analysis:

In this stage, we analyzed and performed requirement analysis, as per the project requirements in the three tiers which are as follows:

a) The client tier: The front-end users namely customers, farmers and the admin will interact with our applications and sends responses using the UI interface.

b) The middle tier/middleware: The majority of the processing of the inputs takes place in this tier, it performs the sending and receiving of the messages between distributed systems. It is at this most critical point that it needs to integrate new components or to scale existing ones as efficiently as possible.The modules have been implemented using RabbitMQ which implements the Advanced Message Queuing Protocol.

c) The third tier: This tier consists of database to store data of the system. At this tier, we also have constraints and relations that define the data. Most of the modules use MySQL as the relational database and the videos and images are stored using MongoDB as the NoSQL database to store data.

3.2 Consideration of the Functional Requirements:

As per the project requirement, we implemented different modules namely Farmers, Customers, Billing, Admin, Billing and Trucks. Each of the modules have an associated schema which are responsible for showing how object should be stored in database. Each objects exhibits their respective functionalities as per the project requirements.

3.3 Development:

According to the project requirement, we have created the UI and have implemented different functionalities. Additionaly, We have also implemented key features like maps using Google maps API and live tracking of the trucks by the customer track the exact location of the truck.We have used connection pool to efftectively manage our connections on huge load and Rabbit MQ to increase the sustainability while the load



in high.



3.4 Testing the System :

After the development of our application, on the basis of requirement, we have used testing frameworks to test our application namely Jmeter for testing the performance of the system when subjected to varying load of the clients and the other namely mocha testing framework.We have also drafted the test cases which were checked if all the requirement functionality is working fine or not.

4. Handling of Heavy Weight Resources

We have used the following to efficiently manage the heavy weight resources :

4.1 Usage of Connection Pooling:

Whenever an application needs an access to database it connects to the database. Hence, it will be disadvantageous if it connects every time a user sends a request for a particular data.so our application makes use of connection pooling where we maintain a pool of connection instances.Here we can create any number of connection objects and keep it in a pool so if a request comes it will take a instance from the pool and serve. After accessing the data and we no longer want the connection we release it back into the pool and it will be reused by some other function.In this way, it makes the access faster improves the performance of an application.



4.2 Usage of Rabbit MQ

Rabbit MQ is a message passing service which uses the AMQP protocol. It serves request from client to server and sends response from server to client. Messaging enables software applications to connect and scale. We have made separate queues for handling the data from different modules viz. admin, truck,farmers,product customer, billing etc. Applications can connect to each other, as components of a larger application, or to user devices and data. Messaging is asynchronous, decoupling applications by separating sending and receiving data.Hence by making use of RabbitMQ in our system we are making it reliable and highly available.



4.3 Usage of Redis Caching :

Redis is a remote data structure server and is extremely useful for cache management. Redis can be accessed by all the processes of your applications, possibly running on several nodes (something local memory cannot achieve). Redis memory storage is quite efficient, and done in a separate process. If the application runs on a platform whose memory is garbage collected (node.js, java, etc ...), it allows handling a much bigger memory cache/store. Redis can also persist the data on disk if needed. Redis can replicate its activity with a master/slave mechanism in order to implement high- availability. It allows for fine-grained control over eviction though a choice of six different eviction policies. Redis employs more sophisticated approaches to memory management and eviction candidate selection. Redis gives us much greater flexibility regarding the objects we can cache. Redis allows key names and values to be as large as 512MB each, and they are binary safe. Redis has six data types that enable more intelligent caching and manipulation of cached data, opening up a world of possibilities to the application developer.Hence By making use of Redis in our application we are making it highly available and efficient.



5. Policy used to decide when to write to data

In our application we have writen or have updated into our database only after the processing of our request is complete so that the data inserted into the database is correct and not incomplete.

6. IMPLEMENTATION DETAILS

1. Performance Testing Updates

Response Time

25

20

15

10

5

0100 200 300 400 500 1000

Number of Concurrent Users

Without Connection Pooling With Connection Pooling

Rabbit MQ + Connection Pool Rabbit MQ + Connection Pool + Redis



Note: Performed on heaviest call – getBillDetails

Num

ber o

f est

imat

ed tr

ansa

ction

s / se

c

Performance improvement Vs Concurrency

45%40%35%30%25%20%15%10%5%0%

100 200 300 400 500 1000

Concurrency

Connection PoolConnection Pool + Rabbit MQConnection Pool + Rabbit MQ + Redis Cache



2. Mocha Test Cases:

Pefo

rman

ce Im

prov

emen

t in

%



Source Code for Mocha test cases:

describe('http tests', function(){

it('Location Statistics API', function(done){ http.get('http://localhost:3000/api/admin/trips/locationStats',

function(res) {

})});

assert.equal(200, res.statusCode); done();

it('Revenue Statistics API', function(done){ http.get('http://localhost:3000/api/admin/trips/revenueStats',

function(res) {

})});


it('Get Passport Session', function(done){ http.get('http://localhost:3000/api/getSessionInfo', function(res) {


})});it('Get Pending Trips', function(done){

http.get('http://localhost:3000/api/admin/trips/getPendingTrips',function(res) {

})});


it('Get Bills', function(done){ http.get('http://localhost:3000/api/admin/trips/getBills', function(res) {


})});



});

7. Use Cases/UI Flows Screenshots. Welcome Page

Click Signin



Register User (Customer and farmer)

Validating the fields in Sign up page



Password Encryption(AES Algorithm)

Now Approve the farmer through Admin Page.



Enter Admin Details

Logging in as Admin



Go to Farmer Approval Page

Approve the Farmer



Approved and Logout

Admin Tracking



Now checking the Login Functionality



View the Farmer Welcome Page & click on edit farmer profile



Updated farmer profile

Click new product in homepage and add new product(Apple)



The new product created

Logging in as admin



Creating trips

Checking pending trips



Checking trip statistics

Checking Revenue statistics



Checking all bills placed

Approving a farmer



Approving a product

Viewing dynamic pricing



Entering the price with reference to dynamic price & updating

Signing in with customer



Entering zipcodes

Editing Customer profile

Click Edit



Home page part-1



Part-2

Clicking the fruits sub category



Clicking the apples product category

Clicking the specific apple (Product)

Searching a product



Farmer Products



Products Information Page with ratings and review



Inserting Rating (1 Star, 2 Star, 3 Star, 4 Star, 5 Star)



Adding Reviews and ratings



Showing average ratings



Cart Information

Getting Farmer Details on the Cart page.



Looking at the reviews in the product page



Cart page after adding product to cart

Clicki proceed to checkout and filling shipping details



Clicking continue and going to payment info page

Click pay now and going to confirmation page



Clicking profile button

Profile view



Clicking all placed orders

View placed orders



Signing out

Back to login page



8. Dynamic Pricing Algorithm.

Compute the average price of the same product for all quantity in the Data base. Compute the average price of the same product sold by the Amazon Fresh. Then find the average of step1 and step2 in the database which will shown as

the dynamic price based on demand supply model. If no similar products are available, DPA will show the result as null. The admin has an option to enter the price from the DFA Algorithm.

Implementation Example:

select sum(quantity) as sumOfQuantity from Products where category_id = 1 and subcategory_id = 1;

select quantity, price from Products where category_id = 1 and subcategory_id = 1 order by price desc;

select price from Products where category_id = 1 and subcategory_id = 1;

select max(price) from Products where category_id = 1 and subcategory_id = 1;

UI Implementation and How to apply the same:

amazon-fresh-project-report (1)

Documents