Course : COP5615 - Distributed Operating Systems Principles

TwitterClone

Distributed Twitter-like engine that simulates various functionalities such as registering accounts, sharing tweets, and querying tweets based on hashtags, mentions and user profiles.

Problem Statement

The goal of this project is to create a Twitter-like engine along with a web interface to allow users to subscribe, tweet and search. Along with this a client simulator is developed in order to mimic the actions of thousands of user to measure performance of the distributed engine. The project is divided into 2 milestones.

Part 1:

• Focuses on developing the backend of the twitter engine.
• Simulator engine is developed that emulate multiple users and invokes the actions of a normal user like registering itself, subscribing to other users, sending tweet posts, and retweeting other users’ tweets.

Part 2 :

• Integrate WebSockets to provide a web interface

Project Features

• Register account
• Send tweet. Tweets can have hashtags (e.g. #topic) and mentions (@bestuser)
• Subscribe to user's tweets
• Re-tweets
• Display the new tweets by subscribed users live (without querying).
• Allow querying tweets subscribed to, tweets with specific hashtags, tweets in which the user is mentioned

Web Inteface GUI

TwitterClone_Demo1.mov

Snapshot of feed

System

Language : Erlang

Installation and Execution

Installation

• Erlang : (on mac)

brew install erlang

• Project code : Download files

Execution

There are two modes of execution. One is with the GUI and another is running just the backend with simulator

Web interface GUI

make run 

Copy paste the link in browser : http://localhost:8080/registerpage.html

Run simulator / backend only

make simulator

Start simulator server

PID = twitterserver:start_server().

Simulate different actions through simulator

simulator:simulateUsers(PID , 5). % Creation of N user profiles : Creating 5 user profiles

simulator:subscribe(PID,"user1","user2"). % User1 subscribing to content of user2 

simulator:tweet(PID, "user2", "Hello #topic @user3"). % User2 tweeting

simulator:tweet(PID, "user1", "Hello #topic @user3"). % User1 retweeting

% simulator:search(ServerPID,Query,User,QueryKeyword) % Searching tweets by Hashtag/ Mention / Keyword

simulator:search(PID, "hashtag", "user4", "#topic"). % User4 Searching by Hashtag

Note : PID is a variable that stores the process id of the twitterserver (referred as the server). During initalization, the function start_server() returns the process id. Refer command {Link}

Implementation

TwitterClone is a system designed to mimic the functionality of Twitter. It follows the Actor model paradigm, where each user profile is treated as an independent actor referred to as the client. The system includes a main engine acting as a server that keeps track of user profile names and their associated process IDs. To create a realistic simulation of the Twitter engine, a simulator engine is implemented. The simulator replicates the actions performed on a regular Twitter platform and invokes the backend functions of the client actors to simulate different actions. This simulator is useful for measuring performance metrics and scaling out the Twitter engine by generating a large number of users in bulk. The backend engine is then integrated with a web interface. The web interface uses websockets to pass messages between the independent actors.

Technology and Concepts

What is the actor model?

The actor model is a programming paradigm that provides a conceptual framework for building concurrent and distributed systems. In the actor model, an actor represents a fundamental unit of computation. Each actor encapsulates its own state, behavior, and communication mechanism. Actors can asynchronously send messages to other actors, receive messages, and modify their internal state based on incoming messages. Actors cannot access each others data or state, they can only share information through messages.

Responsibilites of each Actor/process

Server Engine

1. Keep track of registered accounts
2. Keeps track of tweets
   a. map for key:hashtag ; value:list of tweets with that hashtag and the tweeter’s usernames
   b. map for key:mention ; value:list of tweets with that mention and the tweeter’s usernames
3. Responds to client requests for search queries, allowing users to search for specific hashtags, mentions, or keywords within subscribed tweets.

Client : Tweet, Retweet, Request for search by hashtag, mention, or search for a tweets of a user, Display Feed

Simulator :

1. Create N users
2. Simulate following functions : Subscribe, Tweet, Retweet, Search

Handler : An addition actor proecss is created for each client actor to handle the communication between the backend and frontend.

Websockets and Cowboy server

Websocket connection allows the server to send messages to the client irrespective of client sending a request. HTTP on the other hand only works with request response pairs and so the server cannot send any messages to the client without a request using http.

Cowboy is server used for Erlang. It allows to create a whole web application using Erlang.

Different modules

Code snippets

Web Interface Program flow

1. Open url -> localhost:8080/registerpage.html
2. Enter Details and click on register
   Program redirects to next page
   HTTP request sent to create websocket connection
   Server responds and websocket connection established
3. User can now subscribe, tweet, retweet, search etc...
   User 1 subscribes to user 2
   User 2 tweets “First tweet”
4. User1’s feed is automatically updated with its subscripted user’s tweets

System Design Diagram

Flow Diagrams : Simulator Mode

Output : Simulator Mode

Below are sample outputs when executed through the simulator.

Complete Project Demo

https://www.youtube.com/watch?v=UzKbOn5s6aM