Skip to content

A Framework to run user-defined functions for Map&Reduce on a Distributed Cluster on GCP in one line. Uses a self-implemented version of Memcached to communicate and store intermediate data.

Notifications You must be signed in to change notification settings

rgrishigajra/Uni-CourseWork-Cloud-Project

Repository files navigation

This is an overall project readme. Please go inside each folder for assignment relevant readmes and reports!

1.Assignment 1: A persistent Key value map that handles concurrent clients over TCP.

key_value_pair_cache

2.Assignment 2: A map reduce frame work with master co-ordinating workers running map and reduce workers over sockets with fault tolerance.

map_reduce_master

gcloud beta compute instances create key-value-server --zone us-central1-a --source-machine-image base-map-reduce --metadata startup-script='#! /bin/bash git clone https://github.com/rgrishigajra/Uni-CourseWork-Cloud-Project.git cd Uni-CourseWork-Cloud-Project export VAR=abc
python3 server_init.py'

gcloud compute instances delete key-value-server --zone us-central1-a --quiet

sudo kill -9 sudo lsof -t -i:9001

sudo google_metadata_script_runner --script-type startup --debug

gcloud compute instances set-service-account master-map-reduce --zone=us-central1-a --service-account cloud-computong-service-acc@rishabh-gajra.iam.gserviceaccount.com --scopes compute-rw,storage-ro

About

A Framework to run user-defined functions for Map&Reduce on a Distributed Cluster on GCP in one line. Uses a self-implemented version of Memcached to communicate and store intermediate data.

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published