This is a proof-of-concept implementation of lazy storage syncing between Azure Blob Storage and AWS S3. The simple idea is to replicate the requested files to the storage where they don't exist. For instance, if the file requested only exists in AWS S3, it will be uploaded to Azure automatically after downloaded from S3.
This project is based on the initial work by Eric Lai - he created and implemented a unified interface for Azure Blob Storage and AWS S3 client wrappers.
You need to have Node installed. If you want to build the source, you will
also need TypeScript. The built JavaScript files can be found
in dist folder.
In order to use this sample you need to collect the following credentials and details from your accounts:
Azure Blob Storage
- Account name
- Access key
- Container name
AWS S3
- Access key ID
- Secret access key
- Region (where your S3 storage is hosted, e.g.
eu-west-2if London) - Bucket name
-
Insert your account details and credentials to /test/multi-storage-client-test.js
-
Run the following commands in the root folder of the project:
$ npm install $ LOG_LEVEL=info NODE_ENV=local-dev ./node_modules/.bin/nodeunit ./test/multi-storage-client-test.js
You can also use the provided bash script (build_and_run_tests.sh) to run the tests. If you are running this project in Windows, you can use the following bash shell options to run the script:
- Git bash that comes with Git (the most lightweight option)
- Bash on Ubuntu on Windows - For this you need to enable Windows Subsystem for Linux, see https://msdn.microsoft.com/en-us/commandline/wsl/about
- Cygwin (super heavy duty stuff)
The beef of the project lies in the multi-storage-client.ts class.
Furthermore, the method downloadFileToDiskAndReplicateIfNecessary
is the one that does the lazy replication/syncing; if the desired file is only found from one of
the storage, it will copied to the one where it's missing. See the documentation in
multi-storage-client.ts for all the methods.