Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feature(director): s3 update, make it compatible with other providers #806

Draft
wants to merge 1 commit into
base: master
Choose a base branch
from
Draft

feature(director): s3 update, make it compatible with other providers #806

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
7 changes: 6 additions & 1 deletion packages/director/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,12 @@
"@sorry-cypress/common": "1.0.0",
"@sorry-cypress/logger": "1.0.0",
"@sorry-cypress/mongo": "1.0.0",
"aws-sdk": "^2.814.0",
"@aws-sdk/credential-providers": "^3.362.0",
"@aws-sdk/hash-node": "^3.357.0",
"@aws-sdk/protocol-http": "^3.357.0",
"@aws-sdk/s3-request-presigner": "^3.362.0",
"@aws-sdk/url-parser": "^3.357.0",
"@aws-sdk/util-format-url": "^3.357.0",
"axios": "^0.21.3",
"clone-deep": "^4.0.1",
"date-fns": "^2.21.3",
Expand Down
6 changes: 2 additions & 4 deletions packages/director/src/screenshots/s3/config.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,11 +1,9 @@
export const S3_URL = process.env.S3_URL || 'https://s3.amazonaws.com/';
export const S3_BUCKET = process.env.S3_BUCKET || 'sorry-cypress';
export const S3_REGION = process.env.S3_REGION || 'us-east-1';
export const S3_ACL = process.env.S3_ACL || 'public-read';
export const S3_FORCE_PATH_STYLE = process.env.S3_FORCE_PATH_STYLE || false;
export const S3_READ_URL_PREFIX = process.env.S3_READ_URL_PREFIX || undefined;
export const S3_IMAGE_KEY_PREFIX = process.env.S3_IMAGE_KEY_PREFIX || undefined;
export const S3_VIDEO_KEY_PREFIX = process.env.S3_VIDEO_KEY_PREFIX || undefined;
export const UPLOAD_EXPIRY_SECONDS = parseInt(
process.env.UPLOAD_EXPIRY_SECONDS || '90',
10
);
);
108 changes: 47 additions & 61 deletions packages/director/src/screenshots/s3/s3.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,78 +1,64 @@
import { AssetUploadInstruction } from '@sorry-cypress/common';
import aws from 'aws-sdk';
import { sanitizeS3KeyPrefix } from '../utils/';
import { fromEnv } from "@aws-sdk/credential-providers";
import {AssetUploadInstruction} from '@sorry-cypress/common';
import {HttpRequest} from "@aws-sdk/protocol-http";
import {S3RequestPresigner} from "@aws-sdk/s3-request-presigner";
import {parseUrl} from "@aws-sdk/url-parser";
import {formatUrl} from "@aws-sdk/util-format-url";
import {Hash} from "@aws-sdk/hash-node";
import {sanitizeS3KeyPrefix} from '../utils/';
import {
S3_ACL,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@p-8-z can you please elaborate - was it deprecated or smth?

S3_BUCKET,
S3_FORCE_PATH_STYLE,
S3_IMAGE_KEY_PREFIX,
S3_READ_URL_PREFIX,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@p-8-z I believe some users wants the read url to be completely different from the write S3 bucket because, probably we'd keep it?

S3_REGION,
S3_VIDEO_KEY_PREFIX,
UPLOAD_EXPIRY_SECONDS,
S3_URL,
S3_BUCKET,
S3_IMAGE_KEY_PREFIX,
S3_REGION,
S3_VIDEO_KEY_PREFIX,
UPLOAD_EXPIRY_SECONDS
} from './config';
import { S3SignedUploadResult } from './types';

const BUCKET_URL = S3_FORCE_PATH_STYLE
? `https://s3.amazonaws.com/${S3_BUCKET}`
: `https://${S3_BUCKET}.s3.amazonaws.com`;
import {S3SignedUploadResult} from './types';

const ImageContentType = 'image/png';
const VideoContentType = 'video/mp4';

const s3 = new aws.S3({
region: S3_REGION,
signatureVersion: 'v4',
const presigner = new S3RequestPresigner({
credentials: fromEnv(),
region: S3_REGION,
sha256: Hash.bind(null, "sha256"),
});

interface GetUploadURLParams {
key: string;
ContentType?: string;
Expires?: number;
key: string;
ContentType?: string;
Expires?: number;
}

export const getUploadUrl = async ({
key,
ContentType = ImageContentType,
Expires = UPLOAD_EXPIRY_SECONDS,
export const getUrls = async ({
key,
ContentType = ImageContentType,
}: GetUploadURLParams): Promise<S3SignedUploadResult> => {
const s3Params = {
Bucket: S3_BUCKET,
Key: key,
Expires,
ContentType,
ACL: S3_ACL,
};
const url = parseUrl(`${S3_URL}/${S3_BUCKET}/${key}`);

const signedGetUrlObject = await presigner.presign(
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@p-8-z This one is going to be stored in the DB and used for all subsequent reads. does it have an expiration?

new HttpRequest({...url, method: "GET"})
);

return new Promise((resolve, reject) => {
s3.getSignedUrl(
'putObject',
s3Params,
(error: Error, uploadUrl: string) => {
if (error) {
return reject(error);
}
return resolve({
readUrl: `${S3_READ_URL_PREFIX || BUCKET_URL}/${key}`,
uploadUrl,
});
}
const signedPutUrlObject = await presigner.presign(
new HttpRequest({...url, method: "PUT", headers: {'Content-Type': ContentType}}),
{ expiresIn: UPLOAD_EXPIRY_SECONDS }
);
});

return {
readUrl: formatUrl(signedGetUrlObject),
uploadUrl: formatUrl(signedPutUrlObject),
};
};

export const getImageUploadUrl = async (
key: string
): Promise<AssetUploadInstruction> =>
getUploadUrl({
key: `${sanitizeS3KeyPrefix(S3_IMAGE_KEY_PREFIX)}${key}.png`,
});
export const getImageUploadUrl = async (key: string): Promise<AssetUploadInstruction> =>
getUrls({
key: `${sanitizeS3KeyPrefix(S3_IMAGE_KEY_PREFIX)}${key}.png`,
});

export const getVideoUploadUrl = async (
key: string
): Promise<AssetUploadInstruction> =>
getUploadUrl({
key: `${sanitizeS3KeyPrefix(S3_VIDEO_KEY_PREFIX)}${key}.mp4`,
ContentType: VideoContentType,
Expires: UPLOAD_EXPIRY_SECONDS,
});
export const getVideoUploadUrl = async (key: string): Promise<AssetUploadInstruction> =>
getUrls({
key: `${sanitizeS3KeyPrefix(S3_VIDEO_KEY_PREFIX)}${key}.mp4`,
ContentType: VideoContentType,
});
1 change: 1 addition & 0 deletions tsconfig.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
"target": "es2019",
"types": ["jest"],
"noEmit": true,
"skipLibCheck": true,
"esModuleInterop": true,
"strictNullChecks": true,
"resolveJsonModule": true,
Expand Down