Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

update aws sdk and image uploader #7555

Merged
merged 22 commits into from
May 30, 2024

Conversation

RileySeaburg
Copy link
Member

@RileySeaburg RileySeaburg commented May 10, 2024

Summary

Replaces outdate gulp-s3-upload package with awspublish package and closes #7554.

Solution

How To Test

  1. Add an image(s) and static file to /content/uploads/_inbox/
  2. Run npx gulp upload
  3. Upload to S3 after processing

Dev Checklist

  • PR has correct labels
  • A11y testing (voice over testing, meets WCAG, run axe tools)
  • Code is formatted properly

@RileySeaburg RileySeaburg self-assigned this May 10, 2024
@RileySeaburg
Copy link
Member Author

This is just to trigger a build with AWS credentials

@RileySeaburg RileySeaburg force-pushed the rs-update-aws-sdk-and-image-uploader branch from 12d3565 to 01be580 Compare May 10, 2024 21:04
@RileySeaburg
Copy link
Member Author

Ready for review

@nick-mon1 nick-mon1 self-requested a review May 14, 2024 14:51
config/gulp/file-upload.js Outdated Show resolved Hide resolved
config/gulp/file-upload.js Outdated Show resolved Hide resolved
config/gulp/file-upload.js Show resolved Hide resolved
- fix js code
- add test images
@RileySeaburg
Copy link
Member Author

Addresses the feedback, tested and verified to work locally. @nick-mon1

@RileySeaburg RileySeaburg reopened this May 15, 2024
@RileySeaburg RileySeaburg added Dev: frontend ideas and issues related to the presentation layer of the site Dev: Workflow labels May 15, 2024
@nick-mon1 nick-mon1 self-requested a review May 17, 2024 19:44
config/gulp/file-upload.js Outdated Show resolved Hide resolved
data/images/clock-iconcop.yml Outdated Show resolved Hide resolved
data/images/free.yml Outdated Show resolved Hide resolved
data/images/pexels-photo-2829031.yml Outdated Show resolved Hide resolved
config/gulp/file-upload.js Show resolved Hide resolved
@RileySeaburg
Copy link
Member Author

This pull request also contains the fix for this Security Issue

https://github.com/GSA/digitalgov.gov/security/dependabot/43

$ npm list lodash.trim
[email protected] /home/riley/programming/digitalgov.gov
└── (empty)

@RileySeaburg RileySeaburg force-pushed the rs-update-aws-sdk-and-image-uploader branch from 1535878 to c6988a7 Compare May 17, 2024 21:51
mejiaj

This comment was marked as resolved.

@RileySeaburg
Copy link
Member Author

I'll dive into this tomorrow. Thanks @mejiaj

@nick-mon1 nick-mon1 self-requested a review May 24, 2024 15:55
Copy link
Contributor

@nick-mon1 nick-mon1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ I was able to successfully upload an image and see it on digitalgov bucket. But no go for uploading files, see my suggestion for getting around the InvalidBucketName: Bucket name shouldn't contain '/', received 'digitalgov/static' error.

config/gulp/file-upload.js Outdated Show resolved Hide resolved
config/gulp/file-upload.js Outdated Show resolved Hide resolved
@RileySeaburg
Copy link
Member Author

@RileySeaburg

I've updated my .env file with the new key names.

Following the testing steps in description:

How To Test

  1. Add an image(s) and static file to /content/uploads
  2. Run npm gulp upload
    3.Upload to S3 after processing

I get the following error:

➜ npx gulp upload
[15:25:36] Using gulpfile ~/web/digitalgov.gov/gulpfile.js
[15:25:36] Starting 'upload'...
[15:25:36] Starting 'fileTidy'...
[15:25:36] Finished 'fileTidy' after 383 μs
[15:25:36] Starting 'cleanInbox'...
[15:25:36] Finished 'cleanInbox' after 4.53 ms
[15:25:36] Starting 'writeDataFile'...
[15:25:36] Finished 'writeDataFile' after 6.52 ms
[15:25:36] Starting 'processImages'...
[15:25:36] Finished 'processImages' after 228 μs
[15:25:36] Starting 'removeProcessedImage'...
Removing processed images
No images to process
Error processing file: [undefined]. ENOENT: no such file or directory, scandir './content/uploads/_working-images/to-process'
[15:25:36] Finished 'removeProcessedImage' after 882 μs
[15:25:36] Starting 'uploadImage'...
[15:25:36] Finished 'uploadImage' after 3.62 ms
[15:25:36] Starting 'uploadFile'...
[15:25:36] Finished 'uploadFile' after 1.08 ms
[15:25:36] Starting 'cleanup'...
cleanup
[15:25:36] Finished 'cleanup' after 990 μs
[15:25:36] Finished 'upload' after 20 ms

The image needs to be in /content/uploads/_inbox. That runs successfully:

➜ npx gulp upload
[15:29:23] Using gulpfile ~/web/digitalgov.gov/gulpfile.js
[15:29:23] Starting 'upload'...
[15:29:23] Starting 'fileTidy'...
[15:29:23] Finished 'fileTidy' after 380 μs
[15:29:23] Starting 'cleanInbox'...
[15:29:23] Finished 'cleanInbox' after 5.78 ms
[15:29:23] Starting 'writeDataFile'...
[15:29:23] Finished 'writeDataFile' after 9.38 ms
[15:29:23] Starting 'processImages'...
[15:29:23] Finished 'processImages' after 234 μs
[15:29:23] Starting 'removeProcessedImage'...
Removing processed images
[15:29:23] Finished 'removeProcessedImage' after 2.7 ms
[15:29:23] Starting 'uploadImage'...
file is written
[15:29:24] [create] test-dg-s3-upload-pr-7498.png
[15:29:24] [create] test-dg-s3-upload-pr-7498_w200.png
[15:29:24] [create] test-dg-s3-upload-pr-7498_w400.png
[15:29:25] [create] test-dg-s3-upload-pr-7498_w800.png
[15:29:25] Finished 'uploadImage' after 1.42 s
[15:29:25] Starting 'uploadFile'...
[15:29:25] Finished 'uploadFile' after 3.25 ms
[15:29:25] Starting 'cleanup'...
cleanup
[15:29:25] Finished 'cleanup' after 3.8 ms
[15:29:25] Finished 'upload' after 1.45 s

Unfortunately, I still don't see my files in the S3 bucket.

Hey @mejiaj.
I was under the impression images to be uploaded should be dropped in

content/uploads/_inbox/__add image or static files to this folder__

Should we be putting them in

content/uploads/

@RileySeaburg
Copy link
Member Author

RileySeaburg commented May 28, 2024

@mejiaj I did manage to find you uploads.
In order to use the search box on S3 for some reason you need to type it out from left to right.
I was able to reproduce the not found issue by doing a partial search but they are there.

mejiaj
mejiaj previously approved these changes May 29, 2024
Copy link
Contributor

@mejiaj mejiaj left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@RileySeaburg the directory content/uploads/_inbox/ is correct. The original description had it in content/uploads, but I updated it for others testing.

I was able to upload successfully without any issues, thanks for looking into the issues!


I've approved PR and requested another review for @nick-mon1.

@mejiaj mejiaj requested a review from nick-mon1 May 29, 2024 13:44
@nick-mon1
Copy link
Contributor

@RileySeaburg I added some fixes for uploading files. I was able to successfully upload an file:

[10:15:58] Using gulpfile ~/Documents/gsa/digitalgov.gov/gulpfile.js
[10:15:58] Starting 'upload'...
[10:15:58] Starting 'fileTidy'...
[10:15:58] Finished 'fileTidy' after 571 μs
[10:15:58] Starting 'cleanInbox'...
[10:15:58] Finished 'cleanInbox' after 11 ms
[10:15:58] Starting 'writeDataFile'...
[10:15:58] Finished 'writeDataFile' after 25 ms
[10:15:58] Starting 'processImages'...
[10:15:58] Finished 'processImages' after 290 μs
[10:15:58] Starting 'removeProcessedImage'...
Removing processed images
No images to process
Error processing file: [undefined]. ENOENT: no such file or directory, scandir './content/uploads/_working-images/to-process'
[10:15:58] Finished 'removeProcessedImage' after 1.96 ms
[10:15:58] Starting 'uploadImage'...
[10:15:58] Finished 'uploadImage' after 7.78 ms
[10:15:58] Starting 'uploadFile'...
file is written
(node:41977) [DEP0180] DeprecationWarning: fs.Stats constructor is deprecated.
(Use `node --trace-deprecation ...` to show where the warning was created)
[10:16:10] [create] static/s3-test-uswds-monthly-call-may-2024.pptx
[10:16:11] Finished 'uploadFile' after 12 s
[10:16:11] Starting 'cleanup'...
cleanup
[10:16:11] Finished 'cleanup' after 5.42 ms
[10:16:11] Finished 'upload' after 12 s
Screenshot 2024-05-29 at 10 16 42 AM

Give this a quick test and then I can approve.

@RileySeaburg
Copy link
Member Author

@nick-mon1 I got a successful file upload and a consistent file name.

image

Copy link
Contributor

@mejiaj mejiaj left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you both,

I was able to successfully test uploads (both files & images) via npx gulp upload.

@ToniBonittoGSA ToniBonittoGSA merged commit f5603f5 into main May 30, 2024
8 checks passed
@ToniBonittoGSA ToniBonittoGSA deleted the rs-update-aws-sdk-and-image-uploader branch May 30, 2024 00:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Dev: frontend ideas and issues related to the presentation layer of the site Dev: Workflow
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Update gulp-s3-upload to newer package
4 participants