Add S3 Object list commands

[politas]

This adds two new commands for listing S3 storage objects

s3-ls is equivalent to a standard Posix "ls" command, simply listing the object "Keys"

s3-lsl is equivalent to a standard posix "ls -l" command, making a table including:

• Key
• Last Modified Date
• Storage Class
• Owner (Display Name)
• ETag
• Size

[mbailey]

Thanks for your submission.

I need to better document the command naming convention. This would be bucket-objects and I would be inclined to only include the version that lists more detail. People often discover the commands using tab completion (bucket-TABTAB).

Do you see a lot of value in having a second command (like bucket-objects-short)? I try to avoid adding separate commands that don't add much value.

One concern is the cost (API charges) and amount of traffic if someone lists a bucket with millions of items. I've not looked into this but it's a reason I paused on adding a function that returned the total size of objects in a bucket.

[politas]

Ah, I was assuming that the naming was based on the AWS API command names.

My reasoning for the two commands is that bash users are accustomed to using "ls" to just get the filenames and "ls -l" (or some alias thereof) to get a full listing. If you want it to be "bucket-objects", thus breaking the semantic link to "ls", I see no reason to keep the short version.

I admit that I don't have a good grasp of the API charging structure, as I only have objects in the thousands. I can see this being really useful when implementing a new system in order to check how uploaded files are being handled. If you think the cost issue is sufficiently dire, I could add a warning/confirm step, but that would break piping.

[mbailey]

Thanks, I think this looks like a really desirable addition.

I'll try to find time to have a play with it on the weekend. There are some other conventions I need to document (like sorting by date by default with newest shown last) but this looks so close to ready I can make any changes when I get a chance to review.

[nitrocode]

This is pretty cool but I did notice a couple issues.

• The empty lines seem to contain a character that is causing weird issues when running the command. Using :set list in vim, it looks like it's just due to a space char.

  /Users/snip/.bash-my-aws/lib/s3-functions: line 78: : command not found
  /Users/snip/.bash-my-aws/lib/s3-functions: line 59: : command not found
  /Users/snip/.bash-my-aws/lib/s3-functions: line 66: : command not found

  To get around it, I deleted the empty lines and recreated them.
• I added column -s$'\t ' -t to the end to improve the output

Looks nice!

$ bucket-objects company-tfstate | grep -i test
snip/terraform.tfstate                           2020-04-08T13:53:20+00:00  STANDARD                   snip  "5ee438f5242fe184dbc0d1cc1102bb64"  35629
snip/terraform.tfstate                           2020-09-02T04:12:35+00:00  STANDARD                   snip  "8df992d22ca2c8eb7b8034497b3df83d"  157
snip/terraform.tfstate                           2020-10-14T22:48:49+00:00  STANDARD                   snip  "72a6dfcebf5bbdf18d86cf7d8189fa93"  158
snip/terraform.tfstate                           2020-02-10T15:39:23+00:00  STANDARD                   snip  "317b9a0007078c52f358fdd2316791c4"  17532
snip/terraform.tfstate                           2019-10-30T00:31:10+00:00  STANDARD                   snip  "8d6d90c5ea5b1f2aa79d845794205b93"  21534
snip/terraform.tfstate                           2019-10-12T03:16:25+00:00  STANDARD                   snip  "fcfca3cf41476ac691f67a3d17895d64"  158
snip/terraform.tfstate                           2020-01-31T17:48:50+00:00  STANDARD                   snip  "31c3a872bc236d147ace17bee7b807e6"  158
snip/terraform.tfstate                           2020-07-02T18:45:16+00:00  STANDARD                   snip  "f776f920d9e5c1ad5b6f77f3fb51b6b2"  157
snip/terraform.tfstate                           2018-05-25T14:28:13+00:00  STANDARD                   snip  "799c0d84018caee7440d6687d536ca33"  317
snip/us-east-1/terraform.tfstate                 2020-07-15T22:00:44+00:00  STANDARD                   snip  "da443f76deb8f4413e336604f60b87df"  1015

[mbailey]

I tried to sneak in a review yesterday and ran out of time. This week is unlikely to see me have much time to review PRs.

…

On Mon, 22 Feb 2021 at 07:34, nitro ***@***.***> wrote: This is pretty cool but I did notice a couple issues. - The empty lines seem to contain a character that is causing weird issues when running the command. Using :set list in vim, it looks like it's just due to a space char. /Users/snip/.bash-my-aws/lib/s3-functions: line 78: : command not found /Users/snip/.bash-my-aws/lib/s3-functions: line 59: : command not found /Users/snip/.bash-my-aws/lib/s3-functions: line 66: : command not found To get around it, I deleted the empty lines and recreated them. - I added column -s$'\t ' -t to the end to improve the output Looks nice! $ bucket-objects company-tfstate | grep -i test snip/terraform.tfstate 2020-04-08T13:53:20+00:00 STANDARD snip "5ee438f5242fe184dbc0d1cc1102bb64" 35629 snip/terraform.tfstate 2020-09-02T04:12:35+00:00 STANDARD snip "8df992d22ca2c8eb7b8034497b3df83d" 157 snip/terraform.tfstate 2020-10-14T22:48:49+00:00 STANDARD snip "72a6dfcebf5bbdf18d86cf7d8189fa93" 158 snip/terraform.tfstate 2020-02-10T15:39:23+00:00 STANDARD snip "317b9a0007078c52f358fdd2316791c4" 17532 snip/terraform.tfstate 2019-10-30T00:31:10+00:00 STANDARD snip "8d6d90c5ea5b1f2aa79d845794205b93" 21534 snip/terraform.tfstate 2019-10-12T03:16:25+00:00 STANDARD snip "fcfca3cf41476ac691f67a3d17895d64" 158 snip/terraform.tfstate 2020-01-31T17:48:50+00:00 STANDARD snip "31c3a872bc236d147ace17bee7b807e6" 158 snip/terraform.tfstate 2020-07-02T18:45:16+00:00 STANDARD snip "f776f920d9e5c1ad5b6f77f3fb51b6b2" 157 snip/terraform.tfstate 2018-05-25T14:28:13+00:00 STANDARD snip "799c0d84018caee7440d6687d536ca33" 317 snip/us-east-1/terraform.tfstate 2020-07-15T22:00:44+00:00 STANDARD snip "da443f76deb8f4413e336604f60b87df" 1015 — You are receiving this because your review was requested. Reply to this email directly, view it on GitHub <#286 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAABCOIE4KJL2TLJNRBQV7TTAFU7FANCNFSM4KOWZDRA> .

[nitrocode]

@mbailey no worries! I reviewed on your behalf 😄

[politas]

Gosh, a review! No idea how those odd characters got in there.

[politas]

Whoops!

[ninth-dev]

I probably need to find some time to document this too. #313

To rebuild the aliases and functions and docs, run :

$ scripts/build

For macOS users I did notice that the build-docs requires gnu-sed installed.
Otherwise it will not generate the docs correctly.