AWS S3 Overview

Commandeer allows you to completely manage your S3 environment and view your files on AWS and well as LocalStack. You can view previews of your images, movies, audio recordings. You can also edit your text files inline and save them back to S3.

Amazon Simple Storage Service (Amazon S3) is an object storage service that offers industry-leading scalability, data availability, security, and performance. This means customers of all sizes and industries can use it to store and protect any amount of data for a range of use cases, such as websites, mobile applications, backup and restore, archive, enterprise applications, IoT devices, and big data analytics. Amazon S3 provides easy-to-use management features so you can organize your data and configure finely-tuned access controls to meet your specific business, organizational, and compliance requirements. Amazon S3 is designed for 99.999999999% (11 9's) of durability, and stores data for millions of applications for companies all around the world.

- Amazon

Previewing an image on S3

Commandeer Key Features

• Seamless switching between local and AWS
• See all your files in one place in a tree view.
• View and edit your text files with syntax highlighting for common programming languages.
• Preview your images and gifs (.png, .jpg, .gif, etc.)
• Watch your movies and music
• Download files
• Upload multiple files with Drag and Drop
• Create buckets
• Upload and delete files
• Drag and drop file upload
• Manage permissions
• Edit text files and save back to S3

Helpful Links

• AWS S3 Dashboard
• S3 Wikipedia Page
• AWS S3 API

Tutorials

• Create S3 Bucket
• Test S3 Bucket Invoking a Lambda
• Adding files and folders to S3 using Commandeer
• Adding files and folders to S3 using Ansible
• Create S3 Bucket with Ansible
• Delete S3 Bucket using Commandeer
• Edit files directly on S3 with Commandeer

Coming Soon

• Connect Lambda to an S3 Bucket in the GUI
• Move files
• Rename files
• Duplicate files

What is an S3 Bucket?

An S3 bucket is an AWS resource allowing you to store files organized by folders in the cloud. A bucket is global. It isn't tied to any specific AWS region. The bucket has a set of permissions allowing certain users to read, write or change permissions on the file. Once the files are uploaded into the bucket, you can serve the files from your client application or use the files from your backend services using the AWS S3 API.

What is an S3 folder?

Folders are represented as files with a slash at the end of the filename. AWS does not have the concept of folders like on your computer system. The file name actually has the folder(s) names in the file path like /folder_name/file_name.txt. Commandeer, treats these paths like a folder though, so you will see your files organized in a folder structure. But, it is helpful to understand how this works under the hood. The same permission rules for files apply to folders.

What is an S3 file?

An S3 file is a file with some content in the form of text, media, image, zip or anything else stored on S3. Each file has some permissions associated with it. A file can be fetched, created or deleted from S3 using the S3 API, Commandeer, or the AWS Console.

Bucket Permissions Explained

Both buckets and files have permissions. Bucket permissions control how the files and folders in the bucket are accessed. File permissions control access to just the file in question. Changing permission on a file only changes how this particular file is accessed. Folders are represented as files with a slash at the end of the filename. AWS doesn’t have the concept of folders like on a computer system. The file name actually has the folder(s) names in the file path like /folder_name/file_name.txt. In Commandeer, we treat these paths like a folder though, so you will see your files organized in a folder structure. But, it is helpful to understand how this works under the hood. The same permission rules for files apply to folders.

Each bucket or file has an owner which can be an individual user or a group defined in IAM. In addition to having an owner, each bucket or file has a list of grants. Each grant has a grantee and the permission level. In it's simplest form, the permissions can be represented as the following interfaces:

export interface GetBucketAclOutput {
  owner: Owner;
  grants: Grant[];
}
    
export interface Grant {
  grantee: Grantee;
  permission: Permission;
}

export interface Grantee {
  displayName: string;
  emailAddress: string;
  id: string;
  type: GranteeType;
  uri: string;
}

export enum Permission {
  PRIVATE = 'private',
  PUBLIC_READ = 'public-read',
  PUBLIC_READ_WRITE = 'public-read-write',
  AUTHENTICATED_READ = 'authenticated-read',
}

export enum GranteeType {
  CANONICAL_USER = 'CanonicalUser',
  AMAZON_CUSTOMER_BY_EMAIL = 'AmazonCustomerByEmail',
  GROUP = 'Group',
}

As you can see, it can be time-consuming to infer the overall permission level based on the list of existing permissions each time you would like to see what the permissions are. In Commandeer, we make it easier for you by displaying an overall permission level in a dropdown. This way you can easily see it and change it by simply choosing a different value. We also show you each grant and the owner's name for a bucket or a file on the detail page.

Bucket Permission Levels

Each bucket has a set of permissions associated with it which controls how the files and folders in the bucket are accessed. Here are the current bucket permission levels and their explanation.

Public Read and Write

This is the most permissive permission level allowing anyone in the world to read or write any file in the bucket. Be careful with this permission level because it allows anyone to update your files. Public read and write buckets are highlighted differently on the Commandeer UI so you can avoid having some accidental buckets with wide-open permissions.

Public Read

Public read allows anyone on the Internet to read the files from your bucket. This permission level restricts writing into the bucket. This permission level is also highlighted in Commander UI to make you aware of it. This is useful if you want to share images with people on the internet, as you can make it available and then give them the URL to it, and they can access it on their computer.

Any AWS User Read

It allows any authenticated AWS user to read the files from the bucket with such permission level. Be careful with this permission level too. Even though it requires a user to be authenticated with AWS in order to read your files. Nothing prevents any person out there to sign up for an AWS account to gain access to any bucket with this permission level.

Private

Private access restricts all read and write operations. This is the most restrictive permission level Commandeer has available. If you don’t want anyone seeing your files, this permission level is for you. Commandeer uses Private permission by default when creating new buckets to lean on the safe side.

File Permissions Levels

A file permission level controls how an individual file can be accessed. Let’s go over some file permission levels.

Public Read and Write

This is the most permissive permission level allowing anyone in the world to read or write into your file without any authentication.

Public Read

Public read allows anyone on the Internet to read the file. This permission level restricts writing into the file. Public read is great for hosting some files which need to be publicly accessed.

Any AWS User Read

It allows any authenticated AWS user to read the file. Be careful with this permission level also. Even though it requires a user to be authenticated with AWS in order to read your file. Nothing prevents any person out there to sign up for an AWS account to gain access to the file with this permission level.

Private

Private access restricts all read and write operations to a specific file. This is the most restrictive permission level Commandeer has available. Commandeer uses Private permission by default when creating new files.

EC2 AMI Read

This permission level is great if you would like to launch an EC2 machine using an AMI stored on S3. When EC2 AMI Read permission gets applied to a file, it grants Amazon EC2 read access to get an Amazon Machine Image (AMI) bundle from Amazon S3.

S3 and Localstack

Localstack allows you to run S3 locally on your machine without having an AWS account. Which is a fast and cost-efficient solution for local development.

Commandeer talks directly to the LocalStack HTTP API the same way it talks to the real S3 AWS ARN API. Once you have the S3 service started on the Localstack page, you can switch your account to 'local', which points to LocalStack on port 4572 on localhost. This allows you to manage your buckets, files, and folders inside LocalStack the same way you would manage it in your AWS account.

S3 with Lambda Triggers

The Lambda S3 Tester allows you to upload a file to an S3 bucket that is connected to a Lambda on the other end. You can then view the results of the lambda invocation.

S3 as a Data Lake

S3 can be used as a Data Lake containing some loosely structured files in the formats like .csv, .json, etc. The advantage of this technique is that you can store your data in a semi-formatted fashion and figure out the end result later when you query it with tools like AWS Athena, Google Big Query or doing Map Reduce.

• How to Set Up a Data Lake Architecture With AWS open_in_new
• Data Lake vs. Data Warehouse open_in_new
• Tips for Enhancing Your Data Lake Strategy open_in_new

Screenshots

Previewing an image on S3

Seeing all buckets with their permissions and the treeview of the buckets on the left

Watching a video on S3

Editting some code on S3