Version 1.5.0 - 11/27/2020
... About 5 min
# Version 1.5.0 - 11/27/2020
We are excited to announce our newest version of Commandeer. We have a ton of exciting enhancements and a bug fix for Dynamo filtering.
# Two New Services - AWS AppSync and AWS Glue
This includes two new services AppSync and AWS Glue that are going to be really helpful for people using GraphQL and AWS ETL functionality.
AWS AppSync Breakdown Banner (Click to view)
AWS Glue Breakdown Banner (Click to view)
# Table of Contents
- Introductory Tour
- Revamped Web Portal
- Team Management
- ER and System Diagrams Dashboard Card
- AppSync
- AWS Glue
- AWS Glue Crawlers
- AWS Glue Jobs
- LocalStack
- S3 Bucket and Folder Filter
- S3 Treeview Enhancements
- DynamoDB Treeview Enhancements
- PostgreSQL Updates
# Introductory Tour
Now when you download the app for the first time, we help guide you through all the core pieces of functionality. For all of you that already have the app, you can go to the Help icon in the upper right of the app, and click the Introductory Tour button to get guided through the system. We plan on rolling out many more of these over the next few months starting with Account Management, Docker, and LocalStack.
Start a tour of Commandeer
# Revamped Web Portal
As we have been building out our team offering, which is a way to manage a team of engineers on Commandeer under one billing invoice. Because we were moving our team administration from within the app to the website, we also decided to start building out the site a bit more. In future releases, you will start to see your stats from the app here, as well as your team stats, and some other great web-centric pieces of functionality.
Commandeer Website Dashboard
# Team Management
To manage teams in Commandeer, you would go to Commandeer --> Teams ---> Dashboard. As we are expanding on this offering, and because of the fact that it is normally an engineering manager managing this aspect of the system, we are moving this to our web portal. Stay tuned over the next few releases as we will be rolling out resource, user, and tag statistics for you to better understand actual usage by your team within your system.
Commandeer Team Management Portal
# ER and System Diagrams Dashboard Card
Visualizing your system in one of the most important things you can do so that you and your team can understand what is going on in distributed serverless and container based systems. As we build out this offering, we needed a way to organize them all. You can see the card below. You can always still get to them under each service.
ER and System Diagrams Dashboard Card
If you don't recall, our system diagrams are a great way to see your system, and we offer the ability to download them all in .png, so you can include them in meetings.
Docker System Diagram
# AppSync
We are framing out the AppSync service. For those of you that are unfamiliar with AppSync, it is the GraphQL system offered by AWS. Those of you may remember Scaffold.io, which was a GraphQL tool. The developer of it was aqui-hired by AWS and brought in to build out AppSync. The first version was released about two years ago. A little later, Bill Fine, who was formerly a VP at Serverless Framework was brought in to head the business unit. I am excited to see what they show at re:invent this year.
Personally, I have mixed feeling about GraphQL. It does offer some cool pieces of functionality, and does make it more about the frontend developer than is the case with standard REST. But, setting it up, is certainly not trivial, and the templating engine is pretty crazy. So, we at Commandeer are working hard to try to make sense of all the functionality, and get it into a UI that helps manage it a bit better.
In this first incarnation, we will be releasing the general layout of AppSync, which consists of Data Source, Functions, and Resolvers. We will then rollout the Query Runner as well, which we think will be a very powerful way to test running queries of your data. Below you can see the high-level dashboard for AppSync.
AppSync Dashboard
The navigation is coming together nicely. Below you can see the different pieces of the system broken down on the left-hand side.
AppSync Navigation
# AWS Glue
Moving data between S3 and DynamoDB into Athena Data Lakes can be made much easier by running Crawlers and Jobs in AWS Glue. Before our new tool however, this was a very cumbersome task in the AWS Console. The underlying API as is typical in AWS-land is top notch. So, really, what it needed was some commandeering of it into our app. Below you can see the four main sections of Glue. Crawlers, Classifiers, Jobs, and Triggers. There will also be a Workflows section that combines the Crawlers and Jobs to make a full fledged ETL process.
AWS Glue Navigation
# Crawlers
A crawler has a source location such as S3 or DynamoDB and then a destination which is the Athena database. The crawler will run, and produce the correct tables in your database. This is very powerful as you can have all your data setup instantly. For instance, we run this on our S3 data lake analytics data and it automatically creates orur 20 or so tables instantly. Below you can see an example of a crawler that takes the analytics data and saves it to your database..
Glue Crawler Detail
The crawlers can also be viewed in the S3 and Dynamo navigation system.
S3 Crawlers
# Jobs
AWS Glue Jobs allow you to run a piece of python or scala code for doing transformations on your data. Below you can see three different jobs that are set to run.
Glue Jobs
# LocalStack
LocalStack allows you to connect to a local mocked version of AWS. We have two new enhancements. The first is that the newer version of LocalStack uses just one edge port to connect to. This greatly simplifies managing it. You can simply set the port on the LocalStack settings.
Localstack Url in Settings
We also added a state on the LocalStack page when your current account is not set to local, you will be presented with this message on the LocalStack dashboard. This makes it a bit easier to handle the differences between the local and cloud environments within Commandeer.
LocalStack state when not connected to a local account
# S3 Bucket and Folder Filter
You can now filter results of buckets and folders on the main dashboards. This is really helpful for finding files fast. We are looking into building this into the treeview as well, but for now, this is the quickest way to get to a file.
Filtering for cats files in a bucket
# S3 Treeview Enhancements
In the S3 treeview we are now showing you the CloudWatch Alarms, AWS Glue Crawlers, and Lambda Triggers that can all be connected to an S3 Bucket.
S3 Treeview
# DynamoDB Treeview Enhancements
In the DynamoDB treeview on the left, you will now find attached CloudWatch Alarms, AWS Glue Crawlers, and DynamoDB Streams that are connected to each table. This helps to quickly see what is connected to your tables.
DynamoDB Treeview
# PostgreSQL Updates
We have added the Query History and Favorites to the query runner on PostgresSQL. This works the same way as in Athena, where you can see previous queries, and delete and favorite ones you have run. We also now show you if the query was succcessful and how many records were brought back.
History and Favorites