Top 10 Services to use for Serverless Cloud Development
... About 5 min
# Top 10 Services to use for Serverless Cloud Development
Using serverless to build product is a true joy. But it requires the correct tools. Below you will see our top 10 list of tools you need to build and manage a serverless system properly.
# 1. AWS (Amazon Web Services)
AWS is the dream cloud for Serverless development. You can create amazing event-driven programs using S3, SNS, SQS, DynamoDB, DynamoDB Streams, API Gateway, AppSync, Athena, and Lambda. The toolbox of products that AWS can be staggering, but finding the right combination for Serverless isn't impossible.
I have built numerous products on a 100% serverless system, and we have wittled it down to the above setup.
- Setup your API in either API Gateway or AppSync calling a Lambda or saving directly to DynamoDB.
- Create a DynamoDB Stream that calls a Lambda to process the data for events and also saves it to S3 as a data lake
- User SNS and SQS with Lambda triggers to do data processing
- Setup Athena to read from your S3 data lake you have setup in step 2
# 2. LocalStack
LocalStack is a mock AWS environment. It allows you to run an AWS system locally, and access it in the same way as you would the services on the AWS cloud. This has a number of advantages, from speed of development, to ability to be offline, to saving costs for your development. If your team sets up their serverless system correctly, they should be able to run and test all their code paths locally.
# 3. Serverless Framework
Serverless Framework allows you to deploy serverless connections between the various AWS resources, such as S3 or DynamoDB Streams and the Lambda that it invokes, you need to have this done in code. While Serverless Framework does not handle things like creating S3 buckets or DynamoDB tables correctly, it does the job of connecting all the services really well. The also have the best maintained and active plugin library around. It is IaC done right! But, you will most likely need to accompany it with one of the following.
# Honorable Mentions
Each of these five tools have + pluses and - minuses to developing your IaC code. I have heard the argument that Ansible is not IaC. But I still find it the best way to handle creating S3 buckets, DynamoDB tables, and SQS queues. But, each have their own quirks, like Terraform will just wipe away S3 buckets or DynamoDB tables if you have something made outside the Terraform stack, and then try to do an auto-deploy of your Terraform code.
Anyway, you more than likely will need to choose one of these in addition to Serverless Framework, so just know, you might not be 100% satisfied with any of them. We are hoping these really start to pan out over the next 2 years, as IaC is a critical piece of serverless development.
# 4. Docker
Docker allows you to create containers and volumes. Even though we are using serverless, Docker is still crucial for a number of reasons. The first is that you will want to manage your LocalStack in a docker container. It is their preferred way, so that is already one use case. You can then have it running on your CI/CD and in your dev teams local environment. Lastly, it helps when pulling in other services for your system like Postgres or Redis.
# 5. Commandeer
Commandeer is the new comer on the block for serverless management. It is a desktop IDE that let's you manage your Docker, LocalStack, Serverless Framework, Ansible, S3, SNS, SQS, API Gateway and many more services from your computer.
You can setup multiple accounts within the app, and instantly switch between your local environment and many cloud environments. It has all the desktop bells and whistles like right-click context menus, side treeview navigation, universal search, a top menu system, and breadcrumbs.
It's real sweet spot is viewing all your S3 data as well as editing it, viewing all your DynamoDB data, querying data in Athena, RedShift, and Postgres, and testing out all the different connections using tools to test S3 -> Lambda, DynamoDB -> Lambda, SQS -> Lambda and much more.
There are also over 15 ER and System diagrams that you can view to see how your system is really laid out. We are of course biased, because we built this tool. But, we use it daily because we needed to actually see AWS systems in a manageable way.
# 6. Postman
Postman is the ultimate tool for managing your API layer. Most systems in the cloud have a backend in the cloud, and many apps out there in the wild are consuming this data API. Postman lets you organize all your API's into collections, allowing your front end team to see the system the same way the backend team does. It also let's you create complex tests like adding a company, with a POST, updating them with a PATCH, and then deleting them with a DELETE request.
Through their use of simple tests written directly into their UI in javascript, and exporting the environments and collections to flat json files you can setup continuous integration tests. You simply add these two files to your repo, and then use their newman NPM to have it run in Circle.ci or your preferred CI/CD.
# 7. Dashbird
Dashbird handles the battle of observability into your system. When you go live into production, this becomes crucial. The most important way to figure out problems is with logs and alerts. Dashbird consumes your CloudWatch Logs, and gives you broken down invocation information about each call. Turn on your VERBOSE logs on your Lambda, and bask in the glory of seeing logs organized. They are built from the ground up to manage a Lambda based system, this is the real key to their success.
# 8. Slack
Slack is crucial for observability into your system. Setting up many different rooms that get written to by your system is an invaluable way to handle seeing data and errors. You will want to make sure your rooms aren't too chatty. But having CRON running to give your team summaries in a slack channel, or writing all errors to slack are just some of the ways, that instant notifications can help you build a better system.
# 9. Circleci
Circleci enables you to run your serverless system in the cloud. By stitching together your docker-compose, terraform, and serverless framework IaC code for into one helpful pipeline to test locally, deploy, then test in the cloud. We setup it up that the first step will stand up LocalStack and run through all our unit tests and Postman tests. Then we have a step to deploy our Terraform or Ansible code. Then our Serverless Framework code. Lastly, we have a smoke test step that will use the same Postman tests, but run them against the newly created endpoints. All of this is done on every commit to our dev, staging, and master branches in our GitHub repo, and the results are written to a Slack channel.
# 10. Firebase
Firebase is a pub/sub NoSql database. Or at least that is how it started, it was then acquired by Google, and has become the backbone of their mobile system with more than 20 products under their ecosystem.
But the original product is still the best out there if you need realtime data over sockets to mobile devices. It is a true marvel to use, and just always works. They have SDK's for all major backend languages, so if you do want to venture out of the AWS ecosystem, this is one tool I would highly recommend. It isn't exactly a serverless offering, but a managed service, but if you are building state of the art serverless systems, you may have a real need for a pub/sub system.
# Conclusion
Well, there you have it! The serverless paradigm is an exciting evolution in software development. Choosing the right tools to handle it all is not easy. But luckily, the ecosystem is maturing rapidly. It can seem daunting, but once you start to see your entire system it will become an absolute joy to build in.