Retina.ai Case Study - Visualizing and Accessing Your Data
... About 2 min
# Retina.ai Case Study - Visualizing and Accessing Your Data
# How Retina.ai Bridges The Gap Between Data Engineers and Data Scientists using Commandeer
When it comes to data analysis, having all relevant data at your fingertips is everything. Making data-driven business decisions is key in today’s world. Retina.ai, a venture-backed startup with a founding team from Facebook and PayPal helps businesses to maximize their customer lifetime value using both ML (Machine Learning) and AI (Artificial Intelligence.
"By allowing our data engineers to see the system in the same way that our data scientists do, we are able to visualize and explain complex data pipelines in minutes. Quickly viewing S3 and querying Athena is truly magical." - Emad Hasan, CEO - Retina.ai
# Problem
To setup and process data efficiently, Retina.ai data engineers need to deliver the relevant data to data scientists and data analysts using some sophisticated data pipelines. Data engineers and scientists come from two different backgrounds. Engineers write the code to move the data, and the scientists write queries and sophisticated coding models against the data. Where the data needs to be translated from one world into another quickly and efficiently, is one of the biggest bottlenecks in setting up a scaling system.
# Solutions
# Access Data Efficiently
Today, most of the data is stored in the cloud in multiple locations. For AWS, the data can be stored on S3 in basic storage or in a proper data lake format, DynamoDB, RDS, etc. Having a birds-eye view into each piece of data reduces the amount of time needed to start doing some productive data analysis.
# Visualize Data Pipelines
Modern data sources contain hundreds, sometimes thousands of database tables. Being able to see your data structures and the connections between them visually allows engineers to create data pipelines quicker. It also decreases the ramp-up time for new engineers to hit the ground running as fast as possible.
# Reduce Communication Overhead
Having a way to see the data infrastructure and the data itself visually eases the communication between data engineers and data scientists. Lower communication overhead means more energy spent on doing some productive work.
# Use-Cases
# View the content of S3 Files
With Commandeer, viewing files on S3 is effortless. You can see the content of each file while quickly navigating between files. You can view, edit, and download text files with the syntax highlighting for most common data formats like CSV, JSON, YAML, parquet, etc. Under the hood, it is using the Monaco Editor which is the open-source implementation of the editor used in Visual Code. So, your team is getting a world-class IDE experience for every file in their system. They can also instantly preview images, gifs, videos, and audio files.
# Querying Data in Athena and PostgreSQL
Navigating through tables and columns, as well as querying the data in a PostgreSQL database and Athena data lake using the Commandeer UI feels very natural. Commandeer allows you to drill down into your database structure from the side navigation with the ability to see table dashboards with some useful summaries of your data. You can also query the data using a simple and elegant query editor.
# Visualize Your Data Structures
A picture is worth a thousand words. Seeing your data structure visually allows you to connect the dots in the fastest way possible. Walking the teammates through a data diagram is so much better than explaining it in any other way.
Commandeer offers ER diagrams for Algolia, DynamoDB, PostgreSQL, and Athena. It also offers system diagrams for the connections between your service and the Lambdas, CloudWatch Alarms, and CloudWatch Logs between them. For instance, there are diagrams for S3, DynamoDB tables and streams, SNS, SQS, and more.
# Outcome
Retina.ai is growing rapidly, helping more and more companies optimize their Customer Lifetime Value. With Commandeer, data engineers deliver the most relevant data to data scientists for analysis quickly and efficiently. Being able to see the data visually allows the data scientists to perform some more comprehensive analysis unlocking more precise LTV optimizations.