If me for some time, you in all probability know I began my profession as a QA engineer earlier than transitioning into the world of information analytics. I didn’t go to high school for it, didn’t have a mentor, and didn’t land in a proper coaching program. Every little thing I do know in the present day—from SQL to modeling to storytelling with information—is self-taught. And imagine me, it’s been a journey of trial, error, studying, and re-learning.
The Dilemma That Modified My Profession
A couple of years in the past, I began serious about switching organizations. Like many individuals in fast-evolving tech roles, I confronted a surprisingly troublesome query:
What function am I really doing? Which roles ought to I apply for?
On paper, I used to be a Knowledge Analyst. However in actuality, my function straddled a number of features: writing SQL pipelines, constructing dashboards, defining KPIs, and digging into product analytics. I wasn’t positive whether or not I must be making use of for Analyst roles, BI roles, or one thing completely totally different.
To make issues worse, again then, job titles had been obscure, and job descriptions had been bloated with buzzwords. You’d discover a posting titled “Knowledge Analyst” that listed necessities like:
- Construct ML pipelines
- Write advanced ETL scripts
- Keep information lakes
- Create dashboards
- Current executive-level insights
- And oh, by the best way, be nice at stakeholder administration
It was overwhelming and complicated. And I do know I’m not alone on this.
Quick ahead to in the present day: fortunately, issues are evolving. There’s nonetheless overlap between roles, however organizations have began to outline them extra clearly. On this article, I need to break down the actual variations between information roles, via the lens of a real-world instance.
A Actual-World Situation: Meet Quikee
Let’s think about a fictional quick-commerce startup referred to as Quikee, launching throughout a number of Indian cities. Their worth proposition? Ship groceries and necessities inside 10 minutes.
Clients place orders via the app or web site. Behind the scenes, there are micro-warehouses (additionally referred to as “darkish shops”) throughout cities, and a fleet of supply companions who make these lightning-fast deliveries.
Now, let’s stroll via the info wants of this firm—from the second an order is positioned, to the dashboards executives use of their Monday morning conferences.
Step 1: Capturing and Storing Uncooked Knowledge
The second a buyer locations an order, transactional information is generated:
- Timestamps
- Order ID
- Gadgets ordered
- Value
- Low cost codes
- Buyer location
- Fee technique
- Assigned supply companion
Let’s assume Quikee makes use of Amazon Kinesis to stream this information in actual time to an S3 information lake. That stream is high-volume, time-sensitive, and essential for enterprise monitoring.
However right here’s the catch: uncooked information is messy. You possibly can’t use it straight for decision-making.
So what occurs subsequent?
Step 2: Constructing Knowledge Pipelines
Enter the Knowledge Engineers.
They’re chargeable for:
- Ingesting real-time information
- Validating schema consistency
- Dealing with failures and retries
- Writing pipelines to maneuver information from S3 into an information warehouse (say, Snowflake or Redshift)
That is the place ETL (Extract, Rework, Load) or ELT pipelines come into play. Knowledge engineers clear, format, and construction the info to make it queryable.
For instance, an order desk may get break up into:
- Orders → One row per order
- Order_Items → One row per merchandise in an order
- Funds → One row per cost try
At this stage, uncooked logs are became structured tables that analysts can work with.
Step 3: Dimensional Modeling & OLAP
As management begins asking strategic questions like:
- “Which metropolis brings in essentially the most income?”
- “Which retailer is underperforming?”
- “What’s our common supply time by zone?”
…it turns into clear that querying transactional information straight gained’t scale.
That’s the place dimensional modeling is available in.
As a substitute of flat, uncooked tables, information is structured into Reality and Dimension Tables.
🔸 Reality Tables
- Giant, quantitative information tables which include international keys together with measures and metrics (Effectively, more often than not. There are factless reality tables as effectively which would not have any measures).
- Examples:
fact_orders,fact_payments,fact_deliveries - Include metrics like income, order depend, supply time
🔹 Dimension Tables
- Smaller, descriptive tables that assist perceive the info in a reality desk
- Examples:
dim_store,dim_product,dim_customer,dim_delivery_agent - Assist filter, group, and be a part of info for deeper insights
This construction allows OLAP—quick, analytical querying throughout a number of dimensions. For instance, now you can run queries like:
“Present me common supply time by retailer and hour of day, over the past 7 days.”
This step is finished by Knowledge Engineers at a lot of the organisations however I did construct few Dim and Reality tables once I was working as a Business Intelligence Engineer at Amazon.
Step 4: Defining KPIs and Metrics
That is the place Analytics Engineers (or BI Engineers) shine.
They sit between the technical information layer and enterprise customers. Their obligations usually embrace:
- Defining KPIs (e.g., churn price, repeat buy %, time-to-fulfillment)
- Writing logic for advanced metrics (e.g., cohort retention, energetic customers)
- Creating semantic fashions or metrics layers in instruments like dbt or Looker
- Guaranteeing constant definitions throughout the corporate
For instance, at Amazon, our workforce didn’t question uncooked information to calculate income each time. As a substitute, we created pre-aggregated reality tables at day by day, weekly, and month-to-month grains. That manner, dashboards loaded quicker, and metrics stayed constant throughout groups.
Analytics Engineers act as translators between engineering and the enterprise—defining what we measure and how we measure it.
Step 5: Evaluation, Reporting & Storytelling
Now comes the function of the Data Analyst.
Armed with clear, modeled information, they give attention to answering actual enterprise questions like:
- “Why did retention drop in Bangalore final month?”
- “Which coupon codes drive essentially the most new customers?”
- “What are the highest merchandise reordered within the first 30 days?”
They construct dashboards in instruments like Tableau, Energy BI, or Looker. They run ad-hoc SQL queries. They dive into A/B take a look at outcomes, person habits traits, and marketing campaign effectiveness.
However above all, they inform tales with information—making advanced numbers comprehensible and actionable for stakeholders.
Who’s Who?
TL;DR: The place Do You Match?
Right here’s how I give it some thought:
- Love constructing strong pipelines and fixing scalability issues? → You’re a Knowledge Engineer
- Love defining enterprise metrics and organizing advanced datasets? → You’re an Analytics Engineer
- Love uncovering insights and storytelling with information? → You’re a Knowledge Analyst
In fact, real-world roles usually mix these. Particularly at smaller corporations, it’s possible you’ll put on a number of hats. And that’s okay.
The secret is not the title—however the place you add essentially the most worth and what energizes you.
Closing Ideas
It took me a very long time to grasp what I really do—not simply what my job title says. And for those who’ve ever felt that confusion, you’re not alone.
Right this moment, I can clearly say I function on the intersection of information modeling, enterprise logic, and storytelling—a candy spot between analytics and engineering. And I’ve realized that the flexibility to attach the dots is extra essential than becoming into an ideal field.
For those who’ve walked an analogous path—or put on a number of hats in your function—I’d love to listen to your story.
Drop a remark 👇 or share this with somebody figuring it out too.
