Deduplication sounds simple until you actually try to do it on real data. If you have a few thousand short texts, almost any solution works. When you move to millions of long documents, multiple languages, noisy data, and the requirement…
Hey there! A couple of weeks ago I hit a classic data-engineering wall. I had a pipeline that needed to: This process has to run for about 3K libraries data. if the whole process takes 1 seconds it takes about…
Spoiler Alert: This article contains key plot details from Whiplash (2014) In every ambitious person’s journey, there’s a moment when the excitement of learning slowly turns into an obsession with being perfect. It’s the moment when “I want to get…
There’s a moment in every data engineer’s journey when the excitement of building pipelines meets a difficult, quiet question: Should this run in batch, or should it be real-time? It sounds technical — but it’s actually philosophical. Behind it lies…
Welcome back to our deep dive into Elasticsearch! So far, we’ve mastered the art of finding the right documents. We’ve become experts in queries, filters, and the mighty bool query. But what happens after you’ve retrieved your results? How do you make sense of…
Introduction If you’ve been following this series, you already know: Now it’s time for the real workhorse: the bool query.Why? Because no real-world search problem is solved by just one condition. Users expect relevance and restrictions: The bool query is…
In Part 1 of this series I walked through the foundations of Elasticsearch queries: the mental model, why mapping is your best friend, and how to choose between filters and matches. Now it’s time to roll up our sleeves and…
When I first got to know Elasticsearch, I told myself: “Well, this is just another database… right?”But I was wrong. Elasticsearch is actually different. It kind of feels like a mix between a search engine and a database.To be honest,…
Let me be honest: when I sit down with a fresh dataset, I’m not looking for ceremony. I’m looking for clarity. That first hour matters more than most people admit. I want to get a feel for the terrain—what’s messy,…
You’ve probably heard the term batch processing before.In this post, we’ll talk about what it means and how to use it in Apache Airflow. These days, with the mind-boggling amount of data growing by the second, one of the most…
