Data engineering

Most enterprise leaders dip their toe into AI, only to realize their data isn’t ready—whether that means insufficient data, legacy data formats, lack of data accessibility, or poorly performing data infrastructure. In fact, Gartner predicts that through 2026, organizations will abandon 60% of AI pr...

Explore how AI, Agentic workflows, and cloud-native technologies are reshaping insurance underwriting, claims, and risk management in 2025.

An average company adopting cloud today could achieve 180% ROI in business benefits, although few are getting close to these returns.  Migrating or replatforming to a cloud data platform is a complex process. Even a basic "lift-and-shift" migration requires careful planning—designing the target...

How telemedicine, the IoMT, integrated healthcare data, digital health commercialization, and GenAI-powered drug R&D will shape the year ahead.

With the recent advancements in generative AI, and the ongoing explosion of multimodal biomedical data from large biobanks, electronic health records, medical imaging, and biosensors, it is now possible to capture the complexity of human health and disease more accurately. In this whitepaper, w...

Without data observability in their production pipeline, this Fortune 500 manufacturer encountered multiple data issues that drastically impacted time-to-market, added significant development overhead, and complicated the product development roadmap. A data observability solution simplified, accele...

Originally published in Forbes Enterprise companies accumulated massive amounts of data over the years in pursuit of the promised rewards of big data. The question that now often comes from management is, “There must be some value, somewhere—can’t you find it?” The data democratization moveme...

Data quality control is a mission-critical capability for virtually any modern enterprise. Why? Because data quality issues can disrupt business processes and services, invalidate any type of analytics performed in the company, and damage a company's reputation. However, despite the importance of d...

Having complete control over data quality can seem daunting and prohibitively expensive. But is that really true? Is it as difficult as it seems? There are many tools out there in the marketplace to help with data quality, but it’s true that many of them are pricey, have a steep learning curve,...

It’s no secret that leading a healthy lifestyle improves your overall quality of life. A healthy lifestyle involves following a good diet, limiting excessive alcohol or smoking, taking the time to exercise, and getting enough sleep. This same philosophy can be applied to the way your business t...

After the initial excitement that data lakes would help companies maximize the utility of their data, many companies became disillusioned by rapidly diminishing returns from their big data efforts. While it was easy to put large volumes of data in the lakes, turning that data into insights and real...

In an era marked by growing velocity, volume, and variety in data, managing this constant influx of competing data points and information is crucial for businesses aiming to stay ahead. Data platforms help achieve this—but building and customizing them is key to success. In our experience workin...

The quarterly report showed a 40% drop in sales over the last quarter. The executive team panicked. The entire company is collapsing! Or is it? After the investigation concluded, they discovered that the sales were okay. The report was incorrect because an automated system had failed. The infrast...

In early 2016 one of our online media customers came to us with a problem. Our customer, a media giant, hosts articles from its newspapers and magazines on its websites. Each of the articles’ web pages has three ad blocks, and the customer buys paid redirects to the article pages. They then analyze...

In this blog post we share our experience delivering deduplicated data during In-Stream Processing for a large-scale RTB (real time bidding) platform. A common problem in such systems is the existence of duplicate data records that can cause false results when processed by the analytic queries. One...

In the previous blog post we went through the details of how to set up the Docker infrastructure with Mesos and Marathon, and how to bootstrap the environment to get it ready to host application services. Now it is time to deploy some applications. Our blog’s purpose is not only to show you a fi...

In the previous post we discussed which models we tried for sentiment classification and which one has demonstrated the best performance. In this post, we’ll show you how to visualize our under-the-hood findings so that others can see the results of our analysis. You can see our twitter senti...

In the previous post we discussed how we created an appropriate data dictionary. In this post we’ll address the process of building the training data sets and preparing the data for analysis. The training process aims to reveal hidden dependencies and patterns in the data that will be analyzed...

In the previous post we discussed the structure of the tweet data. In this post we’ll address the process of selecting or building the right data dictionary for our purpose. What constitutes a good dictionary? A crucial data set for any kind of text mining is a dictionary. As for sentiment...

In the previous post we outlined the basic scientific method used and formalized the problem statement we are solving, which is, “Based on of the tweets of English-speaking population of the United States related to selected new movie releases, can we identify patterns in the public’s sentiments t...

This article introduces the Grid Dynamics blueprint for in-stream processing. It is based on our experience and the lessons we have learned from multiple large-scale client implementations. We have included cloud-ready configuration examples for Apache Kafka, Spark...

This post contains a brief survey of better-known products related to in-stream processing that are available on the market at the time of this writing. In this survey, we focus specifically on critical architectural differentiations, rather than functional differences, that affect why custome...

Now that we have introduced the high-level concepts behind In-Stream Processing and how it fits into the Big Data and Fast Data landscapes, it is time to dive deeper and explain how In-Stream Processing works. As we already know, In-Stream Processing is a service that takes events as input and...

In-stream processing is a powerful technology that can scan huge volumes of data coming from sensors, credit card swipes, clickstreams and other inputs, and find actionable insights nearly instantaneously. For example, in-stream processing can detect a single fraudulent transaction in a stream...