Building a Seamless Search-Driven Analytics Experience Across Multiple Data Sources With Apache Spark

Search-Driven Analytics From Multiple Data Sources

Gone are the days where all the data a company wants to analyze comes from a single relational database. In today’s world, most companies have data stored in not only different sources but a variety of different types of sources, from relational databases, NoSQL stores, Hadoop repositories, as well as application sources.

As data sources keep growing and evolving, reconciling data across these disparate resources becomes a considerable challenge. Each of these sources have different storage engines, schema types, and interaction protocols. Providing analytics and actionable intelligence using data across sources is highly valuable but challenging.

The Need for Multiple Data Sources

Different storage systems provide different data storage capabilities, and so it is natural for an organization to choose the best tool for the specific application or problem at hand.

Let’s take an example of e-commerce website. The typical data architecture contains the following components:

  • Relational databases are used to hold product details and customer transactions.
  • Big data warehousing tools like Hadoop/Hive/Impala are used to store historical transactions and ratings for analytics. This reduces the pressure on relational databases.
  • Google Analytics to analyze the behavior of customers on the website.
  • Log data on S3 or Azure blob storage.

In this example, the organization is using different data systems which are optimized to store the different kind of data.

The Need for Multi-Source Data Analysis

To derive the maximum value from data, an organization typically wants to complete view that connects all the different sources of data. In our e-commerce example, it will be good to combine Google analytics data with transactional data in databases to understand the patterns in users interest.

This kind of multi-source data analysis drives greater value by providing a more complete view compared to single source analysis which will only give one perspective of the overall story.

The Problem With Traditional Multi-Source Data Analysis

Traditionally, multi-source data analysis needed all the data to be moved to single data warehouse. This data warehouse could be a relational database such as Oracle or Teradata, or it could be a big data store such as Hadoop. But these types of systems require costly ETL operations to move the data.

The primary reasons for this ETL is to normalize the data from different sources, and simply to have common storage for running analytics across multi-source data.

This movement of data often result in slowness in analysis. Also, the ETL process creates latency from when data is updated at the original source until new data makes its way into the analytical systems, so the latest data is not always available. For many organizations, they end up performing incomplete data analysis from either fewer sources or delayed data, even though they have access to much more data.

Multi Source Data Support in Tellius

tellius insights

The picture above shows many of the popular connectors supported in the Tellius platform. Tellius can connect to multiple sources and perform multi-source analysis without requiring complete movement of the data. One question we are often asked is: how do we support a seamless search-driven analytics experience across multiple data sources?

Structured Data Analysis for Big Data

Before Apache Spark became popular, it was assumed that “big data” is meant for unstructured data. But in reality, that is not true. From our experience, many of the customers who are big data (and certainly for non-big data customers), most of their data is structured or semi-structured. So any data processing platform we choose has to support structured data processing out of the box.

Apache Spark is the first major big data processing platform which has embraced structured data analysis as native abstraction. This abstraction can support structured, semi-structured, and even unstructured data. Being structured-first allows Spark to combine data from sources such as relational databases, Google analytics, and MongoDB seamlessly. Even for unstructured data, Spark has capabilities to turn that into structured data.

Multi-Source Data Analysis in Spark

From Spark 2.0, Spark has chosen Dataset/DataFrame abstraction, which is built for structured data. This normalizes data read from different sources to a single DataFrame abstraction. This native abstraction combined with Spark SQL gives us the ability to seamlessly unify the data across different sources and make the available for search-driven query and analytics.

Spark loads only the needed data on-demand from the sources. This allows analysis to be done in an ad hoc manner rather than waiting for complete movement of the data in the traditional way. This greatly speeds up the time of analysis.

Natural Language Query on Spark

All the datasets loaded in Tellius are represented as Dataframes on Spark. This gives us the ability to provide an intuitive search experience to the user, where he/she can ask query across different sources with a single question. The queries initiated by the user are internally represented as SparkSQL queries, which then accesses the data across sources. This analysis can use the latest data without the need to make a copy of data in a warehouse system.


Combining the Tellius Search Engine with the versatile Spark engine allows our customers to ask questions of their data irrespective of where the data comes from. This capability gives our customers clear advantages over their traditional systems as they now can leverage all their organizational data for analysis to make better informed decisions.


Read Similar Posts

  • Tellius Kaiya: Welcome to the Next Generation of Self-Service
    Analytics & Insights

    Tellius Kaiya: Welcome to the Next Generation of Self-Service

    Introducing Tellius Kaiya: A suite of GenAI-powered tools designed to democratize data-driven decision-making for the entire organization.

  • Navigating Generative AI in Pharma: Insights from Top Industry Leaders
    Analytics & Insights

    Navigating Generative AI in Pharma: Insights from Top Industry Leaders

    During a PMSA 2024 Q&A session, leaders from Novartis, Pfizer, UCB Biopharma, and Boehringer-Ingelheim covered the current state of GenAI and its immense potential to transform the pharmaceuticals industry. So, what's in store?

  • 7 Takeaways from the Gartner Data & Analytics Summit 2024
    Analytics & Insights

    7 Takeaways from the Gartner Data & Analytics Summit 2024

    The 2024 Gartner Data & Analytics Summit was a jam-packed three days of sessions and networking opportunities for data & analytics leaders. Here's what Tellius took away from the event.