5 Challenges of Building Data Applications

July 13th, 2020

This article is by Snowflake and originally appeared on the Snowflake Blog here: https://www.snowflake.com/blog/5-challenges-of-building-data-applications/

 

Fast-growing software companies are building data applications for a variety of uses, from marketing apps that provide customer insights, to IoT apps that handle device feedback, and data analytics apps that process both historical and near real-time data.

But developers often face obstacles when building, designing, and supporting applications that need to parse large volumes of information. To address these challenges, companies are building their data applications on a cloud data platform, which can provide virtually unlimited and automatic scalability, concurrency, near instant elasticity, and support for structured and semi-structured data. With Snowflake Cloud Data Platform, data applications do not contend for resources, and customers have continuous access to the data and analysis they need.

Below are five of the most common challenges for building highly performant data applications. For more details on how Snowflake helps address and solve the challenges faced by application builders, download our ebook: The Product Manager’s Guide to Building Data Apps on a Cloud Data Platform.

  1. Developing a 360-degree view of the customer 

Customer data is typically fragmented across many systems and consists of different data types. For example, recent customer purchase history is stored in operational databases. In contrast, real-time event data, such as a clickstream, isn’t structured and requires complex data pipelines. Intent and demographics data is typically acquired from third parties and requires difficult integrations and expensive data duplication, which results in stale data.

  1. Handling IoT device data in near real-time

Ingesting, storing, and analyzing massive amounts of device data for IoT applications often overwhelms traditional data pipelines and that data is costly to store. Because device data is unstructured, it requires transformations that disrupt the ingestion process. Additional delays and expenses can also arise because raw device data is often stored in cloud object storage and must be moved to a data warehouse for analysis. Adding to the challenges, data may arrive out of order and requires manipulation before it can be analyzed.

  1. Combining historical and current data for analysis

Developers of security and application health analytics apps often need to evaluate data to predict anomalies, detect threats, and respond proactively. But combining sizable amounts of historical and current data for analysis presents several challenges. Storing large volumes of data for long periods is costly, and ingesting real-time event data is difficult to do reliably. And staging data in cloud object storage rather than streaming it results in delayed time to insight and additional storage costs.

  1. Bringing data together for machine learning (ML) models

Constructing accurate ML models requires a large volume of clean, historical data, and an iterative process of continuous improvement. Copies of entire data sets must be made to support each experiment—a costly and lengthy process that can limit the number of experiments conducted and hurt model accuracy. To support near real-time predictions, application developers need to query fresh data with very fast response times, which may require expensive overprovisioning of compute resources.

  1. Embedding analytics in data-intensive applications 

Developers face a number of challenges in embedding analytics in data-intensive applications. They encounter performance issues and concurrency limits as the number of users or query complexity increases, which results in poor user experiences and possible downtime. To alleviate the performance impact, developers often overprovision compute, which hurts profit margins by paying for idle resources. And ingestion of semi-structured data requires dedicated engineers who can maintain the pipeline to avoid disruptions when schemas change.

Download our ebook, The Product Manager’s Guide to Building Data Apps on a Cloud Data Platform, to learn more about how Snowflake Cloud Data Platform can help applications builders solve the challenges they face in five use cases:

  • Customer 360 data apps
  • IoT data apps
  • Application health and security analytics data apps
  • Machine learning and data science data apps
  • Embedded analytics data apps