This article is written by Snowflake and originally written by John Carter on the Snowflake Blog here: https://www.snowflake.com/blog/data-modeling-in-a-post-covid-19-world/
As a result of the COVID-19 pandemic, organizations around the world have had to transform overnight. Businesses that had been delaying digital transformation, or that hadn’t been thinking about it at all, have suddenly realized that moving their data analytics to the cloud is the key to coping with and surviving the COVID-19 disruption. The next phase is about rebounding and thriving in a post-COVID-19 world.
Quite frankly, businesses that have invested in data management and data governance are better positioned to not only weather the storm but to chart a path forward because they have accurate, trusted data at their disposal. Without good data, it’s difficult to make good decisions.
Data modeling has always been the best way to understand complex data sources and automate design standards. Today, the role of data modeling has expanded as the central point of collaboration between data producers, data stewards, and data consumers. That’s because data modeling is the best way to visualize metadata, and metadata is now the heart of enterprise data management, as well as data governance and intelligence.
DATA MODELING FOR THE CLOUD: THE NEED FOR SPEED
Companies everywhere are rapidly building innovative business applications to support their customers, partners, and employees in this time of need. But even though speed is critical, businesses must take the time to model and document new applications for compliance and transparency.
Moving data to the cloud? No problem. However, you can’t manage what you can’t see. That’s why data modeling is a critical component of metadata management, data governance, and data intelligence. It provides an integrated view of conceptual, logical, and physical data models to help business and IT stakeholders understand data structures and their meaning.
Data modeling is the first step to ensuring that the enterprise can use, understand, and trust mission-critical information.
erwin Data Modeler (erwin DM) has been the most trusted data modeling solution for more than 30 years. It helps organizations find, visualize, design, deploy, and standardize high-quality enterprise data assets. And it’s intuitive, so you can get new models up and running quickly as you scale to address new business realities.
ERWIN AND SNOWFLAKE
erwin recently announced a partnership with Snowflake to help enterprises accelerate cloud migration while ensuring proper data governance. The native erwin DM integration lets customers automate the creation of Snowflake data models; engineer or generate code for Snowflake database schemas; reverse-engineer existing Snowflake schemas into erwin models; and compare, analyze, and synchronize Snowflake models with the databases they represent.
These integrations provide customers with the necessary data modeling, code generation, data mapping, lineage, documentation, and impact analysis that’s foundational to a sustainable data governance program.
NATIVE SNOWFLAKE CONNECTIVITY
Reverse engineering is the process of creating a data model from an existing database or a script. erwin DM creates a graphical representation of the selected database objects and the relationships between the objects. This graphical representation can be a logical or a physical model.
erwin DM now supports Snowflake as a target database using Codemesh-based JDBC support for Snowflake connectivity.
Figure 1 – Built in connectivity for Snowflake
After the reverse engineering process, you can perform the following tasks:
- Add new database objects
- Create the system documentation to do the following:
- Understand how the objects are related to each other and then to build upon the data structure
- Demonstrate the database structure visually
- Redesign the database structure to accommodate changing requirements
Most of the information that you reverse-engineer is explicitly defined in the physical schema. However, reverse engineering also derives information from the schema and incorporates it into the model. For example, if your existing Snowflake database includes primary and foreign key declarations, the reverse engineering process will derive identifying and non-identifying relationships and default role names based on that metadata. If the physical design does not have all the referential integrity (RI) constraints defined, you can infer some of that logical information. In addition, you can use column names to infer these keys and relationships.
For more information on Codemesh-based JDBC, refer to Database Connection Parameters.
For information on Snowflake objects and data types currently supported, refer to the erwin Snowflake Object Support list.
ACCELERATE YOUR MIGRATION TO SNOWFLAKE
Organizations planning to migrate to Snowflake’s cloud data platform have two goals: they want to quickly and safely migrate their data from legacy on-premises systems, and they want to ensure the quality and overall governance of that data. The first step in this process is converting the physical table structures on the legacy platform to be compliant with Snowflake. Without automation, this is a time-consuming and expensive undertaking.
erwin DM can document your existing legacy data warehouses and data marts automatically by reverse-engineering your DBMS structures and creating a physical data model. You can use this model to create a logical model of your data structures or you can convert the physical model directly to a model that is compatible with Snowflake. erwin DM supports over 20 database management systems natively so you can use the conversion process for most environments. For example, you can auto-document an Oracle database and convert the subsequent data model to Snowflake, using the converted model to generate the relevant DDL for deployment into your Snowflake environment. Data types are mapped within the model to provide for a seamless conversion.
Figure 2 – Wizard to create a new Snowflake-compliant derived model
This erwin DM capability enables an organization to move its legacy model to Snowflake without manually extracting and editing the DDL from the existing data warehouse.
If you already use erwin DM for your current systems, the process is even faster as you don’t need to do the reverse engineering!
A data-driven approach has never been more valuable to addressing the complex yet foundational questions enterprises must answer.
Data now supports decisions few executives thought they’d be making even 90 days ago. It helps business leaders understand fundamental information like their cash flow, and it’s also a guide as their business moves forward in this crisis (or future crises).
Trusted data illuminates what is happening in your business and your business processes. It may even show you where your infrastructure and your critical systems need better support.
Organizations that have good data management, data governance, and data intelligence practices are much better positioned to respond to challenges and thrive in crises. Moving your business critical data to Snowflake’s cloud data platform gives you the agility and flexibility to adapt quickly, responding to new demands faster than with your on-premises legacy platforms. erwin DM helps accelerate that move.