Data Warehouse to Entity Relationship Diagram in Minutes on Magic Data

Intro

A core piece of the Magic Data platform is understanding your entity relationships. When onboarding a data warehouse or other resource onto Magic Data, we deploy a multi agent process to explore your resource efficiently and process that exploration through our workflow to discover entity relationships across all of your data automatically.

There are a few things unique advantages to our approach that will make you smile:

Scale

Magic Data's data discovery process scales to extreme complexity. We can handle THOUSANDS of tables within your data warehouse.

And it's actually fast. You will be shocked at how fast our process can rip through hundreds to thousands of tables.

Semantic Entities and Relationships

Magic Data's entity relationship discovery is semantic in nature. So regardless of strict key relationships, inconsistent naming conventions, or slightly messy data, Magic Data will discover semantic relationships between entities within your warehouse.

Magic Data can even reach inside your data structures and pull out entities that are not a "table" or "collection"; for example, embedded entities in nested noSQL data. Take for example a json structure representing Cars:

Cars Collection:
{
  "car_id": "CAR-238716",
  "make": "Toyota",
  "model": "Camry",
  "year": 2021,
  "vin": "4T1BF1FK7MU123456",
  "color": "Silver",
  "engine": {
    "type": "2.5L I4",
    "horsepower": 203,
    "fuel_type": "Gasoline",
    "transmission": "8-speed automatic",
    "drivetrain": "FWD"
  },
  "features": [
    "Bluetooth",
    "Backup Camera",
    "Adaptive Cruise Control",
    "Lane Departure Warning",
    "Apple CarPlay",
    "Heated Front Seats"
  ],
  "mileage": 38200,
  "registration": {
    "plate_number": "ABC-2047",
    "state": "CA",
    "registration_expiry": "2025-03-31"
  },
  "historical_owners": [
    {
      "name": "James Lee",
      "address": "88 Sunrise Ave, Fremont, CA 94538",
      "phone": "(510) 555-1122",
      "email": "james.lee@example.com",
      "ownership_start": "2021-02-15",
      "ownership_end": "2023-08-01"
    },
    {
      "name": "Maria Thompson",
      "address": "1234 Redwood Dr, San Jose, CA 95129",
      "phone": "(408) 555-2398",
      "email": "maria.thompson@example.com",
      "ownership_start": "2023-08-01",
      "ownership_end": null
    }
  ],
  "insurance": {
    "provider": "State Farm",
    "policy_number": "SF-CA-42983712",
    "expiration_date": "2025-06-01"
  }
}

Magic data may discover and pull our "Owners" as a separate entity.

Step By Step

Let's breakdown exactly how step by step:

1. Navigate to Your Magic Data Resource and click "Diagram" on the left sidebar.

2. Often an ERD may already be present from our Data Discovery process. If it is not, click "Generate Diagram".

3. Your ERD will start to generate and you will see entity cards populate live as Magic Data's process progresses.

4. As the process continues the diagram will also update live with new discovered relationships.

5. Once finished the ERD will load to showing you the entire diagram unfiltered. If you have a complex data schema this can be pretty knarly to look at and consume. But don't worry! Magic Data has automatically discovered "Topics" from your data.

6. Simply select the "Filter by..." dropdown and select one of the discovered topics. The ERD will automatically filter down to a digestible subset of entity relationships.

These relationships and lots of other context around your data power our AI experiences like Brainstorming and Pipelines to deliver you super relevant and accurate suggestions, queries, and code.

Coming very soon, you will also be able to generate Entity Relationship Diagrams across multiple resources and data types. Stay tuned!