Building a Semantic Layer of your Data Platform

BUILDING A SEMANTIC LAYER OF YOUR DATA PLATFORM
DATA SUMMIT WORKSHOP
MAY 2024
Joe Hilger, COO and Principal (jhilger@enterprise-knowledge.com)
Sara Nash, Principal Consultant (snash@enterprise-knowledge.com)

What You Will Learn Today
1. The business case for semantic layers
2. Deﬁnitions of the semantic layer and
its components
3. Real world applications for semantic
layers
4. How to prototype semantic layer use
cases and models
@EKCONSULTING

Introductions
Joe Hilger
COO and Principal Consultant
Enterprise Knowledge
Sara Nash
Principal Consultant
Enterprise Knowledge
@EKCONSULTING

ENTERPRISE KNOWLEDGE
Modern Data
Platforms…
Provide Answers
and Context

Modern Data
Platforms…
Enable Discovery

Modern Data
Platforms…
Increase Access
and Trust in Data

Modern Data
Platforms…
Break Down Silos

How are these experiences enabled?
● Content (knowledge, data, and
information) is managed and
accessible
● Data is connected across
repositories, databases, and
applications
● Context and meaning is
embedded with source data,
making common understanding
of data machine-readable

What is a Semantic Layer?
“A semantic layer is a standardized
framework that organizes and
abstracts organizational
knowledge and data (structured,
unstructured, semi-structured) and
serves as a connector for all
organizational knowledge assets.”

The Semantic Layer – Your Data’s “Rosetta Stone”
A semantic layer encodes
information based on context and
business logic via a broader
understanding of what those data
values mean to an organization.
The components of the semantic
layer (Knowledge Graph, Taxonomy,
Metadata Service, etc.) work together
to translate data across systems,
enabling comprehensive analytics,
reporting, and search applications.

INCONSISTENT
METADATA
NON-INTUITIVE USER
INTERACTIONS
INEFFICIENT
DATA ANALYSIS
PROCESSES
What Problems Does a Semantic Layer Solve?
POOR DATA QUALITY &
GOVERNANCE
VENDOR LOCK
AI HALLUCINATIONS

Standardizing Metadata
Uniﬁed language used to
talk about information
across the organization
Connecting Information
Structured and
unstructured data and
content are both human
and machine-readable.
Utilizing AI & LLMs
AI-supported applications
like LLMs perform with
higher ﬁdelity and avoid
hallucinations
Reporting and Analyzing
Search results display
information that exists in
multiple locations and
formats – producing a
one-stop data hub
What Does a Semantic Layer Enable?

Foundations of
Semantic Layers:
Components

Describe
Standardize
Catalog
Metadata
Product Name: iPhone 7
Product Type: Electronics
Product Dataset: prod_data
Date Created: 02-20-2024
What data do we have on our products?
How can we quickly identify and
retrieve product images for our
marketing campaigns?
Which datasets are capturing
proﬁtability from product Sales?

Define
Align
Index
Business Glossary
Term Type Def
Profitability INT The degree to which
a product sale yields
financial gain
How do we define profitability for a
product area?
What is the meaning of "customer
journey" in our company, and how is it
mapped?
What is the agreed-upon definition of
key performance indicator (KPI) in our
organization?

Categorize
Organize
Architect
Taxonomy |
Information Architecture
Electronics
TVs
Cell Phones
Which products fall under this
Product category?
How can we better organize our product
content to enhance customer service
efﬁciency?
How can we improve the organization of
our online product website for a smoother
customer browsing experience?

Relate
Map
Contextualize
Ontology
Product
Region
Customer
Sale
HasCustomer
HasSale
HasLocation
LivesIn
How can we ensure a uniﬁed
understanding of product data across
departments?
What are the sequential steps in our
supply chain process, and how are they
interrelated?
How do Online Sales relate to Regions?

Connect
Analyze
Infer
Knowledge Graph
iPhone7
NorthEast
Joebot
$$$$
What is the most purchased product
across customers in a given region?
What is our revenue from this product
in Store A in 2023?
How does a disruption in one part of our
supply chain affect other components?

Foundations of
Semantic Layers:
Enabling Technology

Semantic
Layer
APIs ETL
Application
Search
Research &
Analytics
Recommendations
& Chatbots
Admin &
Governance
Knowledge Graph
Metadata Service
Business Glossary
Taxonomy
RDF Ontology / LPG Schema
Content
Management System
Data Lake / Data
Warehouse
Subscriptions External Sources
Data Sources
Presentation
Layer
APIs
High Level Architecture

Semantic
Layer
APIs ETL
Application
Search
Research &
Analytics
Recommendations
& Chatbots
Admin &
Governance
Knowledge Graph
Metadata Service
Business Glossary
Taxonomy
RDF Ontology / LPG Schema
Content
Management System
Data Lake / Data
Warehouse
Subscriptions External Sources
Data Sources
Presentation
Layer
APIs
Semantic Layers and AI
Fact-based natural language query
Relationship-based recommendations
Advanced analytics
Extract entities and relationships
Summarize unstructured data
Recognize patterns and curate facts

APPLICATIONS
What products
support the
Western Blot
process?
Natural Language
Search
Categorization and
Classiﬁcation
Advanced Analytics
Recommender System
Entity Recognition
Clustering and Similarity
Algorithms
Link Detection
Categorization
Semantic Layers & AI

What Is NOT a Semantic Layer?
Not Just for
Data
Not A Single
Product
Not Just a
Graph
Not All of Your
Content
Not an Automated/
AI-Generated
Solution

Why Our Clients are
Investing in
Semantic Layer

Top Enterprise
Use Cases

Reporting and Insights
Use Case 1

The Challenge
The Solution
● The solution provided a 360° view of data
used for analysis across all systems
● The knowledge graph enabled
automated creation of reports for
regulatory ﬁlings and analysis of lots of
different process details in depth.
The Results
An organization’s team of scientists needed
to quickly ﬁnd and get insights about drug
development processes. However, their
insights were limited to what the scientists
could manually aggregate from siloed legacy
systems with different naming conventions.
● Develop a comprehensive ontology to
model the drug development process
and standardized nomenclature.
● Build a knowledge graph that
aggregated and normalized disparate
data from legacy systems.
Use Case 1: Global Biotechnology Company

What is the dataset we used for
this Experiment?
How do we deﬁne a
Product Run?
What Materials did we use
to create this Product?
What was the Average Cell
Viability for this Product?
What are the Measurements
used in this Experiment?
What are our Products?
Which Business Unit led this
Project?

Use Case 2
Enterprise Data Architecture

The Challenge
The Solution
● The semantic model allowed consistent
categorization of risk data across 15+
systems.
● Deduplicated risk descriptions by 40%
using AI-based approaches.
● The centralized and standardized
structure for data makes it easier to
connect and handle data on a large scale.
The Results
A multinational financial services firm
needed strategy support in implementing a
data modernization program to solve their
organizations problems with risk
identification and the user experience
around reporting.
● Build central taxonomy network to
provide consistent categorization of risk
data across systems.
● Develop semantic model for data
connectivity to drive knowledge panels,
recommendation engines, and semantic
search.
● Leverage LLMs to standardize risk
category descriptions.
Use Case 2: Financial Services Firm

Assessment
Risks
Process
Organization
Business
Units
Legal
Entities
Regions
Products
/Services
Controls
Incidents
Compliance
Rules Policies
Complaints
Regulatory
Matters
Monitoring &
Testing
Priority Risk
Programs
Identify
Mitigate
Governance, Procedures, and Training
Data, Metrics, and Reporting

Key Takeaways
● Enterprise Data-warehouse to
drive data aggregation for
reporting and analytics
● Shared Services for
authentication, entitlements,
and logging
● Semantic Services and
Storage to support graph use
cases such as semantic search,
personalization, data quality
and connectivity
● Data Provider Model for
semantic data exchange
between business applications
and semantic layer
Enterprise Architecture

What is the Rule relevant to this
Risk?
What is a Business Unit vs.
a Product?
What Controls are
mitigating this Risk?
What are the Risk Ratings for
this Assessment?
What are our Products?
How should we describe our
Controls?

Use Case 3
Data
Modernization &
User Journeys

The Challenge
The Solution
● Enabled creation of a semantic layer to
support organization-wide data
discoverability
● Provided a deep understanding of the
targeted improvements to data
accessibility that would enhance user
satisfaction, engagement, and efficiency
● Identified short, medium, and long term
goals for reaching the defined target state
The Results
A global retail chain needed support in
confronting challenges with finding,
connecting to, and understanding existing
data related to tracking the health of their
stores.
● Create business glossary and taxonomy
to tag business assets for increased
findability and understanding
● Create journey maps describing current
state and target state persona
experiences.
● Design detailed roadmap outlining key
steps for improving organizational
semantic maturity
Use Case 3: Global Retail Chain

End State
Independent
understanding the
business meaning
Minutes to see results
Data democratization
and ownership
Current State
Heavy reliance on the IT
data analytics team
Weeks of discussion to
get results
High volume of data
USER STORY: As an Executive…want to understand the business meaning of data …so that I can
make a quick and informed decision about what business strategy my team should use.
Step 1
Executive reads a data
analytics report.
Step 2
Executive doesn’t
understand what the
data is trying to
represent.
Step 3
Executive approaches
data analytics team
asking for explanation.
Step 4
Executive waits for data
analytics team to
consult about the
meaning of data
included in the report.
Step 5
Executive receives an
explanation from the
data analytics team
about data meaning.
Step 6
Executive can make a
strategic decision based
on the meaning of the
data in the report.
Step 1
Executive reads a data
analytics report.
Step 2
Executive looks up the
report in a UI connected
to the Semantic Layer.
Step 3
Executive finds business
definitions of the data
in the report.
Step 4
Executive can make a
decision based on the
meaning of the data in the
report.
User Journey Map

End State
Minutes to find and
understand tables
Self-directed discovery
to vet data
Current State
Weeks to find and
understand tables
Meetings to determine
if the tables can be used
USER STORY: As a Data Analytics Team Member…I want to find and understand relevant data
tables…so that I can easily reuse existing data for a project.
Step 1
DA Team Member is
assigned a project that
requires data analytics.
Step 2
DA Team Member
spends hours to days
looking through familiar
data repositories for
data tables that could
be useful.
Step 3
DA Team Member is
unable to find what
they need, and asks a
SME for instructions on
where to look.
Step 4
DA Team Member
spends weeks looking
through recommended
data repositories, and
finds tables that seem
relevant.
Step 5
DA Team Member does
not understand the
table’s columns, and
asks a SME for
definitions and
calculation rules.
Step 6
DA Team Member is
able to determine if the
existing data table is
suitable for their
project.
Step 1
DA Team Member is
assigned a project that
requires data analytics.
Step 2
DA Team Member
searches for data tables
in a UI connected to the
Semantic Layer.
Step 3
DA Team Member finds
data tables with
metadata and column
definitions.
Step 4
DA Team Member is able to
determine if the existing
data table is suitable for
their project.
User Journey Map

Where do I ﬁnd the latest
Performance Reports?
What is the deﬁnition of a
Data Product?
Who are the domain
Experts?
What caused a Store
Outage?
Which Metrics does a given
Store need to track?
What Topic do I use to tag my
Data?

Use Case 4
Data Consistency
and Usability

The Challenge
The Solution
● The contribution model enabled 10+
departments to contribute and lead
semantic model development and
governance.
● Drove the implementation of data
standards through the publication of the
enterprise ontology.
● Increased data awareness, consistent
understanding, and alignment for users
across departments and technologies.
The Results
A large ﬁnancial corporation had a lack of
alignment around the meaning, format, and
intent of data elements across organizational
divisions, reducing the ability of data
producers and consumers to ﬁnd, use, and
and trust data.
● Develop an enterprise ontology to
standardize data from multiple systems
and migrate from an existing physical
data model.
● Implement a federated ontology
governance and contribution model.
● Leverage standardized ontology
concepts throughout the data lifecycle.
Use Case 4: Financial Services Company

stateCode
CountrySubdivisionCode
Subdivision_Code
state_postal_code
USPS_Code
STATE
US AL
US AK
US AZ
… …
countryCode
IsoCountryCode
alpha_code_country
geo.country_code2
iso_3166_country_code_2
CTRY
@EKCONSULTING

Where can I access Compliant
Data Models?
How do we deﬁne
Credit Risk across
departments?
What Legacy Systems
were integrated?
How is risk data Classiﬁed? How is Data structured for
Regulatory Compliance?

Activity 1: Use Case Deﬁnition

Establishing Your Prototype Use Case:
Sample
Semantic Layer
Use Case
User Story
As an online learner, I want to see course
recommendations when I get an assessment
question incorrect, so that I can upskill in that
area of weakness.
Speciﬁcations Source Data Assets
● Course Library
● Question and
Assessment Library
● Healthcare Topic
Taxonomy
Key Knowledge Concepts
● Course
● Question
● Topic
● Healthcare
Setting
@EKCONSULTING

Activity Instructions
1. As a group, complete 3-5 use case
worksheets (15 minutes).
You can select a use case from
your experience or make up
examples.
1. Each group will present their a few
of the use cases we will select
favorites to model.
2. In our next activity, you will be
modeling your selected use case.
@EKCONSULTING

Activity 2: Modeling

How to Model Your Knowledge Graph
@EKCONSULTING
▪ Entity – A unique type of thing that you want to deﬁne and relate in your model.
▪ Attributes – or properties. Aspects, features, characteristics, descriptors, or
parameters that describe and differentiate instances of an entity.
▪ Relationships – The types of connections that can be deﬁned between entities.
Person
Works For
Company
Address
Founding
Date
Annual
Revenue
Sells

Activity Instructions
Define
Entities
Write the Entity names on
hexagons.
Define
Relationships
Using string and tape,
define Relationships by
connecting two Entities.
Define
Attributes
Write the Attribute names
on sticky notes and stick to
the relevant Entity.
With your group, build out the semantic layer model for your use case using the materials provided.
Definition of Done: The data concepts required to support your use case are incorporated.
Customer
Custome
r
Product
Customer
Name Age
Buys
Customer
Name Age
Product
Size Color
Buys
Product
Family
Size Color
p
a
r
t
O
f
@EKCONSULTING

JHILGER@ENTERPRISE-KNOWLEDGE.COM
Joe Hilger
WWW.LINKEDIN.COM/IN/JOSEPH-HILGER/
SNASH@ENTERPRISE-KNOWLEDGE.COM
Sara Nash
WWW.LINKEDIN.COM/IN/SARA-G-NASH
Contact Us
@EKCONSULTING

Building a Semantic Layer of your Data Platform

More Related Content

What's hot

Similar to Building a Semantic Layer of your Data Platform

More from Enterprise Knowledge

Recently uploaded

Building a Semantic Layer of your Data Platform