Master of Science in Management & Systems
Data Warehousing and Data Mining

MASY1-GC3510

COURSE DESCRIPTION:

The course addresses the concepts, skills, methodologies, and models of data warehousing. The course addresses proper techniques for designing data warehouses for various business domains, and covers concpets for potential uses of the data warehouse and other data repositories in mining opportunities.

COURSE LEARNING GOALS:

1. Course Objectives:

In today's organization, the data warehouse is the center of the information systems' knowledge repository. Data warehousing supports informational processing by providing a solid platform of integrated, historical data from which to perform enterprise-wide data analysis. This helps improve profit and guide strategic decision making

Data mining is a recent advancement in data analysis. Data mining exploits the knowledge that is held in enterprise data warehouses and other data stores by examining the data to reveal untapped patterns that suggest better ways to improve quality of product, customer satisfaction and retention, and profit potentials

This course will cover the concepts and methodologies of both data warehousing and data mining.

• Data warehousing topics include: modeling data warehouses, concepts of data marts, the star schema and other data models, Fact and Dimension tables, data cubes and multi-dimensional data, data extraction, data transformation, data loads, and metadata.
  
  
• Data mining topics include: concepts and techniques for exploiting untapped business intelligence potentials from real business model data warehouses

       The focus of the course will be on the following topics:

• Techniques for Developing Proper Data Warehouses
• Dtabase design for OLTP, ODS, Data Warehouses and Data Marts
• Dimenstion and Facts tables
• The Star Schema and other Data Warehouse models
• ETL - Data Extract, Translation, and Load
• Retail, inventory and order management data warehousing
• CRM, human resouces, and accounting data warehouses
• Clickstream and web traffic data warehousing
• Data Mining concepts and techniques

2. Student Learning Outcomes:

• Understand the purpose of an OLTP (transactional application/database)
• Understand the purpose of an ODS (operational reporting database)
• Understand the purpose of an DW or DM (data warehouse or data mart)
• Learn proper techniques for data normalization and de-normalization
• Distinguish between various database design models for OLTP, ODS, and DW
• Ability to create well structured dimension tables
• Ability to create well structured fact tables
• Ability to model STAR and Snowflake schemas
• Learn techniques for integrating various DM STAR schemas to create a data warehouse
• Clearly articulate the various steps of an ETL process (extract, transform and load)
• Understand the various issues of data transformation and integration
• Define dimension hierarchies and aggregates facts
• Understand the various techniques for data mining
• Use data mining tools to help segment/classify your data

BOOKS:

Required Reading & Materials -

• The Data Warehouse Lifecycle Toolkit (2nd Edition)
  • Authors - Kimball, Ross, Thornthwaite, Mundy & Becker
  • Publisher - Wiley

• The Data Warehouse Toolkit: The Definitive Guide to Dimensional Modeling (3rd Edition)
  • Authors - Ralph Kimball, Margy Ross
  • Publisher - Wiley

• Teach Yourself SQL in One Hour a Day (5th Edition)
  • Authors - Ryan Stephens and Ron Plew
  • Publisher - Sams

• Building the Data Warehouse (4th edition)
  • Author - William Inmon
  • Publisher - Wiley

GRADE ASSIGNMENT AND EVALUATION

Contributing factors for determining your course grade include:

• Class Attendance & Participation - 10%
• Homework - 20%
• Team Project - 20%
• Midterm Exam - 25%
• Final Exam - 25%

• Class Attendance and Participation: To receive full credit for class attendance and particitation, you must attend all classes since much of the learning occurs during class presentation and discussions. You must also participate, engage, and contribute to class discussion during every class session. Please contact the instructor if you anticipate missing any part of the class. Participation grade will be based on:
  • Involvement in class discussions, dialogue and activities
  • Participation which demonstrates integration of reading, class work, relevance and application.
  • Quality/quantity of providing effective and balanced feedback to Professor or other students.
  • You must ask a question, or respond to professor or other student question (at least once) during every session.
  
  
• Homework: Homeworks must be submitted on time within 1 week after date assigned. Late submission will severely impact your homework grade, or may not be accepted altogether at instructor discretion.
  • All homeworks must be uploaded/posted to NYU Brightspace
  • Professor will not accept homework via email
  • Where appropriate, proper code indentation is a must. If not properly indented I may return it without grading it.
  • Where appropriate, upload/post a screenshot of your output
  
  [Homework Answers]
  
  
• Final team Project: There will be a group/team project. The team project will be a culmination of organizational and presentation skills. It will be the culmination of topics, concepts and competencies learned in this class.
  
  
• Midterm Exam: There will be a midterm exam. The exam will be an open internet style exam. Course notes will not be allowed. The exam will test the student's acquisition of topics, concepts and competencies learned by midterm.
  
  
• Final Exam: There will be a final exam. The exam will be an open internet style exam. Course notes will not be allowed. The exam will test the student's acquisition of topics, concepts and competencies learned in this class. The final exam will not be cumulative. It will only cover topics discussed since the midterm.

Please Note: Professor will not entertain any request for an assignment "redo" or extra credit assignment to improve grade

Statement on Academic Integrity:

New York University is a top level academic institution that takes academic integrity very seriously. All students suspected of violating this policy including cheating and/or plagiarism and/or copying from others or published materials on assignments or exams will be severely penalized for their action.

• All violations of NYU policies on Academic Integrity and Misconduct will be reported to the appropriate academic department chairman for disciplinary action.
• In addition, students suspected of Academic Misconduct will receive a grade of zero ("F" or "0") for the assignment or exam in question.

Statement on Usage of Mobile Devices:

Usage of smartphones is not allowed during class. If you are using a tablet or a laptop to support class learning, these devices must only be used strictly for class purposes. No social media, web surfing or usage of any kind is allowed outside the needs for class consideration.

COURSE OUTLINE:

DATE	SESSION	TOPIC[s] COVERED
[Week 1]	1	Introduction to Data Warehousing Relationship of Data Mining and Data Warehousing What is a Data Warehouse? Data Warehousing ROI DSS - Decision Support Systems Operational vs. Analytical Systems Evolution of DSS and Data Warehousing OLTP - Online Transaction Processing Characteristics of a Data Warehouse What is a Data Mart? Creating a Data Mart Data Comparison Chart OLAP - Online Analytical Processing
	Reading:	Chapter 1 (both DW Toolkit, and Building the DW), Skim thru Glossary (DW Lifecycle Toolkit)
[Week 2]	2	Planning & Building the Data Warehouse Sponsorship and Cost Justification Project Prerequisites Barriers, Challenges and Risks Preparing for Implementation Developing the Data Warehouse SDLC Methodologies - Waterfall vs. RUP Approach Planning & Project Management Analysis Logical & Physical Design Implementation and Deployment Operations
	Reading:	Chapter 1, 2 (The Data Warehouse Lifecycle Toolkit)
[Week 3]	3	Data Warehouse Design Drivers for Multi-Demensional Analysis Limitations of Relational Models The Data Cube What is dimensional modeling? Advantages of Dimensional Models Logical and Physical Design Data Normalization Benefits and Drawbacks of Data Normalization De-Normalizing of Data Characteristics of a Data Warehouse Subject Oriented, Integrated, Time Variant, Non-Volatile The Star Schema
	Reading:	Chapter 6 (The Data Warehouse Lifecycle Toolkit)
[Week 4]	4	Data Warehouse Schemas Dimensions and Dimension Tables Facts and Fact Tables The Star Schema The Snowflake Schema Degenerate and Junk Dimensions The Data Warehouse Bus Architecture Conformed Dimensions and Standard Facts Data Granularity Changing Dimensions
	Reading:	Chapter 6 (The Data Warehouse Lifecycle Toolkit)
[Week 5]	5	Components of a Data Warehouse Source Systems, Staging Area, Presentation, Access Tools Building the Data Matrix The Four Steps Process Multiple Fact Tables in a single Data Mart Chain, Heterogeneous, Transaction/Snapshot & Aggregate Facts Fact and Dimension Table Detail Identifying Source for each Fact & Dimension Mapping from Source to Target
	Reading:	Chapter 7, 4 (The Data Warehouse Lifecycle Toolkit)
[Week 6]	6	The ETL Process Extracting the Data into the Staging Area The Challenge of Extracting from Disparate Platforms Full vs. Incremental Extracts Detecting Changes to Data Transforming the Data Complexity of Data Integration Dealing with Missing & Dirty Data Data Transformation Tasks Loading the Data Timing and Job Control of Data Loads
	Reading:	Chapter 9 (The Data Warehouse Lifecycle Toolkit)
[Week 7]	7	Midterm Exam Covers sessions 1 through 6
[Week 8]	8	Aggregating Data Goals and Risks of Data Aggregation Deciding What to Aggregate Data Sparsity Design Requirement for Aggregates The problem with Aggregates Aggregate Navigators
	Reading:	Chapter 8 p353-357(The Data Warehouse Lifecycle Toolkit)
[Week 9]	9a	Self-Study Selecting the Business Subject Declaring the Grain Choosing the Dimension Identify the Fact Avoiding Null Keys Retail Market Basket Analysis Additive and Semi-Additive Facts The Value Chain Integrated Inventory Model Order Management Data Marts Date and Other Dimension Role Playing Allocation to Lower Level Facts Profit and Loss Data Marts
	Reading:	Chapter 2, 3, 5 (The Data Warehouse Toolkit)
	9b	Self-Study CRM Overview Customer Dimension Demographic Dimension Outriggers Date Dimension Outriggers Large Changing Customer Dimension Mini-Dimensions Commercial Customer Hierarchies Fixed vs. Variable Level Hierarchies General Ledger Accounting OLAP role in G/L and Chart of Accounts Time Stamped Employee Dimensions
	Reading:	Chapter 6, 7, 8 (The Data Warehouse Toolkit)
[Week 10]	10	Clickstream/Web Data Warehouses & Analytics Overview of Web Based Interaction Challenges of Tracking Data Creating Persistent State on the Web Techniques for Tracking States Working with Cookies User Registration Web Server Log Files Online Advertising Online Page Tracking and Analytics User Dimension and Page Hits Facts
	Reading:	Chapter 15 (The Data Warehouse Toolkit)
[Week 11]	11	Data Mining What is Data Mining Good For? Statistics, Artificial Intelligence & Machine Learning Data Mining Examples and Tools Connection between Data Mining and Data Warehousing Retrospective Reporting vs. Predictive Data Mining Applications Data Mining vs. Statistics vs. OLAP Data Mining Statistical Techniques (Sampling, Regression & Decision Trees) Clustering, Segmentation and Nearest Neighbor Techniques Keys to commercial success of Data Mining
	Reading:	Online
[Week 12]	12	Data Mining Techniques - Part I Hands-on Presentation and Lab Classification, Regression, Similarity Matching, Co-occurence Grouping Predictive Modeling Clustering/Segmentation Data Mining and Statistics Terminologies Supervised vs. Unsupervised Tree Induction Entropy and Information Gain The ID3 and C4.5 classifier process Clustering Association and Co-occurrence grouping
	Reading:	Online
[Week 13]	13	Data Mining Techniques - Part II Hands-on Presentation and Lab Classification, Regression, Similarity Matching, Co-occurence Grouping Predictive Modeling Clustering/Segmentation Data Mining and Statistics Terminologies Supervised vs. Unsupervised Tree Induction Entropy and Information Gain The ID3 and C4.5 classifier process Clustering Association and Co-occurrence grouping
	Reading:	Online
[Week 14]	14	Data Mining using Weka Practice excersises
	Reading:	Online
[Week F]	F	Final Exam Covers sessions 8 through 13 Final Project Due