4. 基本概念—数据仓库Data warehouse is a subject oriented, integrated,non-volatile and time variant collection of data in support of management’s decision —— [Inmon,1996].
Data warehouse is a set of methods, techniques,and tools that may be leveraged together to produce a vehicle that delivers data to end-users on an integrated platform —— [Ladley,1997].
Data warehouse is a process of crating, maintaining,and using a decision-support infrastructure —— [Appleton,1995][Haley,1997][Gardner 1998].
11. 体系结构 [Pieter ,1998]Source
DatabasesData Extraction,
Transformation, loadWarehouse
Admin.
ToolsExtract,
Transform
and LoadData
Modeling
ToolCentralMetadataArchitected
Data MartsData Access
and AnalysisEnd-User
DW ToolsCentral Data
WarehouseCentral
Data
WarehouseMid-
TierMid-
TierData
MartData
MartLocal
MetadataLocal
MetadataLocal
MetadataMetadata
ExchangeMDBData
Cleansing
ToolRelationalAppl. PackageLegacyExternalRDBMSRDBMS
12. 带ODS的体系结构Source
DatabasesHub - Data Extraction,
Transformation, loadWarehouse
Admin.
ToolsExtract,
Transform
and LoadData
Modeling
ToolCentralMetadataArchitected
Data MartsData Access
and AnalysisCentral Data Ware-
house and ODSCentral
Data
WarehouseMid-
TierRDBMSData
MartMid-
TierRDBMSData
MartLocal
MetadataLocal
MetadataLocal
MetadataMetadata
ExchangeODSOLTP
ToolsData
Cleansing
ToolRelationalAppl. PackageLegacyExternalMDBEnd-User
DW Tools
13. 现实环境—异质性[Douglas Hackney ,2001]Custom
Marketing
Data
WarehousePackaged
Oracle
Financial
Data
WarehousePackaged
I2 Supply Chain
Non- Architected
Data MartSubset
Data MartsOracle Financialsi2 Supply ChainSiebel CRM3rd Partye-Commerce
14. 联合型数据仓库/数据集市体系结构Real Time
ODSFederated
Financial
Data
WarehouseSubset
Data MartsCommon
Staging
AreaOracle Financialsi2 Supply ChainSiebel CRM3rd PartyFederated
Packaged
I2 Supply
Chain
Data MartsAnalytical
Applicationse-CommerceReal Time
Data Mining
and AnalyticsReal Time
Segmentation,
Classification,
Qualification,
Offerings, etc.Federated
Marketing
Data
Warehouse
15. ETL tools & DW templatesData profiling & reengineering toolsDemand-driven data acquisition & analysisMetadata InterchangeFederated data warehouse and data mart systemsDecision engine models, rules and metricsOLAP & data mining tools,
Analysis templatesAnalytic application development tools & componentsAnalytic applicationsFront- and back-office OLTPe-Business systemsExternal information providersCRM Analytics & ReportingSupply Chain Analytics & ReportingEKP - Enterprise Knowledge Management PortalEPM Analytics & ReportingBusiness information & recommendationsInformed decisions & actionsFinancial Analytics & ReportingHR Analytics & Reporting闭环的联合型BI体系结构
25. Top-down ApproachBuild Enterprise data warehouse
Common central data model
Data re-engineering performed once
Minimize redundancy and inconsistency
Detailed and history data; global data discovery
Build datamarts from the Enterprise Data Warehouse (EDW)
Subset of EDW relevant to department
Mostly summarized data
Direct dependency on EDW data availabilityLocal Data MartExternal DataLocal Data MartOperational DataEnterprise Warehouse
27. 数据仓库建模 — 星型模式Example of Star SchemaDate
Month
YearDateCustId
CustName
CustCity
CustCountryCustSales Fact Table Date Product Store Customer unit_sales dollar_sales Yen_salesMeasurementsProductNo
ProdName
ProdDesc
Category
QOHProductStoreID
City
State
Country
RegionStore
28. 数据仓库建模 — 雪片模式Date
MonthDateCustId
CustName
CustCity
CustCountryCustSales Fact Table Date Product Store Customer unit_sales dollar_sales Yen_salesMeasurementsProductNo
ProdName
ProdDesc
Category
QOHProductMonth
YearMonthYearYearCity
StateCityCountry
RegionCountryState
CountryStateStoreID
CityStoreExample of Snowflake Schema
40. 数据仓库应用 — DW用户数的调查“DW系统的用户
在100-500以内或以上
是未来一段时期内
的主要部分“DW用户的调查
最近一年
Meta Group Survey
调查对象：3000+ 用户或意向用户
41. DW数据规模的调查DW规模的调查
最近一年
Meta Group Survey
调查对象：3000+ 用户或意向用户
42. How Much?$3-6m for mid-size company, less if smaller, more if larger
$10m+ for large organizations, large data sets
10-50+% annual maintenance costs
33% Hardware / 33% Software / 33% Services
43. How Long?2-4 years for 80/20 of full system for mid-size company
6-12 months for initial iteration
3-6 months for subsequent iterations
44. How Risky?For EDW Projects, 20% (Meta) to 70% (OTR, DWN) fail
High failure rate for non-business driven initiatives
Very few systems meet the expectations of the business
Failure not due to technology, due to “soft” issues
Massive upside to successful projects (100% - 2000+% ROI)
99% politics - 1% technology
45. 参考文献Inmon,W.H.,” Building the Data Warehouse” ,Johm Wiley and Sons,1996.
Ladley,John,”Operational Data Stores:Building an Effective Strategy”,Data warehouse:Pratical Advice form the Experts,Prentice Hall,Englewood Cliffs,NJ,1997.
Gardmer,Stephen R., “Building the Data warehouse”,Communication of ACM, September 1998, Volume 41, Numver 9, 52-60.
Douglas Hackney , Http:// www.egltd.com, DW101: A Practical Overview, 2001
Pieter R. Mimno, “The Big Picture - How Brio Competes in the Data Warehousing Market”, Presentation to Brio Technology - August 4, 1998.
Alex Berson, Stephen Smith, Kurt Therling, “Building Data Mining Application for CRM”, McGraw-Hill, 1999
Martin Stardt, Anca Vaduva, Thomas Vetterli, “The Role of Meta for Data Warehouse”, 2000
W.H.Inmon, Ken Rudin, Christopher K. Buss, Ryan Sousa, “Data Warehouse Performance”, John Wiley & Sons , 1999
50. Discovery of previously unknown relationships, trends, anomalies, etc.
Powerful competitive weapon
Automation of repetitive analysis
Predictive capabilitiesData Mining Upsides
51. Knowledge discovery technology immature
Long learning and tuning cycles for some technologies
“Black box” technology minimizes confidence
VLDB (Very Large Data Base) requirementsData Mining Downsides
52. Data Mining UsesDiscover anomalies, outliers and exceptions in process data
Discover behavior and predict outcomes of customer relationships
Churn management
Target marketing (market of one)
Promotion management
Fraud detection
Pattern ID & matching (dark programs, science)
53. Data Mining Industry and ApplicationsFrom research prototypes to data mining products, languages, and standards
IBM Intelligent Miner, SAS Enterprise Miner, SGI MineSet, Clementine, MS/SQLServer 2000, DBMiner, BlueMartini, MineIt, DigiMine, etc.
A few data mining languages and standards (esp. MS OLEDB for Data Mining).
Application achievements in many domains
Market analysis, trend analysis, fraud detection, outlier analysis, Web mining, etc.
54. Data Mining CostsDesktop tools: $500 and up (MSFT coming at low price point)
Server / MF based: $20,000 to $700,000+
Must also add cost of extensive consulting for high end tools
Don’t forget long training and learning curve time
Ongoing process, not task automation software
57. 历史回顾1989 IJCAI Workshop on Knowledge Discovery in Databases
Knowledge Discovery in Databases (G. Piatetsky-Shapiro and W. Frawley, 1991)
1991-1994 Workshops on Knowledge Discovery in Databases
Advances in Knowledge Discovery and Data Mining (U. Fayyad, G. Piatetsky-Shapiro, P. Smyth, and R. Uthurusamy, 1996)
1995-1998 International Conferences on Knowledge Discovery in Databases and Data Mining (KDD’95-98)
Journal of Data Mining and Knowledge Discovery (1997)
1998 ACM SIGKDD, SIGKDD’1999-2001 conferences, and SIGKDD Explorations
More conferences on data mining
PAKDD, PKDD, SIAM-Data Mining, (IEEE) ICDM, DaWaK, SPIE-DM, etc.
58. Data Mining: Confluence of Multiple Disciplines Data MiningDatabase
TechnologyStatisticsOther
DisciplinesInformation
ScienceMachine
Learning (AI)Visualization
59. A Multi-Dimensional View of Data MiningDatabases to be mined
Relational, transactional, object-relational, active, spatial, time-series, text, multi-media, heterogeneous, legacy, WWW, etc.
Knowledge to be mined
Characterization, discrimination, association, classification, clustering, trend, deviation and outlier analysis, etc.
Techniques utilized
Database-oriented, data warehouse (OLAP), machine learning, statistics, visualization, neural network, etc.
Applications adapted
Retail, telecommunication, banking, fraud analysis, DNA mining, stock market analysis, Web mining, Weblog analysis, etc.
60. Research Progress in the Last DecadeMulti-dimensional data analysis: Data warehouse and OLAP (on-line analytical processing)
Association, correlation, and causality analysis
Classification: scalability and new approaches
Clustering and outlier analysis
Sequential patterns and time-series analysis
Similarity analysis: curves, trends, images, texts, etc.
Text mining, Web mining and Weblog analysis
Spatial, multimedia, scientific data analysis
Data preprocessing and database compression
Data visualization and visual data mining
Many others, e.g., collaborative filtering
61. Research Directions — [Han J. W. , 2001]Web mining
Towards integrated data mining environments and tools
“Vertical” (or application-specific) data mining
Invisible data mining
Towards intelligent, efficient, and scalable data mining methods
62. Towards Integrated Data Mining Environments and ToolsOLAP Mining: Integration of Data Warehousing and Data Mining
Querying and Mining: An Integrated Information Analysis Environment
Basic Mining Operations and Mining Query Optimization
“Vertical” (or application-specific) data mining
Invisible data mining
63. Querying and Mining: An Integrated Information Analysis EnvironmentData mining as a component of DBMS, data warehouse, or Web information system
Integrated information processing environment
MS/SQLServer-2000 (Analysis service)
IBM IntelligentMiner on DB2
SAS EnterpriseMiner: data warehousing + mining
Query-based mining
Querying database/DW/Web knowledge
Efficiency and flexibility: preprocessing, on-line processing, optimization, integration, etc.
64. “Vertical” Data MiningGeneric data mining tools? —Too simple to match domain-specific, sophisticated applications
Expert knowledge and business logic represent many years of work in their own fields!
Data mining + business logic + domain experts
A multi-dimensional view of data miners
Complexity of data: Web, sequence, spatial, multimedia, …
Complexity of domains: DNA, astronomy, market, telecom, …
Domain-specific data mining tools
Provide concrete, killer solution to specific problems
Feedback to build more powerful tools
65. Invisible Data MiningBuild mining functions into daily information services
Web search engine (link analysis, authoritative pages, user profiles)—adaptive web sites, etc.
Improvement of query processing: history + data
Making service smart and efficient
Benefits from/to data mining research
Data mining research has produced many scalable, efficient, novel mining solutions
Applications feed new challenge problems to research
66. Towards Intelligent Tools for Data MiningIntegration paves the way to intelligent mining
Smart interface brings intelligence
Easy to use, understand and manipulate
One picture may worth 1,000 words
Visual and audio data mining
Human-Centered Data Mining
Towards self-tuning, self-managing, self-triggering data mining
67. Integrated Mining: A Booster for Intelligent MiningIntegration paves the way to intelligent mining
Data mining integrates with DBMS, DW, WebDB, etc
Integration inherits the power of up-to-date information technology: querying, MD analysis, similarity search, etc.
Mining can be viewed as querying database knowledge
Integration leads to standard interface/language, function/process standardization, utility, and reachability
Efficiency and scalability bring intelligent mining to reality
68. 数据挖掘与标准化进程CRISP—DM
过程标准化（CRoss-Industry Standard Process for Data Mining）
XML
与数据预处理相结合
SOAP（Simple Object Access Protocol ）
数据库与系统互操作的标准
PMML
预言模型交换标准
OLE DB For Data Mining
数据挖掘系统基于API的接口
