By using a data warehouse you are in effect providing a common data model for all data that is of interest without having to be concerned about the origin of the source. This ability makes it far easier to compile a report and analyze all available information than it would be if you had to retrieve information, invoices, ledgers, orders and so on from a multitude of data models.
Before the data is loaded into the database it has to be cleansed, in effect this involves identifying and resolving all the dirty/inconsistent data. By doing this before the data is loaded into the database the actions of reporting and analyzing data are greatly simplified.
Once it has been loaded into the data warehouse the information is under the control of the user. This allows for the information in the warehouse to be stored safely for extended periods of time, even if the source system data is purged over time,
Another great advantage of using a data warehouse is that they are separate from any operational systems. This means that they can perform retrieval of data operation without interfering with any operational requirement of the system.
Data warehouses is also able to work in conjunction with, and by doing so enhance the value of, operational business applications, the most notable of which being customer relationship management (CRM) systems.
Data warehouses are generally used to facilitate applications associated within the decision support system. These applications will deal with reports concerning trends, for example what was item had the best sales in what area over the last year, exception reports, and reports that show actual performance versus goals.
These days a company lives and dies on how fast it can store, retrieve and analyze information. As such, database administration (DBA) has become one of the most important aspects of a corporation's MIS department. Because of this, a fast growing occupation inside the department is the Data Warehouse Manager.
The DB manager is the person responsible for the loading, storing, maintenance and extraction of all files stored within a business intelligence system. People in this field have done their share of time in the working world, and it's becoming quite common for them to not only have a few years experience under their belts, but also have gotten an MS in some form of Computer Science, with a strong emphasis on DBA.
Anyone interested in pursuing this line of work should be of a meticulous nature. They should also possess strong analytical and logical skills. Good communications capability is also essential as part of the job is explaining how the warehouse system works to those who aren't as adept at storing and retrieving data.
There are many good on campus and online colleges that offer good undergraduate programs in Computer Science and Information Technology. As businesses are the primary employers of warehouse managers, taking some basic courses in the field is oft times beneficial. Once past the basic core a student should try to put in some courses in on basic database construction and architecture.
Upon achieving a Bachelor's, it's time to specialize in Database Architecture and Network information systems. Another Master's level degree that is quite popular is an MBA with an MIS core. At the same time, it should not be too difficult to find a solid job as a database analyst or assistant DBA position. With some time at the company it should be more than willing to help pay the tuition for you sticking around.
Upon achieving either an MS or MBA, there will be one last step in the education process, and that's attending the last few courses or seminars needed to be certified as a Database Warehouse Analyst or Manager. Again, most companies include continued education benefits as part of their employment package. They really have to as this is a position where professionals have to constantly go back for the latest updates and innovations in the field.
Salaries are commensurate with this level of education. According to a recently published report inside the industry, a Warehouse Analyst earns $77,000 to $106,000 a year, depending on industry, location and experience. A Warehouse Manager earns even more, from $92,000 to $125,000 a year. Both jobs are expected to see their salaries grow over 4% in the next year alone. Those with knowledge of IBM DB2, SQL and Oracle are at a premium, and can earn an extra 10% for their knowledge.
Make Informed Decisions With Data Warehousing and Business Intelligence
0 comments Posted by Ram at 2:52 AMUse of business intelligence solutions and software will help you collate various data from various sections of your business so being able to make more informed decisions can be carried out a lot easier.
Although you may find initially when it comes to setting up data warehousing and business intelligence systems is a challenge. It is time and money spent that you will have invested in your business wisely as it can help to impact positively on how your business does in relation to your competitors.
Today there are many different types of data warehousing systems that a business can employ. These provide an effective all in one solution to gathering business intelligence and then allowing you to analyze it.
So just what are the benefits to be thinking about installing data warehousing and business intelligence systems for your business? Below we take a look at a just a number of these benefits.
Benefit 1 - These systems actually allow you to identify and resolve any inconsistencies in the data being collated before it is loaded. As a result it makes the whole process of reporting and analysing the business intelligence gathered much simpler.
Benefit 2 - Another benefit of using such systems is that the information being collected is controlled by the people using the system. Even so the system can still be purged over time and any information that has been collected can be safely and easily stored for a lot longer.
Benefit 3 - As this differs from other operational systems you will find that it allows the users to retrieve data without it causing the operating system to slow down.
Benefit 4 - Businesses that choose to install any kind of data warehousing and business intelligence gathering systems will find that they can improve the relationship not only between the various departments, but also their customers. Such improvements will then ensure that they find it easy to identify the needs of their customers and can work on ways to make sure that this will be met.
Benefit 5 - Through the use of such systems a business will be able to make sure that support system applications like execution and trend reports are functioning correctly. So the production of the reports that they then use to analyse their businesses performance are more accurate.
Benefit 6 - As mentioned above the use of data warehousing and business intelligence gathering systems can help a business to be more competitive. This is because it allows them to identify areas of the business where improvements can be made much more quickly.
I have to admit the term "strategic decision making" can conjure up a lot of different meanings to a lot of people. And the first one that springs to mind, but probably not of the decision makers themselves, is that it is one that involves spending a lot of money hiring a lot of people who are going to cause a lot of pain to a lot of people until the next strategic decision is made.
Now you must appreciate that strategic decision making is not one of the primary uses of data warehouses, thankfully because strategic decision making is not done that often. Most data warehouses are used primarily for post decision monitoring of the effects of these decisions. However, some data warehouses do get involved in strategic decision making and are usually very effective.
So how would you use use a data warehouse in a decision making exercise. Before you start one thing to bear in mind is that the life-span of systems for strategic decision making tend to be relatively short and the creation of "special" databases, modelling and formal reporting are very time consuming tasks.
The days spent using these systems can be counted on one hand, though the payoff can be better than those reporting system that have been used for years.
The time-scale you are given to produce the system can be anything from a long afternoon to several weeks. And with no time for formal interviews you become the business analyst. Requirements are usually gleaned from "business" meeting minutes and are usually ambiguous. You will be required to aggregate data differently, and combine data that never been combined before.
You are doing this so the decision makers can see a point of view that is not the common view of the business. Basically to see the business in a different perspective.
Much of the use of data warehousing for making decisions ultimately involves the use of user maintained spreadsheets. Data cleanliness is much less of a concern in strategic decision making. Analysis is often done with highly summarized data and the need for speed lowers the need for extremely clean data.
The information generated by the data warehouse has to be understood by people who do not have direct access to the data warehouse. Most users will want to communicate the information in printed reports created using a word processor, presentation tool, spreadsheet, or generated directly from the database.
Do not try to design your warehouse for every contingency that could occur in a strategic decision making exercise. You cannot possibly foresee everything that will be needed in these exercises.
Data warehousing software's have a great demand in the market because of the ease that they offer to an organization for handling various projects. There are many data warehousing consultants present in the market, who offer customized solutions to clients. But what is the scope of being a data-warehousing consultant? Read the article.
The data warehousing market has many warehousing consultants who provide resources and services to meet the requirements of warehouse owners. These consultants provide totally integrated solutions that will help business owners in handling the company and database more efficiently.
The data warehousing vendors offer tools, technology and methodologies to help you in constructing, managing, using and maintaining software & hardware that are used in a data warehouse. They understand the business of customers and provide them with services that meet and even exceed their expectations.
The daily duties of a data-warehousing consultant are never the same. They design data extraction, transform and load routines. Using a programming language, script, tool or the combination of all the three helps in developing all this. Here are some duties of the consultant:
o A data warehouse consultant creates documents.
o Interfaces with other team members and the supervisors.
o Conduct different tests of data warehouse batches.
o Talk to users, business analysts, technical coordinators, server administrators, database administrators and many other people working on the system.
Do You Want to be a Data Warehouse Consultant?
The must-have technical skills a data warehouse consultant should have include a thorough knowledge of databases that are being used, the operating systems which are used, script that would be required to write code and expertise in a programming language. Another must have skill is the knowledge of data warehouse tool, which the organization would be using. If you have enough information of the tool, it will lead to a seamless merging of one technology with another to finish the ETL tasks. Some of the common tools are DOS batch script, UNIX shell scripts, DataStage, advanced SQL and Informatica.
Apart from this, one should also have keen observations skills to find out fine details or any mismatches in the numbers. Those with an ability to do analysis are also fit for the job of a warehousing consultant. However, there will be separate analysts for creating technical specifications and mapping all the documents.
Those new into the field of data warehousing will love the excitement of working on projects and in facing the challenges that are crucial for the growth of an organization. Most of the data warehousing projects will be fast paced, so you need to invest a lot of time and manage things properly. Handling a data warehousing project means working on multiple technologies and teams. A data-warehousing consultant gets good exposure and there is a lot of scope for newcomers in this field.
Data warehousing in many ways has become an industry buzzword that seems to promise much and deliver little. A good part of this is due to the often lofty and nebulous benefits that data warehousing vendors tout. What most businesses really want to understand is can data warehousing help them and if so, in what concrete way.
Offload Resource Intensive Processing
One of the ways that data warehousing can help your business is by offloading the resource intensive processes of generating reports onto database systems that are not also responsible for real time transaction processing. I have seen many web applications completely bogged down by the management reports being generated by users. Often companies solve this by throwing bigger and more expensive hardware at the database when offloading the reporting functions would allow them to gain more benefit for less cost. This also makes it possible for users to write ad hoc reporting queries without the concern that they might bring the transaction processing system down from a poorly written query.
Preserve Historical Data
Data warehousing allows reporting systems to analyze and report on a larger volume of historical data. Most transaction processing systems periodically purge transaction data to save space and maintain performance. Moving this data into a data warehouse allows your business to generate reports over a larger period than what may be saved in the transaction processing systems. This allows you to spot trends over a longer period or analyze the state of various transactions at a point in time.
Merge External Data
This ability to analyze longer time periods to spot trends makes data warehousing particularly helpful to marketing departments. In fact, its most immediate benefits are actually realized by those responsible for marketing. Another way data warehousing makes this possible is by allowing for the merging of data from external data sources for analysis. You would not want to merge this external data in your transaction processing system but in a data warehouse, external data may provide better intelligence about your customers. For example, you might use purchased mailing lists to find which of your customers recently obtained new mortgages. If you marketed products that were of interest to new home owners, this would be a highly useful metric.
Restrict Data Access
Another way that data warehousing can help your business is by segregating roles and permissions in a more concrete way. You may have employees that should have access to data for reporting purposes but perhaps they should not have permissions or access to the transaction processing system. A data warehouse makes it easy to separate those users that only need reporting access versus those that have access to "real time" transaction data.
An integrated set of software and hardware that is designed to meet a specific use is what constitutes a data warehouse appliance. This generally is made up of many servers, data storage devices, operating systems etc being very affordable and effective has emerged as a vital part of the data warehousing market. This appliance can be used to optimize different areas of data processing. Many appliances use languages like the SQL for interacting with the appliance on a database request level. Generally a true appliance requires no indexing or fine tuning and like other ordinary household devices is very easy to use and maintain. This makes it possible to set up a big data center warehouse in just a short span of time.
A data warehouse tool draws power from Massive Parallel Processing nodes and can deploy countless query processing nodes in a single appliance package. An appliance is capable of giving performance advantage that is practically a hundred times faster than general-use data warehouses. This amounts to low costs and low maintenance and automatically lesser power and cooling requirements since processors are not made to handle voluminous data. Data warehouse appliances are advantageous because they allow big companies to staff their warehouses better and help smaller organizations to resolve business challenges. Data center warehouse is therefore largely responsible for the manner in which businesses operate today.
Business intelligence implies activities that a company undertakes to get data about their competitors covering areas like market analysis, industry analysis and competition analysis. Even industrial espionage, it is believed, is a part of business intelligence. Here either an organization hires an outside agency or builds its own intelligence group to get inside information about the company's performance and areas that need improvement. It may then go through records of other businesses in the same field and customer surveys and at times also employ a spy to discreetly gather data. Unlike classic information gathering techniques, business intelligence systems make use of advanced technologies in data mining. Here all segments are interconnected and help to inform each other about their insights to get the complete picture. Business agility grows with business intelligence allowing an organization to exploit constantly changing market conditions.
Business intelligence in Australia is highly developed with the country ranking amongst the top five IT nations in the world. It can boast of good broadband connectivity, great internet security and strong government backing. It services are found to be taking control over nearly all spheres of the economy here ranging from social services and education to business, engineering projects to media and computing applications.
Data Warehousing was an innovation from the 90's that promised to change the data landscape for good. How far have we come? Many vendors have entered the marketplace because it makes sense to bring together data from throughout the organization, and this will continue to make sense in the future.
How large the Data Warehouse market will grow nobody knows yet. But for sure it is still growing fast, and currently is estimated at 4,5 billion dollar per year (IDC).
1. Why Do Data Warehouse Projects Run Into Scope Creep?
To quote Bill Inmon (guru and author of several great books on Data Warehousing) "Traditional projects start with requirements and end with data. Data Warehousing projects start with data and end with requirements." As soon as the project gets under way, users will find new applications, and with it will come new requests for data. Interestingly, these projects often are justified by moving Q&R work away from the 'data people'. What we've seen is that the first thing that happens as soon as the project delivers is that more requests for special queries are submitted to these same 'data people'. This may appear to undermine the initial business case but actually signals the onset of value creation from the DWH project.
2. Star Schema Versus Entity Relation Model?
There has been enormous debate in the community about the merits of different data models. At the risk of over simplifying: ER models tend to have better performance (processing time) for the end user, and are often perceived as "easier" to understand by end users. Drawbacks are that ER models require more disk space, and, because of the intrinsic redundancy in the data, have consistency problems from a maintenance perspective. Having said this, the practice seems to be that often some combination of the two is unavoidable in the practical setting, despite preferences (ER or Star) of the chief architects. Overall, Star models seem to have gained the most ground.
3. The Importance of a Data Warehouse Business Case
Much has been written about the business case for a Data Warehouse. What goes in to a good business case? IT savings are ubiquitous in DWH business cases. The important point is to not limit this to 'pure' savings, but to connect to primary business processes as much as possible. As an example, faster turnaround cycles for list selections are fine (when quantified in hourly rates), but it is even better if the revenue from more customer acquisitions that follow from these selections can be tied in. Not only will the relation to revenue growth rather than savings make for a more balanced business case, more important is the intrinsic business buy-in that results from a direct connection to the company bottom line. These days, changes in legislation (in particular Sarbanes-Oxley) play a major role in justifying business cases. This may be either through a higher company valuation for its transparent information gathering, or, less sleepless night for the CEO, which is of course priceless...
4. Why Do Data Warehouse Projects 'Never' Go Wrong?
Actually, Data Warehouse projects do sometimes fail. But, they fail so rarely, that it is actually very hard to believe... Especially after having talked to so many disgruntled end-users. And there are many ways a Data Warehouse project can go wrong. Delivering on time, data administration issues, and unavoidable data quality issues in feeding systems. Corporate politics (see Tip 7) are probably the best explanation for this phenomenon of near 100% success rates on DWH projects. In my experience, the reason why a failure or 'semi-failure' can go unnoticed is either because senior management is not aware, or, let's say "unmotivated" to talk about misspending of company funds. As a result, not enough is learned. Maybe we as consultants have a stake in this as well, as this assures the industry plenty of ongoing business... J
5. What is Different About Warehousing Web Data?
Kimball & Merz (2000): "Although this clickstream data in many cases is raw and unvarnished, it has the potential of providing unprecedented detail about every gesture made by every human being using the Web medium". The subatomic nature of clickstream data poses unique challenges. There are fewer built in feedback mechanisms to ensure data quality, compared to other data streams. The relation between user mouse clicks and server log records is not as tight as in "traditional" transaction processing due to technical issues like proxy servers and caching. Because of these differences, IT people need to adapt to the web process flow, rather than having the process adapt to IT needs as is common for most other DWH interfaces.
6. Which Data Should Be loaded In The Data Warehouse?
The data that enter the DWH ultimately determine its place in the organization. A "let's load all data, to be safe"-attitude is a sure fire way to derail your DWH project. Choices as to what should and should not be included need to be made early on, to keep the project manageable. After proven success of the delivered, deployed, and profitably exploited DWH, there always will be funding somewhere to include previously ignored interfaces. Given the anticipated lifecycle of the DWH, it makes perfect sense to consciously exclude certain sources. The choice as to what data to include needs to be driven by business considerations, and in particular reference to the company bottom line. If it can't be shown how data will be put to use profitably, they stay out! See also tip #3.
7. Data Warehousing & Company Politics
Data Warehouses have an impact on the company bottom line. Hence, they are likely candidates for turf battles, and are also at risk of becoming "small change" in budget allocation negotiations. None of these considerations benefit corporate long term goals. Managing a DWH project is hard enough as it is, and budget issues shouldn't make it any harder than it already is. Because DWH investments are in the present and revenues lie in the future, it is even more important to secure funding through a sound business case and buy-in from the appropriate (high) management level. See also Tip #3. Access to data means power, and talking about power is one of the greatest management taboos, still around. Sensitive as they are, even budgets are more readily discussed...
8. Data Warehouse Projects Traps
Some commonly recurring 'roadblocks' on the path to timely delivery of a Data Warehouse project:
- ETL processes have eaten up so much time (and still need "babysitters"), that little if any time is left to develop applications needed to exploit the DWH
- Some data are needed, but turn out not to be unavailable, or not in a timely fashion
- Maintenance required for tuning, indexing, and backup and recovery is severely underestimated
- Different ways of calculating the same phenomenon lead to different results, and nobody is able to conclusively explain the difference(s)
- The data that is loaded (and recombined) turn out to contain previously unknown inconsistencies in the source systems, the 'classic' data quality issues that trip DWH projects
- Metadata were lacking, and developers spend inordinate amounts of time finding out what a field really 'means'
9. DWH Hardware and Software Go Hand in Hand
In Data Warehousing, it is not about hardware, and not about software: it is about the perfect integration of these two. Those who begin their project from either end, will pay dearly for this mistake. Reasons are:
· in terms of price/performance, new, pre-integrated hardware-software combinations are taking the lead
· from a project management perspective, you never want to be caught between vendors when a proposed solution doesn't work as expected
· database tuning and indexing is very important and a hugely complex job, necessarily left to specialists (in-house trained)
10. Performance is Key
Although I don't often find technology factors to be this important, in Data Warehouse acceptance, no other factor will be as important as performance. As size increases over time, this factor becomes even more important. There are three reasons for this:
- performance has a huge impact on the development speed (initial load is always very time consuming), and hence the overall maturity of the DWH at delivery time
- performance can make or break end-user acceptance, in particular the predictability of performance
- performance has a tremendous impact on end user productivity, the ultimate driver of the business pay-off
Tom Breur: Biographical Sketch
Tom Breur is a consultant out of deep passion for his work.
He can be profoundly analytic, in his passionate quest to drive out the deepest business issues and the nexus point of a business model. It’s all about finding where the least effort will generate the most results.
Once the business challenge becomes clear Tom loves to roll up his sleeves and get his ‘hands dirty’.
What makes a data warehouse important for a company is its ability to gather information from different parts and then making them a single part of a
1 comments Posted by Ram at 2:50 AMIf you want to get information on all the techniques of designing, maintaining, building and retrieving data, Data warehousing is the ideal method. A data warehouse is premeditated and generated for supporting the decision making process within an organization. When the production databases are copied in the warehouse, it becomes easier to answer all the queries without hampering the consistency of the production system.
A data warehouse is actually a set of new concepts and important tools evolved into a technology. With the help of data warehousing, it becomes easy for an organization to counter all the problems faced during providing key information to concerned people.
Over the last two decades, a number of experiences and technologies incorporated together to evolve the new field of Data warehousing. You can say it as a well organized and resourceful method of managing & reporting data non uniform and scattered sourced throughout an organization.
Because of hundreds of gigabytes of transactions, it is necessary for a data warehouse to be vast. Therefore, "data marts" are often designed for individual department or a product line. A data warehouse system is an influential and necessary platform for merging data from old and new applications. You can transfer rules to a warehouse without making much efforts. The prime features of a data warehouse is that it records, collects, filters and provides basic data to different systems at higher levels.
Here are some of the benefits of a data warehouse:
o With data warehousing, you can provide a common data model for different interest areas regardless of data's source. In this way, it becomes easier to report and analyze information.
o Many inconsistencies are identified and resolved before loading of information in data warehousing. This makes the reporting and analyzing process simpler.
o The best part of data warehousing is that the information is under the control of users, so that in case the system gets purged over time, information can be easily and safely stored for longer time period.
o Because of being different from operational systems, a data warehouse helps in retrieving data without slowing down the operational system.
o Data warehousing enhances the value of operational business applications and customer relationship management systems.
o Data warehousing also leads to proper functioning of support system applications like trend reports, exception reports and the actual performance analyzing reports.
Precisely, a data warehouse system proves to be helpful in providing collective information to all its users. It is mainly created to support different analysis, queries that need extensive searching on a larger scale.
What makes a data warehouse important for a company is its ability to gather information from different parts and then making them a single part of a centralized database. It is a collection of data, which is further used by employees for an easy and smooth working process. Know more about data warehousing by reading the article.
Data warehouse is an asset for an organization because it maintains the efficiency, profitability and competitive graph. A company collects data from sources like inventory manageable, call center, sales leads etc., which is then passed through the Data Life Cycle Management policy. It is this policy of the organization that determines the design and methodology of the data warehouse.
The main motive of a data warehouse is to create front-end analytics that will support the operation staff and other employees of the organization. Here are some of the elements of a data warehouse:
Pre-Data Warehouse
This zone provides data for the data warehousing and the team of designers filters out the data that contains business value for insertion. Operational data is stored in OLTP database, which resides in transactional software applications like supply chain, ERP etc. OLTP's are designed for high transaction speed and accuracy.
It is the metadata that ensures accuracy of data that will be entered into the warehouse. Most of the organizations reduce cost for the ETL stage by opting for a metadata policy.
Data Cleansing
Data cleansing is the extraction, transformation and cleaning process that are done to ensure the quality of the data before it is entered in the warehouse.
Data Repository
Data repository is a database where active data of an organization is stored. It is then optimized for data analysis.
There are two types of data warehouses - ODS and Data Marts. Although data marts are no different from data warehouses in physical terms but they can be smaller and are built on departmental level instead of company level.
One drawback of data warehouse is that it collects data and has older data as well, which means you will not get an up-to-date analysis. Operational Data Stores can be useful when it comes to storing recent data before migrating to the data warehouse.
Front-End Analysis
Front-end application that will be used by employees is the most critical part of a data warehouse. They will use it to extract information and interact with the data stored in the repositories.
Data Mining
It is the discovery of many useful patterns in the data. Data mining is used for analyzing and the classification process.
Data Visualization Tools
These tools are used for displaying the data from the data repository. Designers often combine it with data mining and OLAP tools. The process of data visualization helps users in manipulating data as per its relevancy and pattern.