DatatWare Housing

Informatica Training - What Is Informatica?

2011-02-18T03:34:00.000-08:00

Informatica is something that you may have heard of before or maybe have not. However, if you have not heard of it you are truly missing out on this great bit of knowledge. Informatica is a software which allows the user to create a data warehouse with ease. It makes the process of designing, ETL and maintaining the data warehouse a breeze as well. Informatica training brings a new, simple approach to the term "Data warehousing."

Informatica training: What is data warehousing?

The use of data warehousing is an innovative way for a business to lower their operating costs, raise the production rates and push their sales through the door by using the management of information. It allows for the support of intelligence through the business, management of the relationship with the customer plus lots of other beneficial business uses.

Informatica training: How can Informatica help with data warehousing?

Data warehousing is not always the simplest of tasks especially when it comes around to accountability however, Informatica consists of an easy to use interface that allows you to create and control your warehouse effectively leaving, next to no room for the possibility of error. The bulk of the warehouse design is done by performing simple click and drag methods. This method, allows you to easily comprehend the tasks which you are performing. Informatica also has a unique ability to connect with any other major database that exists. This means that information is easily transported in between each of the databases including, the large volume information files.

Informatica has the ability to do this by performing throttle tasks or, performing huge tasks broken into smaller ones so as not to bog down the log. This is all done in a very effective yet secure way. Additionally, thanks to the server which is used by Informatica and a plug in server manager application, you are able to connect tables from any other server or database in existence. This in turn, helps you to effectively create, run and manage your data warehouse without frustration.

Informatica training: How does the software work?

The design is created in the designer portion of the program. It clues you into where you will put the tables and sources, what the specific targets which you will have are as well as how you should go about moving the information. With Informatica, you get your own repository manager which helps you to maintain the repository portion of the program. The repository is the database which is used by Informatica in order to keep track of all the data which is stored there for use.

Informatica training: What do I get with Informatica?

You have several different options when it comes to Informatica. This includes a number of varied packages and extras that are available. The Informatica Power Center allows you access to every option available which includes meta data. With this, you are able to keep all of your repositories in one place which is, a domain through the data mart. This also allows you to share the meta data throughout other repositories. The Power Mart option is a license which is slightly more limited however, still provides you the ability to do everything with the exception of the meta data.

Data Warehousing and Business Intelligence

2011-02-18T03:30:00.000-08:00

An integrated set of software and hardware that is designed to meet a specific use is what constitutes a data warehouse appliance. This generally is made up of many servers, data storage devices, operating systems etc being very affordable and effective has emerged as a vital part of the data warehousing market. This appliance can be used to optimize different areas of data processing. Many appliances use languages like the SQL for interacting with the appliance on a database request level. Generally a true appliance requires no indexing or fine tuning and like other ordinary household devices is very easy to use and maintain. This makes it possible to set up a big data center warehouse in just a short span of time.

A data warehouse tool draws power from Massive Parallel Processing nodes and can deploy countless query processing nodes in a single appliance package. An appliance is capable of giving performance advantage that is practically a hundred times faster than general-use data warehouses. This amounts to low costs and low maintenance and automatically lesser power and cooling requirements since processors are not made to handle voluminous data. Data warehouse appliances are advantageous because they allow big companies to staff their warehouses better and help smaller organizations to resolve business challenges. Data center warehouse is therefore largely responsible for the manner in which businesses operate today.

Business intelligence implies activities that a company undertakes to get data about their competitors covering areas like market analysis, industry analysis and competition analysis. Even industrial espionage, it is believed, is a part of business intelligence. Here either an organization hires an outside agency or builds its own intelligence group to get inside information about the company's performance and areas that need improvement. It may then go through records of other businesses in the same field and customer surveys and at times also employ a spy to discreetly gather data. Unlike classic information gathering techniques, business intelligence systems make use of advanced technologies in data mining. Here all segments are interconnected and help to inform each other about their insights to get the complete picture. Business agility grows with business intelligence allowing an organization to exploit constantly changing market conditions.

Business intelligence in Australia is highly developed with the country ranking amongst the top five IT nations in the world. It can boast of good broadband connectivity, great internet security and strong government backing. It services are found to be taking control over nearly all spheres of the economy here ranging from social services and education to business, engineering projects to media and computing applications.

Legacy Applications Migration Into a New Data Warehouse Environment Through ETL Tools

2011-02-18T03:29:00.001-08:00

Most of the enterprises used custom coding processes or first or second generation ETL solutions to increase access to their data. Nevertheless, custom scripting does not endorse litheness and continued development in the significant business processes essential for enhanced efficiencies. General problems with a legacy ETL solution comprise:

High operating prices - Most likely are needed to pay for each mainframe processing cycle you consume, and using a mainframe for ETL processes will consequence in excessive cycle utilization by your business element.
Use of outdated technology - no vendor support, busy skillfulness in the job market, and no platform support
Insufficient documentation - legacy code is text-based, the comments in code are for developers, and the knowledge rests (and leaves) with the developers
Probable performance issues - older technologies be unsuccessful to make use of new technology capabilities such as parallelism

ETL solutions responsible for running batch jobs at scheduled periods to confine data from flat files and relational databases and merge it into a data warehouse database directed by a relational DBMS. Over few years, commercial ETL solution vendors have made a broad range of enhancements and additions in to their ETL software product.

Most of the ETL Tool are specializes in data warehousing solutions, with profound proficiency and practical skills, every company need the leading ETL tools to solve business problems. ETL tool help the companies to migrate the ETL processes from a legacy mainframe setting to a current ETL solution.

An ordinary enterprise ETL tools need is to extract data from apps like Oracle, PeopleSoft, and SAP and migrate it in to other applications like ERP, CRM etc. These ETL software products comprise enterprise application connectivity free of charge. These connectors - if available - are usually a paid option. They are not compatible with open source principles but common revenue source.

Make Informed Decisions With Data Warehousing and Business Intelligence

2011-02-18T03:28:00.002-08:00

Use of business intelligence solutions and software will help you collate various data from various sections of your business so being able to make more informed decisions can be carried out a lot easier.

Although you may find initially when it comes to setting up data warehousing and business intelligence systems is a challenge. It is time and money spent that you will have invested in your business wisely as it can help to impact positively on how your business does in relation to your competitors.

Today there are many different types of data warehousing systems that a business can employ. These provide an effective all in one solution to gathering business intelligence and then allowing you to analyze it.

So just what are the benefits to be thinking about installing data warehousing and business intelligence systems for your business? Below we take a look at a just a number of these benefits.

Benefit 1 - These systems actually allow you to identify and resolve any inconsistencies in the data being collated before it is loaded. As a result it makes the whole process of reporting and analysing the business intelligence gathered much simpler.

Benefit 2 - Another benefit of using such systems is that the information being collected is controlled by the people using the system. Even so the system can still be purged over time and any information that has been collected can be safely and easily stored for a lot longer.

Benefit 3 - As this differs from other operational systems you will find that it allows the users to retrieve data without it causing the operating system to slow down.

Benefit 4 - Businesses that choose to install any kind of data warehousing and business intelligence gathering systems will find that they can improve the relationship not only between the various departments, but also their customers. Such improvements will then ensure that they find it easy to identify the needs of their customers and can work on ways to make sure that this will be met.

Benefit 5 - Through the use of such systems a business will be able to make sure that support system applications like execution and trend reports are functioning correctly. So the production of the reports that they then use to analyse their businesses performance are more accurate.

Benefit 6 - As mentioned above the use of data warehousing and business intelligence gathering systems can help a business to be more competitive. This is because it allows them to identify areas of the business where improvements can be made much more quickly.

Three Reasons Why Your General Ledger Should Not Be Your Data Warehouse

2011-02-18T03:28:00.001-08:00

Many companies today rely on the general ledger as key part of their management reporting, well beyond the obvious financial information.

The current practices in many organisations, and the architecture of their systems, and even the very structure of the software they buy have often been shaped by the history of the adoption of information technology in the firm.

In many firms, their management reporting systems reflect the fact that as information technology began to be used extensively by business, often the very first functional area to be automated was accounting.

Because finance and accounting are of course at the heart of any enterprise often the first automated reports and the first database within an enterprise was the general ledger.

In many companies, the general ledger became the clearing house for all information- not just financial, and in effect became a data warehouse before the concept of data warehousing had even evolved.

Lets look at the example of a manufacturing enterprise.

The company invested in a mainframe computer at some time in the seventies. Management was thrilled with the new capability they had in financial reporting. It didn't take long for them to ask to have manufacturing data in the reports as well.

Eager to please, the accounting department added lots of additional accounts into the Chart of accounts (COA), adding entries that were "non-financial" storing sales quantities, volumes consumed, things like energy consumption, raw material quantities, wastage and defect counts.

This made sense at the time, because otherwise the information would not have been stored digitally. The manufacturing plants used hard wired relay logic to control their equipment, and recorded instrumentation readings on chart recorders.

A chart recorder is a device that uses a physical pen to record temperatures, pressures, position, speed, etc. of equipment on paper which is physically pulled past the pen at a predefined rate. As these rolls of paper were used up, the operator would change them. The rolls of paper with the information on them then got stored in filing cabinets.

The shift workers wrote information into formated pages in a shift book, and then at the end of the month, administrative staff added journal entries to capture the information. Many factories didn't have a computer, or if they did, it was a mini-computer that was specified by and operated by the information technology department- which was part of the finance organisation. Companies didn't have CIOs, only CFOs.

The final result was, it was possible to generate management reports with both financial and manufacturing information. How many liters of paint did we buy? How many kilowatt hours of electricity, for how many units produced. In some ways, this might have been the brief golden age of management reporting. (Or is it just that time makes memory blur?)

Then, as computers started to arrive everywhere, no longer just the domain of the finance and accounting department, the trouble started.

The manufacturing plants installed automation systems. Chart recorders gave way to distributed control systems, and SCADA (supervisory control and data acquisition) systems, and pretty soon they had their own databases. They kept having to supply numbers for the bean counters to enter into the now aging mainframe, but they used their own reports and eventually spreadsheets to actually manage their process.

Manufacturing organisations began to include "automation engineers" which, in fact, were information technology professionals, and multiple IT departments began to form in all but name. Standards for data format, coding and methods for calculating key performance indicators evolved slowly, or not at all. Finances definition of how to calculate things tended to win because they held the keys to the general ledger, where the report that went to the CEO came from.

Then came the ERP. The ERP may or may not have included the manufacturing operations, but it almost always included the general ledger. Regardless of if the manufacturing modules of the ERP were used or not, the trouble with getting management reports just got worse. Manufacturing had all sorts of detailed information they needed, and keeping the central, general ledger reports up to date meant creating more and more accounts, more and more cost centers. The concept of a separate data warehouse where information from multiple systems (finance, manufacturing, sales) could be combined was born, and the general ledger, in theory, returned to its roots as a repository for financial transactions.

The problem is, in some organisations, the data warehouse didn't come. The general ledger kept its place as the central repository for not just financial, but also management reporting.

Huge amounts of non-financial information is still stored in many general ledgers. Here are just three important reasons why your general ledger should NOT be your data warehouse.

1) It forces you to compromise on level of detail and drill down, and history

No general ledger can hold the level of detail available in many source systems. As a result, any interface from the sales system, manufacturing system etc. feeding into the GL will have to create journal entries that summarize a great deal of information.

While the detail of course will still exist in the source system, if your management reporting is all from a general ledger based system, upper management will tend to use this single source- and as a result important granularity may be lost to the decision making process.

This summarization also makes it very difficult (or impossible) to have drill down into the details, giving up some of the greatest benefits of modern business intelligence systems.

Finally, general ledger based data storage does not usually allow for the tracking of reference data changes over time. As sales regions are modified, and territories shift, comparing one period to another becomes increasingly difficult. Data warehouses, designed from the beginning to store this type of slowly changing reference information, can provide a much more insight and historical analysis.

2) It results in an overly complex chart of accounts and may even affect month end close

As the source systems become more and more capable of collecting data, the tendency is to want to increase the amount of management reporting. If this is being done in the general ledger, it means that further charts of account must be added, and an increased number of journal entries need to be done. Depending how the overall process is setup, its even possible that the increased complexity might affect the speed at which month end closing can be completed, if for no other reason that the same finance resources must both tend to the financial and the management reporting needs.

3) It discourages cross functional definitions and collaboration on analysis

By making one of the functional areas (finance) the center and owner of management reporting, a general ledger based reporting architecture can actually increase the severity of the information silos it is most likely trying to eliminate.

Because the general ledger reporting does not require all the detail available, each department only needs to provide the summarized information required by finance. While every department has to coordinate with finance, there is no requirement for sales and manufacturing, for example to compare or coordinate their information definitions. While at a high level data is integrated, any benefit from more tightly integrating information across silos that a data warehouse can bring is lost.

In a very real way, a successful general ledger based management reporting system is in fact an impediment to progress for an enterprises business intelligence and data analysis evolution.

Because management reporting is available, the justification or need for a data warehouse is not felt as strongly. However, as needs continue to evolve, the effort expended in the constantly growing general ledger, and its impact on the financial processes, and the companies overall information management culture will become increasingly damaging.

Ironically, companies who failed to ever establish a general ledger based management reporting system could leapfrog their more financially focused competitors, as they embrace the modern data warehouse, the the tools available for data analysis.

A true data warehouse is not an easy road, and is only one component of a broader data analysis strategy. In the short term, using the general ledger for management reporting can seem easier, and could put off the challenges of additional hardware and software, as well as the need to coordinate between departments.

However, despite its historical place as the center of all data storage, in a modern architecture the general ledger should be used only for accounting as it was originally intended.

Tips to Consider Before Choosing a Data Warehouse Vendor

2011-02-18T03:27:00.000-08:00

Data warehousing has become such an essential issue in an organization that you will find the market flooded with several data warehousing vendors. The warehousing vendors provide technologies, tools and methods that help in managing, using, constructing and maintaining software's used for a data warehouse. In most of the cases, these tools themselves are used for the data.

The vendors give ample options with their product that will help you in developing and enhancing the functions of a data warehouse. Companies get benefited from a data warehouse because it leads to the involvement of different discreet technologies together. A well-structured data warehouse will help in the following:

o Data aggregation.
o System identification of data source.
o Data acquisition.
o Data cleaning.
o Business intelligence.
o Data mining.

If you want to get the best of the above and many more features it is necessary to select the data-warehousing vendor carefully.

The tough market competition has made the data warehousing industry very complicated. When it comes to buying a data warehouse you will find hundreds of vendors offering thousands of products. The technology used for making a data warehouse is complicated because it involves numerous software applications and tools. Because of this fact one should do a thorough evaluation of the technology being offered by the data warehouse vendor.

Although there are complexities in choosing the right vendor but you must remember that a data warehouse will provide a competitive edge to your business as compared to other organizations that don't have such a system.

Business Intelligence Strategy

Business intelligence has always been the top priority of middle-sized organizations. A business owner emphasizes more on visualizing the marketing ideas and getting returns, thus requiring a business intelligence strategy to boost the investment. Data warehousing is one such BI technology.

In recent years we have seen that BI strategies are somehow loosing their position in the market. The reason behind this is the coming of age of data warehousing vendors in the market who are providing technology that will make data warehouses more effective.

Making the Right Choice

If you want to come up with better business intelligence strategies you need to make the right choice while dealing with data warehousing vendors. There are several out there that vary greatly in price. Price is obviously a consideration, but should not be the main determining factor in whih vendor you ultimately choose. Before choosing the technology or the vendor there are a few things you must consider:

o Find out if the vendor has the necessary experience of working with small, medium and big sized companies.
o Ask the vendor about his track record in generating great revenues for clients.
o Inquire about the method used by the vendor in providing the service.
o You will find vendors who will let you run the data warehouse on your own server license, while others will host software tools on their server. Choose the best option as per your need and budget.
o Never fall for a vendor providing limited services, as you would always wish him to perform with the broad data sources of the company

Analytics Jobs - Booming Careers

2011-02-06T21:25:00.001-08:00

Analytics jobs are in great demand in today's industry. Some of the popular careers in this field include positions of a Data Analyst, Business Analyst, Marketing Analyst and Systems Analyst.

Data Analyst

The demand for data analysts is high in the Information Technology market. The main job of a data analyst is to search the particular needs of the client and evaluate data that meets those needs. A data analyst is required to search raw data and make comparisons with other related data, statistically. The solution to a particular problem has to be presented to the client in a simple and effective manner.

A freshman in this industry starts out as a Junior Analyst and moves up the ladder to Senior Data Analyst and Data Analysis Project Manager. Both technical skills as well as strong interpersonal skills are required in order for a person to succeed in this career.

Business Analyst

Business analysts are required in all spheres - healthcare, marketing, biotechnology and Information technology. Their main function is to understand that needs of the specific businesses and enhance the areas that need improvement. These professionals test and gauge the technical issues and implement changes, if required, keeping the company's objective in mind.

The qualifications required to apply for this position include a Bachelors' degree in business administration with a specialization in the concerned field such as Marketing, IT, biotechnology and so on. Other than these qualifications, these individuals need to have other analytical and problem solving skills, communication skills and must be able to work as a team and under high pressure.

Marketing Analyst

A Marketing Analyst is also vital for the growth of a company. Professionals in this field analyze, compare product prices, study the spending pattern of customers and other competitors to help develop a marketing strategy for a company. The changes in the marketing, strategy, product line and sales force are made based on the report provide by Marketing Analysts and thus is important for the growth of a company.

Systems Analyst

A System Analyst is responsible for analyzing the present system in the company and determining the future requirements. These include determining the limitations in the softwares and suggest alternatives to the company. These professionals usually act as a liaison between the IT group of a company and the vendor.

A person aspiring to make a career in this filed should be prepared to interact with technical team of the vendor and be familiar with different programming languages, have a knowhow about computer hardware and operating systems.

Negotiating an Oil and Gas Lease

2011-02-06T21:24:00.004-08:00

Lease Negotiation

An oil and gas lease is a legally binding contract for both parties involved. In most cases, a lease is set for a duration of 2-5 years if the energy company doesn't drill. Once a well is drilled and found economically viable, the lease continues in perpetuity so long as the wells inside the lease are economically viable. If the well is not viable, the lease expires after 2-5 years depending on the lease terms.

Terms of an oil and gas lease can be difficult to understand. There are many clauses and stipulations that must be addressed.

Entering into a contract with a large oil company can be daunting especially when a binding contract is involved. For a smoother facilitation of establishing a contract, energy companies usually assign a landman to the initial negotiations between a mineral owner and their company. The landman is also responsible for performing the basic groundwork of determining the correct mineral right owners.

It is imperative that the mineral owner review the oil and gas lease in detail to make sure that the terms set forth are agreeable. If there are items in the contract that are unsatisfactory to the mineral owner, it is necessary to negotiate terms with the landman that will be acceptable. Sometimes these terms are simply adjusted while other times the items are altered completely. There are even times when an agreement can not be reached by either party.

Mineral owners are highly encouraged to seek legal counsel when entering an oil and gas lease. Energy companies have entered into many oil and gas leases and are very familiar with the process. However, a mineral owner may be completely unfamiliar with a lease and the negotiations necessary to create a lease that is mutually beneficial. It is important to remember that at times a mineral owner may feel distrust or hostility after reviewing the lease or during the negotiation process. Again, It is highly encouraged for the mineral owner to retain an experienced oil and gas lawyer to review the lease in order for the negotiation process to run more smoothly.

Lease Components

At the top, usually the right hand corner there is a date. This date is known as the date clause. It establishes the commencement date of the lease.
The names of both parties that are legally bound to the lease are in the first paragraph. The lessor, who is the mineral rights owner, and the lessee, which is the energy company.

A legal description of the land is outlined. So that both parties may establish the exact plot of land that will be bound by this agreement.

The duration of the primary term is mentioned in months. A clause that allows for a secondary term can also be added to a lease.

The royalty clause is usually a lengthy paragraph. This clause states the percentage or share of production proceeds the leasor receives. How the royalty is received is also mentioned in this paragraph as well.

A granting clause is included in all leases. The granting clause outlines the rights of the lessee and the property that is binding to the lease. The lessee's rights include drilling, delay rental, pooling, shut-in royalty, unitization and additional drilling clauses.

The lease also outlines the course of action that will be taken by the lessee if any such problems arise during the term of the lease like what happens when a dry hole is drilled within the primary terms. A damage clause is also included in the lease as well.

Another important clause in an oil and gas lease is an assignment clause. During the term of the lease if either party should choose to transfer ownership, the assignment clause outlines the stipulations that must be met. This is important to the energy company, as many energy companies transfer ownership of their leases.

The force majeure clause touches upon the state and national laws that it is necessary for any drilling rig to adhere to. They are clearly outlined and give the Lessee freedom from non-performance that could possibly be implicated in the lease.

Like many standard contracts there is a warranty clause. A warranty clause states that the mineral owner guarantees their legal right to the land to the Lessee if he or she should later be discovered to not be the true legal mineral right owners.

What is Your CIO Priority 2010? Learn What Your CIO Colleagues See As Their Technology Priorities

2011-02-06T21:24:00.003-08:00

IBM and CIO Insights have recently conducted surveys with CIOs regarding their top priorities for 2010. Top Analysts from Gartner gave their opinion about IT budgets and IT spending for 2010 and beyond. The IBM study revealed that Business Intelligence and Analytics are the top priorities for more than 83% of all interviewed CIOs. The CIO Insights survey show business productivity, cost reduction and IT/Business Alignment as top priorities. According to Peter Sondergaad, VP Research, Gartner, 2009 was the most challenging and interesting year for CIOs. He quotes "The IT market is exiting its worst year ever".

In 2009 enterprise IT spending will end up to be 2,3 trillion dollars compared with 2,5 trillion dollars in 2008. Spending has actually declined in all markets, so CIOs have saved across hardware, software, telecommunication services and IT services. 2010 will not see a huge increase in IT spending, as budgets are still tight. The largest increase spending in IT will happen in health care, utilities and government.

The 2010 CIO spending budget is therefore done on the background of the worst year ever in the IT industry. 2010 is about balancing cost, risk and growth. IT has to be prepared to carefully plan for growth. So CIOs are facing a dilemma: How can they reduce cost while improving services? How can they balance the need to influence business strategies with the need to provide top notch IT support?

Using information technology to produce greater business value is vital, bundled with an ongoing focus on lower cost and higher efficiency. This is why an improved business intelligence solution combined with a great analytics tool with excellent visualization is so important.

One of the new generations' business intelligence tools is "BDA Business Intelligence" which comes with a modern data warehouse, pre-packaged business solutions like sales analysis, financial analysis, inventory analysis and with a web based reporting and analytics tool. The typical implementation time is less than one week and BDA Business Intelligence even offers the front-end solution for unlimited amount of end-users.

Storekeeper Jobs the Caretakers of the Stores

2011-02-06T21:24:00.001-08:00

Storekeeper jobs are in demand in those organizations where material management is required. Storekeepers, also known as watchdogs, are the in charge of the store department and are expected to perform various warehousing duties of receiving, arranging and issuance of goods. They are not only accountable for receiving and issuing of goods but sometimes they also have to act as a inspector, supervisor and manager to ensure that there is proper handling/stacking of goods, inspection of stocks, warehouse maintenance and cleaning, wastage disposal and proper documentation and record-keeping. Store administrator is the one, who efficiently look/supervises after all the things at the store from inspecting stocks to warehouse cleaning and maintenance.

Candidates looking for such kind of role must have sound knowledge of policies and procedures related to receipt, storage and issuance of materials, good communication skills, physically strong etc. Computer literacy would be an added advantage. Material Management Jobs plays a critical role in an organization because production of goods is primarily dependent on the availability of materials. A Material Manager has to identify the vendors for materials, supplies and equipments well in time, maintain records of goods ordered and received, identify quality goods at the lowest possible price etc. An aspiring candidate in this field should have adequate knowledge of the market to identify the suitable vendors, must have good communication skills for getting better deals, and must be well versed with the computer skills as they are expected to work on the inventory database.

Likewise storekeeper jobs, Warehouse Management Jobs also plays an important role in a production oriented organization. A warehouse manager has to plan, organize and control overall operations of the warehouse, setup layout and space management, manage stock control, plans out for development of staff through trainings etc. These jobs require individuals, who have the ability to work effectively without much supervision and communicate effectively. These individuals must also possess good health and physical stamina to handle the daily warehousing activities.

Resource Management jobs play an integral role in every organization because every organization hire resources/employees who work for them and for managing them resource management jobs comes into picture. They are the link between the management and employees who help resolving work related problems, administers benefits and performance management systems, analyze and modify compensation and benefit policies to establish competitive programs and ensure compliance with legal requirements etc.

Store keepers are not only the caretakers but they are also the supporters who have the ability to establish and maintain effective working relationships to ensure smooth running of the business. Surely storekeeper jobs have become an indispensable part of work culture in any organization.

Developing a High Performance Web Application - The Art of Scalability

2011-02-06T21:23:00.002-08:00

Writing a web application is straight forward once you've mastered the fundamentals of a server-side scripting language and a database system. Though it is not obvious, developing a website that caters to thousands of users and one that caters to millions of users require a different set of tricks.

This new scenario must take into careful account the server resources that is utilized to ensure that only optimal coding practices are used when information storage and retrieval goes up to a massive scale. Instead of thinking in terms of a single web server, you must use variable configuration in your application to cater for multiple web servers that can be added-on at any future date. Your programming logic must know how to distribute content and processing work across the added servers.

More servers are needed when the CPU load utilization spikes to a high level that eventually affects the performance of your website. Your web server takes longer time to respond to user request and pages that took seconds to load are now taking much longer. This could happen when the number of simultaneous website visitors increase or that a certain processing tasks in your application is highly intensive, such as video conversion process, real-time image file manipulation or sorting of huge datasets that are fully loaded into memory.

One strategy is to clearly separate different aspects of your application functionality to different servers, such as news.yourserver.com and video1.yourserver.com or music3.yourserver.com. Hence, when your users are using different services of your website, they are bounced to different subdomains that are pointed to different web servers. If it's a membership page, there will be a challenge to maintain user session across the different servers, as session information are usually stored in each local server in the /tmp folder. In PHP, the session handler needs to be configured to utilize a common database instead of the local filesystem to cater for cross server session tracking.

This strategy also require that 'shared files' are stored in a common area so that the different servers has access to them and that there are APIS written for reading or writing to these common files. When multiple users on multiple servers are able to write to a common file, the file access API must ensure that proper file locking mechanism is in place to maintain file integrity.

Since the database backend is usually accessed by a local socket/network connection, it is by default that all web servers are able to access a common data pool without further complexities. However in almost every case, the bottleneck of a website lies with the database storage, especially when the amount of data stored has gone into tens of millions of rows and that heavy server usage is causing thousands of database records to be updated every second.

Highly matching database indexes must already be in place to ensure an instantaneous retrieval of data. This being said, it is never as easy as it seems and because we are discussing scalability, there must also be a solid plan in place to add more database servers when necessary. As with the multiple web servers, your application logic must also which database server to retrieve data from, if you're splitting different partitions of data into different database servers.

The other trick is to deploy Database Replication, whereby every database system contains the exact copy of data (as opposed to partitioning different sets of data to different db servers), and that the overall load of each database server is spread across the group of servers. More than just a method of database load balancing, this method also ensures data redundancy, and that your application would be running smoothly in the event of one or more database crash.

The other methods to think about is the caching of often read data that is rarely updated. File based caching or memory based caching can be used, and it has proven that these methods can improve performance by nearly two thousand percent. This is due to the high cpu utilization when sockets communication is involved and that local file or memory access takes a tiny fraction of the cpu resource instead.

When planning for a high performance system, do think of using lookup tables that has values already pre-calculated for frequent use, as well has utilize hashing algorithms for the lookups of cached data. Avoid data intensive real-time processing at all cost and try to use pre-generated tables that is incrementally built instead of a full regeneration every day. Reduce access to the database as much as possible and use local file access instead.

Lastly, always take a peek into your database process list, to get a glimpse of which query is taking too long to respond. These are the queries that locks the database tables for a long periods and denying access to other queries from completing its task. These are the major points of planning a high performance and scalable web application. It's something that's not widely documented in books, and you will need to experiment and to create benchmarks for comparison.

Useful skills to have include the understanding of file read/write locking access, sockets communication, what a hash is compared to a sequential search and memory based caching - Memcache. Lastly, always think of delegating tasks to different servers and plan for future data partitioning. Good luck with your tasks at hand, for it's not an entirely easy one.

The Value of ERP As an Organizing Force

2011-02-06T21:23:00.001-08:00

In concept, the more robust enterprise resource planning solutions (ERP) combine data and processes by using integrated software and hardware components and a unified database to capture information from all areas of an organization. Of course, the most immediate use of ERP solutions in business is found in applications for manufacturing (discrete and process). An ERP solution must be many things to many people within an organization, but at its base it should be both scalable and flexible, as well as transparent to end users even as it continuously works in the background.

ERP functions through the ordering of business processes that are themselves organized according to common business flows (i.e., lead-to-order, order-to-cash, purchase-to-pay, etc.). Such unified and consistently flowing business processes are produced through core applications that are usually built, in ERP, on a single platform to help consolidate information and reduce IT costs. Doing business today-particularly manufacturing business-without ERP is tough, and is at best much less efficient when viewed through the very focused lens of bottom-line metrics.

What does ERP do? It helps to organize the various factors found within emerging and present opportunities, and can:

Improve productivity
Enhance financial performance
Streamline processes and workflows
Improve reliability of tracking and forecasting
Lower production costs
Improve customer service (CRM)

To these ends, a good ERP solution offers a strategic link between business objectives and IT. As well, it can be justified to management in very measurable returns. A robust ERP solution is configurable at the business-process level so that as company objectives change, the ERP solution changes right along with them. Therefore, ERP implementation can be seen as way organizations build relationships; to perform effective, repeatable processes; to drive efficiency into every process; and to share information accurately and as needed. And, of course, to drive profits to greater success.

In short, top-tier ERP applications will directly link the specific attributes that inform the positive financial performance of the business. By connecting the application functionality to business-process improvements in all areas of the manufacturing operation (including finance, production, supply chain management, customer relationship management, data warehousing, and human resources), the entire organization benefits in terms of operational and actionable intelligence. In this regard, ERP is more than just business communication.

With the power of the comprehensive inclusion of business data, ERP becomes an organizing force for manufacturing, and ultimately provides efficiency solutions that are otherwise difficult to imagine. To the extent that ERP is a totalizing tool designed that easily integrates diverse data, the single source solution approach offers the best and most efficient method for organizing and interpreting such data.

Summary: A good ERP solution can offer a strategic link between business objectives and IT, and do it in such a way that drives profitability and productivity. ERP organizes real time data from all areas of the manufacturing operation. In doing so, it provides the capability of making informed decisions rapidly, and with information that is both timely and relevant. In turn, such decision-making capabilities resulting from ERP produce improved productivity, improved financial performance, lower production and administration costs, enhanced forecasting, and improved customer service.

Oil and Gas Royalties - Leasing Mineral Rights

2011-02-06T21:22:00.005-08:00

Royalties vs Mineral Rights

Oil and gas royalties are not as complex as most people think. They are actually fairly simple, and I'll explain clearly what they are and how they generate cash.

If you own a farm, then you own the land also known as the surface rights. Often, when you bought the farm, your deed conveyed the mineral rights under the farm along with the surface rights. Owning the mineral rights means you legally have the right to explore, extract, and sell any oil, gas, coal, uranium, helium or other mineral that rests beneath your land.

Most landowners, however, don't have the geological knowledge or training to understand the potential minerals under their land. In fact, many landowners forget they own the mineral rights under their land. Further, the average landowner does not have the multi-million dollar budgets to explore for hydrocarbons, or the social networking skills to raise a multi-million dollar exploration fund.

Energy companies do have the knowledge and funding to explore for oil and gas. So when they identify a region that likely contains hydrocarbons, they negotiate with the landowners to lease their mineral rights for exploration. This lease gives the energy companies permission to explore for petroleum and to produce and sell it if they find petroleum in economic quantities.

The Bonus and the Royalty

The mineral owner receives two forms of compensation for leasing his mineral rights. The first is called a 'Bonus Payment' which is a signing bonus that is paid on a per acre basis. Typically $200-$500 per acre. The bonus will be paid once at the time of the signing of the lease, and it may be the only money the owner will get.

The second is the royalty which is the percent of the money generated by the oil and gas from his property. Traditionally 12.5%, but more recently around 18% - 25%. The percentage varies upon how well the mineral owner negotiated and how expensive the oil company expects the extraction of oil and gas to be.

However, if the oil company finds no oil or gas, or not any in economic quantities, then they abandon the prospect, and the lease expires which reverts the mineral rights back to the mineral owner. In this case, the Bonus was the only money the owner received.

In the event hydrocarbons are found and the wells produce, then the royalties kick in. So if the well produces 100 barrels a day, and the price of oil is $80 per barrel that month, then the cash flow is 100x$80 = $8,000/day The royalty owner, who agreed to 15% royalty, would receive $8,000 x 0.15 = $1,200/day. Over a month, that brings in $36,000 per month to the mineral owner, who in this case, is the landowner. Now you see why oil is a big business!

Royalties Dwindle Over Time

Royalties paid to the mineral rights owner will often last for decades. The wells will deplete, however, so over time the money received from royalties will drop considerably. The average well is thought to last 35 years. Eventually, the royalty dies, and all the owner has is the mineral rights. Which may get leased again in the future.

Finding Mineral Rights to Buy is Hard

Because of the reliable cash flow stream, oil and gas royalties make for a good investment. Finding mineral right owners who want to sell their royalties is the tough part. The only available data on royalty owners currently is Blackbeard Data Services, and they have all the owners in Texas and Kansas.

Advantages of BI Dashboard Consulting

2011-02-06T21:22:00.003-08:00

Most companies make use of their own dashboard in order for them to gather and manage real time information. Their dashboards are often connected to the Internet or their own network so that they can disperse their data easily to everyone working in their organization. With the dashboard they can determine how well they are performing based on different issues that they want to concentrate on. These include the customers and the processes in their company most of the time. At times, they find this a hard thing to do and this is why they seek the help of the solutions presented to them by BI dashboard consulting.

Such consulting service can vary from one company to another as well as with the service provider. In general though, they are well known for giving guidance, advice and recommendations to their clients regarding the use, design and the function of the business intelligence dashboard that they are exploiting. Availing of the said service will truly benefit your company. First is the fact that the pieces of advice that you gain are those that come from experts in the field. Even though this is the case, you should keep in mind that there are predators out there waiting for you to pay for their bogus services. This is why you have to be careful and research first so that you will not be tricked by these kinds of people.

The people behind the reliable company that offers BI dashboard consultancy can help you particularly for those that make use of an executive dashboard, which is the most common dashboard for businesses. They focus on the needs of your company including both for the business and the technology of the organization. On your part, it is important that you know the demands and the requirements of your company so that the consulting services provider will be able to help you deal with your business situation.

After providing them with the details of your company, you can now choose the kind of service that you need and also discuss other matters including the cost and the team that will be working with you. Among the services that are commonly offered by them include the assessment of your current business intelligence solutions, designing the data warehouse, identification of the data source, data cleaning, assessment of the quality of the data and the management of metadata. With the great number of services that they present to their clients, you will be able to find the ones that suit your needs.

With the consulting services for business intelligence, you will be able to explore and ensure that the key areas of your company will be taken care of. These include the identification of the key performance indicators and the other supporting indicators that will keep you up to date regarding the condition of your company. You will also benefit from the fact that they ca help you optimize the data gathering processes and also assist you in assessing the systems so that you can manage your company's performance effectively.

Interview With Avinash Kaushik, Author of Occam's Razor

2011-02-06T21:22:00.001-08:00

Sonicko President Jeff Lawrence recently sat down with Avinash Kaushik, author of the popular web analytics blog Occam's Razor about his views on web analytics, what he hopes to see from Microsoft's upcoming web analytics application, and Web 2.0 technologies. Avinash has also authored an upcoming book Web Analytics: An Hour A Day which you can preorder on Amazon.

1. You've been in the field of web analytics for quite some time, did you just wake up one day and think to yourself that this is something that you wanted to do, or were you thrown into the role and simply adapted to it?

At my last job with DirecTV, Sr. Manager for Enterprise Analytics, I had small amounts of exposure to Web Analytics (someone supplied log file parsed numbers into the dashboard). When I interviewed for the job at Intuit (Manager for Web Analytics) I was quite excited about the possibility of taking all my experience in Decision Support and apply it to a 100% exclusive web environment.

There is something so beautiful and scary and challenging and fun about data on the web. It was too hard to pass up. But it would be fair to say that when I took the job at Intuit I had no idea what "web analytics" was, I had not yet had the fortune to have used any web analytics application. Blaire Hansen, my hiring manager, certainly made a huuuge leap of faith in hiring me.

It has been a amazing ride and yes to answer your question I have simply adapted to it, but since my post MBA experience has been almost solely focused on Decision Support Systems I think I have brought all the learnings from traditional data warehousing and business intelligence and applied it to my current role.

2. What problems if any do you foresee with the implementation of Web 2.0 technologies such as AJAX and the explosion of tab based browsing? Are you concerned about problems of people keeping tabs open when they are not actively browsing the site?

I have blogged about the fact that slowly but surely the page paradigm is dying. That is not saying that the big problem is that the page view metric is going to be crap. It is more that currently almost all web analytics applications are constructed, from an architecture perspective, on the fact that a page view has to happen and all things go from there.

Benefits of Data Warehousing

2011-01-19T02:54:00.001-08:00

By using a data warehouse you are in effect providing a common data model for all data that is of interest without having to be concerned about the origin of the source. This ability makes it far easier to compile a report and analyze all available information than it would be if you had to retrieve information, invoices, ledgers, orders and so on from a multitude of data models.

Before the data is loaded into the database it has to be cleansed, in effect this involves identifying and resolving all the dirty/inconsistent data. By doing this before the data is loaded into the database the actions of reporting and analyzing data are greatly simplified.

Once it has been loaded into the data warehouse the information is under the control of the user. This allows for the information in the warehouse to be stored safely for extended periods of time, even if the source system data is purged over time,

Another great advantage of using a data warehouse is that they are separate from any operational systems. This means that they can perform retrieval of data operation without interfering with any operational requirement of the system.

Data warehouses is also able to work in conjunction with, and by doing so enhance the value of, operational business applications, the most notable of which being customer relationship management (CRM) systems.

Data warehouses are generally used to facilitate applications associated within the decision support system. These applications will deal with reports concerning trends, for example what was item had the best sales in what area over the last year, exception reports, and reports that show actual performance versus goals.

Data Warehouse Manager Is A Proficient Career To Pursue

2011-01-19T02:53:00.000-08:00

These days a company lives and dies on how fast it can store, retrieve and analyze information. As such, database administration (DBA) has become one of the most important aspects of a corporation's MIS department. Because of this, a fast growing occupation inside the department is the Data Warehouse Manager.

The DB manager is the person responsible for the loading, storing, maintenance and extraction of all files stored within a business intelligence system. People in this field have done their share of time in the working world, and it's becoming quite common for them to not only have a few years experience under their belts, but also have gotten an MS in some form of Computer Science, with a strong emphasis on DBA.

Anyone interested in pursuing this line of work should be of a meticulous nature. They should also possess strong analytical and logical skills. Good communications capability is also essential as part of the job is explaining how the warehouse system works to those who aren't as adept at storing and retrieving data.

There are many good on campus and online colleges that offer good undergraduate programs in Computer Science and Information Technology. As businesses are the primary employers of warehouse managers, taking some basic courses in the field is oft times beneficial. Once past the basic core a student should try to put in some courses in on basic database construction and architecture.

Upon achieving a Bachelor's, it's time to specialize in Database Architecture and Network information systems. Another Master's level degree that is quite popular is an MBA with an MIS core. At the same time, it should not be too difficult to find a solid job as a database analyst or assistant DBA position. With some time at the company it should be more than willing to help pay the tuition for you sticking around.

Upon achieving either an MS or MBA, there will be one last step in the education process, and that's attending the last few courses or seminars needed to be certified as a Database Warehouse Analyst or Manager. Again, most companies include continued education benefits as part of their employment package. They really have to as this is a position where professionals have to constantly go back for the latest updates and innovations in the field.

Salaries are commensurate with this level of education. According to a recently published report inside the industry, a Warehouse Analyst earns $77,000 to $106,000 a year, depending on industry, location and experience. A Warehouse Manager earns even more, from $92,000 to $125,000 a year. Both jobs are expected to see their salaries grow over 4% in the next year alone. Those with knowledge of IBM DB2, SQL and Oracle are at a premium, and can earn an extra 10% for their knowledge.

Make Informed Decisions With Data Warehousing and Business Intelligence

2011-01-19T02:52:00.004-08:00

So just what are the benefits to be thinking about installing data warehousing and business intelligence systems for your business? Below we take a look at a just a number of these benefits.

Benefit 3 - As this differs from other operational systems you will find that it allows the users to retrieve data without it causing the operating system to slow down.

Data Warehousing Plays a Part in Strategic Decision Making

2011-01-19T02:52:00.003-08:00

I have to admit the term "strategic decision making" can conjure up a lot of different meanings to a lot of people. And the first one that springs to mind, but probably not of the decision makers themselves, is that it is one that involves spending a lot of money hiring a lot of people who are going to cause a lot of pain to a lot of people until the next strategic decision is made.

Now you must appreciate that strategic decision making is not one of the primary uses of data warehouses, thankfully because strategic decision making is not done that often. Most data warehouses are used primarily for post decision monitoring of the effects of these decisions. However, some data warehouses do get involved in strategic decision making and are usually very effective.

So how would you use use a data warehouse in a decision making exercise. Before you start one thing to bear in mind is that the life-span of systems for strategic decision making tend to be relatively short and the creation of "special" databases, modelling and formal reporting are very time consuming tasks.

The days spent using these systems can be counted on one hand, though the payoff can be better than those reporting system that have been used for years.

The time-scale you are given to produce the system can be anything from a long afternoon to several weeks. And with no time for formal interviews you become the business analyst. Requirements are usually gleaned from "business" meeting minutes and are usually ambiguous. You will be required to aggregate data differently, and combine data that never been combined before.

You are doing this so the decision makers can see a point of view that is not the common view of the business. Basically to see the business in a different perspective.

Much of the use of data warehousing for making decisions ultimately involves the use of user maintained spreadsheets. Data cleanliness is much less of a concern in strategic decision making. Analysis is often done with highly summarized data and the need for speed lowers the need for extremely clean data.

The information generated by the data warehouse has to be understood by people who do not have direct access to the data warehouse. Most users will want to communicate the information in printed reports created using a word processor, presentation tool, spreadsheet, or generated directly from the database.

Do not try to design your warehouse for every contingency that could occur in a strategic decision making exercise. You cannot possibly foresee everything that will be needed in these exercises.

The Scope in Data Warehousing Consultancy

2011-01-19T02:52:00.001-08:00

Data warehousing software's have a great demand in the market because of the ease that they offer to an organization for handling various projects. There are many data warehousing consultants present in the market, who offer customized solutions to clients. But what is the scope of being a data-warehousing consultant? Read the article.

The data warehousing market has many warehousing consultants who provide resources and services to meet the requirements of warehouse owners. These consultants provide totally integrated solutions that will help business owners in handling the company and database more efficiently.

The data warehousing vendors offer tools, technology and methodologies to help you in constructing, managing, using and maintaining software & hardware that are used in a data warehouse. They understand the business of customers and provide them with services that meet and even exceed their expectations.

The daily duties of a data-warehousing consultant are never the same. They design data extraction, transform and load routines. Using a programming language, script, tool or the combination of all the three helps in developing all this. Here are some duties of the consultant:

o A data warehouse consultant creates documents.
o Interfaces with other team members and the supervisors.
o Conduct different tests of data warehouse batches.
o Talk to users, business analysts, technical coordinators, server administrators, database administrators and many other people working on the system.

Do You Want to be a Data Warehouse Consultant?

The must-have technical skills a data warehouse consultant should have include a thorough knowledge of databases that are being used, the operating systems which are used, script that would be required to write code and expertise in a programming language. Another must have skill is the knowledge of data warehouse tool, which the organization would be using. If you have enough information of the tool, it will lead to a seamless merging of one technology with another to finish the ETL tasks. Some of the common tools are DOS batch script, UNIX shell scripts, DataStage, advanced SQL and Informatica.

Apart from this, one should also have keen observations skills to find out fine details or any mismatches in the numbers. Those with an ability to do analysis are also fit for the job of a warehousing consultant. However, there will be separate analysts for creating technical specifications and mapping all the documents.

Those new into the field of data warehousing will love the excitement of working on projects and in facing the challenges that are crucial for the growth of an organization. Most of the data warehousing projects will be fast paced, so you need to invest a lot of time and manage things properly. Handling a data warehousing project means working on multiple technologies and teams. A data-warehousing consultant gets good exposure and there is a lot of scope for newcomers in this field.

Can Data Warehousing Help My Business?

2011-01-19T02:51:00.002-08:00

Data warehousing in many ways has become an industry buzzword that seems to promise much and deliver little. A good part of this is due to the often lofty and nebulous benefits that data warehousing vendors tout. What most businesses really want to understand is can data warehousing help them and if so, in what concrete way.

Offload Resource Intensive Processing
One of the ways that data warehousing can help your business is by offloading the resource intensive processes of generating reports onto database systems that are not also responsible for real time transaction processing. I have seen many web applications completely bogged down by the management reports being generated by users. Often companies solve this by throwing bigger and more expensive hardware at the database when offloading the reporting functions would allow them to gain more benefit for less cost. This also makes it possible for users to write ad hoc reporting queries without the concern that they might bring the transaction processing system down from a poorly written query.

Preserve Historical Data
Data warehousing allows reporting systems to analyze and report on a larger volume of historical data. Most transaction processing systems periodically purge transaction data to save space and maintain performance. Moving this data into a data warehouse allows your business to generate reports over a larger period than what may be saved in the transaction processing systems. This allows you to spot trends over a longer period or analyze the state of various transactions at a point in time.

Merge External Data
This ability to analyze longer time periods to spot trends makes data warehousing particularly helpful to marketing departments. In fact, its most immediate benefits are actually realized by those responsible for marketing. Another way data warehousing makes this possible is by allowing for the merging of data from external data sources for analysis. You would not want to merge this external data in your transaction processing system but in a data warehouse, external data may provide better intelligence about your customers. For example, you might use purchased mailing lists to find which of your customers recently obtained new mortgages. If you marketed products that were of interest to new home owners, this would be a highly useful metric.

Restrict Data Access
Another way that data warehousing can help your business is by segregating roles and permissions in a more concrete way. You may have employees that should have access to data for reporting purposes but perhaps they should not have permissions or access to the transaction processing system. A data warehouse makes it easy to separate those users that only need reporting access versus those that have access to "real time" transaction data.

Data warehousing is one of those technologies that is often poorly communicated. In a concrete way, data warehousing can help a business by providing better marketing intelligence and by segregating the data for reporting from the data used for real time transaction processing. This is a strong benefit and one that is simple to understand. Why data warehouse vendors could not just say so, I will never understand

Data Warehousing and Business Intelligence

2011-01-19T02:51:00.001-08:00

Data Warehousing - Tom's Ten Data Tips

2011-01-19T02:50:00.002-08:00

Data Warehousing was an innovation from the 90's that promised to change the data landscape for good. How far have we come? Many vendors have entered the marketplace because it makes sense to bring together data from throughout the organization, and this will continue to make sense in the future.

How large the Data Warehouse market will grow nobody knows yet. But for sure it is still growing fast, and currently is estimated at 4,5 billion dollar per year (IDC).

1. Why Do Data Warehouse Projects Run Into Scope Creep?

To quote Bill Inmon (guru and author of several great books on Data Warehousing) "Traditional projects start with requirements and end with data. Data Warehousing projects start with data and end with requirements." As soon as the project gets under way, users will find new applications, and with it will come new requests for data. Interestingly, these projects often are justified by moving Q&R work away from the 'data people'. What we've seen is that the first thing that happens as soon as the project delivers is that more requests for special queries are submitted to these same 'data people'. This may appear to undermine the initial business case but actually signals the onset of value creation from the DWH project.

2. Star Schema Versus Entity Relation Model?

There has been enormous debate in the community about the merits of different data models. At the risk of over simplifying: ER models tend to have better performance (processing time) for the end user, and are often perceived as "easier" to understand by end users. Drawbacks are that ER models require more disk space, and, because of the intrinsic redundancy in the data, have consistency problems from a maintenance perspective. Having said this, the practice seems to be that often some combination of the two is unavoidable in the practical setting, despite preferences (ER or Star) of the chief architects. Overall, Star models seem to have gained the most ground.

3. The Importance of a Data Warehouse Business Case

Much has been written about the business case for a Data Warehouse. What goes in to a good business case? IT savings are ubiquitous in DWH business cases. The important point is to not limit this to 'pure' savings, but to connect to primary business processes as much as possible. As an example, faster turnaround cycles for list selections are fine (when quantified in hourly rates), but it is even better if the revenue from more customer acquisitions that follow from these selections can be tied in. Not only will the relation to revenue growth rather than savings make for a more balanced business case, more important is the intrinsic business buy-in that results from a direct connection to the company bottom line. These days, changes in legislation (in particular Sarbanes-Oxley) play a major role in justifying business cases. This may be either through a higher company valuation for its transparent information gathering, or, less sleepless night for the CEO, which is of course priceless...

4. Why Do Data Warehouse Projects 'Never' Go Wrong?

Actually, Data Warehouse projects do sometimes fail. But, they fail so rarely, that it is actually very hard to believe... Especially after having talked to so many disgruntled end-users. And there are many ways a Data Warehouse project can go wrong. Delivering on time, data administration issues, and unavoidable data quality issues in feeding systems. Corporate politics (see Tip 7) are probably the best explanation for this phenomenon of near 100% success rates on DWH projects. In my experience, the reason why a failure or 'semi-failure' can go unnoticed is either because senior management is not aware, or, let's say "unmotivated" to talk about misspending of company funds. As a result, not enough is learned. Maybe we as consultants have a stake in this as well, as this assures the industry plenty of ongoing business... J

5. What is Different About Warehousing Web Data?

Kimball & Merz (2000): "Although this clickstream data in many cases is raw and unvarnished, it has the potential of providing unprecedented detail about every gesture made by every human being using the Web medium". The subatomic nature of clickstream data poses unique challenges. There are fewer built in feedback mechanisms to ensure data quality, compared to other data streams. The relation between user mouse clicks and server log records is not as tight as in "traditional" transaction processing due to technical issues like proxy servers and caching. Because of these differences, IT people need to adapt to the web process flow, rather than having the process adapt to IT needs as is common for most other DWH interfaces.

6. Which Data Should Be loaded In The Data Warehouse?

The data that enter the DWH ultimately determine its place in the organization. A "let's load all data, to be safe"-attitude is a sure fire way to derail your DWH project. Choices as to what should and should not be included need to be made early on, to keep the project manageable. After proven success of the delivered, deployed, and profitably exploited DWH, there always will be funding somewhere to include previously ignored interfaces. Given the anticipated lifecycle of the DWH, it makes perfect sense to consciously exclude certain sources. The choice as to what data to include needs to be driven by business considerations, and in particular reference to the company bottom line. If it can't be shown how data will be put to use profitably, they stay out! See also tip #3.

7. Data Warehousing & Company Politics

Data Warehouses have an impact on the company bottom line. Hence, they are likely candidates for turf battles, and are also at risk of becoming "small change" in budget allocation negotiations. None of these considerations benefit corporate long term goals. Managing a DWH project is hard enough as it is, and budget issues shouldn't make it any harder than it already is. Because DWH investments are in the present and revenues lie in the future, it is even more important to secure funding through a sound business case and buy-in from the appropriate (high) management level. See also Tip #3. Access to data means power, and talking about power is one of the greatest management taboos, still around. Sensitive as they are, even budgets are more readily discussed...

8. Data Warehouse Projects Traps

Some commonly recurring 'roadblocks' on the path to timely delivery of a Data Warehouse project:

ETL processes have eaten up so much time (and still need "babysitters"), that little if any time is left to develop applications needed to exploit the DWH
Some data are needed, but turn out not to be unavailable, or not in a timely fashion
Maintenance required for tuning, indexing, and backup and recovery is severely underestimated
Different ways of calculating the same phenomenon lead to different results, and nobody is able to conclusively explain the difference(s)
The data that is loaded (and recombined) turn out to contain previously unknown inconsistencies in the source systems, the 'classic' data quality issues that trip DWH projects
Metadata were lacking, and developers spend inordinate amounts of time finding out what a field really 'means'

9. DWH Hardware and Software Go Hand in Hand

In Data Warehousing, it is not about hardware, and not about software: it is about the perfect integration of these two. Those who begin their project from either end, will pay dearly for this mistake. Reasons are:

· in terms of price/performance, new, pre-integrated hardware-software combinations are taking the lead

· from a project management perspective, you never want to be caught between vendors when a proposed solution doesn't work as expected

· database tuning and indexing is very important and a hugely complex job, necessarily left to specialists (in-house trained)

10. Performance is Key

Although I don't often find technology factors to be this important, in Data Warehouse acceptance, no other factor will be as important as performance. As size increases over time, this factor becomes even more important. There are three reasons for this:

performance has a huge impact on the development speed (initial load is always very time consuming), and hence the overall maturity of the DWH at delivery time
performance can make or break end-user acceptance, in particular the predictability of performance
performance has a tremendous impact on end user productivity, the ultimate driver of the business pay-off

Source Data Warehousing - Tom's Ten Data Tips

Tom Breur: Biographical Sketch

Tom Breur is a consultant out of deep passion for his work.
He can be profoundly analytic, in his passionate quest to drive out the deepest business issues and the nexus point of a business model. It’s all about finding where the least effort will generate the most results.

Once the business challenge becomes clear Tom loves to roll up his sleeves and get his ‘hands dirty’.

Be it data analysis, market research, data mining or database work. Once the hands-on work gets started, his eyes begin to flicker, and he has a tendency to get carried away.

What makes a data warehouse important for a company is its ability to gather information from different parts and then making them a single part of a

2011-01-19T02:50:00.001-08:00

If you want to get information on all the techniques of designing, maintaining, building and retrieving data, Data warehousing is the ideal method. A data warehouse is premeditated and generated for supporting the decision making process within an organization. When the production databases are copied in the warehouse, it becomes easier to answer all the queries without hampering the consistency of the production system.

A data warehouse is actually a set of new concepts and important tools evolved into a technology. With the help of data warehousing, it becomes easy for an organization to counter all the problems faced during providing key information to concerned people.

Over the last two decades, a number of experiences and technologies incorporated together to evolve the new field of Data warehousing. You can say it as a well organized and resourceful method of managing & reporting data non uniform and scattered sourced throughout an organization.

Because of hundreds of gigabytes of transactions, it is necessary for a data warehouse to be vast. Therefore, "data marts" are often designed for individual department or a product line. A data warehouse system is an influential and necessary platform for merging data from old and new applications. You can transfer rules to a warehouse without making much efforts. The prime features of a data warehouse is that it records, collects, filters and provides basic data to different systems at higher levels.

Here are some of the benefits of a data warehouse:

o With data warehousing, you can provide a common data model for different interest areas regardless of data's source. In this way, it becomes easier to report and analyze information.

o Many inconsistencies are identified and resolved before loading of information in data warehousing. This makes the reporting and analyzing process simpler.

o The best part of data warehousing is that the information is under the control of users, so that in case the system gets purged over time, information can be easily and safely stored for longer time period.

o Because of being different from operational systems, a data warehouse helps in retrieving data without slowing down the operational system.

o Data warehousing enhances the value of operational business applications and customer relationship management systems.

o Data warehousing also leads to proper functioning of support system applications like trend reports, exception reports and the actual performance analyzing reports.

Precisely, a data warehouse system proves to be helpful in providing collective information to all its users. It is mainly created to support different analysis, queries that need extensive searching on a larger scale.

Understanding Data Warehousing

2011-01-19T02:47:00.000-08:00

What makes a data warehouse important for a company is its ability to gather information from different parts and then making them a single part of a centralized database. It is a collection of data, which is further used by employees for an easy and smooth working process. Know more about data warehousing by reading the article.

Data warehouse is an asset for an organization because it maintains the efficiency, profitability and competitive graph. A company collects data from sources like inventory manageable, call center, sales leads etc., which is then passed through the Data Life Cycle Management policy. It is this policy of the organization that determines the design and methodology of the data warehouse.

The main motive of a data warehouse is to create front-end analytics that will support the operation staff and other employees of the organization. Here are some of the elements of a data warehouse:

Pre-Data Warehouse

This zone provides data for the data warehousing and the team of designers filters out the data that contains business value for insertion. Operational data is stored in OLTP database, which resides in transactional software applications like supply chain, ERP etc. OLTP's are designed for high transaction speed and accuracy.

It is the metadata that ensures accuracy of data that will be entered into the warehouse. Most of the organizations reduce cost for the ETL stage by opting for a metadata policy.

Data Cleansing

Data cleansing is the extraction, transformation and cleaning process that are done to ensure the quality of the data before it is entered in the warehouse.

Data Repository

Data repository is a database where active data of an organization is stored. It is then optimized for data analysis.

There are two types of data warehouses - ODS and Data Marts. Although data marts are no different from data warehouses in physical terms but they can be smaller and are built on departmental level instead of company level.

One drawback of data warehouse is that it collects data and has older data as well, which means you will not get an up-to-date analysis. Operational Data Stores can be useful when it comes to storing recent data before migrating to the data warehouse.

Front-End Analysis

Front-end application that will be used by employees is the most critical part of a data warehouse. They will use it to extract information and interact with the data stored in the repositories.

Data Mining

It is the discovery of many useful patterns in the data. Data mining is used for analyzing and the classification process.

Data Visualization Tools

These tools are used for displaying the data from the data repository. Designers often combine it with data mining and OLAP tools. The process of data visualization helps users in manipulating data as per its relevancy and pattern.

Oracle Buys Carleton Corporation to Enhance Warehouse Offering

2010-08-24T05:13:00.001-07:00

"REDWOOD SHORES, Calif., and MINNETONKA, Minn., Nov. 9 /PRNewswire/ -- Oracle Corporation (Nasdaq: ORCL) and Carleton Corporation (Nasdaq: CARL) today announced that the two companies have signed a definitive merger agreement for Oracle to acquire Carleton, an early innovator of data quality and mainframe data extraction software for customer-focused data warehousing applications. The acquisition will be effected through a cash merger pursuant to which holders of Carleton common stock will receive approximately $2.45 per share or $8.7 million in the aggregate. The parties anticipate closing the transaction by the end of February 2000, which is subject to approval by Carleton's stockholders and certain other closing conditions". The offer values Carleton shares at $2.45 a share, which is less than the stock's closing price on Monday of $2.56.

According to Michael Howard, vice president of Oracle's Data Warehouse Program Office, "It increases our ability to provide better knowledge to customers; it's a big win for Oracle. There's a lot of dirty data about customers today; it's stored in a fragmented way. The Carleton software can clean it up. This is a top priority for e-businesses."

Carleton was founded in 1979 and marketed a product called Passport, which was an early leader in extract/transform/load tools for data warehousing. It was a strong contender with customers who had mainframe legacy data such as sequential files and VSAM. In 1997 Apertus Corporation acquired Carlton, and the new company became the Apertus Carlton Corporation. Apertus was a supplier of data cleansing and integration software. The product of this merger became Carleton Pure*Extract (data movement) and Carleton Pure*Integrate (data cleansing and validation). In September, Carlton secured a working capital line of credit from Silicon Valley Bank of Santa Clara, California in order to continue its growth strategy. Carlton had problems with their market capitalization, and their 52-week low stock price at the time of the announcement was $1.00 per share. .

Market Impact

TEC predicted Carlton would be acquired in the news analysis "Data Warehouse Vendors Moving Towards Application Suites", September 29, 1999. With this acquisition, Oracle gains access to both the data extraction/transformation and the data cleansing technologies. Oracle has traditionally had their customers use Oracle gateways to access mainframe data, so it is very probable that the data cleansing technology is what they were after for inclusion in Oracle Warehouse Builder. Carleton has been an obvious takeover candidate for some time because of their financial problems. Oracle is the second vendor to acquire data cleansing technology recently, as Ardent Software acquired Prism and its QDB data cleansing technology earlier in the year. More vendors will need to add this feature to their suite, so TEC reiterates the prediction that the remaining data cleansing vendors, such as Vality and Trillium, will be courted vigorously by the remaining extract/transform/load vendors such as Computer Associates and Sagent.

SOURCE:
http://www.technologyevaluation.com/research/articles/oracle-buys-carleton-corporation-to-enhance-warehouse-offering-15494/

SAS/Warehouse 2.0 Goes Live

2010-08-24T05:12:00.004-07:00

"CARY, N.C. (Feb. 24, 2000) - SAS Institute, the market leader in integrated data warehousing and decision support, has announced the production availability of SAS/Warehouse Administrator software, Version 2.0. Demonstrated at the Data Warehousing Institute conference in Anaheim, Calif., this new version provides IT the ability to proactively publish data warehouse information and track its usage, plus aggressively manage the process of change in the data warehouse."

"Data warehouses and data marts have become a vital component of all successful data mining, knowledge management, business portal, e-intelligence, customer relationship management (CRM) and supplier relationship management (SRM) applications today," said Frank Nauta, product manager for SAS/Warehouse Administrator at SAS Institute.

Nauta added, "Successful warehouses are continuously changing to keep pace with the changing business rules that they support. SAS/Warehouse Administrator simplifies change management by providing information delivery through e-enabled viewers like MetaSpace Explorer and integration with business intelligence and reporting tools. It helps make IT professionals more productive by giving them the ability to publish data from the warehouse - putting information in the hands of those who need it and freeing up IT staff for other projects."

Version 2.0 offers proactive information delivery with the addition of publish-and-subscribe tools robust enough for the entire enterprise, and offers enhanced intelligent access to data sources including SAP AG's R3, Baan, Oracle, DB2, Teradata, SQL Server 7.0, Sybase and many others.

SAS Institute was voted No. 1 in data warehousing and business intelligence in DM Review's 1999 Data Warehouse 100. SAS/Warehouse Administrator is a key component of this award-winning solution. Installed at more than 600 sites worldwide, SAS/Warehouse Administrator is the leading tool to help IT professionals meet the demands of administering a warehouse.

"IT and business users will find significant enhancements when building and designing warehouses and extraction, transformation and loading (ETL) processes," Nauta said. "By tracking the usage of information in the warehouse, IT staff can identify and remove data that is not being used. Removing unnecessary data makes the warehouse more efficient and maximizes hardware investments."

Market Impact

Publish/Subscribe metaphors are becoming much more common in the data warehouse arena. The ability for users to subscribe (request information on a regular basis), and for the server to publish ("push") that information automatically to those users is a powerful feature. Platinum Technology had already begun on an effort in this area, known as "Project C", which was delayed by their acquisition by Computer Associates.

It has been made clear by the success of products such as PointCast that push technology is an important aspect of information distribution. In addition, although SAS Institute has not been well known in the Extract/Transform/Load arena, they have a strong offering.

SAS has also formed a business intelligence alliance with IBM which may be leveraged by customers in the general data warehousing arena.

SOURCE:
http://www.technologyevaluation.com/research/articles/sas-warehouse-2-0-goes-live-15410/

Datawarehouse Vendors Moving Towards Application Suites

2010-08-24T05:12:00.003-07:00

During September, two more data warehousing vendors announced product suites that they claim offer broader integration between business intelligence, data movement, data cleansing and metadata management. BI vendor Cognos (NASDAQ: COGN) announced "Cognos platform", a tool to build complete "BI-ready data infrastructures". Data Movement vendor Ardent Software (NASDAQ: ARDT) announced "DataStage XE", which is designed to "simplify integration of multiple data sources and business intelligence tools".

"Cognos platform"

Cognos claims their new product is an "end-to-end platform, the first solution for building, managing and deploying BI solutions for enterprise and e-business needs" which "includes enterprise infrastructure layers that cover data mart building, integrated meta data modeling, and integrated security" as well as content management and distribution.

The product is a suite that contains:

*

Extract/Transform/Load capabilities for populating data marts. This technology was acquired with the purchase of U.K. based Relational Matters, which created the DecisionStream product, "the first OLAP-aware data integration product". DecisionStream is an ETL tool capable of populating OLAP hypercubes, such as the one used by Cognos PowerPlay.
*

Metadata management capabilities to provide centralized metadata and business rules. All applications in the suite share a common meta model.
*

Business intelligence tools in the form of Cognos PowerPlay and Impromptu. These products have made Cognos one of the top three business intelligence vendors.
*

Additional services such as data mart modeling and common security functions.

Ardent "DataStage XE"

Ardent has acquired a number of other software vendors in the last year, and is attempting to integrate the purchased technologies into their existing DataStage 3.6 product. Like Cognos, they are offering a suite that, in addition to providing data movement capabilities, offers metadata management and data quality management to "simplify integration of multiple data sources and business intelligence tools".

The product is a suite that contains:

*

Extract/Transform/Load capabilities in the form of DataStage 3.6. This product has made Ardent one of the top three ETL vendors.
*

Metadata management technology acquired with the purchase of Dovetail Software. This is a proprietary meta model, Ardent promises an XML-compliant repository in a future release. (For further details on metadata repository standards, see News Analysis "Is There Finally a Metadata Exchange Standard on the Horizon?" September 28, 1999).
*

Data quality technologies acquired with the purchase of Prism Solutions. Prism had previously purchased QDB Solutions, which created QDB/Analyze, a tool for complex data cleansing.
*

Improved mainframe functionality, also acquired with the purchase of Prism Solutions. Prism's mainframe based ETL tool provides Ardent with mainframe job scheduling and improved access to legacy data sources.

Market Impact

Data warehouse vendors are flocking to the suite concept. Ardent and Cognos have joined the ranks of larger vendors, such as Microsoft (with SQL Server 7.0) and Computer Associates (with DecisionBase 1.9), in attempting to provide end-to-end data warehousing solutions. Customers and prospects have been complaining for years about how hard it is to integrate ETL and BI tools into a functional whole that includes integrated metadata management. We believe that a truly integrated product would be a better solution than a suite, but suites are a step in the right direction. Users want "one stop shopping" with the ability to procure from a single vendor.

The current market direction indicates that more vendors will merge or be acquired as the larger players in the data warehousing space attempt to buy technology and fill in holes in their product offerings.

*

Ardent is the only ETL vendor that possesses a data cleansing tool, so we expect that data cleansing vendors such as Vality and Trillium will be targets for other ETL vendors. We believe the company most at risk is Carleton, which acquired data cleansing vendor Apertus in October of 1997. Carleton's (NASDAQ:CARL) stock price was recently at $1.75, and they recently had to secure a working capital line of credit with Silicon Valley Bank.
*

Cognos includes a complete BI solution with their product. Ardent has no business intelligence offering in the DataStage suite. However, Ardent has partnered with Business Objects in the U.K to provide a more complete solution in Europe. We expect to see this partnership extended to North America.
*

Neither vendor has adhered to the XML standard in their metadata repository. Both Ardent and Cognos will have to address this issue.
*

Ardent now has stronger mainframe capabilities because of the Prism acquisition, and already had a market-leading ETL tool. Cognos will have to prove the strength of its ETL solution and ensure that the tool can extract legacy data.

SOURCE:
http://www.technologyevaluation.com/research/articles/datawarehouse-vendors-moving-towards-application-suites-15485/

Can You Add New Life To an Old ERP System

2010-08-24T05:12:00.001-07:00

Recently, TEC featured an article by Olin Thompson titled, "The 'Old ERP' Dilemma: Replace or Add-on" which discussed options available to companies who want to add business functionality to their "Old ERP" systems. Certainly, there are many options now available in new business functionality that run the gamut from Supply Chain Planning (SCP) to Customer Relationship Management (CRM). The pros and cons of replacing or adding on to your existing ERP system were set forward in Thompson's article. But before you look to new ERP functionality, you should see if you are getting the full benefit out of your existing system. If not, are there ways to add new life to your current ERP system without going into an extensive development project.

Whether you have an old or new ERP system you have probably learned that to maximize its value, you have to work hard at getting information from the ERP system to key users. According to Thompson, " the data checks in, but the information can't check out of many ERP systems". You also may be finding that as e-business strategies are emerging in your supply chains, you could need access to more externally generated information than your ERP system, in its current configuration, can handle. For an Information Technology manager, both situations are problematic. Many companies should take another look at data warehousing before deciding that what to do with the "old ERP" system.

*In Memoriam

Does Data Warehousing Really Work?

Bob Cramer, Director of IT for Appleton WI based Anchor Food Products has found that, " lots of the pain we have with our old ERP system is based on users not having access to information. We see data warehousing addressing most of the problems our users have with the old ERP system". Today, most reporting from older ERP systems is directly from the ERP transaction processing (OLTP) system. Typically, users take ERP transactional data and input it to an Access database or a spreadsheet to generate the reports they need to make business decisions. From a user perspective, the extraction and re-inputting of information is both time consuming and potentially error prone. From an IT perspective there are no opportunities to build in validation checks to ensure that the information is either reliable or the most current available.

Data warehousing provides another way of getting information from legacy systems. Many companies have found it necessary to "build around" their ERP system to some extent. For example, Advanced Planning and Scheduling (APS) systems have often been added after the ERP installation. Companies find that they can report from either their ERP or their APS systems, but have difficulty combining data from both systems without having to create new databases or spreadsheets. Once the data is extracted from the systems, it is very difficult to ensure its integrity. James F. Dowling pointed out in the TEC article, "Business Basics: Unscrubbed Data is Poisonous Data" data should be managed as a corporate asset that appreciates in value over time. Historical data must be addressed with as much care as current database content".

The Data Warehousing alternative uses a better approach. It "packages" the information in data cubes that are customized for each group of users. Once the information is packaged in a data cube, users can extract the information using an On Line Analytical Processing (OLAP) tool. Today, OLAP tools are available as client-server applications or can be operated from a Web browser.

The data warehouse also can include information that is not in your ERP system. By adding information from outside the ERP system, IT can provide users access to ALL the transaction information that the company collects as well as whatever information they might want to collect from OUTSIDE the company. This is a significant difference and a potentially powerful advantage. Pat Clifford, Director of Business Consulting at the Boise ID agri-business giant the J. R. Simplot Company, found after installing a data warehouse comprised of company information from their ERP and several legacy systems, " it not only gave more information to our employees, but allowed them to move from just reading reports to performing managerial analysis."

What is the Best Way to Integrate Your Old ERP with Data Warehousing?

There are two basic strategies that can be used to start a data warehousing project. For certain ERP systems third party providers have developed 'off the shelf" data warehousing solutions that are pre-built to the fit the features of your ERP system. If you have an old ERP system that is supported by a data warehousing "solution", you should seriously consider this option. Data warehouse solution products are usually based on the ERP modules you have installed. You can roll out the data warehouse to one module at a time making it easier for IT to manage. One major advantage of using a data warehousing solution is that it can be done in a significantly shorter timeframe than if you have to buy an entire data warehousing tool set.

If your old ERP system is not supported by a data warehousing solution product, you will need to "build your own" using a tool set provided by a data warehousing vendor. At Simplot, Clifford found there were advantages in defining the project by functional areas instead of trying to create one big project: "Different functional areas look at information in different ways, so it's important to work with each group as you build the data warehouse". The advantage of a data warehousing tool set is that it gives you total control over what kind of information you want to present to your users. The disadvantage is that it will take more time and internal resources to implement.

Conclusion

IT managers are under increasing pressure to deliver information that can be used to perform managerial analysis. Decision makers in companies are no longer content to read the simple reports that are generated by old ERP systems. They need to have access to multi-dimensional information based on transactions generated both inside and outside your company. A well thought out data warehousing project can address many of the user issues behind their perceived need for a new ERP system.

SOURCE:
http://www.technologyevaluation.com/research/articles/can-you-add-new-life-to-an-old-erp-system-16444/

Computer Associates Splashes Into the Data Warehousing Market with Platinum Technology Acquisition

2010-08-24T05:11:00.000-07:00

In 1993, a vendor of artificial intelligence software named Trinzic acquired Channel Computing of Portsmouth, NH and inherited a product called InfoPump. InfoPump was a script-based data movement tool for the portion of the data warehousing market known as Extract/Transform/Load (ETL) tools, and was a market leader in the early 1990's. In 1995, Platinum technology International, inc. purchased Trinzic. The large influx of research & development capital from Platinum allowed the InfoPump developers to greatly enhance the 3.0 version of the product. Unfortunately, at the same time, market analysts began to predict the demise of scripted data-movement tools. The belief was that graphical tools were necessary to reduce the need for programmers and increase use by the business analysts who actually owned the data. In 1997 Platinum began development of DecisionBase 1.0, a combination of a GUI mapping tool (developed in-house), Platinum's Open Edition Repository version 1.6 (for metadata management, technology acquired by Platinum's purchase of the Reltech and Brownstone companies), and InfoPump 3.2 (for data movement).

DecisionBase 1.0 was released in March of 1998, but GUI mapping functionality was severely limited in the initial release. For instance, the mapper always assumed a row that was being moved to a target database was an insert and the code had to be manually modified to allow an update. By March of 1999, with release 1.9, a significant amount of new functionality had been added, including the ability to bulk-load data to Oracle and IBM UDB, support for Microsoft SQL Server 7.0, the ability to do pre- and post-processing, and the ability to modify the generated SQL from within the mapper.

InfoWorld Magazine favorably reviewed the product in March of 1999. The major drawback was the DecisionBase price tag, which was listed at approximately $200,000, depending on the number of metadata scanners and database interfaces necessary. This was the major contributor in the size of the installed base, which at last count was approximately 40 customers.

The market for ETL tools is expected to grow from $327 million dollars in 1996 to $620 million dollars by 2001, an increase of almost 90%. The number of vendors in the ETL market in the mid-1990's was small, comprised of basically four companies (Prism, Carleton, Evolutionary Technologies, Trinzic) plus some modest offerings from IBM. In the past four years, the space has become very crowded, with over fifty vendors competing in various market niches (e.g. specializing in access to VSAM databases). Four vendors still primarily control the general market, including Ardent, Computer Associates, Informatica, and Sagent, with some offerings from IBM and Oracle. Prism has merged with Ardent, Carleton with Apertus, and Platinum with Computer Associates. The major vendors are now working furiously to find a competitive differentiator, with the most popular differentiator being integration with Enterprise Resource Planning packages such as SAP and PeopleSoft.

Product Strategy and Trajectory

Computer Associates' DecisionBase strategy is to create a new product, to be known as "DecisionBase TND" (The Next Dimension). This product will include:

*

Integration with CA Unicenter TNG (The Next Generation) to provide integrated workflow services and replace Platinum's "Synergy" product. Unicenter will also provide high-speed data transport services, probably via CA's "TransportIT" product (probability 80%).
*

Integration with CA Jasmine TND Object Oriented Database. This relates to CA's "Aspen" repository effort, which is the new Microsoft Repository that Platinum co-developed with Microsoft.
*

Integration with Platinum's Forest & Trees product for data visualization. We believe portions of this integration are a result of an ongoing internal Platinum effort known as "Project C" (an integrated decision support infrastructure suite which would provide services that unify multi-vendor decision support tools)

Platinum technology's main competitive advantage is the metadata management aspect of the product, as it is one of only two true enterprise metadata repositories on the market. For customers interested in a true fully functional repository capable of functions like impact analysis and data rationalization, this is a strong selling point. However, for customers only interested in data movement, the price tag associated with the inclusion of the repository is a difficult sell. The other advantage is that the underlying data movement engine is the InfoPump product, which is feature-rich and has a very large installed base. Computer Associates appears to be positioning the product as a strategic end-to-end data warehouse solution. As stated in their press release: "A Data Warehousing strategy must be based on the most powerful technology for collecting all the information that might possibly be relevant, regardless of its form and its location - in-house, the Web or other public data - through a powerful infrastructure, and through intelligent data collection tools. And to collect, understand, to correlate and leverage this information, through comprehensive metadata management. And to analyze it, using intelligence technologies. And to present the results effectively, through sophisticated visualization technology. And to deliver it, through modern user interfaces and the Web." It is likely that this strategy will be refined over the next few years as CA acquires additional software companies and their technologies.

Product Strengths

*

The product exhibits strength in its metadata management layer, especially for companies with on-going metadata management projects. Platinum's repository technology was industry-leading, as evidenced both by its large installed base and the fact that it was chosen by Microsoft to co-develop the repository included in Microsoft SQL Server 7.0 (see http://msdn.microsoft.com/repository/). The repository technology is a significant competitive advantage with customers interested in metadata, since none of CA's closest competitors have a competitive offering.
*

Data Movement Engine: the InfoPump data movement technology is well known in the industry for its robustness, is very flexible and has a strong development language. It has the capability to handle virtually any datatype, including exact numeric and support for Binary Large Objects (BLOB's).
*

Database interfaces: Another key differentiator is the fact that the interfaces which interact with the source and target databases are written to the native C API ("Application Programming Interface" for the C/C++ programming languages) supplied by the vendor, where possible. Many other vendors use generic ODBC (Open DataBase Connectivity, a Microsoft technology that is becoming a de-facto standard) for connectivity. Using the C API (for example Microsoft and Sybase call theirs "DBLib" and "CTLib", Oracle's is "OCI") allows the interface to take advantage of specific vendor features, some of which are quite powerful. ODBC is the least common denominator, so these specific features become unavailable. The C API also allows for support of non-standard datatypes that the vendor may implement, while the ODBC standard only supports a limited number of datatypes.
*

Integration with CA Unicenter will provide CA with access to its large installed base. Depending on the pricing model CA chooses, customers may regard it as just another Unicenter "plug-in".
*

CA will probably improve the method currently used to access legacy "flat files". A more intuitive approach to mainframe and UNIX file systems would be a big improvement.

Product Challenges

*

Computer Associates only recently completed the acquisition of Platinum technology. Since CA previously had virtually no presence in the data warehousing space, and a great number of Platinum employees either were not retained by CA or left on their own, it will likely be some time before the CA sales force can effectively market the DecisionBase product.
*

Another consequence of CA's new arrival to data warehousing will be market perception. Customers about to set out on expensive, long-term data warehouse projects will want strong assurances that CA is committed to data warehousing, both from a research and development perspective, and from a product line perspective. Customers who have doubt about CA's commitment to the product will go elsewhere.
*

Platinum was tardy in porting DecisionBase to the most popular UNIX platforms (HP/UX, Sun Solaris, and IBM AIX), while DecisionBase's closest competitors are already strong on those platforms. CA's position on UNIX ports has not been publicly announced, and DecisionBase is only available currently on Sun Solaris, and only in beta.
*

CA also lags behind the competition in Enterprise Resource Planning (ERP) integration. The product does have the ability to read SAP data, but there is no SAP BW integration (already present in the Ardent product), and no integration with PeopleSoft (present in both Ardent DataStage and Informatica PowerMart). If industry rumors are correct and CA acquires PeopleSoft, integration may occur sooner, but as with all CA acquisitions, integration of the corporate cultures and products would be long and painful.
*

Release dates have been pushed back and many customers are frustrated. Since the CA acquisition of Platinum, a number of key employees have left the company or had their positions redefined. This, in conjunction with the fact that the product future has been completely changed, will make it difficult for CA to meet the announced "Q4 1999" beta release date.
*

Cost. DecisionBase costs a great deal more than many of the competing technologies (from 50% to as much as 500%).
*

Widgets and Wizards. Competing products have pre-defined "widgets" (code snippets to perform more complex processing) which can be dragged and dropped into the GUI. DecisionBase is currently woefully short in this area, and provides no pre-defined widgets with the shipping product. DecisionBase is also short in the area of wizards, assistants that walk the user through a "question and answer" session to help fill out complex dialogs.
*

Mainframe data access. Some competing products have embedded technology to access non-DB2 data residing on IBM MVS mainframes, as well as other systems, often without the need to convert the data into a "flat file". Ardent's acquisition of Prism should give them a leg up on CA in this area.
*

InfoPump upgrade path. There is no way to migrate existing InfoPump code into the repository. This means that existing InfoPump customers can not upgrade to DecisionBase unless they are willing to re-write all of the work they have already done.

SOURCE:
http://www.technologyevaluation.com/research/articles/computer-associates-splashes-into-the-data-warehousing-market-with-platinum-technology-acquisition-15224/

A Definition of Data Warehousing

2010-08-24T05:09:00.000-07:00

Bill Inmon
Bill Inmon is universally recognized as the "father of the data warehouse." He has over 26 years of database technology management experience and data warehouse design expertise, and has published 36 books and more than 350 articles in major computer journals. His books have been translated into nine languages. He is known globally for his seminars on developing data warehouses and has been a keynote speaker for every major computing association. Before founding Pine Cone Systems, Bill was a co-founder of Prism Solutions, Inc.

Ralph Kimball
Ralph Kimball was co-inventor of the Xerox Star workstation, the first commercial product to use mice, icons, and windows. He was vice president of applications at Metaphor Computer Systems, and founder and CEO of Red Brick Systems. He has a Ph.D. from Stanford in electrical engineering, specializing in man-machine systems. Ralph is a leading proponent of the dimensional approach to designing large data warehouses. He currently teaches data warehousing design skills to IT groups, and helps selected clients with specific data warehouse designs. Ralph is a columnist for Intelligent Enterprise magazine and has a relationship with Sagent Technology, Inc., a data warehouse tool vendor. His book "The Data Warehouse Toolkit" is widely recognized as the seminal work on the subject.

In order to clear up some of the confusion that is rampant in the market, here are some definitions:

Data Warehouse:

The term Data Warehouse was coined by Bill Inmon in 1990, which he defined in the following way: "A warehouse is a subject-oriented, integrated, time-variant and non-volatile collection of data in support of management's decision making process".

He defined the terms in the sentence as follows:

*

Subject Oriented: Data that gives information about a particular subject instead of about a company's ongoing operations.
*

Integrated: Data that is gathered into the data warehouse from a variety of sources and merged into a coherent whole.
*

Time-variant: All data in the data warehouse is identified with a particular time period.
*

Non-volatile: Data is stable in a data warehouse. More data is added but data is never removed. This enables management to gain a consistent picture of the business.

(Source: "What is a Data Warehouse?" W.H. Inmon, Prism, Volume 1, Number 1, 1995). This definition remains reasonably accurate almost ten years later. However, a single-subject data warehouse is typically referred to as a data mart, while data warehouses are generally enterprise in scope. Also, data warehouses can be volatile. Due to the large amount of storage required for a data warehouse, (multi-terabyte data warehouses are not uncommon), only a certain number of periods of history are kept in the warehouse. For instance, if three years of data are decided on and loaded into the warehouse, every month the oldest month will be "rolled off" the database, and the newest month added.

Ralph Kimball provided a much simpler definition of a data warehouse. As stated in his book, "The Data Warehouse Toolkit", on page 310, a data warehouse is "a copy of transaction data specifically structured for query and analysis". This definition provides less insight and depth than Mr. Inmon's, but is no less accurate.

Data Warehousing:

Components of Datawarehousing

Data warehousing is essentially what you need to do in order to create a data warehouse, and what you do with it. It is the process of creating, populating, and then querying a data warehouse and can involve a number of discrete technologies such as:

*

Source System Identification: In order to build the data warehouse, the appropriate data must be located. Typically, this will involve both the current OLTP (On-Line Transaction Processing) system where the "day-to-day" information about the business resides, and historical data for prior periods, which may be contained in some form of "legacy" system. Often these legacy systems are not relational databases, so much effort is required to extract the appropriate data.
*

Data Warehouse Design and Creation: This describes the process of designing the warehouse, with care taken to ensure that the design supports the types of queries the warehouse will be used for. This is an involved effort that requires both an understanding of the database schema to be created, and a great deal of interaction with the user community. The design is often an iterative process and it must be modified a number of times before the model can be stabilized. Great care must be taken at this stage, because once the model is populated with large amounts of data, some of which may be very difficult to recreate, the model can not easily be changed.
*

Data Acquisition: This is the process of moving company data from the source systems into the warehouse. It is often the most time-consuming and costly effort in the data warehousing project, and is performed with software products known as ETL (Extract/Transform/Load) tools. There are currently over 50 ETL tools on the market. The data acquisition phase can cost millions of dollars and take months or even years to complete. Data acquisition is then an ongoing, scheduled process, which is executed to keep the warehouse current to a pre-determined period in time, (i.e. the warehouse is refreshed monthly).
*

Changed Data Capture: The periodic update of the warehouse from the transactional system(s) is complicated by the difficulty of identifying which records in the source have changed since the last update. This effort is referred to as "changed data capture". Changed data capture is a field of endeavor in itself, and many products are on the market to address it. Some of the technologies that are used in this area are Replication servers, Publish/Subscribe, Triggers and Stored Procedures, and Database Log Analysis.
*

Data Cleansing: This is typically performed in conjunction with data acquisition (it can be part of the "T" in "ETL"). A data warehouse that contains incorrect data is not only useless, but also very dangerous. The whole idea behind a data warehouse is to enable decision-making. If a high level decision is made based on incorrect data in the warehouse, the company could suffer severe consequences, or even complete failure. Data cleansing is a complicated process that validates and, if necessary, corrects the data before it is inserted into the warehouse. For example, the company could have three "Customer Name" entries in its various source systems, one entered as "IBM", one as "I.B.M.", and one as "International Business Machines". Obviously, these are all the same customer. Someone in the organization must make a decision as to which is correct, and then the data cleansing tool will change the others to match the rule. This process is also referred to as "data scrubbing" or "data quality assurance". It can be an extremely complex process, especially if some of the warehouse inputs are from older mainframe file systems (commonly referred to as "flat files" or "sequential files").
*

Data Aggregation: This process is often performed during the "T" phase of ETL, if it is performed at all. Data warehouses can be designed to store data at the detail level (each individual transaction), at some aggregate level (summary data), or a combination of both. The advantage of summarized data is that typical queries against the warehouse run faster. The disadvantage is that information, which may be needed to answer a query, is lost during aggregation. The tradeoff must be carefully weighed, because the decision can not be undone without rebuilding and repopulating the warehouse. The safest decision is to build the warehouse with a high level of detail, but the cost in storage can be extreme.

Now that the warehouse has been built and populated, it becomes possible to extract meaningful information from it that will provide a competitive advantage and a return on investment. This is done with tools that fall within the general rubric of "Business Intelligence".

Business Intelligence (BI):

A very broad field indeed, it contains technologies such as Decision Support Systems (DSS), Executive Information Systems (EIS), On-Line Analytical Processing (OLAP), Relational OLAP (ROLAP), Multi-Dimensional OLAP (MOLAP), Hybrid OLAP (HOLAP, a combination of MOLAP and ROLAP), and more. BI can be broken down into four broad fields:

*

Multi-dimensional Analysis Tools: Tools that allow the user to look at the data from a number of different "angles". These tools often use a multi-dimensional database referred to as a "cube".
*

Query tools: Tools that allow the user to issue SQL (Structured Query Language) queries against the warehouse and get a result set back.
*

Data Mining Tools: Tools that automatically search for patterns in data. These tools are usually driven by complex statistical formulas. The easiest way to distinguish data mining from the various forms of OLAP is that OLAP can only answer questions you know to ask, data mining answers questions you didn't necessarily know to ask.
*

Data Visualization Tools: Tools that show graphical representations of data, including complex three-dimensional data pictures. The theory is that the user can "see" trends more effectively in this manner than when looking at complex statistical graphs. Some vendors are making progress in this area using the Virtual Reality Modeling Language (VRML).

Metadata Management:

Throughout the entire process of identifying, acquiring, and querying the data, metadata management takes place. Metadata is defined as "data about data". An example is a column in a table. The datatype (for instance a string or integer) of the column is one piece of metadata. The name of the column is another. The actual value in the column for a particular row is not metadata - it is data. Metadata is stored in a Metadata Repository and provides extremely useful information to all of the tools mentioned previously. Metadata management has developed into an exacting science that can provide huge returns to an organization. It can assist companies in analyzing the impact of changes to database tables, tracking owners of individual data elements ("data stewards"), and much more. It is also required to build the warehouse, since the ETL tool needs to know the metadata attributes of the sources and targets in order to "map" the data properly. The BI tools need the metadata for similar reasons.

SOURCE:
http://www.technologyevaluation.com/research/articles/a-definition-of-data-warehousing-16730/

The Necessity of Data Warehousing

2010-08-24T05:08:00.002-07:00

Data warehousing is an integral part of the "information age". Corporations have long known that some of the keys to their future success could be gleaned from their existing data, both current and historical. Until approximately 1990, many factors made it difficult, if not impossible, to extract this data and turn it into useful information. Some examples:

* Data storage peripherals such as DASD (Direct Access Storage Device) were extremely expensive on a per-megabyte basis. Therefore, much of the needed data was stored offline, typically on magnetic tape.

* Processing power was very expensive as measured in MIPS (Millions of Instructions per Second). Mainframes had to reserve most of their processing power for day-to-day operations, reports could only be run overnight in batch mode (without interaction from the user).

* Relational database technology was still in its infancy, and server engines were not powerful enough to support the data loads required.

* The type of programming that had to be done with third generation languages (3GL's) was tedious and expensive. Fourth generation languages were needed to abstract some of the required coding, but 4GL's were still in their infancy.

Most operational data is stored in what is referred to as an OLTP (On-Line Transaction Processing) system. These systems are specifically designed for high levels of transaction volume with many concurrent users. If the database is relational, it has probably been "normalized" (the process of organizing data in accordance with the rules of a relational database). If the database is non-relational, custom programs have to be written to store and retrieve data from the database. (This is often accomplished with the COBOL programming language). Whether relational or non-relational, the very design that makes an OLTP system efficient for transaction processing makes it inefficient for end-user queries. In the 1980's, many business users referred to their mainframes as "the black hole", because all the information went into it, but none ever came back out - all requests for reports had to be programmed by the Information Systems staff. Only "pre-canned" reports could be generated on a scheduled basis, ad-hoc real-time querying was virtually impossible.

To resolve these issues, data warehousing was created. The theory was to create a database infrastructure that was always on-line, contained all the information from the OLTP systems, including historical data, but structured in such a way that it was fast and efficient for querying. The most common of these schemas (logical and physical database designs) is known as the star schema. A star schema consists of facts (actual business facts) and dimensions (ways of looking at the facts). One simple way to look at a star schema is that it is designed such that the maximum amount of information can be derived from the fewest number of table reads. Another way to reduce the amount of data being read is to pre-define aggregations (summaries of detail data, such as monthly total sales) within the star, since most queries ask questions like "how many were sold last month?"

Data warehousing also led to the development of the concept of metadata management. Metadata is data about data, such as table and column names, and datatypes. Managing metadata makes it possible to understand relationships between data elements and assists in the mapping of source to target fields. (For more information of Metadata see "Metadata Standards in the Marketplace ")

Next came the creation of Extract/Transform/Load (ETL) tools, which made use of the metadata to get the information from the source systems into the data warehouse.

Additional tools, which made use of SQL (Structured Query Language), were developed to give end-users direct access to the data in the warehouse. As time went by, the query tools became user-friendly, and many now have a parser that can turn plain English questions into valid SQL. These end-user tools are now loosely referred to as "business intelligence" tools. In addition, there are other database constructs used to assist business intelligence tools in multi-dimensional analysis of data in the warehouse. These databases are referred to as hypercubes (also known as cubes, multi-dimensional cubes, or MDB's).

Since the early 1990's, data warehouses have become ubiquitous, technology and methodology have been improving, and costs have been decreasing. In 1998, data warehousing was a $28 Billion (USD) industry, and growing at over 10% per year. In addition, a recent survey of top IT executives indicated that data warehousing would be the number one post-Y2K priority. Data warehousing is now recognized as an important way to add business value and improve return on investment, if it is properly planned and implemented.

Selection Issues

Selecting a set of products for a data warehouse effort is complex. The first and most important issue is to ensure that the Extract/Transform/Load tool that is chosen can effectively and efficiently extract the source data from all the required systems.

The selection of the ETL tool requires an understanding of the source data feeds. The following issues should be considered:

* Many warehouses are built from "legacy" systems that may be difficult to access from the computer network. ETL tools often do not reside on the same machine as the source data.

* The data structures of the legacy systems may be hard to decompose into raw data.

* Legacy data is often "dirty" (containing invalid data, or missing data). Care must be taken in the evaluation of the tool to ensure it has an adequate function library for cleansing the data. Depending on the complexity of the cleansing required, a separate tool designed specifically for cleansing and validation may have to be purchased in addition to the ETL tool.

* The ETL tool should have a metadata ("data about data") repository, which allows the data sources, targets, and transformations to be tracked in an effective manner.

* The tool should be able to access legacy data without the need for pre-processing (usually with COBOL programs) to get the data into sequential "flat files". This becomes increasingly complex when working with filesystems like VSAM (Virtual Sequential Access Method), and files that contain COBOL Occurs and Re-Defines clauses (repeating groups and conditionally defined fields). It should be noted that a large percentage of the world's data is stored in VSAM files.

* A final issue is whether the ETL tool moves all the data through its own engine on the way to the target, or can be a "proxy" and move the data directly from the source to the target.

Selection of the business intelligence tool(s) requires decisions such as:

* Will multi-dimensional analysis be necessary, or does the organization need only generalized queries? Not all warehouse implementations require sophisticated analysis techniques such as data mining (statistical analysis to discover trends in the data), data visualization (graphical display of query results), or multi-dimensional analysis (the so called "slice and dice").

* Will the architecture be two-tiered or three-tiered? Three-tiered architectures offload some of the processing to an "application server" which sits between the database server and the end-user.

* Will the tool employ a "push" or a "pull" technology? ("Push" technology publishes the queries to subscribed users, much like Pointcast works, "pull" requires that the user request the query).

* Will the information be broadcast over a corporate intranet, extranet, or the Internet?

* How will the organization implement data security, especially if information is being broadcast outside the corporate firewalls?

SOURCE:
http://www.technologyevaluation.com/research/articles/the-necessity-of-data-warehousing-15998/

A One-stop Event for Business Intelligence and Data Warehousing Information

2010-08-24T05:08:00.001-07:00

The Data Warehousing Institute (TDWI) hosts its quarterly World Conference in cities across the US to help organizations involved in data warehousing, business intelligence (BI), and performance management, by giving them access to industry experts, and providing impartial classes related to topics pertinent to the industry. As the industry grows, organizations are faced with questions about how to best access their data to drive profits and meet goals and budgets. The need to understand data warehousing and the best means of leveraging data has become essential to developing a forward-looking approach to a BI solution.

This year, TDWI's summer event was held in San Diego, California (US) from August 20 to 26. Participants were able to take advantage of courses given by worldwide BI experts, as well as network with peers, have access to vendors and product demonstrations, and participate in one-on-one sessions with industry experts and instructors. The six-day event provided one-stop shopping for participants, who were able to take advantage of planned networking events, a two-day trade show highlighting various vendor offerings, and classes ranging from data warehousing testing techniques to best practices in performance management. The advantage of this one-stop shopping approach was that organizations had the opportunity to evaluate software, compare vendor offerings, and gain knowledge from other organizations that have implemented their own data warehousing environments.

The conference focused on five main themes, namely business analytics, leadership and management, data analysis and design, data integration, and administration and technology. These themes identify the main areas within data warehousing and BI, and provide the necessary knowledge related to the whole design and implementation process. A series of classes were offered in each area to allow users to focus on a specific industry aspect, or to gain an overall understanding of the sector and the different driving forces within it. Not only does TDWI focus on technology and the drivers associated with technological advances, but a key advantage to participating in the conference series is the additional focus on the business side of technology, and on managing the business processes associated with BI and performance management.

TDWI Overview

TDWI delivers research, education, and news, which enables individuals, teams, and organizations to leverage BI industry information to improve organizational decision-making, optimize performance, and achieve business objectives. One of TDWI's goals is to provide organizations with the impartial information required to make informed decisions. Although the organization runs events sponsored by various vendors, and provides users with product-related information, TDWI touts itself as being a central impartial resource for information. Business and information technology (IT) evaluators of solutions—whether in the requirements-gathering or enhancement phase of a current platform—can access a wide range of information, including classes, webinars, on-site training, and research.

TDWI has an international membership program, and provides industry publications and news, and a comprehensive web site. A division of 1105 Media, TDWI was created in 1995. It has over 5,000 members from Fortune 1000 companies, and includes both business and technology professionals. It is regarded as one of the central organizations for collecting data and providing insight into the world of data warehousing and BI.

TDWI collects and promotes best practices research to educate technical and business professionals about new BI technologies, concepts, and the approaches that have been applied in other organizations. This research also addresses significant issues and problems that organizations have experienced, and how they were handled. Many companies use TDWI's information to identify how they measure up to industry standards, how to take advantage of new or upcoming technologies, and how to address issues that relate to how they conduct business. The benefit of this information is two-fold. First of all, organizations can keep on top of enhancements within the industry, and can gain a wider knowledge base than that provided to them by their service provider (their selected vendor). Secondly, TDWI can help drive industry trends by leveraging the needs of organizations, as well as the way vendors should develop products to meet those needs.

TDWI's annual BI Benchmark Report identifies best practice metrics and compares TDWI's data warehousing maturity model to the industry. Many organizations consult this report to benchmark their BI use to ensure they are optimizing their implemented solutions, and discover ways to continuously improve their technical platforms and BI environments. This can include comparing their current environment with other organizations, or looking at information about other organizations within their vertical markets. TDWI also distributes other industry-related publications:

* The Business Intelligence Journal, published biannually, provides information and resources for BI and data warehousing professionals. The focus is on actionable advice on how to plan, build, and deploy BI and data warehousing solutions.
* Ten Mistakes to Avoid is distributed quarterly, and advises readers on different topics related to building, deploying, or maintaining a data warehouse, or managing a data warehouse team.
* What Works: Best Practices in Business Intelligence and Data Warehousing, also distributed quarterly, gives readers a comprehensive collection of case studies, questions and answers, and lessons learned from the experts.
* TDWI e-mail newsletters provide up-to-date news and industry commentary.

These publications provide users with continual information on the industry, and can help identify pitfalls in order to prevent them from making those same mistakes. Also, organizations that are in the same situations can gain insights on how to solve issues, as well as learn from other organizations and industry experts.

TDWI also develops webinars to discuss pertinent issues in the BI and data warehousing industry, and gives training at customer sites. TDWI seminars deal with the skills and techniques used to ensure successful implementations of BI and data warehousing projects. Overall, TDWI leverages its decade of experience within the data warehousing and BI industries to provide organizations with the information needed to make the best decisions possible. This way, organizations can access information that is industry-specific (without a bias towards one vendor versus another), and benchmark their own BI and data warehousing environments against organizations that have more experience implementing and growing these solutions. Also, organizations can compare and contrast challenges and issues as they arise.

TDWI Conference Tracks

Each of the quarterly conferences focuses on different tracks. These tracks present business and IT users with classes and seminars that highlight main industry trends, and provide a basis for enhancing their current data warehousing and BI environments (or aid in the requirements and selection process to implement such an environment). Not only do classes provide a wealth of information that can be justly described as verging on information overload, but in-class exercises, depending on the class, allow users to internalize the information to which they are being exposed. Aside from diverse and in-depth topics, the instructors are experts (whether within their respective industries, or their consultancy practices). Not only can users learn about the topics being presented, but they can also meet with experts to gain additional insight into topics directed specifically to their organizations.

Over fifty classes were offered during this summer's six-day conference. Topics ranged from data warehousing testing techniques to performance management benchmarking practices, in either full-day or half-day sessions. This allowed participants to learn about the latest trends, best practices, and industry insights on how to improve their current structure or enhance their technical platforms. Five tracks were presented during the event:

* Business Analytics
The business analytics track focused on both business and technical aspects of analysis. Topics included performance management, the definition and delivery of business metrics, data visualization, and the deployment and use of technology solutions. Solutions discussed included online analytical processing (OLAP), dashboards, scorecards, and data mining, as well as analytic applications. This focus allows organizations to gain insight into areas within BI and the different aspects of insight that analytics can provide. Organizations that require a subset of BI can identify how their needs can be met, by identifying requirements based on the topics presented. Additionally, they can take advantage of the trade show to identify those vendors that meet their needs, or those that (while not all-encompassing BI vendors) play in a specific space within the industry, such as data mining or data integration.

* Leadership and Management
The leadership and management track provided users with the insights needed to take a project from inception through to completion. Aside from identifying process and project management methodologies related to data warehousing and BI projects, emphasis was placed on the overall management of these projects. Ideas presented ranged from team building and the high level technical requirements needed to manage such projects, to other business areas such as customer relationship management (CRM) and supply chain management (SCM). This focus allowed users to identify a broad range of topics and considerations needed to implement and manage a data warehousing project through the systems development life cycle. Additionally, outside markets were identified to show the interrelation between BI and other industries. For example, many operational BI efforts are driven by SCM and the need to manage day-to-day decisions from the shop floor.

* Data Analysis and Design
A key focus of the data analysis and design theme centered on the skills needed to identify business needs and to transform those needs into data structures that are adaptable, extensible, and sustainable to the business unit. Course topics included needs analysis, specifications of business metrics, and data modeling. These topics and surrounding concepts encompass the backbone of developing a data warehousing and BI platform. Identifying business and systems requirements and translating them into the appropriate systems requirements is essential within any project. Within data warehousing and BI, it becomes more important as platforms are designed, and as business needs analysis has to be integrated into the actual design of the platform. Integration questions center on whether the current systems will integrate with the new software—and more importantly, how they will integrate.

* Data Integration
The theme of data integration included all the topics related to implementing a data warehouse solution. Included were data profiling; data transformation; data cleansing; source and target mapping; data cleansing and transformation; and extract, transform, and load (ETL) development. It is important not to underestimate the importance of data integration, as the way data is integrated into a data warehouse or BI solution is the essence of that system. If a scorecard is developed to measure an organization's sales metrics and the source data is not accurate, the key performance indicators (KPIs) set and reported on will be meaningless.

* Administration and Technology
The administration and technology track identified and covered topics related to infrastructure management, and the continued successful operation of data warehousing and BI solutions. The focus was on technology architecture, planning and configuration, system and network administration, database administration, and access and security administration. Maintenance of the implemented architecture and platform is essential to continued success in the data warehousing and BI environment. This section helped bridge the gap between administration and technology, and identifies the complexity of managing these two aspects of a data warehouse.

SOURCE:
http://www.technologyevaluation.com/research/articles/a-one-stop-event-for-business-intelligence-and-data-warehousing-information-18778/

Microsoft Goes Their Own Way with Data Warehousing Alliance 2000

2010-08-24T05:02:00.000-07:00

"REDMOND, Wash., Nov. 30 /PRNewswire/ -- Microsoft Corp. (Nasdaq: MSFT) today announced that 47 applications and tools from 39 top vendors throughout the industry have qualified for Microsoft Data Warehousing Alliance 2000. Alliance members and partners are committed to delivering tools and applications based on the Microsoft Data Warehousing Framework 2000, an open architecture for building business intelligence and analytical applications based on the open standards and services built into the Windows 2000 operating system, Microsoft SQL Server 7.0 and Office 2000. Application vendor membership for the Data Warehousing Alliance has more than doubled since it was originally announced in October 1998."

According to the release "organizations leveraging the framework and using alliance member products are better able to align local decision-making around key business drivers and harness the full potential of the web to win new customers, retain and extend customer relationships, and work more effectively with partners."

The architecture is based on OLE DB and the Open Information Model (OIM), in "recognition of the value and competitive advantage provided by the data warehousing services built into Microsoft products."

According to Microsoft, this technology is based on the Microsoft Data Warehousing Framework, which "is based on open, published protocols for interoperability and integrated end-to-end data warehousing services. It utilizes technologies provided in Microsoft Office 2000 and Microsoft SQL Server 7.0 products, and a partnership with Data Warehousing Alliance members for complementary tools and applications. The DWF enables data warehousing solutions where the data comes from virtually any source and where any type of information can be delivered to any compliant client interface or application."

Market Impact

Once again, Microsoft is using proprietary standards (OLE DB and OIM) to achieve its data warehousing goals. The more widely accepted standards are under the stewardship of the Object Management Group (OMG), which has over 800 members. OIM is a standard developed by Microsoft and turned over to the MetaData Council (MDC) which has "close to 50" members. For more information on the dueling standards bodies see "Is There Finally A Metadata Exchange Standard on the Horizon?", (http://technologyevaluation.com/news_analysis/09-99/NA_DW_MFR_9_28_99_1.asp,September 28, 1999). The alliance criteria require compliance with OLE DB for data access and the Open Information Model for sharing metadata. According to Colin White, president of DataBase Associates International Inc., "The Microsoft Data Warehousing Framework 2000 makes it easy to build Digital Dashboard applications integrating business intelligence, collaboration, and Web content right into the environment many knowledge workers live in: Outlook 2000."

This effort should make it easier for customers to integrate and use tools from multiple vendors, as long as their database is Microsoft's, and the other vendors are members of this alliance. The web component is to be provided by Microsoft's SQL Server 7.0, a component of the Windows DNA platform (Distributed interNet Architecture, introduced in 1997, Microsoft's umbrella term for its enterprise network architecture based on COM and Windows 2000 (NT 5.0)). The Windows DNA platform is advertised as "Microsoft's comprehensive platform for building Web applications."

We believe this will only serve to further fragment the data warehousing market. Obviously, Oracle is not a member of this alliance, and other applications show spotty representation. For example, in the enterprise resource planning space, Baan NV is represented, but SAP AG and PeopleSoft are not. In the area of supply chain analytics, the only vendor represented is Manuguistics Inc.

SOURCE:
http://www.technologyevaluation.com/research/articles/microsoft-goes-their-own-way-with-data-warehousing-alliance-2000-15412/

Netherlands

2009-12-03T22:19:00.001-08:00

Eindhoven, home of the Eindhoven University of Technology, has come a long way since its medieval past. It is the home of a cutting-edge biometrics facility owned by Royal Philips Electronics where Philips has incorporated finger-print recognition scanning into its line of flat-display monitors. Philips employs many expert cryptography professionals and earning a Ph.D. in cryptography at Einhoven University requires no tuition. Eindhoven's city slogan is "Leading in Technology."

Eindhoven (big round circular dark red area), which became a city in 1232, is a hotbed of technology in the Netherlands and is home to 32,000 companies. [See Figure 1]

Figure 1.

Source: TNO

Eindhoven is an important hub of European Internet Backbone connectivity and is one of the biggest technological centers in Europe. [See Figure 2]

Figure 2.

Source: Eindhoven ICT

8 CRM Essentials: An Executive Guide to the Must-have Elements of Every Successful CRM Initiative

2009-12-03T22:17:00.002-08:00

White Paper Description
It’s no secret that winning and retaining customers is the key to growth and success. But that’s no small feat, with ever-increasing customer demands, as well as the difficulty of implementing and enforcing processes to support your interactions with prospects, customers, and partners. Eight proven customer relationship management (CRM) best practices can help you create a customer retention strategy. Learn more.

Related Topics: Customer Relationship Management (CRM), Management Practices, Best Practices, Decision Making, Software Selection

Related Keywords: Salesforce, crm on demand, siebel crm on demand, customer service retention, customer retention strategies, customer retention marketing, customer retention programs, customers retention, customer retention, crm implementation

Ramco HCM Gets TEC’s Seal of Approval

2009-11-09T05:52:00.001-08:00

Ramco returned its request for information (RFI) for HCM, and as analysts, we compared it against the combined responses of other vendors within the HR space. Ramco responded to the RFI with almost total support against all criteria. There were several flags against various supported entries. (Flags are more often signs of superior functionality.)

TEC’s way of certifying any vendor—including Ramco—is fair and impartial. Over the years, TEC has created an RFI form tailored to HCM, which has been completed by many vendors of HCM products. TEC has a database of the responses that correspond to the RFI. Through the gathering of such data, we know statistically when an RFI criterion is industry “supported,” whether it is a “customization,” or if it holds some other status. In cases where the general industry response is different from the respondents, the software “flagged” that response.

The next step in the certification process is to verify the RFI by constructing a validation suite of criteria that the vendor must demonstrate. The “supported flag entries” mentioned above are also included. TEC sends this validation list (or demo script) to the vendor two weeks prior to the certification meeting. This validation list is required in order for the vendor to demonstrate that the level of support for a particular functionality is as it has claimed. We also include other items that are essential to HCM functionality. As analysts, we play the role of examiner, with a “show me the functionality; I want to validate your responses” approach.

Overall, the Ramco certification went very well. Ramco’s analysts demo’ed the product. There was not one aspect of HCM that they did not know or demo. Not once was there the response of “I’ll get back to you.” Every “flagged” item that we noted as supported passed the test.

In validating over 200 entries, we noted the following:

* The entire user interface is browser-based and very easy to use.
* The HCM product is very rich in functionality.
* The menus are well organized, with common functionalities grouped together.
* The detail screens were not cluttered, making them easy to read.
* Horizontal scrolling was used when an entry required more data than what a line could contain.
* Data screens were user-friendly and easily searchable.
* The general layout and functionality was impeccable.

Managing the Aches and Pains of Long Cycle Times: Automating Controls for Pharmaceutical Manufacturers

2009-11-09T05:51:00.001-08:00

One of the biggest challenges (or business pain points) for pharmaceutical manufacturers (or life sciences companies) is the long cycles that are required for research and development (R&D) and product approval. This is particularly a challenge for manufacturers of generic drugs, for which cycle times can average 20 months or more (and the full time-to-market period upwards of 12 years).

Why are long cycles a problem?

Simply put, it comes down to the familiar equation that “time = money.” More time needed means more capital spent, and manufacturers watch their bottom lines slip farther and farther away. To begin to formulate a plan to address the issue of long cycle times, it’s important to understand the factors that contribute to this challenge.

Long R&D cycles happen for a number of reasons. One is that there has been increasing need to comply with regulations, including the Food and Drug Administration’s (FDA’s) Title 21 Code of Federal Regulations (CFR) Part 11, for pharmaceutical manufacturers that are employing methods for electronic record-keeping and electronic and digital signatures.

This increasing need often means that additional administrative time must be spent on ensuring that the technical and procedural protocols are set up correctly and doing what they are supposed to do.

Another reason for long cycle times has to do with the need to ensure that all stages of product development are adequately documented for audits. Whether a manufacturer is using paper or electronic methods of data storage, there must be a reliable, consistent, secure, and accessible method of storing all documents related to the research, development, manufacture, and release of all drugs.

Every change to a document must be retained, and the integrity of the versions kept intact. For manufacturers straddling the line between paper-based and electronic methods, all paper-based documents need to be transferred and saved in digital form, a process that can require considerable time for scanning or manually entering data.

What are the business risks involved in longer R&D cycles and product approval?

Fewer products can be developed or manufactured concurrently, which means fewer products get to market. And fewer products to market can mean a decrease in the company’s in-coming cash flow (i.e. decreased profits). Additional worry may come from the fact that with this increase in time-to-market, other competing manufacturers may develop a similar drug and release it sooner, thereby further diminishing profits due to lost market share and a shortened product life cycle. A delayed or lengthened cycle time can seriously affect the return on investment (ROI) for a given new drug or product.

What can help?
A software solution that implements automated controls that address compliance issues, including 21 CFR Part 11.

How does 21 CFR Part 11 relate to product R&D and approvals?

For all of the processes involved in getting a drug to market, strict policies must be established and followed by a company regarding the use of electronic records. Each step of product R&D and approval processes must be, according to the dictates of 21 CFRR Part 11, consistent, reliable, and repeatable—in other words, each version of every document must be archived and easily retrieved for the purposes of inspection or auditing.

But this thorough documentation means that the approval process can be streamlined with automated functionality, as the time needed to send documents to the approving individual(s) will be reduced (with a centralized system, all users may have access to documents, providing they are authorized to do so according to level-specific electronic signatures; also, the system can be configured to send automatic notifications). Consequently, document turnaround time can be reduced, while the authenticity, integrity, non-repudiation, and confidentiality of documents is assured.

Ask the Experts: Data Purging and System Migration

2009-11-09T05:45:00.000-08:00

When migrating between systems, it is crucial to define the scope of implementation, as well as to outline each stage of the project and the resources that will be needed. A failed implementation will paralyze the operational capabilities of an organization, but the right methodology will help ensure a successful implementation.

In the issues related to the areas of ERP and PLM integration, we’ll highlight relevant areas of consideration. Furthermore, you’ll learn what steps can be taken to safeguard purging and data retention. This is a legal and mandatory business consideration.

We’ll assume for the purposes of this blog post that a new system exists, and that we are migrating data from an existing legacy system to a new ERP/PLM system. This can be viewed as an in-house system upgrade, or as migration of data from a purchased company.

Purging and Data Retention

When production databases become too large, they impact productivity by slowing access to information, and by extending the time required for system backups or for system restores.

Depending upon the industry (for example, medical, government, etc.), the need for data retention varies based on regulatory compliance. Some industries have long duration product guarantees, which results in the necessity to retain data.

Archiving has evolved into a discipline known as information lifecycle management (ILM). ILM helps organizations maximize the business management of storage from creation to disposal. Management is understandably reluctant to perform data purges due to the unknown operational risks, and it is therefore often done in stages.

Unstructured data populates file servers and typically includes e-mails, drawings, and user- and application-generated files in hundreds of unique formats. Purging can be by date, by type (internal or external), and by inbound or outbound status. Nevertheless, while many IT shops are only archiving e-mail to a less expensive tier of storage, they are still unwilling to permanently purge e-mail for legal or operational reasons.

The usual approach consists of transfer of data from active tables to online historical backups on a monthly basis. Since historical data is essentially invariant for long periods, it does not require being re-backed up if it had no changes. The backup facility may also make a second copy to non-rewritable storage. In the process of creating archives, an accompanying step is often taken to create summary data into a data-warehousing product for business intelligence studies. Summary data allows a look at a product’s sales figures for a given time period, by examining a single entry in a table rather then summing up individual sales order lines.

Oracle Announces a Database Fire Sale

2009-10-28T06:38:00.000-07:00

As stated by Ellison, "Our real competition is Windows NT, not SQL Server", the competing database product from Microsoft Corporation. Oracle is aiming at the midrange to workgroup market with this move, hoping to take market share away from Microsoft. These changes do not affect Oracle's named-user pricing model according to Ellison, but support pricing has been standardized at 7 percent of the software cost for continuous coverage. In addition, Ellison is hoping that all of Oracle's sales can be booked via the Internet within the next year, which would serve to reduce sales overhead. Oracle's stock has more than tripled within the last three months, reflecting investors' confidence in Mr. Ellison's strategies. The most interesting aspect of Ellison's strategy is to go after customers that don't store their data in a database, but in a traditional file system, such as Windows NT. As stated by Ellison "We're going after the NT file server market." If Oracle's stated strategy is successful, they could pick up a huge piece of business in the low to mid-range customer base.