In the context of big data, one always needs powerful platforms that can efficiently store a large amount of data. Such a platform is also called a data warehouse. This analyzes the information it contains according to certain patterns.
Data warehousing process
The data warehousing process, which is often used to describe how it works, comprises four main main steps for analyzing data by managing the data in the data warehouse and evaluating it for results.
The 4-stage analysis process of a data warehouse
- Acquisition of data from the source system
- Loading the data
- Backup of the data
- Analysis and evaluation of the stored data
This is how a data warehouse is structured
A data warehouse, like a real building, is basically a construct made up of several elements. The foundation is an operational database that contains a large amount of information. The so-called staging area, which has the task of pre-sorting the information, finally rises from the foundation. Only after special ETL processes that collect, extract, transform and load the data according to a predetermined structure does the information finally reach the data warehouse. This enables separate access to data, independent of operational data stores. Finally, the information can be accessed with special data access tools. This is possible on different levels, the so-called data marts.
In order to obtain an even better structure with large amounts of data, so-called OLAP databases can also be used. These enable the consolidation of information from different areas and can efficiently map relationships and hierarchies.
However, it should be noted that every data warehouse is only as high-quality as the data on which it is based. Poor data quality or incomplete data stocks can lead to considerable problems in the analysis processes.
Data warehouse tasks
In the context of big data, it is now essential for companies to have an overview of the mass of information in order to be able to efficiently evaluate the stored data. For this reason, a data warehouse usually has four important tasks.
- Central collection of all data: Data is compressed at a collection point.
- Sorting of the data stocks: Separation into analytical and unprocessed data sets in order to obtain unadulterated results.
- Data integration: Combination of data from different sources in different formats into an evaluable model.
- Long-term storage of the data: Backup of the data in the form of a history for specific query options and time-related analyzes.
Advantages and disadvantages
A data warehouse is used by many companies as a helpful tool when it comes to storing large amounts of data. In addition to numerous advantages, there are also some disadvantages when using it.
benefits
- powerful function for storing large amounts of data
- special tools for the individual areas
- Data quality management
disadvantage
- sometimes long loading times (especially with increasing volumes of data)
- unstructured data cannot be processed (ins. films or audios)
- no possibility of real-time streaming
The following articles also provide more on the subject of data and big data:
- What is big data
- Big data: yesterday, today and tomorrow!
- Big Data Opportunities – Is Data the New Oil?
- Big Data Risks – A Question of Implementation!
Image source: pixabay.com
[fotolia]
