The term data mining is often used when it comes to the storage and management of information in the big data area. Many companies use data mining as a tool by enabling the systematic application of computer-based procedures to find patterns, trends and relationships within large databases. It builds on various findings from the fields of computer science, statistics and mathematics by performing analyzes of databases. These analyzes pursue the goal of finding connections, patterns, trends and relationships between information within large databases and making them usable. Data mining works in a purely automated manner, which results in both cost and time savings. Companies can then use the results provided to make decisions about strategies or problem solving more easily. Functions Data mining is mostly used for the achievement of several goals by companies. In order to achieve these goals, it has to do a variety of tasks. This includes:…

In the context of big data, one always needs powerful platforms that can efficiently store a large amount of data. Such a platform is also called a data warehouse. This analyzes the information it contains according to certain patterns. Data warehousing process The data warehousing process, which is often used to describe how it works, comprises four main main steps for analyzing data by managing the data in the data warehouse and evaluating it for results. The 4-stage analysis process of a data warehouse Acquisition of data from the source systemLoading the dataBackup of the dataAnalysis and evaluation of the stored data This is how a data warehouse is structured A data warehouse, like a real building, is basically a construct made up of several elements. The foundation is an operational database that contains a large amount of information. The so-called staging area, which has the task of pre-sorting the…

Data processing in the area of big data often poses great difficulties for many companies. To counteract this problem, many organizations use tools such as software-based frameworks. These also include Hadoop, which is connected to Java. What is Hadoop The Java-based software framework Hadoop can most easily be imagined as a kind of shell that can be tailored to the most varied of architectures and operated by a wide variety of workers, in this case the hardware. The framework was invented by Doug Cutting, who developed Hadoop into one of the best projects in the field of the Apache Software Foundation by 2008. Cutting developed the software framework for better management of distributed and scalable systems. It is based on the MapReduce algorithm from Google, which uses Hadoop to combine large amounts of data in detailed computing processes on distributed but networked computers. Hadoop is not only so popular, but…

In the age of digitization and big data, a lot revolves around one thing: data. Terms such as data mart and data lineage regularly catch the eye. It is not always clear exactly what the technical terms are, which is why this article is intended to provide a brief overview. What is a data mart? Data marts are a kind of collection point for user-defined data. In doing so, data is extracted from large data stocks and made accessible in isolation for certain user groups. They thus form a sub-segment of a data warehouse and can help to make certain data accessible to the user more quickly and with less effort. This not only saves time but also costs. Data Mart vs Data Warehouse Both data marts and data warehouses are used to store and manage data records until they are used. Data warehouses are specialized in organizing the entire…

Big data analyzes usually require a large amount of data in order to capture and collect all information in its raw state. This data storage resembles a real sea in size, which is why the technical term “data lake” has been established for it. You can find out exactly what this is all about in this article. definition As a large data store, the data lake manages the entire mass of data in its original form, i.e. in its raw format. He makes use of the collection of information from a wide variety of sources. It makes no difference to the data lake whether the data has a structure or not. This large data store also does not require any prior validation or reformatting of the data. However, a data lake cannot manage number or text-based data. In addition, it can also save information from the media area, such as…

Big data is an important topic. That already shows a study by Bitkom . In 2018, the association surveyed over 600 companies on trending topics and found the following results: 57 percent are planning investments in big data or are already being implementedThe five top topics are big data (57%), Industry 4.0 (39%), 3D printing (38%), robotics (36%) and VR (25%)But: New concepts and possibilities such as artificial intelligence and blockchain have only rarely been used so far Reading tip: What is big data Implementation of big data only hesitantly According to the study, the potential of big data is only being used hesitantly. According to the study, the reasons for this are the requirements for data protection (63%) and the technical implementation (54%) as well as a lack of specialists (42%). Reading tip: What is a data scientist I am currently working on the technical implementation. In order to…

“Big data creates mixed feelings for many people. The economic opportunities are obvious. But the possibilities of abuse are also evident ( Computer week ) “. Big data is certainly more than just hype and brings numerous new opportunities with it. However, there are also many risks, which are discussed in this article.Reading tip: What is big data Big Data Risks: Monitoring An example can be found in the Computer week when looking for a perpetrator on the autobahn: “The investigators had installed cameras on seven relevant sections of the autobahn. These read in the license plates of all passing automobiles, including those of the vehicles being shot at. In April 2013, the police received reports of gunfire on trucks again within five days, a total of six. “Of course, this massive storage of data allows a high level of surveillance. The verdict of Computerwoche: “The evaluation of massive amounts…

Whether we drive a car, surf the Internet, take photos or videos with our smartphones or operate machines at work: data is generated. The amount is so gigantic that experts speak of “big data” (source: North Bavarian courier ). The chances of Big Data are great and it is certainly also a hype, but it opens up almost infinite possibilities. This article aims to take a closer look at the opportunities offered by big data. Reading tip: What is big data Data is the new oil – big data opportunities “The collection and analysis of data nowadays corresponds to the same principle as it did 100 years ago, during the oil boom. You try to tap into as many sources as possible in order to make a profit.” (Source Censation ). The magazine Google and Facebook also give examples of which companies buy just to get their data. “We are…