It builds on various findings from the fields of computer science, statistics and mathematics by performing analyzes of databases. These analyzes pursue the goal of finding connections, patterns, trends and relationships between information within large databases and making them usable.

Data mining works in a purely automated manner, which results in both cost and time savings. Companies can then use the results provided to make decisions about strategies or problem solving more easily.

Functions

Data mining is mostly used for the achievement of several goals by companies. In order to achieve these goals, it has to do a variety of tasks.

This includes:

classification : Object data are divided into classes.

Segmentation: Combination of feature-like objects into groups.

Forecast: Prediction of unknown or new features.

Dependency analysis: Knowledge of connections and relationships between features of objects.

Deviation Analysis: Identification of objects whose characteristics are not dependent on other objects.

Significance for big data areas

While big data often serves as a framework for data mining, the latter does not tend to be linked to it. Because data mining only describes an analysis of data stocks for characteristics and relationships of individual objects. It is often used in the context of large databases, such as in the area of big data, but it can also be used for smaller databases.

Nonetheless, it can be found far more frequently in the fume cupboard on big data and uses the technical basis to effectively obtain information from existing data. In addition to artificial intelligence, it also uses statistical algorithms. This enables more structure and transparency to be promoted and more relevant results to be delivered, especially with large databases that are often confusing.

Who can benefit from data mining?

Data mining is already used in practice in a large number of areas, as it offers great potential for users. For example, it is currently widely used in finance, marketing and medicine, and even as a tool for police analysis. But data mining is also used for improved customer service and risk analyzes, for example by banks and insurance companies. It can also be used to analyze the buying behavior of customers and is therefore also very popular in the area of online shops.

The following articles also provide more on the subject of data and big data:

• What is big data
• Big data: yesterday, today and tomorrow!
• Big Data Opportunities – Is Data the New Oil?
• Big Data Risks – A Question of Implementation!

Image source: pixabay.com

Der Beitrag What is data mining? erschien zuerst auf agile Companies.

Data warehousing process

The data warehousing process, which is often used to describe how it works, comprises four main main steps for analyzing data by managing the data in the data warehouse and evaluating it for results.

The 4-stage analysis process of a data warehouse

1. Acquisition of data from the source system
2. Loading the data
3. Backup of the data
4. Analysis and evaluation of the stored data

This is how a data warehouse is structured

A data warehouse, like a real building, is basically a construct made up of several elements. The foundation is an operational database that contains a large amount of information. The so-called staging area, which has the task of pre-sorting the information, finally rises from the foundation. Only after special ETL processes that collect, extract, transform and load the data according to a predetermined structure does the information finally reach the data warehouse. This enables separate access to data, independent of operational data stores. Finally, the information can be accessed with special data access tools. This is possible on different levels, the so-called data marts.

In order to obtain an even better structure with large amounts of data, so-called OLAP databases can also be used. These enable the consolidation of information from different areas and can efficiently map relationships and hierarchies.

However, it should be noted that every data warehouse is only as high-quality as the data on which it is based. Poor data quality or incomplete data stocks can lead to considerable problems in the analysis processes.

Data warehouse tasks

In the context of big data, it is now essential for companies to have an overview of the mass of information in order to be able to efficiently evaluate the stored data. For this reason, a data warehouse usually has four important tasks.

• Central collection of all data: Data is compressed at a collection point.
• Sorting of the data stocks: Separation into analytical and unprocessed data sets in order to obtain unadulterated results.
• Data integration: Combination of data from different sources in different formats into an evaluable model.
• Long-term storage of the data: Backup of the data in the form of a history for specific query options and time-related analyzes.

Advantages and disadvantages

A data warehouse is used by many companies as a helpful tool when it comes to storing large amounts of data. In addition to numerous advantages, there are also some disadvantages when using it.

benefits

• powerful function for storing large amounts of data
• special tools for the individual areas
• Data quality management

disadvantage

• sometimes long loading times (especially with increasing volumes of data)
• unstructured data cannot be processed (ins. films or audios)
• no possibility of real-time streaming

The following articles also provide more on the subject of data and big data:

• What is big data
• Big data: yesterday, today and tomorrow!
• Big Data Opportunities – Is Data the New Oil?
• Big Data Risks – A Question of Implementation!

Image source: pixabay.com

Der Beitrag How does a data warehouse work? erschien zuerst auf agile Companies.

What is Hadoop

The Java-based software framework Hadoop can most easily be imagined as a kind of shell that can be tailored to the most varied of architectures and operated by a wide variety of workers, in this case the hardware.

The framework was invented by Doug Cutting, who developed Hadoop into one of the best projects in the field of the Apache Software Foundation by 2008. Cutting developed the software framework for better management of distributed and scalable systems. It is based on the MapReduce algorithm from Google, which uses Hadoop to combine large amounts of data in detailed computing processes on distributed but networked computers.

Hadoop is not only so popular, but also because it is made available to everyone free of charge as free source code by Apache and is also written in the well-known Java programming language.

What role does Hadoop play in big data?

Hadoop’s expertise in being able to process large amounts of data, no matter what kind, in the area of big data, not only in a structured way, but also quickly, make the software framework an attractive tool for many companies. In particular, the ability to process data from different sources with different structures in parallel in a bundle in a clear and tangible way is a great enrichment, especially for organizations in the business intelligence industry.

In addition, with the help of Hadoop it is also possible to efficiently solve complex computing tasks in the petabyte area and, on the basis of this, for example, to develop new corporate strategies, to collect basic information for important decisions or to considerably simplify the reporting of an organization.

construction

Hadoop is made up of several building blocks which, when combined, make all the basic functions of the software framework possible.

These are:

Hadoop is made up of individual components. The four central components of the software framework are:

• Hadoop Common,
• Hadoop Distributed File System (HDFS),
• MapReduce algorithm
• Yet Another Resource Negotiator (YARN).

Hadoop Common is responsible for the basic functions and thus also serves as the basis for all other tools, such as the Java archive files. Hadoop Common is connected to the other elements via interfaces with defined access rights.

The Hadoop Distributed File System is used to store the individual data stocks on different systems. According to the manufacturer, the HDFS is able to manage data in the hundreds of millions.

Hadoop is powered by Google’s MapReduce algorithm. This enables the software framework to distribute complex computing tasks to various systems, which then process them in parallel. This can enormously increase the speed of data processing.

The MapReduce algorithm is supplemented by the Yet Another Resource Negotiator. The YARN manages the individual resources by assigning them to their tasks in the respective clusters.

functionality

As already mentioned, Hadoop is largely based on Google’s MapReduce algorithm. In addition, central tasks are also controlled by the HDFS file system, which is responsible for distributing the data to the individual bundle components. The MapReduce algorithm from Google, in turn, splits the processing of the data so that it can run in parallel on all bundle components. Hadoop then brings the individual results together to form a large overall result.

Hadoop divides the data volumes independently into individual clusters. Each cluster has a single master (represented by a computer node) while the other computer nodes are subject to the one in slave mode. The slaves serve as storage locations for data, while the master is responsible for replication and thus makes the data available on several nodes. Thanks to its ability to determine the exact location of a data block at any time, the master protects efficiency against data loss. In addition, he takes on the role of a supervisor of the individual nodes, who automatically accesses its data block if a node is absent for a long period of time and replicates and saves it again.

The following articles also provide more on the subject of data and big data:

• What is big data
• Big data: yesterday, today and tomorrow!
• Big Data Opportunities – Is Data the New Oil?
• Big Data Risks – A Question of Implementation!

Image source: pixabay.com

Der Beitrag Big data with Hadoop erschien zuerst auf agile Companies.

What is a data mart?

Data marts are a kind of collection point for user-defined data. In doing so, data is extracted from large data stocks and made accessible in isolation for certain user groups. They thus form a sub-segment of a data warehouse and can help to make certain data accessible to the user more quickly and with less effort. This not only saves time but also costs.

Data Mart vs Data Warehouse

Both data marts and data warehouses are used to store and manage data records until they are used. Data warehouses are specialized in organizing the entire data of a company, while data marts only organize collection points for the data of individual departments. They represent a tool that isolates certain data records and makes them available separately to the respective function field.

species

A basic distinction is made between 3 different categories for data marts.

Dependent

Dependent data marts are always directly related to an enterprise data warehouse in that they are developed according to the top-down principle. For this purpose, data is first combined at a collection point and then certain data records are extracted, which are then distributed in their intended data mart.

Independent

An independent data mart, on the other hand, is not linked to a data warehouse and thus forms an autonomous system. Data is obtained from internal and external data sources of an organization instead of from the collection point of the data warehouse and then specifically distributed to the individual data marts. This type of data marts is thus much easier to implement and particularly helpful when pursuing short-term business goals.

Hybrid

Hybrid data marts describe the connection of dependent and independent data marts in a system by obtaining data from a data warehouse as well as from internal and external sources of a company. This allows you to combine the advantages of both methods and create a complex but clear system.

benefits

Because of their function as an accelerator when accessing special data sets, data marts offer many advantages.

• Minimizing the time it takes to acquire certain data
• Ready for use much faster than an enterprise data warehouse
• Data marts require comparatively little specialist knowledge for implementation
• Inexpensive alternative to an enterprise data warehouse
• Data marts help improve the performance of a data warehouse because they can obtain data with less effort
• Thanks to the data mart, KPIs are easier to monitor
• Data marts support data maintenance by assigning data records to specific departments, which in turn can monitor them independently

What is data lineage?

Data lineage plays a role in connection with the origin of data, which is why the term is often used as a synonym. Data Lineage has the task of recording changes and optimizations of data as well as the development of their elements in a history. It tracks a data record on its journey from creation to adjustments to the final destination and at the same time also documents the associated properties. Simply put, data lineage can be thought of as a kind of biography of a data set.

benefits

With its function, data lineage offers many advantages for the user.

• Data can be fully monitored at any time
• Increased transparency about the development and history of data sets
• The quality of the data is retained
• Helpful when it comes to confidential data that needs to be protected
• Companies can use data lineage to more easily comply with data-based standards and regulations

Beyond data mart and data lineage

The following articles also provide more on the subject of data and big data:

• What is big data
• Big data: yesterday, today and tomorrow!
• Big Data Opportunities – Is Data the New Oil?
• Big Data Risks – A Question of Implementation!

Der Beitrag What is data mart and data lineage? erschien zuerst auf agile Companies.

definition

As a large data store, the data lake manages the entire mass of data in its original form, i.e. in its raw format. He makes use of the collection of information from a wide variety of sources. It makes no difference to the data lake whether the data has a structure or not. This large data store also does not require any prior validation or reformatting of the data. However, a data lake cannot manage number or text-based data. In addition, it can also save information from the media area, such as images and videos.

What appears to be a chaotic collection of data, however, follows a system. Because even if the data lake receives all information in its individual raw states, it structures it as soon as the data is required. Then, if necessary, he also initiates a restructuring of the data.

Use of a data lake

The many different ways of using and applying the information collected by a data lake, such as flexible analyzes, make the large data store extremely attractive. However, the application requires some requirements in order to be able to use the system optimally.

The most important basic function of the data lake is primarily to be able to collect and manage data from a wide variety of sources. By grouping all data in one place, data silos can be avoided and information is available more quickly. However, given the large amount of data, even a single storage space does not guarantee problem-free data management. Therefore, data lakes require common frameworks as well as the creation of protocols of the contained databases in order to bring more structure into the mass of information.

In the course of security and data protection requirements, additional access controls must be implemented and information encryption must be ensured. At the same time, data lakes should always enable a function of backing up and restoring data.

Advantages and disadvantages

The use of a data lake is particularly useful when large amounts of data are repeatedly generated that have to be managed. At the same time, however, such a large collection of information can also pose a number of hurdles.

benefits

• fast and uncomplicated data storage in raw format
• low requirements with regard to the required computing power
• provides the basis for detailed and content-rich analyzes
• many possibilities for the evaluation of data, since all data is collected without prior sorting
• Big data analytics can be a competitive advantage

disadvantage

• High requirements in terms of data protection and security
• Need for a complex data protection system
• Requirement of prior implementation of access rights and regular user controls

Conclusion

As you can see, a data lake is a real asset, especially for companies with large volumes. This is because, when used optimally, real competitive advantages can be achieved thanks to in-depth Big Data analyzes. At the same time, however, sufficient data protection must be ensured with regard to the amount of data. However, this sometimes makes the use of a data lake very complex.

The following articles also provide more on the subject of data and big data:

• What is big data
• Big data: yesterday, today and tomorrow!
• Big Data Opportunities – Is Data the New Oil?
• Big Data Risks – A Question of Implementation!

Image source: pixabay.com

Der Beitrag What is a data lake? erschien zuerst auf agile Companies.

• 57 percent are planning investments in big data or are already being implemented
• The five top topics are big data (57%), Industry 4.0 (39%), 3D printing (38%), robotics (36%) and VR (25%)
• But: New concepts and possibilities such as artificial intelligence and blockchain have only rarely been used so far

Reading tip: What is big data

Implementation of big data only hesitantly

According to the study, the potential of big data is only being used hesitantly. According to the study, the reasons for this are the requirements for data protection (63%) and the technical implementation (54%) as well as a lack of specialists (42%).

Reading tip: What is a data scientist

I am currently working on the technical implementation. In order to really use big data in a meaningful way, numerous technical requirements have to be created. I would like to give a practical example, which should serve as an impulse for practice.

Practical example: building a data lake

Due to my job, I am often involved in customer projects that want to set up big data architectures. In the following I made a kind of blueprint from the majority of the projects. I would like to introduce them to you today. To make the example clearer, I’ll write the whole thing as a case study.

initial situation

The customer is a fictitious large bank. Out different sources For example databases and data streams, the system copies data in raw form into a Staging area. A data stream is a continuous flow of data records, the end of which can usually not be foreseen in advance, e.g. transfers from a bank or payments to an account, as well as the heart rate monitor in the hospital or the temperature measurement of a weather station. Staging is used to ensure that the raw data is saved in its current form with a time stamp. The advantage is that these are still there in the event that the external data source is lost.

The staging area saves the data on different hard drives using a secure and redundant storage format. The data is converted into a uniform format in the Norming Area saved again. With the help of SQL queries data is exported in CSV format and saved in Sharepoint. Numerous external IT consultants use this to prepare weekly reports in PowerPoint and Excel. The reports are sorted in folders with the source data to be found on Sharepoint.

Summary of the architecture:

• Data comes from different source systems
• At the end of the day, the data is processed, standardized and recorded
• A wide variety of queries are made
• Reports are generated weekly by external service providers (MS Office and Sharepoint as storage location)

Now we come into play. The customer asked my team and me to design a new architecture. The reasons for this were:

• The reports require a lot of effort
• Little flexibility (especially AD-HOC requests)
• No versioned raw data
• High costs for external service providers
• Requirements regarding BCBS239 and MARISK can no longer be implemented (principles for the effective aggregation of risk data and risk reporting)

Target situation

Now we have started to redesign the customer’s architecture. In the first step, we revised the charging processes. On the one hand, we loaded new data into our lake every day (at night) through batch loading processes. On the other hand, the data streams are loaded continuously, in contrast.

swell

All of this data is saved on hard drives and given a versioning stamp. This concept is called data lake and metadata management. When it comes to the data lake, we are talking about a very large data storage medium, i.e. an oversized hard drive that accepts data from a wide variety of sources in its raw format.

Data lake

The data lake helped us to carry out so-called data lineage. Think of it this way: Data lineage is like a patient record with important information about when data was created, the current data age and changes. The presentation is usually very clear in the form of a diagram.

Advantage: In reports we were able to prove exactly on which data basis we produced them at time X. With the help of data lineage, we were then able to track the change in the data.

Our data lake was based on Hadoop with modern monitoring by Prometheus, Grafana and Icinga 2. We have set up high-availability clusters for this purpose. The Hadoop Distributed File System (HDFS) is a distributed file system which, using a MapReduce algorithm, can split complex and computationally intensive tasks into many small individual parts on several computers. This means that evaluations based on the raw data are possible during runtime.

By the way: Due to the guidelines of MARISK and BCBS 239 for banks, we have loaded risk data into a separate cluster. This cluster could only be accessed by authorized persons. There are concerns that so many cross-circuits are drawn in the lake through the combination of data that we have saved certain data for security reasons.

Now we want to process the raw data from the brine as well. First we have the data from standardized reports that are used every week or requested by certain departments as Data marts copied to extra hard drives. This enabled us to guarantee access control and improve performance. We thus had fixed (permanent) and volatile data marts (project-based).

Limitation : I realize that data marts are a concept of the data warehouse, but in our context they were really helpful because we cannot rely 100% on the data lake.

With the help of new Algorithms (internal algorithms of the customer) we normalized the data at runtime or for the data marts. With the help of software, we also tried to gain new insights and knowledge by forming clusters using artificial intelligence Models to collect from the data. For example, we had data correlated or investigated departure. This has been done by a special AI service provider. Another element is the data catalog. Of the Data catalog is a catalog of metadata and shows the presentation rules for all data and the relationships between the various data.

Note: The data catalog has the important function that no relationship can be established between certain personal risk data without access to the catalog.

evaluation

Now we come to the logic of evaluation . We want to ensure that the various stakeholders in the company simply adhere to three standards Query server can send. The requests are sensibly distributed by a load balancer and also protected by an access control. The three standards are:

• SQL
• Hive (SQL-like Hadoop compatible language)
• Tableau (software tool)

Reporting

Now we come to the actual report generation for the end user. We have our three groups in the company for this purpose. These are:

• classic controlling,
• the specialist departments (and project managers) and
• the data scientists.

All three roles can send requests to our Query server send. There is a possible evaluation for each role:

• Fixed and automated reports for controlling,
• Interactive and customizable dashboards for the specialist departments and
• Exploratory reports in Tableau for the data scientists.

In summary, there are the standardized automatic reports, which we could view in CSV or Excel, as well as interactive dashboards for individual real-time reports using our own software. Furthermore, a team of data scientists had the goal of using Tableau to gain new knowledge from data (exploratory reports).

Summary of the new architecture :

• Raw data is loaded into the data storage (Hadoop cluster)
• Map-Reduce algorithm intelligently distributes the evaluation
• Evaluations are made directly from the raw data layer
• Transformation always only at runtime
• Data catalog for storing data relationships
• Modeling through AI
• Query Server allows various queries in various languages
• Data lineage for versioning the data
• Automated reports and interactive / exploratory dashboards through in-house development

Conclusion

Big data can significantly change companies and is right at the top of the agenda. The main obstacles, however, are the technical implementation and preparation of the data. Classical concepts are no longer capable of upgrading such data and companies are required to save data in a targeted manner and deliver up-to-date reports under pressure due to legal requirements.

In this case study, I have given an example of technical implementation that can serve as an impetus for practice. I built a data lake in the case study and used various well-known concepts such as Hadoop. My experience shows that the correct implementation of the concepts can help to make the potential of Big Data possible. It is important to draw meaningful reports and insights from the data.

Reading tip: Big data benefits

Image source: Business photo created by mindandi – www.freepik.com

Der Beitrag Case study: building a data lake for the use of big data erschien zuerst auf agile Companies.

Big Data Risks: Monitoring

An example can be found in the Computer week when looking for a perpetrator on the autobahn: “The investigators had installed cameras on seven relevant sections of the autobahn. These read in the license plates of all passing automobiles, including those of the vehicles being shot at. In April 2013, the police received reports of gunfire on trucks again within five days, a total of six. “
Of course, this massive storage of data allows a high level of surveillance. The verdict of Computerwoche: “The evaluation of massive amounts of data has an ambivalent character.
An example can also be the storage of numerous data by wearables by health insurance companies. The health of the individual becomes transparent and it can be understood how often you do sport and move. So it creates a permanent feeling of surveillance.

Big data risks: Sensitive data requires special protection

“Dealing with large amounts of data, especially data about internal company relationships, poses technical challenges for companies and IT departments. The balance between accessibility and security must be precisely balanced. “this can be found in the BigDatablog . Sensitive data must therefore also be specially secured. Not only from abuse but also from manipulation.

Big data risks: manipulation

Another risk of big data is its manipulative use. For example, big data can be misused and voters can be influenced, as is the case with elections. It is therefore about the sensible and ethical use of the large amount of data.
So the magazine warns Propaganda show : “Critical contemporaries have always warned, but only now is it slowly becoming apparent – all the more powerfully – how data collections can and are already being used to target people who previously” thoughtlessly “made their thoughts available to third parties, specifically and unnoticed to manipulate.”
So the magazine continues to believe that “the data is today Not primarily used to analyze political moods and then to make laws and politics in the interests of the citizens’ known opinions and interests, but rather to manipulate the raised moods and opinions with targeted measures in the interests of the elites. “

Big data risks: uselessness

A final risk is that large amounts of data are stored and certainly also evaluated, but you also like to sit in front of it and think: Mhmmm? And what do we do with it now? What does that tell us? When I was still on the road as an external consultant, I experienced this very often. We have evaluated and thought about a lot of things, but hardly came up with a meaningful idea of what that could mean.
So he warns too Havard Business Manager : Ready and open to experiment: “Managers and analysts must be able to apply scientific methods in their business area. You need to know what reasonable working hypotheses look like. You also need to understand the basics of experiments and how they are set up “

Conclusion: Big data but with care!

In addition to the risks mentioned, it is important to use big data sensibly. For this, guidelines are certainly necessary on the one hand to change abuse and manipulation and, on the other hand, further training to prevent uselessness. Because there must also be a data culture in the company.
That’s what he says Havard Business Manager At the end of the day, great efforts are needed in further training so that Big Data leads to more added value. It is about promoting a data-oriented mindset and analytical culture in the company and introducing new technologies.
Tip: Book suggestions too Big data
Of course, not everything is just a risk and big data can offer numerous opportunities such as well-founded prognoses for decisions and new products as well as an individual approach to customers. So read my follow-up article on this next week.
Reading tip: Chances of Big data

Der Beitrag Big Data Risks – A Question of Implementation! erschien zuerst auf agile Companies.

What should a data scientist be able to do?

If you look at the job advertisements, a data scientist should usually be able to do the following:

• Analytical talent
• Expertise
• communication
• Urge to research
• Coordination talent

On the one hand, the scientist must recognize relationships in large amounts of data and be able to analyze them. Furthermore, he should also have business and specialist knowledge in order to understand the problems of the specialist areas. With the help of his communication skills, he can talk extensively with them and be in constant contact. His curiosity also allows him to solve difficult problems and work with hypotheses. But he is also a project manager and has to expand, manage and maintain the database.

What does a data scientist do?

There are many names for the new job profiles. The Computer week has already broken down some of it and defines it as follows:

• The (Big) Data Engineer is the master of the data supply.
• The management scientist is the mediator between the departmental worlds.
• The data scientist provides answers to analytical questions based on data.
• The data steward is responsible for monitoring data quality and integrity.

The data engineer is responsible for merging the data and knows where the data is and how it is merged. The management scientist then analyzes this data and defines the actual problems. She says to the data scientist Computer week : “The main task of the data scientist is to generate answers to analytical questions from data – with the help of analytical methods from the areas of statistics, machine learning or operations research.” The data steward monitors the data going there and ensures that it is technically correct.
So in the end there are numerous job descriptions and these terms are not always used and clearly separated. Therefore this is only used as an orientation. I don’t think that every company separates this so strictly and often everything is summarized under the umbrella of the data scientist or data analyst. The following figure shows the fields of application and provides the answer to the question: What does a data scientist do.

Conclusion: what does a data scientist do?

There are numerous terms and answers to the question: What does a data scientist do but in the end an abstract picture emerges. On the one hand, the data scientist is a scientist who solves business problems with the help of data and, on the other hand, he is a project manager who manages the database in the company.
I hope to have shed some light on the darkness with this article and I look forward to your comments on what the data scientists are doing in your environment. I have already worked with a few during my consulting time and can also see what my big data colleagues at the chair are researching. I also learned something from you and one quantitative analysis using data used in my doctorate.
Tip: Book suggestions too Big data
[werbung]
Verwendete Quellen anzeigen

[fotolia]

Der Beitrag What does a data scientist do? erschien zuerst auf agile Companies.

Data is the new oil – big data opportunities

“The collection and analysis of data nowadays corresponds to the same principle as it did 100 years ago, during the oil boom. You try to tap into as many sources as possible in order to make a profit.” (Source Censation ). The magazine Google and Facebook also give examples of which companies buy just to get their data. “We are trying to evaluate a large number of touchpoints and use the data obtained from them to create prognoses for action and consequently to derive new products and services. ” (Source Censation ).

Addressing customers

Big data makes it possible to understand customers better and then to address them in a more targeted manner. Customers are now on the move on smartphones and tablets. Mobile and at any time. Traditional messages are no longer up to date. A customer on the smartphone (possibly on the train) must be addressed differently than a customer on the PC at home at his desk.
Data can help to guarantee the precision and the addressing of customers in real time. Knowledge of the purchase history can also bring significant added value. These are all data that are often already in the company but are currently not used to address customers.

Forecasts

Big data can be used to make more accurate forecasts. Of course, reports have been created for a long time and decisions are made based on them. But using new methods like qualitative evaluation direct knowledge can now be obtained from this report.
With the help of these forecasts, more informed decisions can be made and, coupled with a pinch of gut feeling and some experience, this decision can often be very good. So it can be a good addition. In particular, the connection between variables can be useful for a decision: Do customers buy more who order in the morning or who came via a landing page? These are connections that can perhaps only be revealed by a big data tool.

development

Big data can be used to create evaluations, the results of which flow into the development of products and services. For example, vehicles now generate a lot of data that improve the driving experience, or Spotify recommends the music you want to listen to. Spotify checks whether you prefer to listen to quiet music in the evening and rock’n’roll in the morning to wake up.
With the help of these examples, a product can be improved, but also completely new products can be created. In this way, apps or products can be completely personalized. But new products such as data analytics tools or distributed data storage systems are also coming onto the market. The job of data analyst is now also sought and if you look at the job profiles, from the Excel table creator to the server administrator to the software developer, almost everything is included.

Conclusion

“Business e-mails, networked company vehicles, incoming orders in online stores, keyword scanning in social media groups, the acquisition of location-based data, video recordings from surveillance cameras, measured values from machines – countless data is generated every second in a company.” (Source T-Systems ).
Tip: Book suggestions too Big data
This potential can be used on the one hand to address customers more specifically, to make forecasts for well-founded decisions and to develop new products. All of these are opportunities for big data and enable companies to stand out from the market.
Reading tip: Big data risks
Big data therefore offers great potential for companies, but of course also harbors risks such as bad investments or misuse. In order to read the other side as well, I therefore recommend taking a look at my follow-up article.

[werbung]
Verwendete Quellen anzeigen

[fotolia]

Der Beitrag Big Data Opportunities – Is Data the New Oil? erschien zuerst auf agile Companies.

What is big data

If you enter the search term on Google, you will find the following definition of ReserachEnterpriseSoftware : Big data is a general term used to describe the large amounts of unstructured and semi-structured data that companies produce on a daily basis. This data takes a lot of time and money to load into a relational database for analysis.
That too Gabler Business Lexicon provides a definition: “Big Data” refers to large amounts of data from areas such as the Internet and mobile communications, the financial industry, energy industry, health care and transport and from sources such as intelligent agents, social media, credit and customer cards, and smart metering -Systems, assistance devices, surveillance cameras as well as aircraft and vehicles originate and which are stored, processed and evaluated with special solutions.
A great explanation can also be found at chip :

• A large amount of data is referred to as big data if the volume is too large or too complex is to process them by hand. This is especially true for data that is constantly changing.
• Big data can be harmless data from climate research. However, data about people are also collected: communication behavior, consumer behavior or surfing behavior of Internet users. You can see the effects of big data analysis every day on the Internet . A typical example is personalized advertising.

In conclusion, one can say that big data is initially large amounts of data that are unstructured and cannot be evaluated by hand.

Big data – pros and cons

Large amounts of data offer our society a number of new possibilities, at least if we are able to evaluate them. So I have at dice found some great examples which I would like to show.
Crimes can also be fought better on the one hand. As Dice says, it may not be quite like the movie Minority Report, which shows a society where police officers successfully arrest individuals for crimes they did not commit. But in fact, massive amounts of data are already helping regional and local authorities identify difficulties before they can become a major problem.
Furthermore, diseases can be predicted. As dice says: Predicting events based on existing data could offer individual medical care for each patient. By analyzing digitally recorded medical data and similar disease courses from patients, personal disease risk profiles can be created. Doctors could then prescribe preventative treatments or review related symptoms.
Another great example is Netflix. At dice, for example, the following can be found: For Netflix, the visualization of data is of the utmost importance in order to be able to continue its success story. Netflix can use data calculations to determine what viewers want to see and how they would like the content to be presented.
So there are numerous great examples. For more you can just click dice Continue reading. In the following, however, big data may not always be a success factor, which is why I would also like to present the downsides.
So says the magazine CIO, the criticism of big data “is probably due to the “algorithm weakness” widespread among machines, i.e. the inability to draw the right conclusions from a lot of collected information. That being said, there are two main reasons why companies do not benefit, or not enough, from big data. The first: With the help of data analysis, you will come to results that you could have had with less big data. “
The second: Big Data produces results and ideas that, for whatever reason, cannot be implemented in practice. A large US retailer had found in a model test that sales increase if you put a special offer product on the shelves a while before it is cheaper and leave it there when the offer price is no longer valid ( CIO ).
In this paragraph we have now seen some good examples of big data, but also some that may not be quite as meaningful. So, as always, it is up to us that we use big data correctly and not just do it because it is just hype.

AI, deep learning and machine learning

One potential that the mastery of large amounts of data brings with it is the ability to give machines an intelligence, i.e. artificial intelligence. A machine could evaluate the controlling report itself and initiate measures, right?
On the one hand, you can find the term artificial intelligence. So says t3n : All technologies used in connection with the provision of intelligence services that were previously reserved for humans can be found under the generic term AI.
There is also the possibility of machine learning: machine learning describes mathematical techniques that enable a system, i.e. a machine, to independently generate knowledge from experience ( t3n ). However, deep learning goes one step further: “Deep Learning” with artificial neural networks is a particularly efficient method of permanent machine learning based on statistical analysis of large amounts of data (big data) and the most important future technology within AI ( t3n ).

Conclusion: what is big data?

Big data is a lot of unstructured data that we cannot evaluate by hand. Mastering such amounts of data can lead to great potential, but it can also “backfire”. The potential of this data mastery are new methods such as artificial intelligence, which can offer an enormous market advantage. I hope to have given an insight into the topic with this article and I am looking forward to my first article on this topic myself.

LEARN MORE ABOUT: What is Big Data

If you would like to find out more about the question: What is Big Data, please contact the Round tables participate and discuss relevant topics with myself and other experts. Or write in the comments how you are dealing with this trend. Also read my article to big data.
Tip: Book suggestions too Big data
[werbung]
Verwendete Quellen anzeigen

[fotolia]

Der Beitrag What is big data definition erschien zuerst auf agile Companies.