INTRODUCTION

What benefits can your organization achieve with Big Data? How to spot patterns in large quantities of enterprise data to improve performance and find underlying pattern that can benefit the organization? How can enterprises obtain a competitive advantage with Big Data? The Enterprise Big Data Analyst (EBDA) course discusses the theoretical concepts of Big Data and subsequently applies this knowledge to a real-world case study.

The Big Data Framework provides a holistic and compressive approach for enterprises that aim to leverage the value of data in their organizations. The framework covers all the essential aspects of Big Data that are necessary to understand and analyse massive quantities of data. The Enterprise Big Data Analyst course is the second level of the Big Data Framework course curriculum and certification program, that is globally recognized and accredited by APMG-International. The curriculum provides a vendor-neutral and objective understanding of Big Data architectures, technologies and processes.

The Enterprise Big Data Analyst qualification is a practitioner course for all professionals that aim to an in-depth understanding of Big Data analysis techniques and models, core data analysis processes steps, and best practices to retrieve value from data.

BENEFITS

In this course, you will learn the following:

• Understand and explain the data analysis process, including all relevant steps included in enterprise big data analysis.
• Understand the difference and structure of common data sources (local, online and database connections) and the way these sources should be imported in order to perform data analysis.
• Apply and utilize fundamental data cleaning operations and the differences between different data cleaning techniques.
• Apply and utilize fundamental data wrangling operations and the differences between different data wrangling techniques.
• Understand and apply exploratory data analysis techniques that are required for model building, model validation and initial visualizations.
• Understand and apply the core concepts of statistical inference, including techniques required for hypothesis testing.
• Formulate and interpret predictive models based on statistical correlation and regression functions, including simple linear regression.
• Formulate and interpret machine learning models for classification, including K-Nearest Neighbour, Naïve Bayes, Logistic Regression and Classification Trees.
• Formulate and interpret machine learning models for clustering, including the Hierarchical clustering and K-means clustering techniques.
• Formulate and interpret outlier detection models, including Grubbs Outlier detection and K-NN Outlier Detection.

ABOUT THE TRAINER

Jan Willem Middelburg from the Netherlands has been in the information technology industry for the past 10 years and is the author and Chief Examiner of the Enterprise Big Data Framework.  Jan-Willem began his writing career in 2013 as one of the co-authors of the book Serious Gaming. Subsequent publications included the Service Automation Framework and the Enterprise Big Data Framework. Both publication have been translated into Chinese, and now form the basis for global education and certification schemes.

ACCREDITATION

Cybiant is an accredited training provider with APMG International and one of the first to deliver the Enterprise Big Data Analyst course. Upon successful completion of the course, participants will receive the official Enterprise Big Data Analyst certification and digital badge. APMG International is a global well-established accreditation and certification body who is accredited by the United Kingdom Accreditation Service (UKAS) for ISO/ IEC17065: 2012 and ISO/ IEC17024: 2012.

COURSE & DATE

In conjunction with launching the certification, first learning class of Enterprise Big Data Analyst will be solely train by the framework author with live delivery in Asia. The following is the joining instructions for you to be the world first batch certified Enterprise Big Data Analyst:

• Certification Name: Enterprise Big Data Analyst Course & Exam
• First Course Date: 11-15 May 2020/ 18-22 May 2020
• Course Format: Virtual Learning
• Learning Hour: Daily 9am-11am & 1-3pm (GMT+8)
• Pre-requisite: Enterprise Big Data Professional Holder
• Investment Level: MYR 5200| SGD 2500| USD 1800

Decision Tree: A powerful and intuitive process to predict churn (Case Study)

There are many techniques available for the analysis of large-scale data. Churn-data of Telco companies can be analysed in various ways. Some focus on understanding the key characteristics of customers who will end their contract in the near future. Others concentrate more on predicting as precisely as possible which customers are expected to churn. The drawback of the second group of methods is that they are often very complex and not easy to follow for people without a data-science background. We believe decision trees fit perfectly in between these two groups. It gives possible to predict churn for large-scale data, with many different attributes. Furthermore, it can be visualized as a tree, which displays very clearly how the is built and how it is used to make predictions.

An example of a decision tree for a sample dataset of a Telco company is displayed below. The target variable of this tree is customer churn. The tree can be seen as a kind of flow-chart, you start at the top and follow some questions to end up in a certain bucket at the bottom. The tree could be followed manually, such that each customer could be placed in one of the buckets. The splits in the tree are questions about the values of each attribute of the customer. All customers that end up in a bucket have certain characteristics in common. In this way, we can assign a probability of churn to each bucket.

The most complex part of the decision tree is the algorithm which is used to generate the tree, although the main idea is quite intuitive. The goal is to make each bucket as pure as possible. In a perfectly pure bucket, all customers would churn or all customers would not. We start with the whole dataset. Then the algorithm strives to find the attribute to split on, which makes the two subsets the purest as possible. After selecting the first split, the same process is repeated for the part of the data one step lower in the tree, and the same for the next step.

When the model is created the most difficult part of the work is already done. We used data from the past to build the tree. Now we can apply all that is learned from historical data to new data of different customers. It enables us to go through the tree for each customer and predict whether they are likely to end their contract in the near future or not. Of course, we are not going to follow the tree by hand but let the computers do the work. This model could be applied to an unlimited quantity of customers, who would all end up in one of the buckets.

It is important to keep in mind that we are working with a model. The goal is to predict as correctly as possible which customers are going to end their contract. We can never achieve a 100% accuracy on the model. A crucial aspect of the tree for the accuracy of prediction is how many nodes it includes. Of course, the tree could be split such that there is a separate bucket for practically every customer. This would give nearly 100% accuracy if one runs it over the same data which is used for building the tree. For new data, this would not be the correct approach. A tree that has too little nodes is not specific enough. This results in a model that performs poorly, both on the old and the new data. The trick is to find a sweet spot in between those ends.

We obtained a maximum accuracy of 79% for predicting churn on a test dataset. In other words, the model predicts in almost eight out of ten cases correctly whether the customer is going to end his/her contract soon or not. This information shows Telco companies which customers who are likely to end their contract. They could use these insights to target customers in order to retain them for a longer period.

Introduction

Modern Telco companies gather lots of data on their customers. Varying from their gender to their monthly payments to the type of contract they have. One aspect that is especially important to keep track of is the group of customers that ends or will end their contract soon. Cybiant can help Telco companies to obtain valuable insights on how to identify customers that are likely to end their contract and to find which factors are indicating such behaviour.  There are two ways this dataset can be used to study contract ends. The first is determining which factors are typical of customers who end their contract, the second is identifying which customers are likely to end their contract. The focus of this article is on the former.  Therefore we built a showcase model with the powerful combination of Power BI and R.

We use a sample dataset of the customer-base of a telco company for our analysis. The variable of interest is ‘churn’, which has two values: ”yes” if a customer ended their contract in the last month, and “no” otherwise. There are various different characteristics of customers in the dataset which can be divided in three main categories: services that each customer signed up for, customer account information and demographic information about each customer.

Figure 1: Summarizing Customer Data in Power BI

After loading and cleaning the data in Power BI. We add a measure which is called the ‘churn-rate’, which is the percentage of the customers that ended their contract in the last month. The data can be visualized in various ways for different purposes. Some example of insightful visualizations are shown in figure 1.  We can observe the churn-rate of all customers in the top left. At the bottom is shown that the churn rate Fiber optic users is a lot higher that of DSL users or users which have no internet connection at all. This last group could have a lower churn rate, since it is more difficult to end you contract without an internet connection. The chart to the right demonstrates that the customers using Fiber-optic account for the largest part of total monthly income for the Telco company. This means that the largest group in income is also the group which has the highest churn rate. This seems to be a problem the company cannot neglect. At the pie chart becomes clear that the Fiber Optic customers are only 44% of the total number of customers, but they account 63% of monthly charges. It must be that they pay more monthly charges than an average customer.

Attribute Selection: which variables indicate customer churn?

The next step is to determine which attributes are the most important in predicting which customers are going to end their contract. Nowadays businesses have access to such large amounts of data and so many different attributes of information, that it is difficult to determine where to start. Figure 2 shows the importance of each attribute for predicting whether a customer will churn or not. The type of contract is the most important attribute as shown in the figure above. This figure provides a direction to start looking for more information. It makes clear that the contract attribute has the highest predictive value for churn.  Not too much time should be spent on analysing the differences between genders since that attribute has no significant predictive value for churn. Figure 2 is just an example and can be easily extended to datasets with many more attributes.

The used measure is called information gain(IG). We can split the data in different sets according to the categories of an attribute. The whole dataset has a certain purity. If all cases in the set have the same value for churn, the set is perfectly pure. In real-life big data perfect purity is rarely possible. , therefore we use a measure for the gain in purity when segmenting, which is IG. The set of customers can be divided into subsets according to the different values of an attribute. IG measures how much in purity is gained by segmenting on that particular attribute. A more in-depth explanation and the mathematical formulas can be found in the full article.

Back to the Telco data. Figure 2 shows that the type of contract is the most important attribute. The bar chart in figure 1 shows us that people who have a more flexible month-to-month contract, are more likely to churn. The line in the bar chart informs us that the group of people with a flexible contract is responsible for the largest part of income.  The Telco company could use these insights in various ways. For instance try to move people from month-to-month to yearly contracts or to keep more in touch with clients in that group, to make sure that they will be less likely to churn.

Figure 2 – Attribute Selection for Predicting Customer Churn

Correlation of predictive variables

Thanks to the information gain values we know more about the coherence of the attributes with the churn variable. To see how the attributes relate to each other, we have created a plot of “correlation” between the attributes, which is shown in figure 3. The plot below shows how much the attributes are related to each other. The values differ from 1(perfect correlation) to 0(no correlation at all). It is not a correlation as one would normally see with numerical variables, but a different measure for categorical variables, which is called Cramer’s V. In the attribute importance chart can be observed that OnlineSecurity and TechSupport are both important attributes. In the chart above we can see that those are highly correlated. Therefore, it is plausible that one causes another or the two go almost always together. Then they can be viewed as a one combined phenomenon. Besides the value for predicting churn, this chart can be used to develop a better understanding of the behaviour of customers. It can be easily observed which attributes are interrelated.

We used the powerful combination of Power-BI to make business-oriented insightful visualizations on the data. Power BI is used to gather everything in one place, and make visual reports. R provides us with the flexibility to perform more custom-made operations on the data, for the needs of the specific company. The next step is to identify customers who are likely to churn beforehand. In our view both uses of the data described above are essential. And the predictive part would not be complete without understanding the data.

Figure 3 – Correlation between attributes

Read more or download the white paper

Want to learn more about how to predict customer churn, or want to dive into the formulas and underlying theoretical model? Our Predicting Customer Churn Case Study provides an even more in-depth look at at the steps that were taken to generate these results.