Text Data Clustering using Natural Language Processing(NLP)

           It is really very important to classify any type of data to analyse and understand the insights. The companies like Twitter, Facebook has a lots of text data which needs to be analysed based on the requirements. The Data Set of tweets of a BBC health news agency(source: UCI Machine learning Repository )has been taken and clustered/ differentiated the tweets based on their category. For example health tweets may consist of different types of health issues, diseases, health checkups, surgeries, deadly diseases, etc. Let us do this project in a step by step process to be clear. 

Details of Data Set:

          The data consists of Twitter ID, Time of the Tweet and the Tweet. We can see the full data information provided in the above link. The data has been saved in a text file named as 'bbchealth.txt'.

Preliminary understanding of Data Set:

          The Data Set is '|' separated text file with no column names. We have taken a series of 3929 instances to lower the processing time but taking more instances increases the accuracy of the results. If the tweets are observed we can distinguish the tweets based on the most repeated words in the each tweet. That is what gonna happen now. But differentiating large amount of data manually is a difficult task hence we take help of Python programming.

Step by Step process in python to execute the aim:

A sound knowledge on libraries that are used in the required area is a good thing which comes by practice. First we import the required libraries.

# Clustering on twitter data

importing the libraries 

import re  

import matplotlib.pyplot as plt  

import pandas as pd  

from nltk.stem.porter import PorterStemmer

#Enable logging for gensim  

import logging  

logging.basicConfig(format = '%(asctime)s : %(levelname)s : %(message)s', level=logging.ERROR)  

import warnings  

warnings.filterwarnings("ignore", category = DeprecationWarning)  

  

# Preparing stop words  

from nltk.corpus import stopwords  

·                        

•     The library re stands for regular expressions which is a special sequence of characters that helps you match or find other strings or set of strings, using specialized syntax held in a pattern.
• ·  The library matplotlib.pyplot is used to draw graphs and visualize the data for analysis.
• ·        Pandas is used for data manipulation and analysis.
• ·   nltk stand for natural language toolkit it consists of all the required functions to process the natural language. The object stem.porter.porterStemmer consists of stem or root words which is used to transform any word into its stem. for example needed will be transformed to need.
• ·        warning filter is used to remove deprecation warnings.
• ·        stop words library consists of words that are not necessary for the analysis for example to, and etc.

Now we import dataset:

# importing dataset  

dataset = pd.read_csv('bbchealth.txt', sep = '|', names = ["TwitterId", "Time", "Tweet"], quoting =3)  

      we have imported the data set using the pandas library, using sep = '|' we have separated the columns in the data and with names attribute we have given the names to the columns respectively.

The imported data set looks like this,

 

Next step is to clean the data:

#Data cleaning  

Tweet = dataset.Tweet.values.tolist()  

corpus = []  

#Remove newline seperator and emails  

for i in range(0, len(Tweet)):  

   review = re.sub(r'http\S+', '', Tweet[i]) #removing the web address in the tweets

   review = re.sub(r'VIDEO', '', Tweet[i])  #removing the most observed non relevant word that may trap us.

   review = re.sub('[^a-zA-Z]', ' ', review)  #removing symbols and special characters and numbers from the tweets.

   review = review.lower()  # converting the whole text into lower case letters to avoid case sensitivity

   review = review.split()  # converting the string in to list of words

   ps = PorterStemmer()  

   review = [ps.stem(word) for word in review if not word in set(stopwords.words('english'))]  

   review = ' '.join(review)  

   corpus.append(review)  

In the above code PorterStemmer library is used to stem the words in the tweets. Remaining explanation of the code is in the code itself as comments. After executing the above code the tweets look like this.

If we observe the tweets they are clean to further processing

          The real analysis of text starts from Bag Of Words model. the output matrix of Bag of Words model consists of each tweet as its row and all the words in the columns. the values of the matrix are count of each word in the respective tweet. let us model it with python

#Creating a Bag of words model  

from sklearn.feature_extraction.text import CountVectorizer  

cv = CountVectorizer()  # we use countvectoriser to count the to create a sparse matrix(bag of words model)

X = cv.fit_transform(corpus).toarray()  # This is the matrix we have discussed above this is also called as count matrix/ sparse matrix.

y = dataset.Tweet # the dependent variable is text of tweets

          The X variable in the above program is considered as independent variable and y as dependent variable. We are using hierarchical clustering to cluster the data and to find the optimal no. of clusters in the hierarchical clustering we use dendograms. The python code for dendograms as follows.

# Using the dendrogram to find the optimal number of clusters  

import scipy.cluster.hierarchy as sch  

dendrogram = sch.dendrogram(sch.linkage(X, method = 'ward'))  

plt.title('Dendrogram')  

plt.xlabel('Customers')  

plt.ylabel('Euclidean distances') 

plt.show()  

The dendogram plot as follows,

          From the above dendogram we have to find the optimal no. of clusters. It is very simple just looking at the plot. if we look at the above plot we can say the optimal no. of clusters are 11 and hence using 11 clusters let us write the python code for clustering.

1.  #From Dendogram the optimal no. of clusters are 11  

2.  # Fitting Hierarchical Clustering to the dataset  

3.  from sklearn.cluster import AgglomerativeClustering  

4.  hc = AgglomerativeClustering(n_clusters = 11, affinity = 'euclidean', linkage = 'ward')  

5.  y_hc = hc.fit_predict(X)   # Applying the hierarchical clustering for the model.

The output y_hc would look like this.

          A series of numbers. Here each no. refer to the cluster number. zero refer to the first cluster, 1 refers to the second cluster and so on. the table above denotes the first tweet belongs to first cluster, the second tweet belongs to fifth cluster and so on. so we write the below code to differentiate the clusters.

GeneralPractioners = y[y_hc==0]  #General practioners tweets  

Ebola = y[y_hc==1]           # High Risk diseases like EBOLA  

Health_Issue = y[y_hc==2]    #Health Issues' warnings and care  

General_Health_Risks = y[y_hc==3] # General daily life health risk statements  

Babies_Death_Reports= y[y_hc==4] # Born babies death reports and meternal death reports  

Health_Care = y[y_hc==5]        # Heath care advices  

Hospital_News = y[y_hc==6]      # Hospital maintenance, staffing news  

NHS = y[y_hc==7]                #National Health Services  

AandE_PatientGroup= y[y_hc==8]  # A category of patients prone to accidents  

Cancer = y[y_hc==9]             # Cancer risk  

AudioFeedback= y[y_hc==10]      # An audio tweets regarding patients feedback 

          Generally giving names to the clusters while writing the program is impossible. After executing the program and looking into the different clusters based on the category I have named the clusters.

          The results of the clusters can be seen by viewing the variable one by one.There is no certain perfect method to measure the accuracy of the NLP algorithms but we can manually guess the accuracy by find the most distinct tweets.