Document or text classification is one of the predominant tasks in Natural language processing. It has many applications including news type classification, spam filtering, toxic comment identification, etc. In big organizations the datasets are large and training deep learning text classification models from scratch is a feasible solution but for the majority of real-life problems your dataset is small and if you want to build your machine learning model you need to be smart. In this article (originally posted by on the ), I will talk about pragmatic approaches towards text representation which make document classification on small datasets doable. Shahul ES neptune.ai/blog Text Classification 101 The text classification workflow begins by cleaning and preparing the corpus out of the dataset. Then this corpus is represented by any of the different text representation methods which are then followed by modeling. In this article, step of this pipeline. we will focus on the “Text Representation” Example text classification dataset We will use the data from kaggle competition. Here, the Real or Not? NLP with disaster tweets task is to predict which tweets are about real disasters and which ones are not. If you want to follow the article step-by-step you may want to install all the libraries that I used for the analysis. Let’s take a look at our data, pandas pd

tweet= pd.read_csv( )
test=pd.read_csv( )

tweet.head( ) import as '../input/nlp-getting-started/train.csv' '../input/nlp-getting-started/test.csv' 3 The data contains of id, keyword, location, text, and target which is binary. We will only consider the tweets to predict the target. print( .format(tweet.shape[ ],tweet.shape[ ]))
print( .format(test.shape[ ],test.shape[ ])) 'There are {} rows and {} columns in train' 0 1 'There are {} rows and {} columns in test' 0 1 The training dataset has less than 8000 tweets. That, combined with the fact that tweets are 280 characters tops make it a tricky, small(ish) dataset. Text data preparation Before we get into any NLP task, we need to do some data preprocessing and basic cleaning. It is not a focus of this article but if you want to read more about this step check out this . article In short, we will: : the process by which sentences are converted to a list of tokens or words. Tokenize : drop words like ‘a’ or ‘the’ Remove stopwords : reduce the inflectional forms of each word into a common base or root ( . Lemmatize “studies”, “studying” -> “study”) new_corpus=[]
	
	lem=WordNetLemmatizer() text df[ ]:
    	words=[w w word_tokenize(text) (w stop)]
  	 
    	words=[lem.lemmatize(w) w words]
  	 
    	new_corpus.append(words) new_corpus

corpus=preprocess_news(df) : def preprocess_news (df) '''Function to preprocess and create corpus''' for in "question_text" for in if not in for in return Now, let’s see how to represent this corpus so that we can feed this into any machine learning algorithm. Text Representation Text cannot be used directly as input to a machine learning model but needs to be represented in the numeric format first. This is known as text representation. Countvectorizer Countvectorizer provides an easy method to vectorize and represent a collection of text documents. It tokenizes the input text and builds a vocabulary of known words and then represents the documents using this vocabulary. Let’s understand it by using an example, text = [ ] vectorizer = CountVectorizer() vectorizer.fit(text) print(vectorizer.vocabulary_) vector = vectorizer.transform(text) print(vector.shape)
print(type(vector))
print(vector.toarray()) "She sells seashells in the seashore" # create the transform # tokenize and build vocab # summarize # encode document # summarize encoded vector You can see that the Coutvectorizer has built a vocabulary out of the given text and then represented the words using a . We can try and transfer another text using this vocabulary and observe the output to get a better understanding. numpy sparse matrix vector=vectorizer.transform([ ])
vector.toarray() "I sell seashells in the seashore" You can see that: The index positions 3 and 4 have zeroes meaning that these two words are not present in our vocabulary and all other positions have 1 meaning that these words are present in our vocabulary. The corresponding words missing from the vocabulary are “sells” and “she”. Now that you understand how Coutvectorizer works, we can fit and transform our corpus using it. vec=CountVectorizer(max_df= ,max_features= )
vec.fit(df.question_text.values)
vector=vec.transform(df.question_text.values) 10 10000 You should know that Countvectorizer has a few important parameters that you should adjust to your problem: : build a vocabulary that only considers the top n tokens ordered by term frequency across the corpus. max_features : When building the vocabulary ignore terms that have a token frequency strictly lower than the given threshold min_df : When building the vocabulary ignore terms that have a token frequency strictly higher than the given threshold. max_df What usually helps with selecting reasonable values (or ranges for hyperparameter optimization methods) is good exploratory data analysis. Check out my other article to read about it. TfidfVectorizer One issue with Countvectorizer is that common words like “the” will appear many times (unless you remove them at the preprocessing stage) and these words are not actually important. One popular alternative is Tfidfvectorizer. It is an acronym for Term frequency-inverse document frequency. : This summarizes how often a given word appears within a document. Term Frequency : This downscales words that appear a lot across documents. Inverse Document Frequency Let’s look at an example: sklearn.feature_extraction.text TfidfVectorizer text = [ , , ] vectorizer = TfidfVectorizer() vectorizer.fit(text) print(vectorizer.vocabulary_)
print(vectorizer.idf_) vector = vectorizer.transform([text[ ]]) print(vector.shape)
print(vector.toarray()) from import # list of text documents "She sells seashells by the seashore" "The sea." "The seashore" # create the transform # tokenize and build vocab # summarize # encode document 0 # summarize encoded vector The vocabulary again consists of 6 words and the inverse document frequency is calculated for each word, assigning the lowest score to “the” which occurred 4 times. Then the scores are normalized between 0 and 1 and this text representation can be used as input into any machine learning model. Word2vec The big issue with the above approaches is that the context of the word is lost when representing it. Word embeddings provide a much better representation of the words in NLP by encoding some context information. It provides a mapping from a word to a corresponding n-dimensional vector. Word2Vec was developed at Google by Tomas Mikolov, et al. and uses a to learn word embeddings. The vectors are learned by understanding the context in which the word occurs. shallow neural network Specifically, it looks at co-occurring words. Given below is the co-occurrence matrix for the sentence “The cat sat on the mat”. Word2vec is composed of two different models: (CBOW) model can be thought of as learning word embeddings by training a model to Continuous Bag of Words predict a word given its context. model is the opposite, learning word embeddings by training a model to Skip-Gram predict context given a word. The basic idea of word embedding is words that occur in similar context tend to be closer to each other in vector space. Let’s check how to implement word2vec in python. gensim gensim.models Word2Vec

model = gensim.models.Word2Vec(corpus, 
                               min_count = , size = , window = ) import from import 1 100 5 Now you have created your word2vec model, some of the important parameters that you can actually change and observe the differences are, : this indicates the embedding size of the resulting vector for each word. size : When building the vocabulary ignore terms that have a document frequency strictly lower than the given threshold min_count : The number of words surrounding the word is considered when building the representation. Also known as the window size. window In this article, we focus on pragmatic approaches for small datasets and we will use pre-trained word vectors instead of training vectors from our corpus. This method is guaranteed to yield better performance. First, you will have to download the trained vectors from . Then you can load the vectors using gensim. here gensim.models.KeyedVectors load_word2vec_format word2vecDict = load_word2vec_format( ,
        binary= , unicode_errors= )
    embeddings_index = dict() word word2vecDict.wv.vocab:
        embeddings_index[word] = word2vecDict.word_vec(word) embeddings_index from import : def load_word2vec () '../input/word2vec-google/GoogleNews-vectors-negative300.bin' True 'ignore' for in return Let’s check the embedding, w2v_model=load_word2vec()
w2v_model[ ].shape 'London' You can see that the word is represented using a 300-dimensional vector. So every word in your corpus can be represented like this and this embedding matrix is used to train your model. FastText Now, let’s learn about fastText which is an extremely useful module available in . FastText has been developed by Facebook and yields great performance and speed in text classification tasks. gensim It supports both Continuous Bag of Words and Skip-Gram models. The main difference between previous models and FastText is that it breaks the word in several n-grams. Let’s take the word orange for example. The trigrams of word orange are,org,ran,ang,nge(ignoring the starting and ending boundaries of the word). The word Rare words or typos can now be properly represented since it is highly likely that some of their n-grams also appears in other words. embedding vector (text representation)for orange will be the sum of these n-grams. For example, for a word like , which might never have been in any corpus, gensim might return any two of the following solutions: a zero vector or a random vector with low magnitude. stupedofantabulouslyfantastic , however, . In this particular case, the final vector might be closer to the vectors of fantastic and fantabulous. FastText can produce better vectors by breaking the word into chunks and using the vectors for those chunks to create a final vector for the word Again, we will use a pre-trained model rather than training our own word embeddings. For this, you can download pre-trained vectors from . here Each line of this file contains a word and it’s a corresponding n-dimensional vector. We will create a dictionary using this file for mapping each word to its vector representation. gensim.models FastText print( )
	embeddings_index = {}
	f = open( ,encoding= ) line tqdm(f):
    	values = line.strip().rsplit( )
    	word = values[ ]
    	coefs = np.asarray(values[ :], dtype= )
    	embeddings_index[word] = coefs
	f.close()
	print( % len(embeddings_index)) embeddings_index

embeddings_index=load_fastext() from import : def load_fasttext () 'loading word embeddings...' '../input/fasttext/wiki.simple.vec' 'utf-8' for in ' ' 0 1 'float32' 'found %s word vectors' return Let’s check the embedding for a word, embeddings_index[ ].shape 'london' GloVe (Global vectors for word representation) GloVe stands for global vectors for word representation. It is an unsupervised learning algorithm developed by Stanford. The basic idea of GloVe is to derive a semantic relationship between words using a co-occurrence matrix. The idea is very similar to word2vec but there are slight differences. Go here to read more. For this, we will use which are trained on large corpora. This is guaranteed to perform better in almost any situation.You can download it from . pre-trained glove vectors here After downloading we can load our pre-trained word model. Before that, you should understand the format in which it is made available. Each line contains a word and its corresponding n-dimensional vector representation. Like this, So, to use this you should first prepare a dictionary that contains the mapping between word and corresponding vector. This can be called an embedding dictionary. Let’s create one for our purpose. embedding_dict = {}
    path = open(path, ) f: line f:
            values = line.split()
            word = values[ ]
            vectors = np.asarray(values[ :], )
            embedding_dict[word] = vectors
    f.close() embedding_dict


embeddings_index = load_glove() : def load_glove () '../input/glove-global-vectors-for-word-representation/glove.6B.100d.txt' with 'r' as for in 0 1 'float32' return Now, we have a dictionary containing every word in the glove pre-trained vectors and their corresponding vector in a dictionary. Let’s check the embedding for a word. embeddings_index[ ].shape 'london' Till now we were dealing with representing words and these techniques are most useful for word-level operations. Sometimes we need to explore operations. These encoders are called sentence encoders. sentence level A good sentence encoder is expected to encode sentences in such a way that the vectors of similar sentences have a minimal distance between them in the vector space. For example, It is sunny today It is rainy today It is cloudy today. These sentences will be encoded and represented so that they are close to each other in the vector space. Let’s move on and check how to implement universal sentence encoder and find similar sentences using it. You can download the pertained vectors from . here We will load the module using the TensorFlow hub. module_url = embed = hub.load(module_url) "../input/universalsentenceencoderlarge4" # Import the Universal Sentence Encoder's TF Hub module Next, we will create the embedding for each sentence in our list. sentence_list=df.question_text.values.tolist()
sentence_emb=embed(sentence_list)[ ].numpy() 'outputs' Here is an article to read more about universal sentence encoder. Elmo, BERT, and others. When using any of the above embedding methods one thing we forget about is the context in which the word was used. This is one of the main drawbacks of such word representation models. For example, the word word “stick” will be represented using the same vector independent of the context in which it was used which doesn’t make much sense. With the recent developments in the field of NLP and models like BERT (bidirectional encoder representation from transformers), this has been made possible. Here is an article to read more. Text Classification In this section, we will prepare the embedding matrix which is passed to the Keras Embedding layer to learn text representations. You can use the same steps to prepare the corpus for any word-level embedding methods. Let’s create a word index and fix a maximum sentence length, pad each sentence in our corpus using and . Keras Tokenizer pad_sequences MAX_LEN= tokenizer_obj=Tokenizer()
tokenizer_obj.fit_on_texts(corpus)
sequences=tokenizer_obj.texts_to_sequences(corpus)

tweet_pad=pad_sequences(sequences,
                        maxlen=MAX_LEN,
                        truncating= ,
                        padding= ) 50 'post' 'post' Let’s check the number of unique words in our corpus, word_index=tokenizer_obj.word_index
print( ,len(word_index)) 'Number of unique words:' Using this word index dictionary and embedding dictionary you can create an for our corpus. This embedding matrix is passed on to the of the neural network to learn word representations. embedding matrix embedding layer num_words = len(word_index)
    embedding_matrix = np.zeros((num_words, emb_size)) word, i tqdm(word_index.items()): i > num_words: emb_vec = embedding_dict.get(word) emb_vec :
        embedding_matrix[i] = emb_vec embedding_matrix : def prepare_matrix (embedding_dict, emb_size= ) 300 for in if continue if is not None return We can define our neural network and pass this embedding index to the Embedding layer of the network. We pass the vectors onto the Embedding layer and set to prevent the weights from being updated. trainable=False inp = Input(shape=(MAX_LEN,))

    x = Embedding(num_words, embedding_matrix.shape[ ], weights=[embedding_matrix],
                  trainable= )(inp)

    x = Bidirectional(
        LSTM( , return_sequences= , name= , 
             dropout= , recurrent_dropout= ))(x)

    x = GlobalAveragePool1D()(x)
    x = Dense( , activation= )(x)
    model = Model(inputs=inp, outputs=x)

    model.compile(loss= ,
                  optimizer= ,
                  metrics=[ ]) model : def new_model (embedding_matrix) 1 False 60 True 'lstm_layer' 0.1 0.1 1 "sigmoid" 'binary_crossentropy' 'adam' 'accuracy' return For example, to run the model using word2vec embeddings, embeddings_index=load_word2vec()
embedding_matrix=prepare_matrix(embeddings_index)
model=new_model(embedding_matrix)

history=model.fit(X_train,y_train,
                  batch_size= ,
                  epochs= ,
                  validation_data=(X_test,y_test),
                  verbose= ) 8 5 2 You can call your desired type of embeddings and follow the same steps to implement any of them. Comparison So which text classification method worked best in our example problem? You can use Neptune to compare the performance of our model using different embeddings by simply setting up an experiment. Glove embeddings performed a little better in test sets when compared to the other two embeddings. You may be able to get better results by doing extensive cleaning on the data and tuning the model. You can here if you want to. explore experiments Final Thoughts In this article, we discussed and implemented different feature representation methods for text classification that you can use for smaller datasets. Hopefully, you will find them useful in your projects. This article was originally posted on the Neptune blog. If you liked it, you may like it there :) You can also find me tweeting @Neptune_a i or posting on LinkedIn about ML and Data Science stuff.

Facebook

Google

ORANGE

Target

Check out our free experiment tracking tool

Too Long; Didn't Read

In your car, at home, or at work — Bosch technology shapes many areas of life.

Document Classification Process: 7 Pragmatic Approaches For Small Datasets

Too Long; Didn't Read

Companies Mentioned

neptune.ai Jakub Czakon

About Author

TOPICS

THIS ARTICLE WAS FEATURED IN...

Document Classification Process: 7 Pragmatic Approaches For Small Datasets

Too Long; Didn't Read

Companies Mentioned

neptune.ai Jakub Czakon

About Author

TOPICS

THIS ARTICLE WAS FEATURED IN...

RELATED STORIES