This past summer I was fortunate enough to be accepted to a Research Experience for Undergraduates (REU) program hosted by the mathematics department of the University of California, Los Angeles. It was my first time doing research at a scale larger than projects for classes, and certainly the most time I have invested into any one project. My team worked full time for two months of the summer learning about and experimenting with new strategies for designing chatbots. For those who are not familiar, you can think of a chatbot as a computer program which you can have a conversation with; in other words, it is like texting with someone except the other person is the computer. Our specific research goal was to create a general-purpose chatbot which provides topical responses more so than general ones. For example, if you ask it something about basketball, it should in turn respond with something at least related to basketball, instead of just giving a generic response. As one can imagine, this is a very challenging and daunting task, but it would also be an extremely useful one to solve. People outside of academia are focused on this problem too, with OpenAI (a research laboratory based in San Francisco) currently having one of the best performing solutions with their GPT2 model. While OpenAI and similar companies have nearly unlimited brainpower as well as computing power, our resources were limited. Because of this, we did not set out to expand on the state of the art models, but instead focus on building off of slightly older (~ 2 years old) methods which are more reasonable in their cost to train.
Team organization
Our project had the two components of text generation and topic modeling, so our team split into two groups each of which focused on one part of the task. The idea was that we would work separately for the first half of project, learning all we can about the part of the problem we want to solve and developing individual models which perform well. Then in the second month we would combine these two concepts together into one coherent model.
Topic modeling
I was on the topic modeling team, and our advisor decided that we would use non-negative matrix factorization (NMF) instead of other methods. None of us had experience with NMF so we began by teaching ourselves about topic modeling in general, and then read about the specifics of NMF. The key aspect of NMF that gives it advantages over other methods is that both the original matrix and its two factor matrices which are produced contain only non-negative entries. This makes the topics that are extracted by the method much more interpretable than topics extracted from other methods. Other methods have negative entries as well, which means that topics can be “added” together but also “subtracted” from each other. When topics are combined in ways other than additive methods, the individual topics are generally less interpretable because the “difference” of two topics can represent something else. We can call these two matrices the dictionary matrix and the code matrix, and their product should approximate the original data matrix.
\[\texttt{Data} \approx \texttt{Dictionary} \times \texttt{Code}\]If we perform NMF on an entire corpus, we can obtain the dictionary matrix for that corpus. The columns of this matrix represent the topics of the corpus. We can then use this dictionary matrix along with a single example to find the code vector for that example. This code vector can then be used to modify the output of a text generation model in order to give topic words higher weights.
Text Generation
The other team examined many different text generative models, beginning with basic recurrent neural networks and eventually arriving at an attention-based model. Attention-based models are advanced versions of RNN models, except when generating output, instead of just basing the output off of one hidden state, it generates the output at each step using all hidden state vectors with a set of weights applied to them. This allows for more information about the input to be used during the generation of the output.
Combining message attention and topic attention
We will refer to the attention from the text generation model as message attention because this attention is the general information about the input example that is used to generate the output. Our method for merging the topic modeling with the text generation model is to also have topic attention which is used to generate the outputs. This method is largely based off of the work of Xing et al. (https://arxiv.org/abs/1606.08340) which proposed a similar strategy except they used latent Dirichlet allocation instead of NMF to create the topic attention.
\[\hat{\mathbf{p}}_{t}(\omega) \propto \exp(\Psi^{(c)}_{t}(\omega)) \left(\sum_{j=1}^{r} k_{j} \mathbf{1}(\omega\in \mathbf{w}_{j}) \right) \exp(\Psi^{(o)}_{t}(\omega)),\]where $\Psi_t^{(c)}(\omega)$ is the contribution from the message attention and $\Psi_t^{(o)}(\omega)$ is the contribution from the topic attention. The vectors $\mathbf{w}_j$ represent words which are contained within the $j^\text{th}$ topic, and $k_j$ is the weight from the code matrix of the input example which corresponds to the $j^\text{th}$ topic. This expression is proportional to our predicted probability distribution which we can then generate output words from by using Markov chain Monte Carlo sampling.
Experiment and results
In order to evaluate the performance of our model, we utilized several datasets. To build dictionary matrices, we compared the performance of three datasets: DeltaAir (Delta Airline customer support records), Shakespeare (all lines in all of Shakespeare’s plays), and 20NewsGroups (Newspaper articles from 20 categories). Using each of the dictionary matrices generated from these datasets through NMF, we then trained the overall chatbot model on the Cornell Movie Dialogue dataset. We also trained one version of the chatbot which did not use topic modeling in order to have a baseline to compare to. Shown in the three tables below are the top 5 topics from each dataset along with each of these topics’ top 10 weighted words.
Topics learned from DeltaAir:
| Topic # | 10 highest-weighted words |
|---|---|
| 1 | thank, welcome, flying, feedback, appreciate, great, delta, loyalty, sharing, day |
| 2 | flight, delayed, delay, sorry, time, crew, gate, hours, hi, hear |
| 3 | seat delta upgrade seats comfort middle available class hi economy |
| 4 | let, know, assistance, need, sorry, amv, rebooking, assist, delay, apologies |
| 5 | bag, baggage, check, claim, airport,bags, lost, checked, luggage, team |
Topics learned from Shakespeare:
| Topic # | 10 highest-weighted words |
|---|---|
| 1 | enter, messenger, king, attendants, servant, gloucester, lords, duke, queen, henry |
| 2 | lord, good, ay, know, noble, say, tis, king,gracious, did |
| 3 | exit, servant, falstaff, messenger, gentle-man, body, lucius, boy, gloucester, cassio |
| 4 | thy, hand, father, heart, hath, love, life,let, master, face |
| 5 | sir, good, know, ay, pray, john, man, say,did, marry |
Topics learned from 20NewsGroups:
| Topic # | 10 highest-weighted words |
|---|---|
| 1 | card, video, monitor, drivers, cards, bus,vga, driver, color, ram |
| 2 | god, jesus, bible, christ, faith, believe,christian, christians, church, sin |
| 3 | game, team, year, games, season, play-ers, play, hockey, win, player |
| 4 | car, new, 00, sale, 10, price, offer, condition, shipping, 20 |
| 5 | edu, soon, cs, university, com, email, internet, article, ftp, send |
We trained the chatbot using a single NVIDIA GeForce GTX 1660 Ti GPU. We use the Adagrad optimizer, with a batch size of 64, dropout rate of 0.1, hidden size for the word embedding of 500, and a total number of training iterations of 64,000 (87 epochs). We also use teacher forcing to accelerate the training, with a teacher forcing rate of 1.
We found that all topic-aware models achieved better training loss with the Shakespeare topic dictionary achieving the best results.
| Model | Training Loss |
|---|---|
Non-topic | 0.2847 |
DeltaAir | 0.1937 |
Shakespeare | 0.1708 |
20NewsGroups | 0.2061 |
For a qualitative comparison of the models, we show the results of example conversations with the chatbots in the table below. A bold response indicates that we found it to be a reasonable, non-generic response to the input.
We observe that in general, topic-aware chatbots give more non-generic and context-dependent responses to various input questions. Moreover, when we ask questions closely related to each topic matrix (DeltaAir, Shakespeare, and 20NewsGroups), we observe that the chatbot with the corresponding NMF-topic filter gives the most appropriate answer using relevant topic words more frequently. Namely, for the question “I lost something at the airport”, the words “lost” and “airport” belong to Topic #5 of the DeltaAir dictionary matrix and the related topic word “check” does appear in the corresponding chatbot’s response. Also, for the question, “Is he good enough to marry her”, “good” and “marry” belong to Shakespeaere’s Topic #5, and the corresponding chatbot’s answer contains associated topic words “good” and “man”. Lastly, for the question “I love your new car”, the words “car” and “new” belong to Topic #4 of the 20NewsGroups dictionary, and both of them as well as additional topic word “years” from the same topic vector appear in the corresponding chatbot’s response.
Conclusion
Our work serves as the first proof of concept for using NMF-based topic modeling to create a topic-aware, RNN-based chatbot. This opens up a number of future research possibilities, which build on recent developments in the literature of NMF. We also attempted to augment Google’s BERT model as well as the OpenAI GPT2 model with NMF topic-awareness. However, due to time constraints we were not able to complete this part of the project. Future work could be done with attempting to add topic modeling to either of these transfer learning models.
Overall, I had an awesome summer living in Los Angeles and doing research at UCLA. Living so far from home was a difficult, yet incredibly fun experience, and living in a proper city for the first time was also a major departure from life in Rhode Island and Vermont.
The arXiv preprint of this paper is available here.