`geom_smooth()` using formula = 'y ~ x'
What Makes a TED Talk Inspiring?: A Text Analysis of TED Talk Transcripts Using R
Introduction
Public speaking has long played an important role in shaping public opinion, spreading knowledge, and inspiring social change. From political speeches to lectures in the classroom, communication influences the way people think about the world and interact with one another. Recently, TED has become one of the most recognizable and influential public speaking platforms hosting talks covering a wide variety of subjects. TED’s mission is to communicate “ideas worth spreading” and many of these talks have gained a lot of traction.
In this project, textual analysis techniques are used to investigate patterns within such TED Talk transcripts. More specifically, this project aims to examine emotional language, and word usage, across TED Talks. The analysis explored whether highly viewed TED Talks tend to use different or stronger emotional language than less popular talks. On top of this, it also analyzes general word usage, comparing across popular and less popular talks in an attempt to get an idea of words or classes of words that may be more consistent with popular vs unpopular talks. To answer these questions, the project applies tokenization, regular expression cleaning, sentiment analysis, tf-idf analysis, and data visualization methods.
By combining transcript text with metadata such as view counts, this project demonstrates how computational text analysis can uncover meaningful communication patterns in public speaking. The results provide insight into the language characteristics of successful TED Talks while also demonstrating how text mining techniques can be applied to real-world datasets.
Data
The project uses two datasets from the Kaggle TED Talks dataset collection created by Rounak Banik. The first dataset, ted_main.csv, contains metadata for TED Talks published on TED.com through September 2017. It includes variables such as speaker names, talk titles, descriptions, tags, publication dates, comments, duration, and total view counts. These variables make it possible to compare transcript features with measurable audience engagement.
The second dataset, transcripts.csv, contains the full transcript text for TED Talks along with the corresponding talk URL. This dataset provides the textual data necessary for tokenization, sentiment analysis, tf-idf analysis, and work with regex’s. By joining the datasets using the shared URL variable, transcript text can be analyzed alongside popularity and topic information.
Sentiment Analysis and TED Talk Popularity
This visualization explores the relationship between emotional language and TED Talk popularity. Each point represents an individual TED Talk, with the x-axis displaying the normalized sentiment score and the y-axis showing the logarithm of total view counts. Point size represents transcript length, allowing larger and more detailed talks to be visually distinguished from shorter talks. The use of a logarithmic scale helps reduce the influence of extremely viral TED Talks and makes overall trends easier to interpret across the dataset.
The graph suggests a slight positive relationship between emotional positivity and popularity. Talks with more optimistic or emotionally uplifting language tend to receive somewhat higher view counts on average. This trend may exist because TED audiences often seek content that is inspiring, motivational, or hopeful. Speakers who use positive emotional language may create stronger emotional connections with viewers, making their talks more memorable and more likely to be shared online. Positive talks may also encourage feelings of curiosity, empowerment, or excitement, which align closely with TED’s goal of spreading impactful ideas.
However, the relationship is not especially strong, and there is substantial variation throughout the graph. Many talks with neutral or negative sentiment still achieve high view counts, suggesting that emotional positivity alone does not determine success. Other factors such as speaker reputation, storytelling ability, humor, topic relevance, production quality, and timing likely play major roles in audience engagement. For example, talks discussing serious issues such as climate change, mental health, or inequality may naturally contain more negative language while still attracting large audiences because of their social importance. Overall, the visualization suggests that emotional tone contributes to TED Talk popularity, but it is only one factor among many that influence audience engagement.
TF-IDF Analysis and Popularity Groups
Sentiment analysis using the methods applied here can only take us so far. There are often words which do not have any sentiment in our prebuilt sentiment datasets, and furthermore we may miss important context of phrases. Because of this, we can also employ some tf_idf analysis to attempt to gain some further insights. To do this, we will further break up our TED Talks in to “popularity groups.” We split the talks in to the top 10% and the bottom 40% of based on views. This allows us to perform tf_idf analysis across each group, to determine what words (or kinds of words) are most important in popular versus unpopular talks.
What we have done is first treated the entire dataset as our corpus, with each TED Talk as a single document. We then calculated the tf, idf, and tf_idf score for each word across the entire corpus, and also filtered out words that only appear in one or two talks to limit these skewing our results. Next, we group by popularity group and word, and take the average tf_idf score for each group. This will give us tf_idf information about words in each popularity group, essentially giving us a rough idea of the influence of that word in both popular and unpopular talks. Finally, we consider only words that show up in both popular, and unpopular talks, and look at the difference in their tf_idf scores. The difference in average tf_idf scores for a word between popularity groups gives us an idea of the importance of that word relative to each group. A high difference tells us that that word is in some sense “more influential” in one group than the other. Using a dumbbell plot, we can plot the 20 words with the largest gaps in tf_idf scores between each group, and visualize which group that word is more important too. This is what we do below.
As described above, this dumbbell plot is comparing the influence of words across each group based on the tf_idf score. Each point represents the average tf_idf score for that word in that respective group. The segment between them visualizes the difference between these average scores. A large difference in scores corresponds to a large difference in importance across groups, with a similar relationship for a small difference in scores. Here we have plotted the 20 words with the largest difference in average scores. This allows us to visualize which words carry a lot of weight in one group, and comparatively small weight in the other.
The words that are the most influential in popular talks, while being less influential in unpopular talks are grit, distractions, cite, kindergarten, tenderness, roommates, illusions, psychotic, public’s, ripples, and nagging. Compare these to words that are primarily influential in unpopular talks, which include hovering, placebo, hesitate, pivot, serotonin, circuitry, intergalactic, strawberry, and pixels. Despite a few outliers, it seems the words that are most influential in these less popular talks involve more technical topics. Serotonin, circuitry, intergalactic, pixels, and even placebo all appear to be words that may pop up in an academic or scholarly talk on a specific subject. For example, astrophysics for intergalactic, or psychology and neuroscience for serotonin and placebo. Several of the words that are influential in popular talks comparatively appear as though they could be things that are more broadly applied to people/communities as whole. Certainly thing like public’s, kindergarten, grit, and distractions are all words that are likely being used in talks geared towards the general public, with some message that is aimed for all ears.
It must be said that these conclusions are indeed somewhat speculative. There are certainly some outliers in each group based on the categories we’ve roughly defined here (technical vs geared towards general audience) and furthermore some words that may seem to fit well in one of those groups could easily be a part of the other. Much additional context and analysis would be needed to make a full conclusion.
Conclusion
This project used natural language processing and text analysis techniques in R to investigate patterns within TED Talk transcripts. The project explored the relationship between emotional language and popularity. While talks with more positive sentiment tended to receive somewhat higher view counts, the relationship was not strong enough to suggest that positivity alone determines success. Instead, the findings suggest that TED Talk popularity is influenced by many interacting factors including topic relevance, storytelling ability, speaker reputation, and audience interest.
On top of analyzing relationship between emotional language and popularity, this project also dug a bit deeper into the use of general language across popular and less popular talks. The goal here was to try and possibly identify words, or groups/classes of words that were more influential in popular/unpopular talks, while being minimally influential in the other. The findings seem to broadly suggest something that makes logical sense - more popular talks are those that are less technical and geared more towards the general public. However, the basis of this conclusion was somewhat speculative. Additional context is certainly needed, and the analysis is limited by the techniques employed. That being said, this conclusion seems consistent with expectations, and could serve as a starting point for further analysis.
Overall, this analysis demonstrates how computational text analysis can uncover patterns that may not be obvious through casual observation alone. By combining transcript text with metadata such as views, the project highlights the value of data wrangling and visualization techniques for understanding communication, audience engagement, and public speaking trends.
References/Sources
- Link to Kaggle Datasets: https://www.kaggle.com/datasets/rounakbanik/ted-talks