How To Use AI To Summarize The Top Posts Of A Sub-Reddit
Forums like Reddit contain vast amounts of text. In this demo, we should you how to summarize that automatically and get valuable insights.
Sites like Reddit contain a huge amount of text information on a variety of different topics, spread out across a huge forum. Because of this, it’s both difficult and time-consuming to manually summarize and obtain key insights. However, with the use of the right AI model, it is possible to accomplish this type of summarization quickly and easily.
Using a pre-trained model allows us to perform this task without the need to construct and train our own model. In this tutorial, you’ll see how to use Cohere’s LLM API to produce summaries of the top 20 posts of all time from Reddit’s literature subreddit. The subreddit of choice can easily be changed in the App.js file, which is explained later on.
Prerequisites
- Node / NPM. For this tutorial, we recommend installing the “LTS” version
- Knowledge of JavaScript/ES6 Basics
- Knowledge of React Basics
- A code editor —this tutorial uses Visual Studio Code
Getting Started
To begin, you’ll bootstrap a React app.
Create a React App
Follow the steps below to create your React application for this guide.
First, run npx create-react-app reddit-posts-summarizer in your terminal. The image below shows the structure of your created project.
Then, change directories using cd reddit-posts-summarizer and launch the dev server with npm start.
Next, you’ll connect the project to the Reddit API.
Connect to the Reddit API
Before summarizing the Reddit posts, you first have to grab them from the /literature subreddit. You can choose whichever subreddit you want by replacing literature in the Reddit link below.
First, replace the contents of src/App.js with the following code.
import React, { Component } from 'react';
class App extends Component {
// fetch the top 20 Reddit posts of all time in a subreddit
getRedditPosts() {
//Fetch extra posts so that posts without text can be filtered (100)
fetch('https://www.reddit.com/r/literature/top.json??&t=all&limit=100').then(res => {
if(res.status !== 200){
console.log("ERROR: unable to retrieve posts from reddit.com")
return;
}
res.json().then(data => {
if(data !== null){
const posts = data.data.children;
console.log("REDDIT POSTS: ", posts);
//Filter posts without text
//Traverse array in reverse order to avoid array issues when removing items
var i = posts.length
while(i--){
if(posts[i].data.selftext == ""){
posts.splice(i,1);
}
}
//Drop excess posts
if(posts.length > 20){
posts.length = 20;
}
}
})
});
}
componentWillMount(){
this.getRedditPosts();
}
render(){
return (
<div className="App">
<p><h2>Top 20 Posts of all Time in <i>r/literature</i></h2></p>
<p>Posts with potentially offensive language have been removed from the list as decided by Cohere, which could reduce the number of posts below 20</p>
<p>Many Reddit posts contain images instead of text - these posts have been skipped</p>
<div className="article">
<ol>
{
(this.state.cohereSummaries.length > 0) ? this.state.cohereSummaries.map((post, index) => <Article key={index} post={post} />) : <p>processing... length: {this.state.cohereSummaries.length}</p>
}
</ol>
</div>
</div>
);
}
}
export default App;
The componentWillMount function invokes the getRedditPosts function, which grabs the top 20 posts in r/literature (with limit=20) of all time (with t=all).
To confirm that you’ve retrieved the posts, visit r/literature and then launch your browser’s console (CTRL + SHIFT + I on Windows). Once you’ve connected, your console will list posts with some of their properties.
Connect to Cohere’s Text Summarizer
Set Up with Cohere
First, register your account, then select the Dashboard tab to navigate to your dashboard.
Next, select Create API Key. Give it a name and safely store the API key — you’ll need it later.
Summarize with Cohere
The application you’ll build in this tutorial connects to Cohere using the JavaScript fetch API. Alternatively, you could use Cohere’s official API client, which lets you use the Cohere Playground to quickly test parameter values for optimal results.
Text summarization with Cohere requires making a request to the Generate endpoint. The endpoint requires two specific parameters in your request:
- The prompt string provides the input — in this case, the text to be summarized. A well-crafted prompt should have at least the following three components: a description of the task, input text, and the output indicator. But to achieve the best results, include several examples that demonstrate successful task inputs and outputs to the model.
- The max_tokens integer parameter sets the length of the model’s output. Although it’s a slight oversimplification, think of the tokens as words.
For example, you can summarize the text, “Home is anywhere I find peace, even if it’s a foreign land”, by providing the following prompt.
prompt = ' "My husband is in the process of redoing our lawn. He recently began killing off all our grass. I didn't want our neighbors to think we were neglectful homeowners, so I made a sign..."
In summary: "My husband is redoing our lawn."
"Home is anywhere I find peace, even if it's a foreign land"
In summary:”'
The first line of the prompt describes the task. Then, the Text and In summary statements give the LLM an example of successful task execution. In summary, is a simple output indicator, but you can also provide more elaborate indicators to tailor your results.
The Generate endpoint also accepts several optional parameters:
- The temperature float accepts values between 0.0 and 5.0, and defaults to 0.75. Higher temperature values increase the degree of randomness of your model’s output. This is the most impactful optional parameter.
- Specifying the model string parameter in your request tells the Cohere Platform which model to use. The Generate endpoint uses the xlarge model by default, but you can also use large, medium, and small models, which are faster but better suited for more straightforward tasks.
- return_likelihoods string — set to one of GENERATION|ALL|NONE. This allows you to specify how you would like to receive the likelihood - a measure of the model’s level of surprise of this token in a sentence.
Next, you’ll add a function that sends requests to the Generate endpoint.
Paste the following code into src/App.js, above the getRedditPosts function inside the previously defined class App extends Component.
constructor(props){
super(props);
this.state = {
cohereSummaries:[]
};
};
getPostsSummaries(posts) {
const summaries = [];
const headers = {
Authorization: "[YOUR-API-KEY]",
"Content-Type": "application/json"
};
posts.forEach(post => {
const postText = post.data.selftext;
const prompt = `"In my teens I developed a deep passion for literature and analyzing texts to discover meaning. Simultaneously, I became interested in philosophy. Obviously, these interests intertwine quite a bit, and I often find that what I learn regarding one topic adds valuable perspective to my reading of the other. However, I recently spoke to a philosophy professor who held the opinion that literature, when it is arguing something, is absolutely worthless. This is because (regardless of where you stand on death of the author) a piece of literature was crafted by an individual who held complete control over the actual narrative events of the text. He went so far as to say it was practically a fallacy to assume that we could learn anything about the world from a piece of fiction, no matter how thoroughly it explores its themes and ideas. For instance, he says that Macbeth is a play that tells us how ambition for its own sake leads us into nihilism. However, he also holds that, even if the point ‘Macbeth’ argues is true, we can in no way LEARN this from Macbeth, as it is a work of fiction. I hate to admit it, but his point has got me stumped. How do you all react and respond to this idea?"
In summary: "I have a deep interest in philosophy and literature, however a professor argued that literature is incompatible with philosophy as literature is purely fiction. I do not agree but I am unable to counter this point and would like assistance."
"Let's say for example a novel was adapted into a graphic novel(an entire novel is adapted, not just a small summary). If the graphic novel just adapts the novel word for word, every word aside from environmental description, using the same storytelling mechanisms of the novel, is it any good as an adaptation, or must an adaptation change some things in order to be good?(let us also assume the novel in question is a great novel and not a mediocre one). Personally I find that such an adaptation is inherently not good, but what are your thoughts?"
In summary:"Can an adaptation of a novel that does not change anything still contain value? Does it need to add something to the original? It is my opinion that it does but I would like other opinions."
"Hi, so I recently fell in love with the stories of a certain author but I lost the book and sadly can not remember his name. Search engines dont help so I thought maybe someone here might know. It is very niche I am afraid but still worth the try. What I remember: The guy was born in Iran around 1950, has a iranian first and last name, but moved to Germany in his late teens. He had a high education and worked in journalism I believe and mostly wrote short stories with fantasy elements in german. The one I remember best is about a tree wich can decide what kind of leaves it grows. His first ones are apple and peach leaves because his neighbours convinced him to. After his first hard winter he gets carried to a new and better place near a river by his bird friends. To show his gratitude he starts growing bird leaves. I will be happy about all kinds of comments!"
In summary: "I love stories from a certain author from Iran who wrote in German. He wrote short stories with fantasy elements. I am unable to remember or discover his name and would like help."
"${postText}"
In summary:"`
const data = {
prompt: prompt,
max_tokens: 150,
return_likelihoods: "GENERATION",
stop_sequences:['"'],
temperature:0.7,
k:0,
p:0.75
};
fetch('https://api.cohere.ai/xlarge/generate', {
method: 'POST',
headers: headers,
body: JSON.stringify(data),
})
.then((response) => {
if(response.status !== 200){
if(response.status == 498){ //498 Implies profanity filtering from Cohere
console.log("post removed due to potentially offensive language");
}
else{
console.log("ERROR: connection refused")
}
return;
}
return response.json()
})
.then((data) => {
if(data !== null && data != undefined){
summaries.push({
permalink: post.data.permalink,
summary: data.text,
likelihood: data.likelihood
})
console.log("Summary: ", data.text);
this.setState({cohereSummaries: summaries});
}
});
});
}
In the code above, you initialized an empty array, cohereSummaries, in your constructor. You then defined the getPostsSummaries function, which accepts a list of Reddit posts.
In the getPostsSummaries function, you configured your request headers. Remember to replace YOUR_API_KEY with your retrieved Cohere API key.
The same function then looped through the list of posts passed to it. For each post, you crafted a prompt following the rules discussed earlier, initialized your request parameters in the data variable, and then initiated a request to Cohere with fetch.
Finally, you updated your empty cohereSummaries array with this.setState({cohereSummaries: summaries}).
Now, with getPostsSummaries in place, invoke it in the getRedditPosts function with this.getPostsSummaries(posts).
Your getRedditPosts() function should now appear as below.
getRedditPosts() {
//Fetch extra posts so that posts without text can be filtered
fetch('https://www.reddit.com/r/literature/top.json??&t=all&limit=100').then(res => {
if(res.status !== 200){
console.log("ERROR: unable to retrieve posts from reddit.com")
return;
}
res.json().then(data => {
if(data !== null){
const posts = data.data.children;
console.log("REDDIT POSTS: ", posts);
//Filter posts without text
//Traverse array in reverse order to avoid array issues when removing items
var i = posts.length
while(i--){
if(posts[i].data.selftext == ""){
posts.splice(i,1);
}
}
//Drop excess posts
if(posts.length > 20){
posts.length = 20;
}
this.getPostsSummaries(posts);
}
})
});
}
Now, run your local development server. If everything goes well, you’ll see the Cohere summaries logged in your browser’s console.
Displaying Post Summaries
Next, you’ll render the post summaries to the DOM using the steps outlined below.
First, create a file called src/components/Article.js and paste the snippet below into the file.
import React, { Component } from 'react';
class Article extends Component {
render() {
return (
<li>
<article>
<p><b>SUMMARY: </b>{this.props.post.summary}</p>
<p><a href={"https://reddit.com" + this.props.post.permalink} target="blank">
Link to original post
</a></p>
</article>
</li>
);
}
}
export default Article
In the above code, you initialized a component: Article. This component displays the summary of a Reddit post and a link to the original post for reference.
Now, open the src/App.js file and add import Article from './component/Article'; to the top of the file. Next, replace the render() block in src/App.js with the snippet below.
render(){
return (
<div className="App">
<p><h2>Top 20 Posts of all Time in <i> r/literature</i></h2></p>
<p>Posts with potentially offensive language have been removed from the list as decided by Cohere, which could reduce the number of posts below 20</p>
<p>Many Reddit posts contain images instead of text - these posts have been skipped</p>
<div className="article">
<ol>
{
(this.state.cohereSummaries.length > 0) ? this.state.cohereSummaries.map((post, index) => <Article key={index} post={post} />) : <p>processing... length: {this.state.cohereSummaries.length}</p>
}
</ol>
</div>
</div>
);
}
In the snippet above, the Article component renders each summary in the cohereSummaries array that you declared in the constructor, then populates it with data in the getPostsSummaries function.
Finally, fire up your local development server. You should then see summaries of the top 20 Reddit posts of all time, along with their links and confidence levels, as shown in the image below.
Conclusion
You've now experienced how large language models are enhancing the capabilities of machines to process language. Moreover, Cohere’s straightforward API makes it even simpler to integrate NLP features into a project.
Because Cohere’s LLMs have been trained with a massive dataset, they are well equipped to perform tasks like text summarization.
To explore all that Cohere can do for you, sign up for an account and get started!