Automating User Feedback Monitoring on Discord Using AI

Cohere Team

• 8 min read
Automating User Feedback Monitoring on Discord Using AI

Monitoring product feedback on social media is challenging—even for businesses with a smaller product portfolio or market share. There can be countless messages sent at all hours of the day, with lots of text to review, and little time to analyze and sort them manually.

In this project tutorial, you’ll learn how to create a Discord bot that harnesses the power of natural language processing (NLP) by using Cohere representation models to automatically analyze feedback and classify it into different product areas.

Prerequisites

To build this bot, you’ll need:

  • Node.js: The latest version of Node.js installed on your computer
  • A Cohere account: A valid Cohere account for fine-tuning your model
  • A Discord account: A Discord account to create and manage a bot account

Getting Started

You’ll start by creating a Cohere API key, which allows you to access large language models from your code.

Log in to the Cohere Dashboard, scroll to the bottom to find the API Keys section, and click Create API Key.

Provide a name for the API Key, such as “Feedback Bot,” and click the Create API Key button.

Copy and save the newly generated key, as you’ll need it to use the Cohere API.

Training the Cohere Model

For this tutorial, you’ll use a pre-trained text classification model and finetune it using a collection of example feedback that refers to four different product areas. You’ll do so using a simple two-column .csv spreadsheet containing the text sample and its label. You can find the sample data on GitHub.

Start by clicking on the Create finetune button in the Cohere Dashboard.

Change the Model type to Representation (Embed, Classify), select the Link to your .csv radio button and enter this link. Then, click Preview data.

On the next menu, check the Remove column headings option to exclude the top row of the .csv file, click Review data, and then name your finetune model and click Start finetuning. The finetuning process can take several minutes, so you may want to get a snack.

Once finetuning is completed, you can click on the Model ID to copy it to your clipboard. Save this ID alongside the API key you generated earlier.

Now, you’ll test and make sure this finetuned model works. Click on Playground at the bottom of the dashboard and select the finetuned model. Enter some example text, such as “current UI design is lacking in aesthetics” (Product area 4) or “how do we perform sentiment analysis on tweets” (Product area 2), to see how they perform.

Notice that, on the right side, there’s a column called Confidence Levels. You’ll set a threshold value of 0.8 to filter out low-confidence predictions, so you only send feedback on text with high-confidence levels. This means that most of the messages unrelated to product feedback will also be ignored.

Setting Up a Discord Bot

Next, you need to configure a Discord application for the project.

Create a Discord Server (Optional)

If you don’t already have a Discord server where you can add a Discord bot application, you’ll need to create one from the Discord app on the browser or the downloaded application. Click the + button on the bottom left of the icon menu.

Select Create My Own or select from one of the templates. Skip the next question and then give a name for the server, such as “Feedback Test Server.”

Create a Discord Application

Open the Discord Developer Portal and select New Application on the top right.

Name the application something that you’d like the bot to display and click Create.

Navigate to the Bot tab, press Add Bot, then click Yes, do it to create a new Discord bot.

Scroll down to the Privileged Gateway Intents section and toggle on the Message Content Intent option, so your bot can read the text of chat messages. Save the changes.

Next, open the OAuth2 > URL Generator page using the left menu and check the following options: bot, Read Messages/View Channels, and Send Messages. Scroll to the bottom and copy the generated link—this is the link to invite the bot into a Discord server.

Open a new browser tab and go to the generated link. Select the Discord server you’ll use, then click Continue. Discord will ask you to authorize the permissions you previously selected, and your bot will be added to the Discord server.

Now you need the bot authentication token. Return to the application page in the Developer Portal, reopen the Bot tab, and click Reset Token.

This step will create an app token that you’ll need for your code. Click the Copy button and save it with your Cohere API key from the earlier step.

The final step for your Discord app is to create a test channel on the server. This is where it will send feedback analysis. You can use the default #general channel to check messages for product feedback.

Create a text channel on the server by right-clicking in an empty area of the channel list section, selecting Create Channel, and naming it #feedback. If you want, you can create it as a private channel and organize it into its own category, so that it’s closer to a real scenario.

In your Discord application, click on User Settings and navigate to Advanced. Toggle the Developer Mode to enable it and exit the settings.  

Right-click on the #general channel and choose Copy ID to get the channel's ID. Then, save it with the other saved keys/tokens. Repeat this step for the #feedback channel.

Time to Code

Open a terminal window or command prompt to a new project folder and initialize the Node.js project with npm init -y.

Next, install the required dependencies for this project using npm install cohere-ai discord.js dotenv.

With your preferred code editor, open the project folder and create a new .env file at the project root. Then, enter your key/token information that you saved during the project setup process as follows.

COHERE_API_KEY=YOUR_API_KEY
COHERE_FEEDBACK_MODEL_ID=YOUR_COHERE_MODEL_ID
DISCORD_API_KEY=YOUR_DISCORD_APP_API_KEY
DISCORD_GENERAL_CHANNEL_ID=YOUR_GENERAL_CHANNEL_ID
DISCORD_FEEDBACK_CHANNEL_ID=YOUR_FEEDBACK_CHANNEL_ID

Save and close the .env file and create an index.js file to contain your bot code.

Initialize the environment variables using dotenv.

require("dotenv").config();

Below that, import the cohere-ai and discord.js modules.

const cohere = require("cohere-ai");
const { Client, GatewayIntentBits } = require("discord.js");

Next, you can set up some constants for your code from the environment variables.

const COHERE_API_KEY = process.env.COHERE_API_KEY
const COHERE_FEEDBACK_MODEL_ID = process.env.COHERE_FEEDBACK_MODEL_ID;
const FEEDBACK_SCORE_THRESHOLD = 0.8;
const DISCORD_API_KEY = process.env.DISCORD_API_KEY
const DISCORD_GENERAL_CHANNEL_ID = process.env.DISCORD_GENERAL_CHANNEL_ID;
const DISCORD_FEEDBACK_CHANNEL_ID = process.env.DISCORD_FEEDBACK_CHANNEL_ID;

Then, create the Cohere client and the Discord client.

// Initialize API clients
cohere.init(COHERE_API_KEY, "2021-11-08");
const discordClient = new Client({ intents: [GatewayIntentBits.Guilds, GatewayIntentBits.GuildMessages, GatewayIntentBits.MessageContent] });

Now, you’ll create a helper function that can take any text message and try to classify it using the finetuned model you trained. This function calls the classifyMessage function in the cohere-api module with your finetuned model ID. Then, it returns the label and score of the best prediction.

// Classify each message and return the label and the highest prediction confidence of
// best label
async function classifyMessage(message) {
    const response = await cohere.classify({
        model: COHERE_FEEDBACK_MODEL_ID,
        inputs: [message]
    });

    if (!!response.body.classifications && !!response.body.classifications[0]) {
        let [prediction, best] = ["", 0];
        for (const { option, confidence } of response.body.classifications[0].confidences) {
            if (confidence > best) {
                prediction = option
                best = confidence
            }
        }

        return [prediction, best]
    }
};

Add a Discord message event handler to check sent messages. If the message is in the channel where you’re checking for product feedback, and the prediction confidence is high enough, it should notify the #feedback channel.

// Listen for new messages in the specified channel. If a message exceeds the threshold for
// feedback, route it to the correct product owner.
discordClient.on("messageCreate", async msg => {
    const { author, channelId, content } = msg;

    if (channelId === DISCORD_GENERAL_CHANNEL_ID) {
        try {
            const [prediction, confidence] = await classifyMessage(content);

            if (confidence > FEEDBACK_SCORE_THRESHOLD) {
                const message = `${author.username} just gave some product feedback for ${prediction} on Discord:\n\n${content}`;
                discordClient.channels.cache.get(DISCORD_FEEDBACK_CHANNEL_ID).send( message );
                console.log(`message had label ${prediction} with score ${confidence}:\n"${content}"`);
            }
            else {
                console.log(`message had label ${prediction} with score ${confidence} so it was ignored:\n"${content}"`);
            }
        }
        catch (err) {
            console.error(err);
        }
    }
});

Finally, at the bottom of the code, you can add an event handler for when your Discord client is ready and have the bot log in using your Discord API key.

discordClient.once("ready", () => {
    console.log("~~~ ready to bring the specifications from the customers to the software engineers ~~~\n")
});

discordClient.login(DISCORD_API_KEY);

For your reference, the full project code is also available to download on GitHub.

Putting the Bot to the Test

You’re ready to run the bot and see how it performs.

From your terminal window or command prompt, run node index.js to start the Discord bot. Try typing some of the product feedback messages into the #general channel, such as “what is this product?” and “I want more than 3 results per endpoint.” You can also try a message like, “this message should be ignored” to check that non-feedback messages return a low confidence value and are ignored.

Cool, right?

What’s Next

As you can see, it’s very easy to analyze and sort natural language text when you use a Cohere representation model. You can finetune it the way you want and add more training data to improve the accuracy of your classification.

While this is only an example project, the code can be extended to connect to other social media, such as Twitter, and analyze text across different sources.

To learn more, you can read about finetuning Cohere representation models or check out the full Cohere documentation. Finally, go out and connect with other members of the Cohere community on Discord.