Last week, our CEO and co-founder Aidan Gomez was featured on stage at the Elevate Festival in Toronto. He was joined by Stephen Marche, a novelist and essayist who has written about NLP for The New Yorker, The Atlantic and The New York Times.

Aidan and Stephen sat down for a discussion on language technology and what the future holds. In case you missed it, here’s part of their conversation:

Stephen: My “holy shit” moment using language models was when I was writing a piece for The New Yorker. I asked GPT-3 to finish famous unfinished poems, and they worked. They truly sounded like Coleridge or Shakespeare. When was your “holy shit” moment?

Aidan: It happened shortly after the Transformer paper came out. I was in Toronto, and I got an email from my colleague back at Google. He sent me what was seemingly a Wikipedia page on the Transformer. I started reading and it went into a story about a Japanese punk band, how the members had broken up. At the end of the email he said, “I just put in ‘the Transformer,’ everything else was written by the model.” I was floored. Up until that point, our models could barely do anything. They couldn’t spell correctly. He trained a language model on Wikipedia and it crafted a super compelling story about the Transformer, the Japanese punk band.

Stephen: There’s so much of this that I don’t understand. Obviously, my PhD is in Shakespeare so I’m not supposed to understand it, but you are. Can you tell me what you do and don’t understand about the process?

Aidan: I’m so close to the nuts and bolts of it that I often just see a bunch of matrix multiplications and floating point numbers. But when I step back and I look at the outputs — at a system we built where you can say “hey solve this problem for me” and it solves the problem — that is so extraordinary. It’s still magical for me. There’s still so much to be understood.

I understand how you source the data. I understand how the model is trained on that data. I understand how to scale up. When you put those three things together and actually get the output — you’re sitting in front of a trained model — I still don’t fully understand why the outputs are the outputs. Why does a model pick one option over another option? Getting into the way it makes those selections, that’s still an area of active research.

Stephen: Why aren’t we seeing more of this technology out in the world?

Aidan: Yeah, there's frustration there. We were promised that AI will change the world, and I’m just not seeing it. I’m a consumer too — I use all the same products that everyone else uses. I know the technology, I know what it’s capable of, but it’s not out there.

For Cohere, the mission is to push it out further. The way we’re doing that is by trying to lower barriers. One of the largest barriers that I’m sure a lot of people are aware of is that the people who know how to do this stuff — MLEs or machine learning engineers — we can’t train enough of them. There’s a supply-demand dynamic where there’s not enough talent on the face of the planet and there won’t be.

Stephen: Why?

Aidan: It’s going to take us so long to meet that demand — decades of education and new students. The way to bring AI into the products of today is not to train a bunch of people with highly specialized knowledge, instead it’s to present the technology in a different way.

At Cohere, we’re creating an interface onto Transformers and onto supercomputers that's accessible to anyone, to any developer. Using this technology should be intuitive and natural. That’s the mission; that’s our product vision.

Stephen: When you imagine where language AI is going to be in 5-10 years, what do you see?

Aidan: In gaming, for instance, today if you’re interacting with a character, there’s a dialogue tree that someone’s written, and there’s maybe ten paths through that dialogue tree. Every single player has the same experience. I imagine a world where games have a dialogue tree of 8 or 10 billion paths, and everyone experiences a different conversation with that character. Every single play-through is unique.

When you introduce this concept to the rest of the world, you have more mundane examples, like in customer support. As soon as I get a customer support chatbot, the first thing I write is “human,” “I want to talk to a human.” If we actually had compelling models of language — if we could actually create a chatbot that a human wants to talk to — it would change the interface of tech. Dialogue would be the interface. You could talk to your technology.

Right now, we must learn the computer’s language. I went to school for five years to learn how to talk to a computer, to tell it what to do, and to code it. I learned to speak its language. We’re not yet in a place where our products speak our language. That will change.