Game theory can make the computer more reliable

Quanta Magazine

That’s exactly what’s happening with many large language models (LLMs), the ultra-powerful machine learning tools that power ChatGPT and other marvels of artificial intelligence.
Dubbed the consensus game, this simple procedure pits an LLM against itself, using the tools of game theory to improve the model’s accuracy and internal consistency.
For decades, he said, language models have generated responses to prompts in the same way.
Posing a far greater challenge for AI researchers was the game of Diplomacy—a favorite of politicians like John F. Kennedy and Henry Kissinger.
The team’s goal, he said, “was to build the best language model we could for the purposes of playing this game.” But what if instead they focused on building the best game they could to improve the performance of large language models?
Again, these options can come from a human, a list, or a search carried out by the language model itself.
To test the effects of the consensus game, the team tried out a set of standard questions on various moderate-size language models with 7 billion to 13 billion parameters.
“The main objective of this project is to make language models more strategic,” he said.

POSITIVE

This story’s initial publication was in Quanta Magazine.

Let’s say you had a friend who, depending on how you posed the question, would respond differently. One response to the question “What is the capital of Peru?” would be given, while another response to the question “Is Lima the capital of Peru?” would be given. You would almost certainly find it difficult to trust any response your friend provided, and you would probably be a little concerned about their mental state.

Large language models (LLMs) are incredibly potent machine learning tools that underpin ChatGPT and other artificial intelligence marvels. This is precisely what’s happening with many LLMs. When a question is discriminative—requiring a choice between options—it typically yields a different answer than an open-ended generative question. Ph.D. candidate at the Massachusetts Institute of Technology Athul Paul Jacob stated, “There is a disconnect when the same question is phrased differently.”.

A game that drives the two modes of a language model to find a consensus answer was developed by Jacob and colleagues to improve consistency in the answers provided by the model and increase its overall reliability. This straightforward process, known as the “consensus game,” pits an LLM against itself and applies game theory techniques to enhance the internal consistency and accuracy of the model.

Shayegan Omidshafiei, chief scientific officer of Field AI, a robotics company, stated that there has been very little research done on self-consistency in these models. “By designing a game that the language model can play with itself, this paper is among the first that addresses this in a clever and methodical way. “.

Ahmad Beirami, a research scientist at Google Research, continued, “It’s really exciting work.”. Language models have produced prompt responses in the same manner for decades, he claimed. “MIT researchers have introduced a completely new paradigm that may lead to a plethora of new applications with their innovative idea of incorporating a game into this process. “.

Making Play a Work.

Unlike previous approaches that determined an AI program’s success based on how well it played games, the new work uses games to improve AI. For instance, IBM’s Deep Blue computer defeated chess master Garry Kasparov in 1997, marking a significant advancement for “thinking machines.”. Nineteen years later, it was revealed that humans were no longer the dominant force in another arena when a Google DeepMind program called AlphaGo defeated former Go champion Lee Sedol four games to one. Additionally, in “zero-sum” games like checkers and two-player poker, where the winner always means the other player loses, machines have surpassed humans.

The game of diplomacy, which politicians like John F. Kennedy loved to play, presented a far bigger challenge for AI researchers. Kennedy and Kissinger, Henry. The game has seven players instead of just two, and it can be challenging to discern their motivations. In order to prevail, a player must bargain, creating cooperative agreements that anyone could break at any time. Since diplomacy is so complicated, a group from Meta was happy when its AI program Cicero achieved “human-level play” after 40 games in 2022. Cicero performed well enough to rank among the top 10% of participants when competing against humans, despite not defeating the world champion.

Jacob, one of the Meta team members, was surprised to learn that Cicero was using a language model to create its player dialogue during the project. He felt there was unrealized potential. He explained that the team’s objective was to “build the best language model we could for the purposes of playing this game.”. “However, what if their attention was directed towards creating the most exceptional game possible to enhance the capabilities of extensive language models?

Consensual Exchanges.

At MIT, Jacob started investigating that question in 2023. He collaborated on the consensus game with Yikang Shen, Gabriele Farina, and his adviser, Jacob Andreas. The main concept originated from the idea that two people could play a cooperative game where the goal is to communicate with each other so that the listener wins when the speaker understands what the other is saying. Specifically, the goal of the consensus game is to balance the two systems in the language model: the discriminator, which manages discriminative questions, and the generator, which manages generative ones.

The team developed this idea into a complete game after a few months of ups and downs. The generator is asked a question first. It may originate from an already-existing list or from a human. For instance, when asked “Where was Barack Obama born?” the generator will return a few potential answers, such as Honolulu, Chicago, and Nairobi. Once more, the language model itself may have searched for these options, or it may have retrieved them from a list.

However, the generator is informed beforehand as to whether it should respond appropriately or incorrectly, contingent on the outcome of a fair coin toss, before it can respond.

The machine tries to provide an accurate response if the result is heads. The discriminator receives the original question from the generator along with its selected response. As a sort of reward, each discriminator receives one point if they can prove that the generator purposefully sent the right answer.

The generator transmits the response it believes to be incorrect if the coin lands on tails. They each receive another point if the discriminator determines that the incorrect response was given on purpose. Here, encouraging agreement is the goal. Jacob compared it to training a dog a trick. When they behave well, you reward them with a treat. “.

Similar to the discriminator, the generator also begins with some basic “beliefs.”. These are presented as a probability distribution pertaining to the various options. Based on data it has gathered from the internet, the generator might, for instance, conclude that Obama has an 80% chance of being born in Honolulu, a 10% chance of being born in Chicago, a 5% chance of being born in Nairobi, and a 5% chance of being born somewhere else. It’s possible for the discriminator to begin with a different distribution. The two “players” lose points for straying too far from their initial beliefs, even though they are still rewarded for coming to an understanding. Because of this setup, the players are encouraged to add to their responses by incorporating their knowledge of the world, which is again derived from the internet, which should improve the model’s accuracy. Without something similar, they could reach a consensus on an entirely incorrect response, as in Delhi, but still score points.

The two systems compete in about 1,000 games for each question. Each side gains knowledge of the other’s beliefs during these many iterations and adjusts its tactics accordingly.

As they eventually reach a state known as Nash equilibrium, the discriminator and the generator start to agree more. In game theory, this is perhaps the most important idea. It symbolizes a certain level of equilibrium in a game—the moment at which changing tactics will not allow any player to improve their own results. For example, in the game of rock, paper, scissors, players perform best when they select each of the three options precisely one-third of the time; they will, without fail, perform worse when using any other strategy.

This can have a variety of outcomes in the consensus game. As the generator outputs the word “Honolulu” for Obama’s birthplace, the discriminator may notice that it receives a point each time it responds with “correct.”. Upon repeated play, both the discriminator and the generator will come to understand that their actions will be rewarded, and as a result, neither will be motivated to change. For this question, there are numerous examples of Nash equilibrium, and this consensus is one of them. In order to maintain the players’ responses grounded in reality, the MIT group also used a modified version of Nash equilibrium that takes into account their prior beliefs.

Overall, the researchers found that regardless of how the question is posed, the language model participating in this game becomes more precise and likely to provide the same response. The group experimented with a set of common questions on a range of moderate-sized language models with 7 billion to 13 billion parameters in order to evaluate the impact of the consensus game. Even much larger models with up to 540 billion parameters routinely performed better than those that hadn’t played in terms of correct response percentage. Internal consistency of a model was also enhanced by playing the game.

The game is theoretically beneficial for any LLM to play against itself, and it would only take a few milliseconds on a typical laptop to complete 1,000 rounds. According to Omidshafiei, “one nice benefit of the overall approach is that it requires no training or modification of the base language model, making it computationally very lightweight.”. “.

Gaming Using Words.

Jacob is currently looking into additional methods to incorporate game theory into LLM research following this initial success. Based on initial findings, a stronger LLM can be enhanced even more by involving an arbitrary number of smaller models in an alternative game that is being dubbed the ensemble game. There would be a minimum of one smaller model acting as an ally and a minimum of one smaller model acting as an adversary in the main LLM. When the primary LLM is asked to name the US president, it receives a point for each time it selects an answer that matches that of its ally and for each time it selects an answer that differs from that of its adversary. Tests indicate that these interactions with much smaller models can improve an LLM’s performance without the need for additional training or parameter adjustments.

That’s just the beginning. According to Ian Gemp, a research scientist at Google DeepMind, “a variety of situations can be viewed as games,” so the tools from game theory can be applied in various real-world settings. In a February 2024 paper, he and associates addressed negotiation scenarios that call for more in-depth discussions than a simple question-and-answer exchange. “This project’s main goal is to make language models more strategic,” he declared.

A journal or conference’s paper review process for acceptance, particularly after a hard review of one’s initial submission, is one example he gave at an academic conference. Researchers can create game trees that chart the options and their potential outcomes, much like those made for poker games, because language models assign probabilities to various responses. After completing this, Gemp stated, “you can begin to compute Nash equilibria and then rank a bunch of rebuttals.”. In essence, the model says to you: This is what we believe you ought to respond with.

scroll to top