{"id":3363,"date":"2021-02-28T16:00:52","date_gmt":"2021-02-28T16:00:52","guid":{"rendered":"https:\/\/thenextweb.com\/?p=1340644"},"modified":"2021-02-28T16:00:52","modified_gmt":"2021-02-28T16:00:52","slug":"most-ads-you-see-are-chosen-by-a-reinforcement-learning-model-heres-how-it-works","status":"publish","type":"post","link":"https:\/\/www.londonchiropracter.com\/?p=3363","title":{"rendered":"Most ads you see are chosen by a reinforcement learning model \u2014 here\u2019s how it works"},"content":{"rendered":"\n

Every day, digital advertisement agencies serve billions of ads on news websites, search engines, social media networks, video streaming websites, and other platforms. And they all want to answer the same question: Which of the many ads they have in their catalog is more likely to appeal to a certain viewer? Finding the right answer to this question can have a huge impact on revenue when you are dealing with hundreds of websites, thousands of ads, and millions of visitors.<\/p>\n

Fortunately (for the ad agencies, at least), <\/span>reinforcement learning<\/a>, the branch of artificial intelligence that has become renowned for <\/span>mastering board and video games<\/a>, provides a solution. Reinforcement learning models seek to maximize rewards. In the case of online ads, the RL model will try to find the ad that users are more likely to click on.<\/p>\n

The digital ad industry generates hundreds of billions of dollars every year and provides an interesting case study of the powers of reinforcement learning.<\/p>\n

Na\u00efve A\/B\/n testing<\/h2>\n

To better understand how reinforcement learning optimizes ads, consider a very simple scenario: You\u2019re the owner of a news website. To pay for the costs of hosting and staff, you have entered a contract with a company to run their ads on your website. The company has provided you with five different ads and will pay you one dollar every time a visitor clicks on one of the ads.<\/p>\n

Your first goal is to find the ad that generates the most clicks. In advertising lingo, you will want to maximize your click-trhough rate (CTR). The CTR is ratio of clicks over number of ads displayed, also called impressions. For instance, if 1,000 ad impressions earn you three clicks, your CTR will be 3 \/ 1000 = 0.003 or 0.3%<\/span><\/span>.<\/p>\n

Before we solve the problem with reinforcement learning, let\u2019s discuss A\/B testing, the standard technique for comparing the performance of two competing solutions (A and B) such as different webpage layouts, product recommendations, or ads. When you\u2019re dealing with more than two alternatives, it is called A\/B\/n testing.<\/p>\n

[Read: How do you build a pet-friendly gadget? We asked experts and animal owners<\/span><\/a>]<\/em><\/p>\n

In A\/B\/n testing, the experiment\u2019s subjects are randomly divided into separate groups and each is provided with one of the available solutions. In our case, this means that we will randomly show one of the five ads to each new visitor of our website and evaluate the results.<\/p>\n

\n
\"normal<\/a><\/figure>\n<\/div>\n

Say we run our A\/B\/n test for 100,000 iterations, roughly 20,000 impressions per ad. Here are the clicks-over-impression ratio of our ads:<\/p>\n

Ad 1: 80\/20,000 = 0.40% CTR<\/p>\n

Ad 2: 70\/20,000 = 0.35% CTR<\/p>\n

Ad 3: 90\/20,000 = 0.45% CTR<\/p>\n

Ad 4: 62\/20,000 = 0.31% CTR<\/p>\n

Ad 5: 50\/20,000 = 0.25% CTR<\/p>\n

Our 100,000 ad impressions generated $352 in revenue with an average CTR of 0.35%. More importantly, we found out that ad number 3 performs better than the others, and we will continue to use that one for the rest of our viewers. With the worst performing ad (ad number 2), our revenue would have been $250. With the best performing ad (ad number 3), our revenue would have been $450. So, our A\/B\/n test provided us with the average of the minimum and maximum revenue and yielded the very valuable knowledge of the CTR rates we sought.<\/p>\n

Digital ads have very low conversion rates. In our example, there\u2019s a subtle 0.2% difference between our best- and worst-performing ads. But this difference can have a significant impact on scale. At 1,000 impressions, ad number 3 will generate an extra $2 in comparison to ad number 5. At a million impressions, this difference will become $2,000. When you\u2019re running billions of ads, a subtle 0.2%<\/span><\/span> can have a huge impact on revenue.<\/p>\n

Therefore, finding these subtle differences is very important in ad optimization. The problem with A\/B\/n testing is that it is not very efficient at finding these differences. It treats all ads equally and you need to run each ad tens of thousands of times until you discover their differences at a reliable confidence level. This can result in lost revenue, especially when you have a larger catalog of ads.<\/p>\n

Another problem with classic A\/B\/n testing is that it is static. Once you find the optimal ad, you will have to stick to it. If the environment changes due to a new factor (seasonality, news trends, etc.) and causes one of the other ads to have a potentially higher CTR, you won\u2019t find out unless you run the A\/B\/n test all over again.<\/p>\n

What if we could change A\/B\/n testing to make it more efficient and dynamic?<\/p>\n

This is where reinforcement learning comes into play. A reinforcement learning agent starts by knowing nothing about its environment\u2019s actions, rewards, and penalties. The agent must find a way to maximize its rewards.<\/p>\n

In our case, the RL agent\u2019s actions are one of five ads to display. The RL agent will receive a reward point every time a user clicks on an ad. It must find a way to maximize ad clicks.<\/p>\n

The multi-armed bandit<\/h2>\n
\n
\n

\"multi-armed<\/a>
<\/i><\/a>The multi-armed bandit must find ways to discover one of several solutions through trial and error<\/figcaption><\/figure>\n<\/p>\n<\/figure>\n<\/div>\n

In some reinforcement learning environments, actions are evaluated in sequences. For instance, in video games, you must perform a series of actions to reach the reward, which is finishing a level or winning a match. But when serving ads, the outcome of every ad impression is evaluated independently; it is a single-step environment.<\/p>\n

To solve the ad optimization problem, we\u2019ll use a \u201cmulti-armed bandit\u201d (MAB), a reinforcement learning algorithm that is suited for single-step reinforcement learning. The name of the multi-armed bandit comes from an imaginary scenario in which a gambler is standing at a row of slot machines. The gambler knows that the machines have different win rates, but he doesn\u2019t know which one provides the highest reward.<\/p>\n

If he sticks to one machine, he might lose the chance of selecting the machine with the highest win rate. Therefore, the gambler must find an efficient way to discover the machine with the highest reward without using up too much of his tokens.<\/p>\n

Ad optimization is a typical example of a multi-armed bandit problem. In this case, the reinforcement learning agent must find a way to discover the ad with the highest CTR without wasting too many valuable ad impressions on inefficient ads.<\/p>\n

Exploration vs exploitation<\/h2>\n

One of the problems every reinforcement learning model faces is the \u201cexploration vs exploitation\u201d challenge. Exploitation means sticking to the best solution the RL agent has so far found. Exploration means trying other solutions in hopes of landing on one that is better than the current optimal solution.<\/p>\n

In the context of ad selection, the reinforcement learning agent must decide between choosing the best-performing ad and exploring other options.<\/p>\n

One solution to the exploitation-exploration problem is the \u201cepsilon-greedy\u201d (\u03b5-greedy) algorithm. In this case, the reinforcement learning model will choose the best solution most of the time, and in a specified percent of cases (the epsilon factor) it will choose one of the ads at random.<\/p>\n

\n

\"exploration<\/a>
<\/i><\/a>Every reinforcement learning algorithm must find the right balance between exploiting optimal solutions and exploring new options<\/figcaption><\/figure>\n<\/p>\n<\/figure>\n

Here\u2019s how it works in practice. Say we have an epsilon-greedy MAB agent with the \u03b5 factor set to 0.2. This means that the agent chooses the best-performing ad 80%<\/span><\/span> of the time and explores other options 20% of the time.<\/p>\n

The reinforcement learning model starts without knowing which of the ads performs better, therefore it assigns each of them an equal value. When all ads are equal, it will choose one of them at random each time it wants to serve an ad.<\/p>\n

After serving 200 ads (40 impressions per ad), a user clicks on ad number 4. The agent adjusts the CTR of the ads as follows:<\/p>\n

Ad 1: 0\/40 = 0.0%<\/p>\n

Ad 2: 0\/40 = 0.0%<\/p>\n

Ad 3: 0\/40 = 0.0%<\/p>\n

Ad 4: 1\/40 = 2.5%<\/p>\n

Ad 5: 0\/40 = 0.0%<\/p>\n

Now, the agent thinks that ad number 4 is the top-performing ad. For every new ad impression, it will pick a random number between 0 and 1. If the number is above 0.2 (the \u03b5 factor), it will choose ad number 4. If it\u2019s below 0.2, it will choose one of the other ads at random.<\/p>\n

Now, our agent runs 200 other ad impressions before another user clicks on an ad, this time on ad number 3. Note that of these 200 impressions, 160 belong to ad number 4, because it was the optimal ad. The rest are equally divided between the other ads. Our new CTR values are as follows:<\/p>\n

Ad 1: 0\/50 = 0.0%<\/p>\n

Ad 2: 0\/50 = 0.0%<\/p>\n

Ad 3: 1\/50 = 2.0%<\/p>\n

Ad 4: 1\/200 = 0.5%<\/p>\n

Ad 5: 0\/50 = 0.0%<\/p>\n

Now the optimal ad becomes ad number 3. It will get 80%<\/span><\/span> of the ad impressions. Let\u2019s say after another 100 impressions (80 for ad number three, four for each of the other ads), someone clicks on ad number 2. Here\u2019s how what the new CTR distribution looks like:<\/p>\n

Ad 1: 0\/54 = 0.0%<\/p>\n

Ad 2: 1\/54 = 1.8%<\/p>\n

Ad 3: 1\/130 = 0.7%<\/p>\n

Ad 4: 1\/204 = 0.49%<\/p>\n

Ad 5: 0\/54 = 0.0%<\/p>\n

Now, ad number 2 is the optimal solution. As we serve more ads, the CTRs will reflect the real value of each ad. The best ad will get the lion\u2019s share of the impressions, but the agent will continue to explore other options. Therefore, if the environment changes and users start to show more positive reactions to a certain ad, the RL agent can discover it.<\/p>\n

After running 100,000 ads, our distribution can look something like the following:<\/p>\n

Ad 1: 123\/30,600 = 0.40% CTR<\/p>\n

Ad 2: 67\/18,900 = 0.35% CTR<\/p>\n

Ad 3: 187\/41,400 = 0.45% CTR<\/p>\n

Ad 4: 35\/11,300 = 0.31% CTR<\/p>\n

Ad 5: 15\/5,800 = 0.26% CTR<\/p>\n

With the \u03b5-greedy algorithm, we were able to increase our revenue from $352 to $426 on 100,000 ad impressions and an average CTR of 0.42%. This is a great improvement over the classic A\/B\/n testing model.<\/p>\n

\n