{"id":33,"date":"2020-01-21T09:30:14","date_gmt":"2020-01-21T08:30:14","guid":{"rendered":"https:\/\/www.codemotion.com\/magazine\/multi-armed-bandits-a-better-way-to-a-b-test\/"},"modified":"2020-02-14T10:34:46","modified_gmt":"2020-02-14T09:34:46","slug":"multi-armed-bandits-a-better-way-to-a-b-test","status":"publish","type":"post","link":"https:\/\/www.codemotion.com\/magazine\/frontend\/design-ux\/multi-armed-bandits-a-better-way-to-a-b-test\/","title":{"rendered":"Multi-armed bandits \u2013 a better way to A\/B test?"},"content":{"rendered":"

A\/B testing<\/span><\/strong> is a well-known approach for determining which of several options will work best for your users<\/span>. However, it also has some well-known issues. In this talk at Codemotion Amsterdam 2019<\/strong>, Claudia Vicol<\/strong> (head of Platform Infrastructure Data Engineering<\/span> at eBay\u2019s Marktplaats), presented an alternative approach.<\/p>\n

eBay Marktplaats<\/h2>\n

Marktplaats<\/strong> is one of the largest online marketplaces in the Netherlands<\/span>. Over half of the population<\/span> has used it to buy or sell in the last year and they get 8.2m unique visitors a month, viewing over 1 billion adverts. Unlike the classic auction-based eBay, Marktplaats is part of eBay Classifieds<\/strong>.<\/p>\n

Since they make money on every sale<\/span>, it is in their interest to encourage users<\/span> to make more purchases. One of the main ways to do this is the recommender system<\/strong> that suggests other products<\/span> when you look for an item. The problem is, testing alternatives is risky<\/strong> \u2013 if you get it wrong, you lose money. Also, the limited screen space<\/strong>, particularly in the application<\/span>, leads to pages becoming \u201csacred\u201d, with product<\/span> owners refusing to allow them to change. So, how can you test alternative recommenders efficiently?<\/p>\n

A\/B Testing<\/h2>\n

The classic approach to testing alternatives in any user<\/span>-facing application<\/span> is A\/B testing<\/span><\/strong>. Here, the user<\/span> base is divided into equal groups, with each group seeing a different version<\/span> of the page. After a couple of weeks, you look at the data<\/span> to decide which page performed best. For instance, you might split users<\/span> into three groups. Each group sees a different version<\/span> of the recommender. As an example, if a user<\/span> looks for a new tablet<\/span>, the recommenders might show other offers for that same tablet<\/span>, other tablets with a similar specification<\/span>, or accessories for the tablet<\/span>. After two weeks, the group that made the most purchases is the \u201cwinner\u201d (eBay measure success as the return per 1,000 clicks). This is the so-called greedy approach because it is greedily trying to maximise the return.<\/p>\n

Sadly, there are a lot of problems with this approach<\/strong>:<\/p>\n

    \n
  1. By definition, most of your users are being shown a suboptimal recommender<\/strong> for the duration of the experiment<\/span>. This is a huge cost for any experiment<\/span>.<\/li>\n
  2. A\/B testing<\/span> isn\u2019t personalised<\/strong>. People are randomly assigned<\/span> to the groups with no effort made to categorise<\/span> them.<\/li>\n
  3. It isn\u2019t repeatable<\/strong>. Often, the next time you run the exact same experiment<\/span>, a different winner will emerge.<\/li>\n
  4. It is really hard to do this across wide geographical areas<\/strong>.<\/li>\n
  5. Only a single experiment<\/span><\/strong> can run at a time. This is a big drawback when there are multiple teams queuing up with new things to try.<\/li>\n<\/ol>\n

    A better way?<\/h2>\n

    The solution must be to stop experimenting. Or rather, to try to minimise the impact of the experiments while maximising their efficacy. In other words, balance the exploratory phase (where you try to find the best recommender) with the exploitation phase where you actually generate sales<\/span> from the recommender. This is actually a very common problem. For instance, it\u2019s the same as the problem facing news sites when choosing the best headline for an article. The solution is to use a concept from probability<\/span> theory known as the multi-armed bandit<\/strong> (or MAB<\/strong>).<\/p>\n

    Conceptually, the multi-armed bandit assumes you enter a casino. You are faced with a number of slot machines with different rules and payouts. Some machines will pay out small sums regularly, others may pay out large sums, but only rarely. You have limited resources, but you want to know how<\/span> to maximise your winnings across the night. Importantly, before playing, you have no prior knowledge of the rules for each machine. If you had infinite time and money, you could easily solve this. But you only have a fixed time and limited money.<\/p>\n

    Epsilon-greedy method<\/h2>\n

    The first MAB solution that Marktplaats looked at is called epsilon-greedy<\/span><\/strong>. Epsilon-greedy<\/span> has an exploration phase where you quickly pick the most successful strategy. During this phase, people are shown a random<\/span> recommender. Then there is an exploitation phase, where you show, say, 90% of users<\/span> the best recommender and the other 10% are shown a random<\/span> choice (where 10% = epsilon = 0.1). You can either go straight to the exploitation phase after one round, or you can be more nuanced and home in on the most successful recommender slowly, as shown below.<\/p>\n

    \u00a0<\/p>\n


    \"Comparing<\/center>\n

    \u00a0<\/p>\n

    While this approach gave eBay double the revenue of the random<\/span> A\/B testing<\/span> method, it still has drawbacks<\/strong>. Firstly, it still isn\u2019t personalised. Secondly, it homes in on a different choice each time it is run.<\/p>\n

    Contextual bandits with Thompson sampling<\/h2>\n

    Ultimately, Claudia\u2019s team focused on a different approach to solving the MAB problem. Namely, Thompson sampling<\/span><\/strong> (so-called after William R. Thompson). Thompson sampling<\/span> isn\u2019t greedy<\/span>. Instead, it uses a Bayesian<\/span> approach<\/strong>. As Wikipedia<\/span> says, it tries to choose the action that\u00a0 \u201cmaximizes the expected reward with respect to<\/span> a randomly<\/span> drawn belief<\/span><\/i>\u201d. Rather than randomly<\/span> assigning users<\/span>, it is able to use complex<\/span> vectors of features when choosing which recommender to assign a given user<\/span>. In eBay\u2019s case, they chose the last five categories the user<\/span> has visited and whether they are on a mobile<\/span> or desktop.<\/p>\n

    \u00a0<\/p>\n


    \"\"<\/center>\n

    \u00a0<\/p>\n

    The process chooses one of five possible recommenders, using a naive Bayesian<\/span> classifier<\/span> to predict which recommender will perform best for a given set of visited categories. In total there are 32 categories, so there are potentially over 200,000 combinations. Initially, the probability<\/span> for each is set to 0.5 (50%). Success is when a user<\/span> clicks on the recommender. Each time it runs, the probabilities<\/span> are updated to favour the variant with the highest reward. Quickly the least successful variants are eliminated. After a while, you are able to generate a sorted list of the performance<\/span> of each variant.<\/p>\n

    Are there any drawbacks?<\/h3>\n

    One big problem with this approach is it takes twice as long to \u201cunlearn\u201d a choice than it does to learn<\/strong>. In the world of classified<\/span> ads, this is a problem, since people\u2019s behaviour changes significantly with the seasons. Another trap (which Claudia herself fell into) is if you reverse the order of the sort. That led to a brief period where people were being shown the least successful recommender! A more technical problem is that many of the probabilities<\/span> soon become vanishingly small. A solution there is to use an exponential weighted moving average<\/span>. Claudia also came across another problem related to how the list was being sorted. Most programming languages<\/span> use a stable sort algorithm. This meant that the same choices were always favoured. Fortunately, the simple solution here is to shuffle the list first.<\/p>\n

    Conclusions and Advice<\/h2>\n

    MAB is clearly a good solution, but it isn\u2019t quite there yet. Claudia still hasn\u2019t solved the problem of what to do when trends change<\/strong>. In other words, it gives good results in the short term, but not in the long term. One key issue when running multiple experiments<\/span> in parallel is knowing how to measure success. How do you equate different measures such as revenue generated and number of reviews? And Claudia will never again forget which order to sort the list in!<\/p>\n

    Claudia ended by answering a couple of key questions. Firstly, how do they know when to unlearn? Having run one of their experiments<\/span> continually for six months, they have seen that they definitely need to be able to unlearn. But it is easier to simply reset the experiment<\/span> periodically. Another key question is related to whether they had tried doing this on historical data<\/span>. The issue here is that the trends<\/span> change, so historical data<\/span> isn\u2019t helpful. It would also require them to create a simulator to replay the data<\/span>. Fortunately, eBay was prepared to let them develop and test this system in production.<\/p>\n\n","protected":false},"excerpt":{"rendered":"

    A\/B test allows to make design choices based on users preferences. To solve its drawbacks, the team of eBay\u2019s Marktplaats uses a different approach.<\/p>\n","protected":false},"author":7,"featured_media":34,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_editorskit_title_hidden":false,"_editorskit_reading_time":0,"_editorskit_is_block_options_detached":false,"_editorskit_block_options_position":"{}","_uag_custom_page_level_css":"","_genesis_hide_title":false,"_genesis_hide_breadcrumbs":false,"_genesis_hide_singular_image":false,"_genesis_hide_footer_widgets":false,"_genesis_custom_body_class":"","_genesis_custom_post_class":"","_genesis_layout":"","footnotes":""},"categories":[76],"tags":[77,4359],"collections":[],"class_list":{"0":"post-33","1":"post","2":"type-post","3":"status-publish","4":"format-standard","5":"has-post-thumbnail","7":"category-design-ux","8":"tag-codemotion-amsterdam","9":"tag-testing","10":"entry"},"yoast_head":"\nMulti-armed bandits \u2013 a better way to A\/B test? - Codemotion Magazine<\/title>\n<meta name=\"description\" content=\"A\/B test allows to make design choices based on users preferences. To solve its drawbacks, the team of eBay\u2019s Marktplaats uses a different approach.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.codemotion.com\/magazine\/frontend\/design-ux\/multi-armed-bandits-a-better-way-to-a-b-test\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Multi-armed bandits \u2013 a better way to A\/B test?\" \/>\n<meta property=\"og:description\" content=\"A\/B test allows to make design choices based on users preferences. To solve its drawbacks, the team of eBay\u2019s Marktplaats uses a different approach.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.codemotion.com\/magazine\/frontend\/design-ux\/multi-armed-bandits-a-better-way-to-a-b-test\/\" \/>\n<meta property=\"og:site_name\" content=\"Codemotion Magazine\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/Codemotion.Italy\/\" \/>\n<meta property=\"article:published_time\" content=\"2020-01-21T08:30:14+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2020-02-14T09:34:46+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.codemotion.com\/magazine\/wp-content\/uploads\/2019\/08\/homeAmsterdam.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1024\" \/>\n\t<meta property=\"og:image:height\" content=\"682\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Toby Moncaster\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@tobym76\" \/>\n<meta name=\"twitter:site\" content=\"@CodemotionIT\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Toby Moncaster\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"6 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.codemotion.com\/magazine\/frontend\/design-ux\/multi-armed-bandits-a-better-way-to-a-b-test\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.codemotion.com\/magazine\/frontend\/design-ux\/multi-armed-bandits-a-better-way-to-a-b-test\/\"},\"author\":{\"name\":\"Toby Moncaster\",\"@id\":\"https:\/\/www.codemotion.com\/magazine\/#\/schema\/person\/8b9f025e7d76754fb3d4ffd428b0813b\"},\"headline\":\"Multi-armed bandits \u2013 a better way to A\/B test?\",\"datePublished\":\"2020-01-21T08:30:14+00:00\",\"dateModified\":\"2020-02-14T09:34:46+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.codemotion.com\/magazine\/frontend\/design-ux\/multi-armed-bandits-a-better-way-to-a-b-test\/\"},\"wordCount\":1299,\"publisher\":{\"@id\":\"https:\/\/www.codemotion.com\/magazine\/#organization\"},\"image\":{\"@id\":\"https:\/\/www.codemotion.com\/magazine\/frontend\/design-ux\/multi-armed-bandits-a-better-way-to-a-b-test\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.codemotion.com\/magazine\/wp-content\/uploads\/2019\/08\/homeAmsterdam.jpg\",\"keywords\":[\"Codemotion Amsterdam\",\"Testing\"],\"articleSection\":[\"Design\/UX\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.codemotion.com\/magazine\/frontend\/design-ux\/multi-armed-bandits-a-better-way-to-a-b-test\/\",\"url\":\"https:\/\/www.codemotion.com\/magazine\/frontend\/design-ux\/multi-armed-bandits-a-better-way-to-a-b-test\/\",\"name\":\"Multi-armed bandits \u2013 a better way to A\/B test? - Codemotion Magazine\",\"isPartOf\":{\"@id\":\"https:\/\/www.codemotion.com\/magazine\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/www.codemotion.com\/magazine\/frontend\/design-ux\/multi-armed-bandits-a-better-way-to-a-b-test\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/www.codemotion.com\/magazine\/frontend\/design-ux\/multi-armed-bandits-a-better-way-to-a-b-test\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.codemotion.com\/magazine\/wp-content\/uploads\/2019\/08\/homeAmsterdam.jpg\",\"datePublished\":\"2020-01-21T08:30:14+00:00\",\"dateModified\":\"2020-02-14T09:34:46+00:00\",\"description\":\"A\/B test allows to make design choices based on users preferences. To solve its drawbacks, the team of eBay\u2019s Marktplaats uses a different approach.\",\"breadcrumb\":{\"@id\":\"https:\/\/www.codemotion.com\/magazine\/frontend\/design-ux\/multi-armed-bandits-a-better-way-to-a-b-test\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.codemotion.com\/magazine\/frontend\/design-ux\/multi-armed-bandits-a-better-way-to-a-b-test\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.codemotion.com\/magazine\/frontend\/design-ux\/multi-armed-bandits-a-better-way-to-a-b-test\/#primaryimage\",\"url\":\"https:\/\/www.codemotion.com\/magazine\/wp-content\/uploads\/2019\/08\/homeAmsterdam.jpg\",\"contentUrl\":\"https:\/\/www.codemotion.com\/magazine\/wp-content\/uploads\/2019\/08\/homeAmsterdam.jpg\",\"width\":1024,\"height\":682},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.codemotion.com\/magazine\/frontend\/design-ux\/multi-armed-bandits-a-better-way-to-a-b-test\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.codemotion.com\/magazine\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Frontend\",\"item\":\"https:\/\/www.codemotion.com\/magazine\/frontend\/\"},{\"@type\":\"ListItem\",\"position\":3,\"name\":\"Design\/UX\",\"item\":\"https:\/\/www.codemotion.com\/magazine\/frontend\/design-ux\/\"},{\"@type\":\"ListItem\",\"position\":4,\"name\":\"Multi-armed bandits \u2013 a better way to A\/B test?\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.codemotion.com\/magazine\/#website\",\"url\":\"https:\/\/www.codemotion.com\/magazine\/\",\"name\":\"Codemotion Magazine\",\"description\":\"We code the future. Together\",\"publisher\":{\"@id\":\"https:\/\/www.codemotion.com\/magazine\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.codemotion.com\/magazine\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/www.codemotion.com\/magazine\/#organization\",\"name\":\"Codemotion\",\"url\":\"https:\/\/www.codemotion.com\/magazine\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.codemotion.com\/magazine\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/www.codemotion.com\/magazine\/wp-content\/uploads\/2019\/11\/codemotionlogo.png\",\"contentUrl\":\"https:\/\/www.codemotion.com\/magazine\/wp-content\/uploads\/2019\/11\/codemotionlogo.png\",\"width\":225,\"height\":225,\"caption\":\"Codemotion\"},\"image\":{\"@id\":\"https:\/\/www.codemotion.com\/magazine\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/www.facebook.com\/Codemotion.Italy\/\",\"https:\/\/x.com\/CodemotionIT\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.codemotion.com\/magazine\/#\/schema\/person\/8b9f025e7d76754fb3d4ffd428b0813b\",\"name\":\"Toby Moncaster\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.codemotion.com\/magazine\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/126cc1a8360e8cfbfa77aefe9160c4cd916e20f2c3a849d91e1df00c48423ccc?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/126cc1a8360e8cfbfa77aefe9160c4cd916e20f2c3a849d91e1df00c48423ccc?s=96&d=mm&r=g\",\"caption\":\"Toby Moncaster\"},\"description\":\"I am an experienced freelance writer. I specialise in making complex topics accessible to wider audiences. My interests include TCP\/IP, data protection and AI. I currently work with B2B startups across the world. I hold 5 patents, edited 3 RFCs and received a PhD in computer science from the University of Cambridge.\",\"sameAs\":[\"https:\/\/www.linkedin.com\/in\/tobymoncaster\/\",\"https:\/\/x.com\/tobym76\"],\"url\":\"https:\/\/www.codemotion.com\/magazine\/author\/toby-moncaster\/\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Multi-armed bandits \u2013 a better way to A\/B test? - Codemotion Magazine","description":"A\/B test allows to make design choices based on users preferences. To solve its drawbacks, the team of eBay\u2019s Marktplaats uses a different approach.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.codemotion.com\/magazine\/frontend\/design-ux\/multi-armed-bandits-a-better-way-to-a-b-test\/","og_locale":"en_US","og_type":"article","og_title":"Multi-armed bandits \u2013 a better way to A\/B test?","og_description":"A\/B test allows to make design choices based on users preferences. To solve its drawbacks, the team of eBay\u2019s Marktplaats uses a different approach.","og_url":"https:\/\/www.codemotion.com\/magazine\/frontend\/design-ux\/multi-armed-bandits-a-better-way-to-a-b-test\/","og_site_name":"Codemotion Magazine","article_publisher":"https:\/\/www.facebook.com\/Codemotion.Italy\/","article_published_time":"2020-01-21T08:30:14+00:00","article_modified_time":"2020-02-14T09:34:46+00:00","og_image":[{"width":1024,"height":682,"url":"https:\/\/www.codemotion.com\/magazine\/wp-content\/uploads\/2019\/08\/homeAmsterdam.jpg","type":"image\/jpeg"}],"author":"Toby Moncaster","twitter_card":"summary_large_image","twitter_creator":"@tobym76","twitter_site":"@CodemotionIT","twitter_misc":{"Written by":"Toby Moncaster","Est. reading time":"6 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.codemotion.com\/magazine\/frontend\/design-ux\/multi-armed-bandits-a-better-way-to-a-b-test\/#article","isPartOf":{"@id":"https:\/\/www.codemotion.com\/magazine\/frontend\/design-ux\/multi-armed-bandits-a-better-way-to-a-b-test\/"},"author":{"name":"Toby Moncaster","@id":"https:\/\/www.codemotion.com\/magazine\/#\/schema\/person\/8b9f025e7d76754fb3d4ffd428b0813b"},"headline":"Multi-armed bandits \u2013 a better way to A\/B test?","datePublished":"2020-01-21T08:30:14+00:00","dateModified":"2020-02-14T09:34:46+00:00","mainEntityOfPage":{"@id":"https:\/\/www.codemotion.com\/magazine\/frontend\/design-ux\/multi-armed-bandits-a-better-way-to-a-b-test\/"},"wordCount":1299,"publisher":{"@id":"https:\/\/www.codemotion.com\/magazine\/#organization"},"image":{"@id":"https:\/\/www.codemotion.com\/magazine\/frontend\/design-ux\/multi-armed-bandits-a-better-way-to-a-b-test\/#primaryimage"},"thumbnailUrl":"https:\/\/www.codemotion.com\/magazine\/wp-content\/uploads\/2019\/08\/homeAmsterdam.jpg","keywords":["Codemotion Amsterdam","Testing"],"articleSection":["Design\/UX"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/www.codemotion.com\/magazine\/frontend\/design-ux\/multi-armed-bandits-a-better-way-to-a-b-test\/","url":"https:\/\/www.codemotion.com\/magazine\/frontend\/design-ux\/multi-armed-bandits-a-better-way-to-a-b-test\/","name":"Multi-armed bandits \u2013 a better way to A\/B test? - Codemotion Magazine","isPartOf":{"@id":"https:\/\/www.codemotion.com\/magazine\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.codemotion.com\/magazine\/frontend\/design-ux\/multi-armed-bandits-a-better-way-to-a-b-test\/#primaryimage"},"image":{"@id":"https:\/\/www.codemotion.com\/magazine\/frontend\/design-ux\/multi-armed-bandits-a-better-way-to-a-b-test\/#primaryimage"},"thumbnailUrl":"https:\/\/www.codemotion.com\/magazine\/wp-content\/uploads\/2019\/08\/homeAmsterdam.jpg","datePublished":"2020-01-21T08:30:14+00:00","dateModified":"2020-02-14T09:34:46+00:00","description":"A\/B test allows to make design choices based on users preferences. To solve its drawbacks, the team of eBay\u2019s Marktplaats uses a different approach.","breadcrumb":{"@id":"https:\/\/www.codemotion.com\/magazine\/frontend\/design-ux\/multi-armed-bandits-a-better-way-to-a-b-test\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.codemotion.com\/magazine\/frontend\/design-ux\/multi-armed-bandits-a-better-way-to-a-b-test\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.codemotion.com\/magazine\/frontend\/design-ux\/multi-armed-bandits-a-better-way-to-a-b-test\/#primaryimage","url":"https:\/\/www.codemotion.com\/magazine\/wp-content\/uploads\/2019\/08\/homeAmsterdam.jpg","contentUrl":"https:\/\/www.codemotion.com\/magazine\/wp-content\/uploads\/2019\/08\/homeAmsterdam.jpg","width":1024,"height":682},{"@type":"BreadcrumbList","@id":"https:\/\/www.codemotion.com\/magazine\/frontend\/design-ux\/multi-armed-bandits-a-better-way-to-a-b-test\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.codemotion.com\/magazine\/"},{"@type":"ListItem","position":2,"name":"Frontend","item":"https:\/\/www.codemotion.com\/magazine\/frontend\/"},{"@type":"ListItem","position":3,"name":"Design\/UX","item":"https:\/\/www.codemotion.com\/magazine\/frontend\/design-ux\/"},{"@type":"ListItem","position":4,"name":"Multi-armed bandits \u2013 a better way to A\/B test?"}]},{"@type":"WebSite","@id":"https:\/\/www.codemotion.com\/magazine\/#website","url":"https:\/\/www.codemotion.com\/magazine\/","name":"Codemotion Magazine","description":"We code the future. Together","publisher":{"@id":"https:\/\/www.codemotion.com\/magazine\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.codemotion.com\/magazine\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.codemotion.com\/magazine\/#organization","name":"Codemotion","url":"https:\/\/www.codemotion.com\/magazine\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.codemotion.com\/magazine\/#\/schema\/logo\/image\/","url":"https:\/\/www.codemotion.com\/magazine\/wp-content\/uploads\/2019\/11\/codemotionlogo.png","contentUrl":"https:\/\/www.codemotion.com\/magazine\/wp-content\/uploads\/2019\/11\/codemotionlogo.png","width":225,"height":225,"caption":"Codemotion"},"image":{"@id":"https:\/\/www.codemotion.com\/magazine\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/Codemotion.Italy\/","https:\/\/x.com\/CodemotionIT"]},{"@type":"Person","@id":"https:\/\/www.codemotion.com\/magazine\/#\/schema\/person\/8b9f025e7d76754fb3d4ffd428b0813b","name":"Toby Moncaster","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.codemotion.com\/magazine\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/126cc1a8360e8cfbfa77aefe9160c4cd916e20f2c3a849d91e1df00c48423ccc?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/126cc1a8360e8cfbfa77aefe9160c4cd916e20f2c3a849d91e1df00c48423ccc?s=96&d=mm&r=g","caption":"Toby Moncaster"},"description":"I am an experienced freelance writer. I specialise in making complex topics accessible to wider audiences. My interests include TCP\/IP, data protection and AI. I currently work with B2B startups across the world. I hold 5 patents, edited 3 RFCs and received a PhD in computer science from the University of Cambridge.","sameAs":["https:\/\/www.linkedin.com\/in\/tobymoncaster\/","https:\/\/x.com\/tobym76"],"url":"https:\/\/www.codemotion.com\/magazine\/author\/toby-moncaster\/"}]}},"featured_image_src":"https:\/\/www.codemotion.com\/magazine\/wp-content\/uploads\/2019\/08\/homeAmsterdam-600x400.jpg","featured_image_src_square":"https:\/\/www.codemotion.com\/magazine\/wp-content\/uploads\/2019\/08\/homeAmsterdam-600x600.jpg","author_info":{"display_name":"Toby Moncaster","author_link":"https:\/\/www.codemotion.com\/magazine\/author\/toby-moncaster\/"},"uagb_featured_image_src":{"full":["https:\/\/www.codemotion.com\/magazine\/wp-content\/uploads\/2019\/08\/homeAmsterdam.jpg",1024,682,false],"thumbnail":["https:\/\/www.codemotion.com\/magazine\/wp-content\/uploads\/2019\/08\/homeAmsterdam-150x150.jpg",150,150,true],"medium":["https:\/\/www.codemotion.com\/magazine\/wp-content\/uploads\/2019\/08\/homeAmsterdam-300x200.jpg",300,200,true],"medium_large":["https:\/\/www.codemotion.com\/magazine\/wp-content\/uploads\/2019\/08\/homeAmsterdam-768x512.jpg",768,512,true],"large":["https:\/\/www.codemotion.com\/magazine\/wp-content\/uploads\/2019\/08\/homeAmsterdam.jpg",1024,682,false],"1536x1536":["https:\/\/www.codemotion.com\/magazine\/wp-content\/uploads\/2019\/08\/homeAmsterdam.jpg",1024,682,false],"2048x2048":["https:\/\/www.codemotion.com\/magazine\/wp-content\/uploads\/2019\/08\/homeAmsterdam.jpg",1024,682,false],"small-home-featured":["https:\/\/www.codemotion.com\/magazine\/wp-content\/uploads\/2019\/08\/homeAmsterdam.jpg",100,67,false],"sidebar-featured":["https:\/\/www.codemotion.com\/magazine\/wp-content\/uploads\/2019\/08\/homeAmsterdam-180x128.jpg",180,128,true],"genesis-singular-images":["https:\/\/www.codemotion.com\/magazine\/wp-content\/uploads\/2019\/08\/homeAmsterdam-896x504.jpg",896,504,true],"archive-featured":["https:\/\/www.codemotion.com\/magazine\/wp-content\/uploads\/2019\/08\/homeAmsterdam-400x225.jpg",400,225,true],"gb-block-post-grid-landscape":["https:\/\/www.codemotion.com\/magazine\/wp-content\/uploads\/2019\/08\/homeAmsterdam-600x400.jpg",600,400,true],"gb-block-post-grid-square":["https:\/\/www.codemotion.com\/magazine\/wp-content\/uploads\/2019\/08\/homeAmsterdam-600x600.jpg",600,600,true]},"uagb_author_info":{"display_name":"Toby Moncaster","author_link":"https:\/\/www.codemotion.com\/magazine\/author\/toby-moncaster\/"},"uagb_comment_info":0,"uagb_excerpt":"A\/B test allows to make design choices based on users preferences. To solve its drawbacks, the team of eBay\u2019s Marktplaats uses a different approach.","lang":"en","_links":{"self":[{"href":"https:\/\/www.codemotion.com\/magazine\/wp-json\/wp\/v2\/posts\/33","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.codemotion.com\/magazine\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.codemotion.com\/magazine\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.codemotion.com\/magazine\/wp-json\/wp\/v2\/users\/7"}],"replies":[{"embeddable":true,"href":"https:\/\/www.codemotion.com\/magazine\/wp-json\/wp\/v2\/comments?post=33"}],"version-history":[{"count":4,"href":"https:\/\/www.codemotion.com\/magazine\/wp-json\/wp\/v2\/posts\/33\/revisions"}],"predecessor-version":[{"id":2815,"href":"https:\/\/www.codemotion.com\/magazine\/wp-json\/wp\/v2\/posts\/33\/revisions\/2815"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.codemotion.com\/magazine\/wp-json\/wp\/v2\/media\/34"}],"wp:attachment":[{"href":"https:\/\/www.codemotion.com\/magazine\/wp-json\/wp\/v2\/media?parent=33"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.codemotion.com\/magazine\/wp-json\/wp\/v2\/categories?post=33"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.codemotion.com\/magazine\/wp-json\/wp\/v2\/tags?post=33"},{"taxonomy":"collections","embeddable":true,"href":"https:\/\/www.codemotion.com\/magazine\/wp-json\/wp\/v2\/collections?post=33"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}