{"id":14802,"date":"2021-05-17T03:25:00","date_gmt":"2021-05-17T01:25:00","guid":{"rendered":"https:\/\/www.codemotion.com\/magazine\/?p=14802"},"modified":"2023-06-23T16:05:31","modified_gmt":"2023-06-23T14:05:31","slug":"popmon-data-quality-monitoring","status":"publish","type":"post","link":"https:\/\/www.codemotion.com\/magazine\/ai-ml\/big-data\/popmon-data-quality-monitoring\/","title":{"rendered":"Population Shift Analysis: Monitoring Data Quality with Popmon"},"content":{"rendered":"\n<p>One of the most significant parts of any data-driven application is <strong>data quality<\/strong> assessment. Before you start using your data, you must understand how good &#8211; or bad &#8211; it is. <\/p>\n\n\n\n<p>This is why <strong>data analysis<\/strong> and <strong>data cleaning<\/strong> activities are performed &#8211; to obtain a dataset that is ready to be used within the training stage of building a machine learning model, for instance. <\/p>\n\n\n\n<p>However, limiting data quality assessment to the initial steps of data analysis can be a hazardous choice. Indeed, depending on the data source you rely on, <strong>data trends may vary over time<\/strong>; consequently, models trained with historical data may perform worse.<\/p>\n\n\n\n<p>To address this kind of issue, the ING WB Advanced Analytics (WBAA) team developed <strong>popmon<\/strong>, an open source <a href=\"https:\/\/www.codemotion.com\/magazine\/tag\/python\/\" class=\"ek-link\">Python<\/a> module that allows data analysts and scientists, as well as machine learning engineers and developers, to carry out <strong>population shift <a href=\"https:\/\/www.codemotion.com\/magazine\/ai-ml\/what-is-synthetic-monitoring-and-5-reasons-your-business-needs-it\/\" target=\"_blank\" aria-label=\"monitoring (opens in a new tab)\" rel=\"noreferrer noopener\" class=\"ek-link\">monitoring<\/a><\/strong>.<\/p>\n\n\n\n<p>In this article we present the key features of popmon, providing an overview of how it works and presenting some use cases in which such a tool can produce significant benefits for data-driven systems and applications.<\/p>\n\n\n\t\t\t\t<div class=\"wp-block-uagb-table-of-contents uagb-toc__align-left uagb-toc__columns-1  uagb-block-d1e0aae6      \"\n\t\t\t\t\tdata-scroll= \"1\"\n\t\t\t\t\tdata-offset= \"30\"\n\t\t\t\t\tstyle=\"\"\n\t\t\t\t>\n\t\t\t\t<div class=\"uagb-toc__wrap\">\n\t\t\t\t\t\t<div class=\"uagb-toc__title\">\n\t\t\t\t\t\t\tTable Of Contents\t\t\t\t\t\t<\/div>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<div class=\"uagb-toc__list-wrap \">\n\t\t\t\t\t\t<ol class=\"uagb-toc__list\"><li class=\"uagb-toc__list\"><a href=\"#population-shift-analysis-and-data-quality-assessment-why\" class=\"uagb-toc-link__trigger\">Population shift analysis and data quality assessment: why?<\/a><li class=\"uagb-toc__list\"><a href=\"#popmon-population-shift-analysis-in-python\" class=\"uagb-toc-link__trigger\">popmon: population shift analysis in Python<\/a><li class=\"uagb-toc__list\"><a href=\"#benefits-of-popmon\" class=\"uagb-toc-link__trigger\">Benefits of popmon<\/a><ul class=\"uagb-toc__list\"><li class=\"uagb-toc__list\"><a href=\"#internal-representation\" class=\"uagb-toc-link__trigger\">Internal representation<\/a><li class=\"uagb-toc__list\"><li class=\"uagb-toc__list\"><a href=\"#support-for-pandas-and-apache-spark\" class=\"uagb-toc-link__trigger\">Support for Pandas and Apache Spark<\/a><li class=\"uagb-toc__list\"><li class=\"uagb-toc__list\"><a href=\"#dynamic-data-quality-boundaries\" class=\"uagb-toc-link__trigger\">Dynamic Data Quality Boundaries<\/a><\/li><\/ul><\/li><li class=\"uagb-toc__list\"><a href=\"#use-cases\" class=\"uagb-toc-link__trigger\">Use cases<\/a><ul class=\"uagb-toc__list\"><li class=\"uagb-toc__list\"><a href=\"#using-popmon-to-monitor-machine-learning-performance\" class=\"uagb-toc-link__trigger\">Using popmon to monitor machine learning performance<\/a><li class=\"uagb-toc__list\"><li class=\"uagb-toc__list\"><a href=\"#using-popmon-for-data-exploration\" class=\"uagb-toc-link__trigger\">Using popmon for data exploration<\/a><\/li><\/ul><\/li><\/ul><\/li><li class=\"uagb-toc__list\"><a href=\"#conclusions\" class=\"uagb-toc-link__trigger\">Conclusions<\/a><\/ul><\/ul><\/ol>\t\t\t\t\t<\/div>\n\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\n\n\n<h2 class=\"wp-block-heading\" id=\"h-population-shift-analysis-and-data-quality-assessment-why\">Population shift analysis and data quality assessment: why?<\/h2>\n\n\n\n<p>Before going into details about popmon\u2019s features, it is important to explain why population shift monitoring and data quality assessment are fundamental steps in any data-driven application.<\/p>\n\n\n\n<p>If an individual or organization needs to <strong>develop a machine learning model<\/strong>, there are some standard steps that need to be followed. Firstly, the business problem that requires a solution needs to be defined, which in turn allows identification of the kind of data required. Then, the dataset needs to be analyzed and cleaned.<\/p>\n\n\n\n<p>The next step is to <strong>choose or define a model that suits the dataset well<\/strong>. Finally, training the dataset can begin. At the end of this process, the model will be complete and can be used to make the prediction(s) needed to solve the original business problem.<\/p>\n\n\n\n<p>What we have just described is often referred to as the <strong>machine learning development cycle<\/strong>. However, the reason it is known as a \u2018cycle\u2019 seems at odds with the sequence of steps described above. <\/p>\n\n\n\n<p>What is lacking is some sort of <strong>feedback,<\/strong> gathered after running the model, which allows <strong>optimization,<\/strong> and enables the whole solution to adapt to unexpected situations. <\/p>\n\n\n\n<p>Part of such an optimization can occur in the design process, by <strong>testing the model<\/strong> and carefully tuning <strong>hyperparameters<\/strong> until a more stable performance and the most generalized model is reached, but even this would be insufficient.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/lh3.googleusercontent.com\/646xMYSkpsZtWYRJ5D7jAD4tYx1qCT-NJi22h9eZeRJw7fUSe2GbOsiTbgsBsVPiyB8-UyYoUVwiyFIElxv2_KtK2sQ_KdN0skg5mwquEGqTUYdYH2zZDVwroLzjDeJWsP6Cbh6a\" alt=\"Image showing the machine learning development lifecycle\"\/><figcaption class=\"wp-element-caption\"><em>The machine learning development lifecycle<\/em><\/figcaption><\/figure>\n\n\n\n<p>What is missing here is a process that continuously <strong>monitors new data<\/strong>. In any production environment, such an input <a href=\"https:\/\/www.codemotion.com\/magazine\/dev-hub\/big-data-analyst\/interview-stefano-pampaloni-big-data-stream-analytics-without-code\/\" target=\"_blank\" rel=\"noopener\" class=\"ek-link\">data stream<\/a> is far from stable. A plethora of possible issues and anomalies may affect the input, and consequently, the model\u2019s performance. Let\u2019s consider a simple example that reflects this issue.<\/p>\n\n\n\n<p>Imagine a business process that needs to analyze the receipts from bank transfers.&nbsp; We can&nbsp; assume that such documents are all formatted in the same way, with a standardized layout. For some unexpected reason, the issuing institution might decide to change their receipt format. <\/p>\n\n\n\n<p>Such a choice would imply a change within the input data stream, with a concrete risk that the previously trained model might perform worse than before. The whole business process would suddenly become ineffective, with all the consequences that follow on from this.<\/p>\n\n\n\n<p>Such issues occur more frequently than you might imagine, especially in production environments, causing what is known as <strong>population shift<\/strong>. Monitoring a model&#8217;s performance is the only way to allow such an issue to be identified, and it\u2019s easy to understand how important a tool that helps to identify this problem could be.<\/p>\n\n\n\n<p>This is where popmon steps in to help.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"h-popmon-population-shift-analysis-in-python\">popmon: population shift analysis in Python<\/h2>\n\n\n\n<p>As already mentioned, popmon is an open-source Python module that allows users to <strong>check the stability of a dataset<\/strong>. In particular, it allows detection of the occurrence of population shift, by analyzing data frames where data are labelled with times.<\/p>\n\n\n\n<p>Going into a little more detail, popmon creates <strong>histograms of features <\/strong>binned in time-slices, and compares the stability of the profiles and distributions of those histograms, over time and with respect to a reference, by using a set of statistical tests. <\/p>\n\n\n\n<p>popmon works with numerical, ordinal, and\/or categorical features and histograms that can be multi-dimensional.&nbsp; The module also supports high dimensionality.&nbsp;<\/p>\n\n\n\n<p>popmon can use this histogram-based representation to <strong>generate reports<\/strong>, or to automatically flag and alert users to changes observed over time, including shifts, peaks, outliers, anomalies and so on, using different business rules for monitoring.<\/p>\n\n\n\n<p>This tool was developed by the ING WBAA team, and its source code is available on <a href=\"https:\/\/github.com\/ing-bank\/popmon\" target=\"_blank\" aria-label=\" (opens in a new tab)\" rel=\"noreferrer noopener\" class=\"ek-link\">GitHub<\/a>. popmon supports Python 3.6 and later, and can be installed using <em>pip<\/em> as follows:<\/p>\n\n\n\n<script src=\"https:\/\/pastebin.com\/embed_js\/fb8AQKBJ\"><\/script>\n\n\n\n<p>Alternatively, you can install it directly from the source:<\/p>\n\n\n\n<script src=\"https:\/\/pastebin.com\/embed_js\/HcZ9pBUu\"><\/script>\n\n\n\n<p>Once installed, popmon can be imported like any other Python module, and used as needed. The following code snippet demonstrates how to load a CSV file into a Pandas data frame, then use its content to generate a report:<\/p>\n\n\n\n<script src=\"https:\/\/pastebin.com\/embed_js\/B5zVyHq6\"><\/script>\n\n\n\n<p>The generated report includes the details of several statistical analyses computed on the dataset, from the histogram-based aggregation to the many statistical tests used. <\/p>\n\n\n\n<p>The statistics provide insights into how the data changes over time, but some are more interesting to look at than others. The report also provides a traffic light visualization that allows users a very quick overview of the kinds of variables that might be affected most by instability. <\/p>\n\n\n\n<p>The following image (taken from popmon\u2019s GitHub repository) shows an example of visualization within a popmon report:<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/lh6.googleusercontent.com\/s0u7OuwlUg3EEe-_lHRko_yojaN8zwq1n2Hbn8CwbNPZavKQZmUcF1Q_R4fFKqN7lDZvc5j-OUA7sUvEL-QabASFTtivLEJjx24czqvIvg2W7Q25USgHz0XKwHp7rQDDHOdDb9oq\" alt=\"Sample report and traffic lights visualization generated by popmon.\"\/><figcaption class=\"wp-element-caption\"><em>Sample report and traffic lights visualization generated by popmon. The red traffic lights on the right indicate a shift in the last two weeks of the data. The user might want to look into that first. The two lines that are mostly red and yellow are probably set to an overly-sensitive calibration &#8211; we can increase the dynamic threshold or simply ignore them for now.<\/em><\/figcaption><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"h-benefits-of-popmon\">Benefits of popmon<\/h2>\n\n\n\n<p>There are several features that makes popmon a great tool for data monitoring. The first, and most important, thing to say is that this tool is probably one of a kind; to the best of our knowledge there is no single alternative tool that implements all the statistical tests used by popmon, along with its histogram-based internal representation. <\/p>\n\n\n\n<p>As <strong>Max Baak<\/strong>, <a href=\"https:\/\/www.codemotion.com\/magazine\/dev-hub\/machine-learning-dev\/local-data-science-is-dead-ai-with-a-devop-mindset\/\" target=\"_blank\" rel=\"noopener\" class=\"ek-link\">data science<\/a> lead at ING WBAA, former CERN researcher, and one of the most active popmon contributors on GitHub explains, before popmon there were \u201cno good open-source packages that allowed us to monitor input data and predictions for such shifts in a straightforward, automated way.\u201d <\/p>\n\n\n\n<p>For this reason, the features offered by popmon can be extremely useful to anyone who needs to monitor data over time &#8211; and everyone working on ML should do this!<\/p>\n\n\n\n<p>Apart from this, there are several other benefits of popmon to consider.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"h-internal-representation\">Internal representation<\/h3>\n\n\n\n<p>The first of these benefits is the previously-mentioned <strong>histogram-based internal data representation<\/strong>. This representation is memory-efficient, as histograms are typically much smaller than the original data. How such a choice allows several statistical tests to be run on data has been explained above. <\/p>\n\n\n\n<p>Moreover, this design decision also addresses a big issue when dealing with data: <strong>privacy<\/strong>. Using histograms is a form of <strong>data aggregation<\/strong>, which removes identifiable information from entries, therefore easing the process of data storage. This is a major plus when dealing with (and storing) sensitive information.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"h-support-for-pandas-and-apache-spark\">Support for Pandas and Apache Spark<\/h3>\n\n\n\n<p>It is also worth mentioning that <strong>Pandas<\/strong> is not the only module supported by popmon, even though, in the previous snippet, you will see that CSV data are loaded into a Pandas data frame. <\/p>\n\n\n\n<p>Although Pandas is probably the most widely supported technology for dealing with data frames, popmon is not limited to this and also supports <strong>Apache Spark<\/strong> data frames. <\/p>\n\n\n\n<p>Apart from imports and data loading, all other popmon API remain almost identical, even if the user opts for Apache Spark. It\u2019s worth noticing that via these backends, any prominent data source format is supported, from CSV, JSON and Excel to Hive, Parquet, HDF5 and Apache Avro.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"h-dynamic-data-quality-boundaries\">Dynamic Data Quality Boundaries<\/h3>\n\n\n\n<p>A final important feature of popmon is its <strong>data quality boundaries<\/strong>. popmon uses a traffic light system to indicate where large deviations from a certain reference have occurred. <\/p>\n\n\n\n<p>The module allows the user to define a set of static thresholds that can automatically identify whether the degree of deviation is high (red light) or low (yellow light), or if there are no significant deviations (green light).<\/p>\n\n\n\n<p>Setting static thresholds automatically is hard, and doing so manually for many datasets is often not feasible. <\/p>\n\n\n\n<p>This concept can be also extended by defining <strong>dynamic thresholds that<\/strong> can be computed automatically, based on mean and variance, reducing the need for additional parameters (known as \u2018pull\u2019 or Z-score). This idea is depicted in the following image:<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/lh5.googleusercontent.com\/Hpd4jTHu0j7UcGq7kPvENJtOqQh0jdsYlcmS4VWuFmUaRhqregkMvoDty4jHZLA7dRV4k5ZyIy8aUMNAsLMvPtl8lmJZCUBQ4W4OnbO1iDrRyBJXNjM9aZSNAWJ-jLZa6_vBv9zq\" alt=\"Graphs ot Dynamic thresholds \"\/><figcaption class=\"wp-element-caption\"><em>Dynamic thresholds are computed based on mean (\u03bc) and variance (\u03c3)<\/em><\/figcaption><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"h-use-cases\">Use cases<\/h2>\n\n\n\n<p>How popmon works and the benefits it offers users should be clear. We now turn to two typical use cases, including some of the real-life scenarios that pushed the ING WBAA team into developing popmon and using it in their everyday activities, including within production environments.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"h-using-popmon-to-monitor-machine-learning-performance\">Using popmon to monitor machine learning performance<\/h3>\n\n\n\n<p>The first class of users that might benefit from using popmon includes <a href=\"https:\/\/www.codemotion.com\/magazine\/Glossary\/data-scientist\/\" class=\"ek-link\"><strong>data scientists<\/strong><\/a> and <a href=\"https:\/\/www.codemotion.com\/magazine\/Glossary\/machine-learning-engineer\/\" class=\"ek-link\"><strong>machine learning engineers<\/strong><\/a>. For both groups, the use of popmon is primarily focused on the monitoring stage of the machine learning development lifecycle.<\/p>\n\n\n\n<p>When a model is developed, it is essential to continuously evaluate how predictions on the same input data stream vary over time. Given the same model, slight variations might be due to prediction noise; however, more significant trend shifts or unexpected peaks might actually represent a warning signal. <\/p>\n\n\n\n<p>In these cases, it is a good practice to investigate what\u2019s going on, and what exactly is affecting the predictions.<\/p>\n\n\n\n<p>popmon is used by ING to <strong>alert developers and data scientists in case of population shift detection<\/strong>. Every time new data comes in, popmon can be used to evaluate changes in data quality, and eventually, to trigger real-time alerts.<\/p>\n\n\n\n<p>When population shifts are detected, the same models that have already been trained with historical data can be enhanced with fresh training data, or completely retrained to adapt their behaviour to new data patterns.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"h-using-popmon-for-data-exploration\">Using popmon for data exploration<\/h3>\n\n\n\n<p>Another use of popmon is particularly interesting for <a href=\"https:\/\/www.codemotion.com\/magazine\/dev-hub\/big-data-analyst\/\" class=\"ek-link\"><strong>data analysts<\/strong><\/a> &#8211; specifically in relation to the initial stage of dataset analysis, even before data preparation and data cleaning activities take place. <\/p>\n\n\n\n<p>Thanks to the histogram-based internal representation used, as well as the many statistical tests that it implements, popmon is a great tool for <strong>data exploration<\/strong>, allowing in-depth investigation of possible data patterns and trends, as well as outliers and seasonality.<\/p>\n\n\n\n<p>From this point of view, popmon might beneficially be incorporated into <strong>data ingestion pipelines<\/strong> to monitor incoming data in order to prevent drops in performance due to poor data quality.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"h-conclusions\">Conclusions<\/h2>\n\n\n\n<p>Tomas \u0160ostak, data scientist with ING WBAA and another principal contributor to popmon, summarized the situation, saying: \u201cpopmon could be very useful to anyone doing <strong>serious monitoring of ML models running in production<\/strong>, as well as simple <strong>exploratory data analysis<\/strong>.\u201d <\/p>\n\n\n\n<p>An additional point that should not be underestimated: popmon is <strong>easy to use and to integrate<\/strong> into other data-driven applications. With a few lines of code, popmon can be configured to meet user needs &#8211; not just to generate reports as shown in the snippet above, but also to create alerts for data scientists to <strong>speed up retraining activities<\/strong>, rendering the whole <a href=\"https:\/\/www.codemotion.com\/magazine\/dev-hub\/backend-dev\/naturally-born-reactive-systems\/\" target=\"_blank\" rel=\"noopener\" class=\"ek-link\">system more reactive<\/a> to new data.<\/p>\n\n\n\n<p>If you are interested in learning more about popmon, you can find more information on <a href=\"https:\/\/github.com\/ing-bank\/popmon\" target=\"_blank\" aria-label=\" (opens in a new tab)\" rel=\"noreferrer noopener\" class=\"ek-link\">GitHub<\/a>, including code examples, tutorials and videos.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>One of the most significant parts of any data-driven application is data quality assessment. Before you start using your data, you must understand how good &#8211; or bad &#8211; it is. This is why data analysis and data cleaning activities are performed &#8211; to obtain a dataset that is ready to be used within the&#8230; <a class=\"more-link\" href=\"https:\/\/www.codemotion.com\/magazine\/ai-ml\/big-data\/popmon-data-quality-monitoring\/\">Read more<\/a><\/p>\n","protected":false},"author":3,"featured_media":14816,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_editorskit_title_hidden":false,"_editorskit_reading_time":8,"_editorskit_is_block_options_detached":false,"_editorskit_block_options_position":"{}","_uag_custom_page_level_css":"","_genesis_hide_title":false,"_genesis_hide_breadcrumbs":false,"_genesis_hide_singular_image":false,"_genesis_hide_footer_widgets":false,"_genesis_custom_body_class":"","_genesis_custom_post_class":"","_genesis_layout":"","footnotes":""},"categories":[16],"tags":[4446],"collections":[],"class_list":{"0":"post-14802","1":"post","2":"type-post","3":"status-publish","4":"format-standard","5":"has-post-thumbnail","7":"category-big-data","8":"tag-data-analysis","9":"entry"},"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v26.9 (Yoast SEO v27.5) - https:\/\/yoast.com\/product\/yoast-seo-premium-wordpress\/ -->\n<title>Using Popmon for Data Quality Monitoring - Codemotion<\/title>\n<meta name=\"description\" content=\"Tracking population shifts and quality changes is now easier thanks to popmon, an open-source Python module developed by ING. Learn more.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.codemotion.com\/magazine\/ai-ml\/big-data\/popmon-data-quality-monitoring\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Population Shift Analysis: Monitoring Data Quality with Popmon\" \/>\n<meta property=\"og:description\" content=\"Tracking population shifts and quality changes is now easier thanks to popmon, an open-source Python module developed by ING. Learn more.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.codemotion.com\/magazine\/ai-ml\/big-data\/popmon-data-quality-monitoring\/\" \/>\n<meta property=\"og:site_name\" content=\"Codemotion Magazine\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/Codemotion.Italy\/\" \/>\n<meta property=\"article:published_time\" content=\"2021-05-17T01:25:00+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2023-06-23T14:05:31+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.codemotion.com\/magazine\/wp-content\/uploads\/2021\/05\/Population-Shift-Analysis-How-to-Monitor-Data-Quality-with-Popmon.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1200\" \/>\n\t<meta property=\"og:image:height\" content=\"628\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Vito Gentile\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@ViGentile\" \/>\n<meta name=\"twitter:site\" content=\"@CodemotionIT\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Vito Gentile\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"10 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/www.codemotion.com\\\/magazine\\\/ai-ml\\\/big-data\\\/popmon-data-quality-monitoring\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.codemotion.com\\\/magazine\\\/ai-ml\\\/big-data\\\/popmon-data-quality-monitoring\\\/\"},\"author\":{\"name\":\"Vito Gentile\",\"@id\":\"https:\\\/\\\/www.codemotion.com\\\/magazine\\\/#\\\/schema\\\/person\\\/f9c99d8dfb99900d709bb8199225d313\"},\"headline\":\"Population Shift Analysis: Monitoring Data Quality with Popmon\",\"datePublished\":\"2021-05-17T01:25:00+00:00\",\"dateModified\":\"2023-06-23T14:05:31+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/www.codemotion.com\\\/magazine\\\/ai-ml\\\/big-data\\\/popmon-data-quality-monitoring\\\/\"},\"wordCount\":1976,\"publisher\":{\"@id\":\"https:\\\/\\\/www.codemotion.com\\\/magazine\\\/#organization\"},\"image\":{\"@id\":\"https:\\\/\\\/www.codemotion.com\\\/magazine\\\/ai-ml\\\/big-data\\\/popmon-data-quality-monitoring\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/www.codemotion.com\\\/magazine\\\/wp-content\\\/uploads\\\/2021\\\/05\\\/Population-Shift-Analysis-How-to-Monitor-Data-Quality-with-Popmon.jpg\",\"keywords\":[\"Data Analysis\"],\"articleSection\":[\"Big Data\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/www.codemotion.com\\\/magazine\\\/ai-ml\\\/big-data\\\/popmon-data-quality-monitoring\\\/\",\"url\":\"https:\\\/\\\/www.codemotion.com\\\/magazine\\\/ai-ml\\\/big-data\\\/popmon-data-quality-monitoring\\\/\",\"name\":\"Using Popmon for Data Quality Monitoring - Codemotion\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.codemotion.com\\\/magazine\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/www.codemotion.com\\\/magazine\\\/ai-ml\\\/big-data\\\/popmon-data-quality-monitoring\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/www.codemotion.com\\\/magazine\\\/ai-ml\\\/big-data\\\/popmon-data-quality-monitoring\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/www.codemotion.com\\\/magazine\\\/wp-content\\\/uploads\\\/2021\\\/05\\\/Population-Shift-Analysis-How-to-Monitor-Data-Quality-with-Popmon.jpg\",\"datePublished\":\"2021-05-17T01:25:00+00:00\",\"dateModified\":\"2023-06-23T14:05:31+00:00\",\"description\":\"Tracking population shifts and quality changes is now easier thanks to popmon, an open-source Python module developed by ING. Learn more.\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/www.codemotion.com\\\/magazine\\\/ai-ml\\\/big-data\\\/popmon-data-quality-monitoring\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/www.codemotion.com\\\/magazine\\\/ai-ml\\\/big-data\\\/popmon-data-quality-monitoring\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/www.codemotion.com\\\/magazine\\\/ai-ml\\\/big-data\\\/popmon-data-quality-monitoring\\\/#primaryimage\",\"url\":\"https:\\\/\\\/www.codemotion.com\\\/magazine\\\/wp-content\\\/uploads\\\/2021\\\/05\\\/Population-Shift-Analysis-How-to-Monitor-Data-Quality-with-Popmon.jpg\",\"contentUrl\":\"https:\\\/\\\/www.codemotion.com\\\/magazine\\\/wp-content\\\/uploads\\\/2021\\\/05\\\/Population-Shift-Analysis-How-to-Monitor-Data-Quality-with-Popmon.jpg\",\"width\":1200,\"height\":628,\"caption\":\"Population Shift Analysis How to Monitor Data Quality with Popmon\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/www.codemotion.com\\\/magazine\\\/ai-ml\\\/big-data\\\/popmon-data-quality-monitoring\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/www.codemotion.com\\\/magazine\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"AI\\\/ML\",\"item\":\"https:\\\/\\\/www.codemotion.com\\\/magazine\\\/ai-ml\\\/\"},{\"@type\":\"ListItem\",\"position\":3,\"name\":\"Big Data\",\"item\":\"https:\\\/\\\/www.codemotion.com\\\/magazine\\\/ai-ml\\\/big-data\\\/\"},{\"@type\":\"ListItem\",\"position\":4,\"name\":\"Population Shift Analysis: Monitoring Data Quality with Popmon\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/www.codemotion.com\\\/magazine\\\/#website\",\"url\":\"https:\\\/\\\/www.codemotion.com\\\/magazine\\\/\",\"name\":\"Codemotion Magazine\",\"description\":\"We code the future. Together\",\"publisher\":{\"@id\":\"https:\\\/\\\/www.codemotion.com\\\/magazine\\\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/www.codemotion.com\\\/magazine\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\\\/\\\/www.codemotion.com\\\/magazine\\\/#organization\",\"name\":\"Codemotion\",\"url\":\"https:\\\/\\\/www.codemotion.com\\\/magazine\\\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/www.codemotion.com\\\/magazine\\\/#\\\/schema\\\/logo\\\/image\\\/\",\"url\":\"https:\\\/\\\/www.codemotion.com\\\/magazine\\\/wp-content\\\/uploads\\\/2019\\\/11\\\/codemotionlogo.png\",\"contentUrl\":\"https:\\\/\\\/www.codemotion.com\\\/magazine\\\/wp-content\\\/uploads\\\/2019\\\/11\\\/codemotionlogo.png\",\"width\":225,\"height\":225,\"caption\":\"Codemotion\"},\"image\":{\"@id\":\"https:\\\/\\\/www.codemotion.com\\\/magazine\\\/#\\\/schema\\\/logo\\\/image\\\/\"},\"sameAs\":[\"https:\\\/\\\/www.facebook.com\\\/Codemotion.Italy\\\/\",\"https:\\\/\\\/x.com\\\/CodemotionIT\"]},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/www.codemotion.com\\\/magazine\\\/#\\\/schema\\\/person\\\/f9c99d8dfb99900d709bb8199225d313\",\"name\":\"Vito Gentile\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/ebcb5b885a4f1669578d08a55deb81064893c07c1b35b7b36eb059cce730ae90?s=96&d=mm&r=g\",\"url\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/ebcb5b885a4f1669578d08a55deb81064893c07c1b35b7b36eb059cce730ae90?s=96&d=mm&r=g\",\"contentUrl\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/ebcb5b885a4f1669578d08a55deb81064893c07c1b35b7b36eb059cce730ae90?s=96&d=mm&r=g\",\"caption\":\"Vito Gentile\"},\"description\":\"I\u2019m a data scientist, tech writer, software developer with experience in mobile, web (full-stack) and Python programming, and former researcher with interests in human-computer interaction. I thus have a multi-faceted experience in the area of software development, and that\u2019s why I love my job(s)!\",\"sameAs\":[\"https:\\\/\\\/vitogentile.it\",\"https:\\\/\\\/www.linkedin.com\\\/in\\\/vitogentile\\\/en\",\"https:\\\/\\\/x.com\\\/ViGentile\"],\"url\":\"https:\\\/\\\/www.codemotion.com\\\/magazine\\\/author\\\/vito-gentile\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Using Popmon for Data Quality Monitoring - Codemotion","description":"Tracking population shifts and quality changes is now easier thanks to popmon, an open-source Python module developed by ING. Learn more.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.codemotion.com\/magazine\/ai-ml\/big-data\/popmon-data-quality-monitoring\/","og_locale":"en_US","og_type":"article","og_title":"Population Shift Analysis: Monitoring Data Quality with Popmon","og_description":"Tracking population shifts and quality changes is now easier thanks to popmon, an open-source Python module developed by ING. Learn more.","og_url":"https:\/\/www.codemotion.com\/magazine\/ai-ml\/big-data\/popmon-data-quality-monitoring\/","og_site_name":"Codemotion Magazine","article_publisher":"https:\/\/www.facebook.com\/Codemotion.Italy\/","article_published_time":"2021-05-17T01:25:00+00:00","article_modified_time":"2023-06-23T14:05:31+00:00","og_image":[{"width":1200,"height":628,"url":"https:\/\/www.codemotion.com\/magazine\/wp-content\/uploads\/2021\/05\/Population-Shift-Analysis-How-to-Monitor-Data-Quality-with-Popmon.jpg","type":"image\/jpeg"}],"author":"Vito Gentile","twitter_card":"summary_large_image","twitter_creator":"@ViGentile","twitter_site":"@CodemotionIT","twitter_misc":{"Written by":"Vito Gentile","Est. reading time":"10 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.codemotion.com\/magazine\/ai-ml\/big-data\/popmon-data-quality-monitoring\/#article","isPartOf":{"@id":"https:\/\/www.codemotion.com\/magazine\/ai-ml\/big-data\/popmon-data-quality-monitoring\/"},"author":{"name":"Vito Gentile","@id":"https:\/\/www.codemotion.com\/magazine\/#\/schema\/person\/f9c99d8dfb99900d709bb8199225d313"},"headline":"Population Shift Analysis: Monitoring Data Quality with Popmon","datePublished":"2021-05-17T01:25:00+00:00","dateModified":"2023-06-23T14:05:31+00:00","mainEntityOfPage":{"@id":"https:\/\/www.codemotion.com\/magazine\/ai-ml\/big-data\/popmon-data-quality-monitoring\/"},"wordCount":1976,"publisher":{"@id":"https:\/\/www.codemotion.com\/magazine\/#organization"},"image":{"@id":"https:\/\/www.codemotion.com\/magazine\/ai-ml\/big-data\/popmon-data-quality-monitoring\/#primaryimage"},"thumbnailUrl":"https:\/\/www.codemotion.com\/magazine\/wp-content\/uploads\/2021\/05\/Population-Shift-Analysis-How-to-Monitor-Data-Quality-with-Popmon.jpg","keywords":["Data Analysis"],"articleSection":["Big Data"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/www.codemotion.com\/magazine\/ai-ml\/big-data\/popmon-data-quality-monitoring\/","url":"https:\/\/www.codemotion.com\/magazine\/ai-ml\/big-data\/popmon-data-quality-monitoring\/","name":"Using Popmon for Data Quality Monitoring - Codemotion","isPartOf":{"@id":"https:\/\/www.codemotion.com\/magazine\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.codemotion.com\/magazine\/ai-ml\/big-data\/popmon-data-quality-monitoring\/#primaryimage"},"image":{"@id":"https:\/\/www.codemotion.com\/magazine\/ai-ml\/big-data\/popmon-data-quality-monitoring\/#primaryimage"},"thumbnailUrl":"https:\/\/www.codemotion.com\/magazine\/wp-content\/uploads\/2021\/05\/Population-Shift-Analysis-How-to-Monitor-Data-Quality-with-Popmon.jpg","datePublished":"2021-05-17T01:25:00+00:00","dateModified":"2023-06-23T14:05:31+00:00","description":"Tracking population shifts and quality changes is now easier thanks to popmon, an open-source Python module developed by ING. Learn more.","breadcrumb":{"@id":"https:\/\/www.codemotion.com\/magazine\/ai-ml\/big-data\/popmon-data-quality-monitoring\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.codemotion.com\/magazine\/ai-ml\/big-data\/popmon-data-quality-monitoring\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.codemotion.com\/magazine\/ai-ml\/big-data\/popmon-data-quality-monitoring\/#primaryimage","url":"https:\/\/www.codemotion.com\/magazine\/wp-content\/uploads\/2021\/05\/Population-Shift-Analysis-How-to-Monitor-Data-Quality-with-Popmon.jpg","contentUrl":"https:\/\/www.codemotion.com\/magazine\/wp-content\/uploads\/2021\/05\/Population-Shift-Analysis-How-to-Monitor-Data-Quality-with-Popmon.jpg","width":1200,"height":628,"caption":"Population Shift Analysis How to Monitor Data Quality with Popmon"},{"@type":"BreadcrumbList","@id":"https:\/\/www.codemotion.com\/magazine\/ai-ml\/big-data\/popmon-data-quality-monitoring\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.codemotion.com\/magazine\/"},{"@type":"ListItem","position":2,"name":"AI\/ML","item":"https:\/\/www.codemotion.com\/magazine\/ai-ml\/"},{"@type":"ListItem","position":3,"name":"Big Data","item":"https:\/\/www.codemotion.com\/magazine\/ai-ml\/big-data\/"},{"@type":"ListItem","position":4,"name":"Population Shift Analysis: Monitoring Data Quality with Popmon"}]},{"@type":"WebSite","@id":"https:\/\/www.codemotion.com\/magazine\/#website","url":"https:\/\/www.codemotion.com\/magazine\/","name":"Codemotion Magazine","description":"We code the future. Together","publisher":{"@id":"https:\/\/www.codemotion.com\/magazine\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.codemotion.com\/magazine\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.codemotion.com\/magazine\/#organization","name":"Codemotion","url":"https:\/\/www.codemotion.com\/magazine\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.codemotion.com\/magazine\/#\/schema\/logo\/image\/","url":"https:\/\/www.codemotion.com\/magazine\/wp-content\/uploads\/2019\/11\/codemotionlogo.png","contentUrl":"https:\/\/www.codemotion.com\/magazine\/wp-content\/uploads\/2019\/11\/codemotionlogo.png","width":225,"height":225,"caption":"Codemotion"},"image":{"@id":"https:\/\/www.codemotion.com\/magazine\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/Codemotion.Italy\/","https:\/\/x.com\/CodemotionIT"]},{"@type":"Person","@id":"https:\/\/www.codemotion.com\/magazine\/#\/schema\/person\/f9c99d8dfb99900d709bb8199225d313","name":"Vito Gentile","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/ebcb5b885a4f1669578d08a55deb81064893c07c1b35b7b36eb059cce730ae90?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/ebcb5b885a4f1669578d08a55deb81064893c07c1b35b7b36eb059cce730ae90?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/ebcb5b885a4f1669578d08a55deb81064893c07c1b35b7b36eb059cce730ae90?s=96&d=mm&r=g","caption":"Vito Gentile"},"description":"I\u2019m a data scientist, tech writer, software developer with experience in mobile, web (full-stack) and Python programming, and former researcher with interests in human-computer interaction. I thus have a multi-faceted experience in the area of software development, and that\u2019s why I love my job(s)!","sameAs":["https:\/\/vitogentile.it","https:\/\/www.linkedin.com\/in\/vitogentile\/en","https:\/\/x.com\/ViGentile"],"url":"https:\/\/www.codemotion.com\/magazine\/author\/vito-gentile\/"}]}},"featured_image_src":"https:\/\/www.codemotion.com\/magazine\/wp-content\/uploads\/2021\/05\/Population-Shift-Analysis-How-to-Monitor-Data-Quality-with-Popmon-600x400.jpg","featured_image_src_square":"https:\/\/www.codemotion.com\/magazine\/wp-content\/uploads\/2021\/05\/Population-Shift-Analysis-How-to-Monitor-Data-Quality-with-Popmon-600x600.jpg","author_info":{"display_name":"Vito Gentile","author_link":"https:\/\/www.codemotion.com\/magazine\/author\/vito-gentile\/"},"uagb_featured_image_src":{"full":["https:\/\/www.codemotion.com\/magazine\/wp-content\/uploads\/2021\/05\/Population-Shift-Analysis-How-to-Monitor-Data-Quality-with-Popmon.jpg",1200,628,false],"thumbnail":["https:\/\/www.codemotion.com\/magazine\/wp-content\/uploads\/2021\/05\/Population-Shift-Analysis-How-to-Monitor-Data-Quality-with-Popmon-150x150.jpg",150,150,true],"medium":["https:\/\/www.codemotion.com\/magazine\/wp-content\/uploads\/2021\/05\/Population-Shift-Analysis-How-to-Monitor-Data-Quality-with-Popmon-300x157.jpg",300,157,true],"medium_large":["https:\/\/www.codemotion.com\/magazine\/wp-content\/uploads\/2021\/05\/Population-Shift-Analysis-How-to-Monitor-Data-Quality-with-Popmon-768x402.jpg",768,402,true],"large":["https:\/\/www.codemotion.com\/magazine\/wp-content\/uploads\/2021\/05\/Population-Shift-Analysis-How-to-Monitor-Data-Quality-with-Popmon-1024x536.jpg",1024,536,true],"1536x1536":["https:\/\/www.codemotion.com\/magazine\/wp-content\/uploads\/2021\/05\/Population-Shift-Analysis-How-to-Monitor-Data-Quality-with-Popmon.jpg",1200,628,false],"2048x2048":["https:\/\/www.codemotion.com\/magazine\/wp-content\/uploads\/2021\/05\/Population-Shift-Analysis-How-to-Monitor-Data-Quality-with-Popmon.jpg",1200,628,false],"small-home-featured":["https:\/\/www.codemotion.com\/magazine\/wp-content\/uploads\/2021\/05\/Population-Shift-Analysis-How-to-Monitor-Data-Quality-with-Popmon.jpg",100,52,false],"sidebar-featured":["https:\/\/www.codemotion.com\/magazine\/wp-content\/uploads\/2021\/05\/Population-Shift-Analysis-How-to-Monitor-Data-Quality-with-Popmon-180x128.jpg",180,128,true],"genesis-singular-images":["https:\/\/www.codemotion.com\/magazine\/wp-content\/uploads\/2021\/05\/Population-Shift-Analysis-How-to-Monitor-Data-Quality-with-Popmon-896x504.jpg",896,504,true],"archive-featured":["https:\/\/www.codemotion.com\/magazine\/wp-content\/uploads\/2021\/05\/Population-Shift-Analysis-How-to-Monitor-Data-Quality-with-Popmon-400x225.jpg",400,225,true],"gb-block-post-grid-landscape":["https:\/\/www.codemotion.com\/magazine\/wp-content\/uploads\/2021\/05\/Population-Shift-Analysis-How-to-Monitor-Data-Quality-with-Popmon-600x400.jpg",600,400,true],"gb-block-post-grid-square":["https:\/\/www.codemotion.com\/magazine\/wp-content\/uploads\/2021\/05\/Population-Shift-Analysis-How-to-Monitor-Data-Quality-with-Popmon-600x600.jpg",600,600,true]},"uagb_author_info":{"display_name":"Vito Gentile","author_link":"https:\/\/www.codemotion.com\/magazine\/author\/vito-gentile\/"},"uagb_comment_info":0,"uagb_excerpt":"One of the most significant parts of any data-driven application is data quality assessment. Before you start using your data, you must understand how good &#8211; or bad &#8211; it is. This is why data analysis and data cleaning activities are performed &#8211; to obtain a dataset that is ready to be used within the&#8230;&hellip;","lang":"en","_links":{"self":[{"href":"https:\/\/www.codemotion.com\/magazine\/wp-json\/wp\/v2\/posts\/14802","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.codemotion.com\/magazine\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.codemotion.com\/magazine\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.codemotion.com\/magazine\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/www.codemotion.com\/magazine\/wp-json\/wp\/v2\/comments?post=14802"}],"version-history":[{"count":17,"href":"https:\/\/www.codemotion.com\/magazine\/wp-json\/wp\/v2\/posts\/14802\/revisions"}],"predecessor-version":[{"id":21588,"href":"https:\/\/www.codemotion.com\/magazine\/wp-json\/wp\/v2\/posts\/14802\/revisions\/21588"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.codemotion.com\/magazine\/wp-json\/wp\/v2\/media\/14816"}],"wp:attachment":[{"href":"https:\/\/www.codemotion.com\/magazine\/wp-json\/wp\/v2\/media?parent=14802"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.codemotion.com\/magazine\/wp-json\/wp\/v2\/categories?post=14802"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.codemotion.com\/magazine\/wp-json\/wp\/v2\/tags?post=14802"},{"taxonomy":"collections","embeddable":true,"href":"https:\/\/www.codemotion.com\/magazine\/wp-json\/wp\/v2\/collections?post=14802"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}