{"id":22040,"date":"2023-07-26T09:16:00","date_gmt":"2023-07-26T07:16:00","guid":{"rendered":"https:\/\/www.codemotion.com\/magazine\/?p=22040"},"modified":"2023-07-25T18:16:40","modified_gmt":"2023-07-25T16:16:40","slug":"data-analysis-made-easy-mastering-pandas-for-insightful-results","status":"publish","type":"post","link":"https:\/\/www.codemotion.com\/magazine\/data-science\/data-analysis-made-easy-mastering-pandas-for-insightful-results\/","title":{"rendered":"Data Analysis Made Easy: Mastering Pandas for Insightful Results"},"content":{"rendered":"\n

When talking about data analysis, Pandas is surely the most powerful and widely used Python<\/a> library for data manipulation, cleaning, and preprocessing.<\/p>\n\n\n\n

Thanks to its features, in fact, we can work with tabular data, retrieving them from SQL databases or Excel spreadsheets, for example.<\/p>\n\n\n\n

Then, we have the possibility to manipulate and clean the data to prepare them for further analyses like plots, and also for Machine Learning<\/a>.<\/p>\n\n\n\n

In this article, we\u2019ll introduce Pandas and the concept of a Pandas data frame.<\/p>\n\n\n\n

Then, we\u2019ll show some of the features of Pandas with hands-on practical examples.<\/p>\n\n\n\n

Introduction to Pandas and data frames<\/h1>\n\n\n\n

Introducing Pandas<\/h2>\n\n\n\n

What actually Pandas is?<\/p>\n\n\n\n

Well, this is what the developers say on the Pandas website<\/a> about their mission:<\/p>\n\n\n\n

\u201cpandas aims to be the fundamental high-level building block for doing practical, real-world data analysis in Python. Additionally, it has the broader goal of becoming the most powerful and flexible open source data analysis\/manipulation tool available in any language.\u201d<\/em><\/p>\n\n\n\n

So, we\u2019re dealing with what aims to be the only library that will pop up in our minds when we think of the word \u201cdata analysis\u201d.<\/p>\n\n\n\n

And, to be honest, it is so.<\/p>\n\n\n\n

How to install Pandas<\/h2>\n\n\n\n

There are a couple of ways to install Pandas on your machine.<\/p>\n\n\n\n

If you already have Python installed, you can install Pandas like so:<\/p>\n\n\n

$ pip install pandas<\/code><\/span><\/pre>\n\n\n
The other way is to install Anaconda on your machine. This is my favorite way because it will install Python, Pandas, all the libraries related to Data Science and Machine Learning, and many more.<\/p>\n\n\n\n
So, by installing Anaconda, you practically won\u2019t need to install other Python libraries in the future. <\/p>\n\n\n\n
To discover how to install it on your machine, you can check out their website here<\/a>.<\/p>\n\n\n\n
Understanding Pandas series and Pandas data frames<\/h2>\n\n\n\n
A data frame is a table, somehow like a spreadsheet, where we can organize and analyze data and information.<\/p>\n\n\n\n
In other words, we can think of a data frame as a container that holds and organizes data in columns and rows like so:<\/p>\n\n\n\n
\"\"\/
A Pandas data frame. Image by Author.<\/figcaption><\/figure>\n\n\n\n
<\/p>\n\n\n\n
So, a data frame is an ordered container of data that can be in the form of text or numbers, organized in columns and rows.<\/p>\n\n\n\n
In particular, any Pandas column is also called a \u201cPandas series<\/em>\u201d. So, another way to see a Pandas data frame is that it is an ordered collection of Pandas series.<\/p>\n\n\n\n
Creating and visualizing a Pandas data frame<\/h2>\n\n\n\n
To work with tabular data we have two possibilities:<\/p>\n\n\n\n
    \n
  1. The data have been created somewhere and stored in a file. We can open them in Pandas and use them.<\/li>\n\n\n\n
  2. We can create a data frame ourselves and use it immediately.<\/li>\n<\/ol>\n\n\n\n
    Here we\u2019re showing the second option, while the first will be shown in the next paragraph.<\/p>\n\n\n\n
    In Pandas, we can create a data frame with the same syntax we\u2019d create a Python dictionary. So, suppose we want to create a data frame that stores the values regarding measured times, in seconds, of some people who made several trials running. We can do it like so:<\/p>\n\n\n
    import pandas as<\/span> pd\n\n# Create data frame<\/span>\ntimes = pd.DataFrame({\"Jhon\"<\/span>:[20<\/span>, 18<\/span>, 36<\/span>], \"Simon\"<\/span>:[15<\/span>, 21<\/span>, 19<\/span>], \"Karen\"<\/span>:[22<\/span>, 19<\/span>, 16<\/span>]})\n\n# Show first 10 values<\/span>\ntimes.head()<\/code><\/span>Code language:<\/span> PHP<\/span> (<\/span>php<\/span>)<\/span><\/small><\/pre>\n\n\n
    So, we import Pandas as \u201cpd\u201d, first. Then, we create a data frame with the method pd.DataFrame(), then we use the method head() to show the first 10 values of the data frame. This is what we obtain:
    <\/p>\n\n\n\n
    \"\"\/<\/figure>\n\n\n\n
    The data frame. Image by Author.<\/em><\/p>\n\n\n\n
    Now, let\u2019s use some actual data to show some features of Pandas.<\/p>\n\n\n\n
    Data manipulation in Pandas: a hands-on tutorial with actual data<\/h1>\n\n\n\n
    The best way to learn Pandas is by getting some data and putting our hands on the keyboard. <\/p>\n\n\n\n
    To show some of its features, we\u2019ll get some data related to the world population (the file is downloadable from Kaggle here<\/a>) to analyze it.<\/p>\n\n\n\n
    As we can see, the data are in the form of a CSV file. This is one of the most typical formats.<\/p>\n\n\n\n
    Supposing we\u2019ve renamed the file \u201cpopulation.csv\u201d this is how we can open it and show the first five values for each column:<\/p>\n\n\n
    import pandas as<\/span> pd\n\n# Read CSV<\/span>\npopulation = pd.read_csv(\"population.csv\"<\/span>)\n\n# Show head<\/span>\npopulation.head()<\/code><\/span>Code language:<\/span> PHP<\/span> (<\/span>php<\/span>)<\/span><\/small><\/pre>\n\n\n
    And we get:<\/p>\n\n\n\n
    \"\"\/<\/figure>\n\n\n\n
    The data frame of our data. Image by author.<\/em><\/p>\n\n\n\n
    So, we have a data frame with \u201ca lot\u201d of columns. But how many columns does it have? And how many rows?<\/p>\n\n\n\n
    To show this data, we can type the following:<\/p>\n\n\n
    # Show<\/span> shape<\/span>\npopulation<\/span>.shape<\/span>\n\n>>>\n\n\u00a0 \u00a0 (234, 17)<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n\n
    So, our data frame has 234 rows and 17 columns.<\/p>\n\n\n\n
    One of the things we have to deal with when analyzing data is null values.<\/p>\n\n\n\n
    To check if we have any of them, we type the following:<\/p>\n\n\n
    # Show Null values<\/span>\npopulation.isnull().sum()\n\n>>>\n\n\u00a0 Rank \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 0<\/span>\n\u00a0 CCA3 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 0<\/span>\n\u00a0 Country\/Territory\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 0<\/span>\n\u00a0 Capital\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 0<\/span>\n\u00a0 Continent\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 0<\/span>\n\u00a0 2022<\/span> Population\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 0<\/span>\n\u00a0 2020<\/span> Population\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 0<\/span>\n\u00a0 2015<\/span> Population\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 0<\/span>\n\u00a0 2010<\/span> Population\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 0<\/span>\n\u00a0 2000<\/span> Population\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 0<\/span>\n\u00a0 1990<\/span> Population\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 0<\/span>\n\u00a0 1980<\/span> Population\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 0<\/span>\n\u00a0 1970<\/span> Population\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 0<\/span>\n\u00a0 Area (km\u00b2) \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 0<\/span>\n\u00a0 Density (per km\u00b2)\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 0<\/span>\n\u00a0 Growth Rate\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 0<\/span>\n\u00a0 World Population Percentage\u00a0 \u00a0 0<\/span>\n\u00a0 dtype: int64<\/code><\/span>Code language:<\/span> PHP<\/span> (<\/span>php<\/span>)<\/span><\/small><\/pre>\n\n\n
    And the result shows us that, for each column, we have 0 cells with null values. So, we can proceed with our analysis without worrying about nulls.<\/p>\n\n\n\n
    Now, suppose we\u2019re not interested in all the columns of the data frame because we believe some of them provide data we\u2019re not interested in. We can create another data frame that has only the columns we want to analyze.<\/p>\n\n\n\n
    For example, say that we\u2019re not interested in the following columns: \u201cRank\u201d and \u201cCCA3\u201d. We can decide to create another data frame without these two by tying the following:<\/p>\n\n\n
    population = population.drop([\"Rank\"<\/span>, \"CCA3\"<\/span>], axis=1<\/span>)<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
    So, with the drop() method, we drop the columns we\u2019re not interested in, but we also need to specify the axis. In Pandas, axis=1 represents the vertical direction, while axis=0 represents the horizontal direction (and we use it when we want to drop rows).<\/p>\n\n\n\n
    Also, we don\u2019t like too much that a column is named \u201cCountry\/Territory\u201d because if we need to filter it we\u2019ll need to write the whole column name. So let\u2019s rename it:<\/p>\n\n\n
    population = population.rename(columns={\"Country\/Territory\"<\/span>:\"Country\"<\/span>})<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
    Now, suppose we want to plot the population trend over time, related to the Asian top three countries for growth rate. In other words: we want to intercept the top three Asian countries for growth rate over time and we want to plot the population trend over time.<\/p>\n\n\n\n
    We can do it like so:<\/p>\n\n\n
    \n# Transpose the columns<\/span>\nasia_population_df = asia_population_df.transpose()\n\n# Select the Country in Asia<\/span>\nasia_population_df.columns = asia[\"Country\"<\/span>]\n\n# Create plot<\/span>\n# Select the top 3 Countries per growth rate in Asia<\/span>\ntop_3_population_df = asia_population_df[top_3_countries[\"Country\"<\/span>]]\ntop_3_population_df.plot(figsize=(10<\/span>, 6<\/span>))\n\n# Label title and axes<\/span>\nplt.title(\"Population in Asia with the max growth rate\"<\/span>)\nplt.xlabel(\"Year\"<\/span>)\nplt.ylabel(\"Population\"<\/span>)\n\n# Show legend<\/span>\nplt.legend(loc=\"upper left\"<\/span>)\n\n#\u00a0 Rotate x-axis values<\/span>\nplt.xticks(rotation = 'vertical'<\/span>)\n\n# Show plot<\/span>\nplt.show()<\/code><\/span>Code language:<\/span> PHP<\/span> (<\/span>php<\/span>)<\/span><\/small><\/pre>\n\n\n
    And we get:<\/p>\n\n\n\n
    \"\"\/<\/figure>\n\n\n\n
    The trend we were searching for. Image by Author.<\/em><\/p>\n\n\n\n
    So, in the above code we have:<\/p>\n\n\n\n