The concept of Data Lakehouse arose only recently (circa 2020) and goes beyond the concepts and structures of Data Warehouses and Data Lakes.<\/p>\n\n\n\n
Let\u2019s see how.<\/p>\n\n\n\n
The concept and structure of Data Warehouses<\/strong> were born in the 1980s and it’s a model that gives us the possibility to have structured data – meaning text and numbers – in a structured environment – to simplify and clarify: a database that uses SQL.<\/p>\n\n\n\n In recent years – circa in 2010 -, considering the need to organize somehow unstructured data – meaning: audio, images, and so on – the architecture of Data Lakes<\/strong> was born. This gives us the possibility to store unstructured data in a unique place for further analysis.<\/p>\n\n\n\n One of the big disadvantages of Data Lakes is that to make BI reports we first need ETLs<\/a> to structure the data. Meaning: we need to create one (or more!) Data Warehouse inside the Data Lake, then we can analyze the data and make BI reports.<\/p>\n\n\n\n A Data Lakehouse, instead, is a new system \u201cthat addresses the limitations of data lakes. A lakehouse is a new, open architecture that combines the best elements of data lakes and data warehouses<\/a>\u201d.<\/p>\n\n\n\n Simplifying, the key advantages of Data Lakehouses with respect to Data Lakes are:<\/p>\n\n\n\n In a few words, DataBricks is:<\/p>\n\n\n\n \u201cOne platform for your data, consistently governed and available for all your analytics and AI<\/em>\u201d.<\/p>\n\n\n\n To start using DataBricks, we first need to create an account here<\/a>. DataBricks is it is a paid service, but they provide you with a 14-day trial license.<\/p>\n\n\n\n If you have an account on a cloud service, like AWS or Azure, you can use it: this comes in handy if you have data stored in one of such services.<\/p>\n\n\n\n If you don\u2019t have one, you don\u2019t need to create it. After filling in the fields for the registration, in fact, you can specify that you don\u2019t have an account on any of the services they list and they\u2019ll manage to make you use their cloud service.<\/p>\n\n\n\n The registration process is easy and fast and, in a couple of minutes (maybe, less), you are in and you can start working with DataBricks. For example, you can launch a new notebook or import data from your local machine:<\/p>\n\n\n\n The very first thing we have to do is load some data in DataBricks before we can actually use it for our purposes. This process in DataBricks is called \u201ccluster creation\u201d because you\u2019ll be assigned some computational resources. The process can be started by clicking on \u201cData\u201d in the left sidebar:<\/p>\n\n\n\n Now, here we can choose between three options:<\/p>\n\n\n\n We\u2019ll use a dataset to predict the pricing of houses retrieved from Kaggle here<\/a>.<\/p>\n\n\n\n We simply download the file, unzip it, and load the CSV file into DataBricks:<\/p>\n\n\n\n As we can see, DataBricks gives us a path that indicates the location of the file.<\/p>\n\n\n\n Before starting our analyses, whether we\u2019re talking about Machine Learning or analytical discovery, we have to create a new Notebook. We can do it from the dashboard like so:<\/p>\n\n\n\n As we can see, the Notebook is graphically similar to Jupyter Notebooks or Notebooks in Google Colaboratory.<\/p>\n\n\n\n If we want, we can see the resources we have available in our cluster. On the left-sidebar we click on \u201cCompute\u201d and this is what we can see:<\/p>\n\n\n\n So, we have a cluster with 2 active cores and 15Gb.<\/p>\n\n\n\n Now, let\u2019s return to our Notebook.<\/p>\n\n\n\n To install a Python library we need to use the magic command \u201c%\u201d.<\/p>\n\n\n\n The important thing to know is that DataBricks is equipped with all the most used Python <\/a>libraries. So, for example, if we want to install Pandas<\/a> by typing %pip install pandas, DataBricks will return to us that Pandas is already installed:<\/p>\n\n\n\n So, if we don\u2019t need any particular library, we can import our data and show what\u2019s inside it.<\/p>\n\n\n\n Now, DataBricks also helps us speed up the process involved in loading the data and analyzing it. What we mean is that DataBricks can directly create a new Notebook and add the data to our cluster when we import them. Then, DataBricks creates a Notebook for us that is ready to use:<\/p>\n\n\n\n So, DataBricks created a ready-to-use Notebook with the first cell written in Spark. This way, when we run it, it displays the data we loaded as we can see.<\/p>\n\n\n\n Now, we want to make some predictions with Machine Learning using DataBricks.<\/p>\n\n\n\n For the sake of simplicity, we can use the \u201cdiabetes\u201d dataset provided by sklearn.<\/p>\n\n\n\n So, let\u2019s open a new Notebook in DataBricks as we\u2019ve shown earlier, and import all the libraries we need:<\/p>\n\n\n And we get:<\/p>\n\n\n\n Now, first of all, let me tell you that I didn\u2019t need to install any of the libraries imported above: they\u2019re all installed in DataBricks.<\/p>\n\n\n\n Now, let\u2019s see if there are any highly correlated features with a correlation matrix like so:<\/p>\n\n\n Now, let\u2019s split the data into the train and the test sets, fit with a linear regression model, and calculate R2<\/sup>:<\/p>\n\n\n And we get:<\/p>\n\n\n So, the results of R2<\/sup> are not very convincing, and we\u2019d try different Machine Learning models to solve this problem.<\/p>\n\n\n\n Anyway, here we\u2019ve shown that a notebook in DataBricks can be used exactly as any other Notebook.<\/p>\n\n\n\n In this article, we\u2019ve described the importance of DataBricks as a Lakehouse that, amongst the others, allows us to manage complicated ML workflows<\/a>.<\/p>\n\n\n\n Anyway, we\u2019ve also shown how to use Notebooks in DataBricks with Python. As we\u2019ve seen, DataBricks has all the most used Python libraries<\/a> already installed.<\/p>\n\n\n\n Also, we can use Notebooks in DataBricks the same we we use Jupyter Notebooks, but with the advantage to manage workflows and huge amounts of data, as well as unstructured data, depending on the actual problem we\u2019re solving.<\/p>\n\n\n\n Concluding, some of the reasons why we may use DataBricks are when we need:<\/p>\n\n\n\n\n
Getting Started with DataBricks<\/h1>\n\n\n\n
![\"data](\"https:\/\/www.codemotion.com\/magazine\/wp-content\/uploads\/2023\/08\/image1-1024x351.png\")
![\"\"](\"https:\/\/www.codemotion.com\/magazine\/wp-content\/uploads\/2023\/08\/Databricks-python-1024x407.gif\")
\n
![\"\"](\"https:\/\/www.codemotion.com\/magazine\/wp-content\/uploads\/2023\/08\/Databricks-and-Python-tutorial.png\")
Python in DataBricks<\/h1>\n\n\n\n
![\"Notebook](\"https:\/\/www.codemotion.com\/magazine\/wp-content\/uploads\/2023\/08\/Notebook-on-databricks-1024x407.gif\")
![\"\"](\"https:\/\/www.codemotion.com\/magazine\/wp-content\/uploads\/2023\/08\/Available-resources.-Databricks-and-Python-1024x231.png\")
![\"notebook](\"https:\/\/www.codemotion.com\/magazine\/wp-content\/uploads\/2023\/08\/Notebook-pandas-1024x295.png\")
So, suppose we want to analyze the data related to the houses, as we said before. This time the name of the file is houses.csv. We can perform the process by clicking on \u201cCreate a table in a Notebook\u201d:<\/p>\n\n\n\n![\"\"](\"https:\/\/www.codemotion.com\/magazine\/wp-content\/uploads\/2023\/08\/image7.png\")
![\"databricks](\"https:\/\/www.codemotion.com\/magazine\/wp-content\/uploads\/2023\/08\/image8-1024x407.gif\")
<\/figure>\n\n\n\nAnalytics and Machine Learning with Python in DataBricks<\/h1>\n\n\n\n
import pandas as<\/span> pd\r\nimport numpy as<\/span> np\r\n\r\n\r\n# Plotting<\/span>\r\nimport seaborn as<\/span> sns\r\nimport matplotlib.pyplot as<\/span> plt\r\n# Images dimensions<\/span>\r\nplt.figure(figsize=(10<\/span>, 7<\/span>))\r\n\r\n\r\n# Sklearn<\/span>\r\nfrom sklearn.datasets import load_diabetes #importing data<\/span>\r\nfrom sklearn.linear_model import LinearRegression\r\nfrom sklearn.model_selection import train_test_split\r\nfrom sklearn import linear_model\r\nfrom sklearn import metrics\r\n\r\nNow, let\u2019s show the dataset:\r\n\r\n# Import dataset<\/span>\r\ndiab = load_diabetes()\r\n\r\n# Definine feature and label<\/span>\r\nX = diab['data'<\/span>]\r\ny = diab['target'<\/span>]\r\n\r\n\r\n# Create dataframe from X<\/span>\r\ndf = pd.DataFrame(X, columns=[\"age\"<\/span>,\"sex\"<\/span>,\"bmi\"<\/span>,\"bp\"<\/span>, \"tc\"<\/span>, \"ldl\"<\/span>, \"hdl\"<\/span>,\"tch\"<\/span>, \"ltg\"<\/span>, \"glu\"<\/span>])\r\n\r\n# Add 'progression' from y<\/span>\r\ndf['progression'<\/span>] = diab['target'<\/span>]\r\n\r\n# Show head<\/span>\r\ndf.head()\r\n\r\n<\/code><\/span>Code language:<\/span> PHP<\/span> (<\/span>php<\/span>)<\/span><\/small><\/pre>\n\n\n![\"\"](\"https:\/\/www.codemotion.com\/magazine\/wp-content\/uploads\/2023\/08\/table-results-python-and-databricks.png\")
# Apply mask<\/span>\r\nmask = np.triu(np.ones_like(df.corr()))\r\n\r\n\r\n# Show correlation matrix<\/span>\r\ndataplot = sns.heatmap(df.corr(), annot=True<\/span>, fmt='.2f'<\/span>, mask=mask)<\/code><\/span>Code language:<\/span> PHP<\/span> (<\/span>php<\/span>)<\/span><\/small><\/pre>\n\n\n![\"Tutorial](\"https:\/\/www.codemotion.com\/magazine\/wp-content\/uploads\/2023\/08\/image10-1-1.png\")
# Define features<\/span>\r\nX = df.iloc[:,:-1<\/span>]\r\n\r\n\r\n# Define label<\/span>\r\ny = df['progression'<\/span>]\r\n\r\n\r\n# Split<\/span>\r\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2<\/span>,random_state=42<\/span>)\r\n\r\n\r\n# Fit lin. regr. the model<\/span>\r\nreg = LinearRegression().fit(X_train, y_train)\r\n\r\n\r\n# Make predictions<\/span>\r\ny_test_pred = reg.predict(X_test)\r\ny_train_pred = reg.predict(X_train)\r\n\r\n\r\n# R^2 on both sets<\/span>\r\nprint<\/span>(f'Coeff. of determination on train set:{reg.score(X_train, y_train): .2f}'<\/span>) \r\nprint<\/span>(f'Coeff. of determination on test set:{reg.score(X_test, y_test): .2f}'<\/span>) <\/code><\/span>Code language:<\/span> PHP<\/span> (<\/span>php<\/span>)<\/span><\/small><\/pre>\n\n\nCoeff. of determination on train set<\/span>: 0.53\r\nCoeff. of determination on test set<\/span>: 0.45\r\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\nConclusion<\/h1>\n\n\n\n