Find out how to get a working cluster with Databricks using Azure and then use the full Pandas API operating in the cluster with Open Datasets and a Python Jupyter Notebook. This video will walk you through creating a workspace in Azure to create the Databricks service, then create the cluster that comes with the Pyspark Pandas API, and finally import the open datasets into the cluster.

Although straightforward to create a Databricks cluster with Azure, it is a bit more involved to run a Python Jupyter Notebook that has Azure ML Open Datasets installed and availabe in the cluster along with the ability to use the full Pandas API you are used to working with and taking advantage of the clustering capabilities from Databricks.

By the end of this video you will understand how to:

• Create an Azure Databricks service and workgroup
• Select Pyspark version to support Pandas API
• Import the azureml-opendatasets PyPI package and install it in clusters
• Run a Jupyter Notebook and attach it to a running cluster
• Verify that the Pyspark Pandas API is available along with the azureml-opendatasets package

• Free Azure credits for Students
• Try Azure for Free
• Azure ML Open Datasets PyPI package
• Introduction to Azure Databricks