databricks capstone project solution github

databricks-ml-examples

databricks/databricks-ml-examples is a repository to show machine learning examples on Databricks platforms.

Currently this repository contains:

• llm-models/ : Example notebooks to use different State of the art (SOTA) models on Databricks .
• llm-fine-tuning/ : Fine tuning scripts and notebooks to fine tune State of the art (SOTA) models on Databricks .

SOTA LLM examples

Databricks works with thousands of customers to build generative AI applications. While you can use Databricks to work with any generative AI model, including commercial and research, the table below lists our current model recommendations for popular use cases. Note: The table only lists open source models that are for free commercial use.

• To get a better performance on instructor-xl, you may follow the unified template to write instructions .

Model Evaluation Leaderboard

Text generation models

The model evaluation results presented below are measured by the Mosaic Eval Gauntlet framework. This framework comprises a series of tasks specifically designed to assess the performance of language models, including widely-adopted benchmarks such as MMLU, Big-Bench, HellaSwag, and more.

Other examples:

• DIY LLM QA Bot Accelerator
• Biomedical Question Answering over Custom Datasets with LangChain and Llama 2 from Hugging Face
• DIY QA LLM BOT
• Tuning the Finetuning: An exploration of achieving success with QLoRA
• databricks-llm-fine-tuning

This browser is no longer supported.

Upgrade to Microsoft Edge to take advantage of the latest features, security updates, and technical support.

Git integration with Databricks Git folders

• 8 contributors

Databricks Git folders is a visual Git client and API in Azure Databricks. It supports common Git operations such as cloning a repository, committing and pushing, pulling, branch management, and visual comparison of diffs when committing.

Within Git folders you can develop code in notebooks or other files and follow data science and engineering code development best practices using Git for version control, collaboration, and CI/CD.

Git folders (Repos) are primarily designed for authoring and collaborative workflows.

For information on migrating from a legacy Git integration, see Migrate to Git folders (formerly Repos) from legacy Git .

What can you do with Databricks Git folders?

Databricks Git folders provides source control for data and AI projects by integrating with Git providers.

In Databricks Git folders, you can use Git functionality to:

• Clone, push to, and pull from a remote Git repository.
• Create and manage branches for development work, including merging, rebasing, and resolving conflicts.
• Create notebooks (including IPYNB notebooks) and edit them and other files.
• Visually compare differences upon commit and resolve merge conflicts.

For step-by-step instructions, see Run Git operations on Databricks Git folders (Repos) .

Databricks Git folders also has an API that you can integrate with your CI/CD pipeline. For example, you can programmatically update a Databricks repo so that it always has the most recent version of the code. For information about best practices for code development using Databricks Git folders, see CI/CD techniques with Git and Databricks Git folders (Repos) .

For information on the kinds of notebooks supported in Azure Databricks, see Export and import Databricks notebooks .

Supported Git providers

Databricks Git folders are backed by an integrated Git repository. The repository can be hosted by any of the cloud and enterprise Git providers listed in the following section.

What is a “Git provider”?

A “Git provider” is the specific (named) service that hosts a source control model based on Git. Git-based source control platforms are hosted in two ways: as a cloud service hosted by the developing company, or as an on-premises service installed and managed by your own company on its own hardware. Many Git providers such as GitHub, Microsoft, GitLab, and Atlassian provide both cloud-based SaaS and on-premises (sometimes called “self-managed”) Git services.

When choosing your Git provider during configuration, you must be aware of the differences between cloud (SaaS) and on-premises Git providers. On-premises solutions are typically hosted behind a company VPN and might not be accessible from the internet. Usually, the on-premises Git providers have a name ending in “Server” or “Self-Managed”, but if you are uncertain, contact your company admins or review the Git provider’s documentation.

If your Git provider is cloud-based and not listed as a supported provider, selecting “GitHub” as your provider may work but is not guaranteed.

If you are using “GitHub” as a provider and are still uncertain if you are using the cloud or on-premises version, see About GitHub Enterprise Server in the GitHub docs.

Cloud Git providers supported by Databricks

• GitHub, GitHub AE, and GitHub Enterprise Cloud
• Atlassian BitBucket Cloud
• GitLab and GitLab EE
• Microsoft Azure DevOps (Azure Repos)

On-premises Git providers supported by Databricks

• GitHub Enterprise Server
• Atlassian BitBucket Server and Data Center
• GitLab Self-Managed
• Microsoft Azure DevOps Server: A workspace admin must explicitly allowlist the URL domain prefixes for your Microsoft Azure DevOps Server if the URL does not match dev.azure.com/* or visualstudio.com/* . For more details, see Restrict usage to URLs in an allow list

If you are integrating an on-premises Git repo that is not accessible from the internet, a proxy for Git authentication requests must also be installed within your company’s VPN. For more details, see Set up private Git connectivity for Databricks Git folders (Repos) .

To learn how to use access tokens with your Git provider, see Configure Git credentials & connect a remote repo to Azure Databricks .

Resources for Git integration

Use the Databricks CLI 2.0 for Git integration with Azure Databricks:

• Download the latest CLI version
• Set up the CLI

Read the following reference docs:

• Databricks CLI global flags and commands
• Set up Databricks Git folders (Repos)
• Configure Git credentials & connect a remote repo to Azure Databricks

Was this page helpful?

Coming soon: Throughout 2024 we will be phasing out GitHub Issues as the feedback mechanism for content and replacing it with a new feedback system. For more information see: https://aka.ms/ContentUserFeedback .

Submit and view feedback for

Additional resources

Get the Reddit app

News & discussion on Data Engineering topics, including but not limited to: data pipelines, databases, data formats, storage, data modeling, data governance, cleansing, NoSQL, distributed systems, streaming, batch, Big Data, and workflow engines.

Databricks Developer Foundations Capstone

I'm really struggling as the material before the capstone does not cover the solutions the capstone asks you to engineer. Anyone have tips on where to look for ingesting a text file with a schema of column names and string length positions?

By continuing, you agree to our User Agreement and acknowledge that you understand the Privacy Policy .

Enter the 6-digit code from your authenticator app

You’ve set up two-factor authentication for this account.

Enter a 6-digit backup code

Create your username and password.

Reddit is anonymous, so your username is what you’ll go by here. Choose wisely—because once you get a name, you can’t change it.

Reset your password

Enter your email address or username and we’ll send you a link to reset your password

Check your inbox

An email with a link to reset your password was sent to the email address associated with your account

Choose a Reddit account to continue

Git Integration Repo in Databricks Workspaces for Developer Collaboration

By: Temidayo Omoniyi   |   Updated: 2023-11-10   |   Comments   |   Related: > Azure Databricks

In today's world, having an environment where developers can collaborate and have code reviews is essential for most software personnel and technological companies. Being able to vet developer code before being pushed to the production environment is immensely important. The issue of manually moving notebooks from one workspace or folder can be tiring, and a solution is needed.

With the introduction of Git Integration Repo in Databricks workspaces, a developer can now collaborate with other developers for their data engineering, science, and analytic project in a single workspace and provides version control for different stages of code.

What is GitHub?

GitHub is a cloud-based hosting platform that enables developers to store and manage their code and monitor and manage changes over time. GitHub is built on top of Git, a distributed version control system that offers an intuitive graphical user interface (GUI).

GitHub Features

GitHub is a version control platform that helps developers improve their code using the best software practices:

• Version Control: This platform makes it easy for developers to keep track of changes in multiple code versions, backtrack, or resort to a previous version and see changes being made.
• Code Review: This feature enables developers to review and modify code before being merged to the main branch.
• Collaboration: This feature makes working together on code projects simple for developers. Developers can create branches to work on new features or problem fixes without affecting the main source. Once their changes are made, they can be merged back to the main branch with the pull request command.
• Continuous Integration and Continuous Delivery (CI/CD): This is by far the most important feature of GitHub. It helps automate code building, testing, and deployment, making it possible to release new features rapidly and safely, always keeping the code in a deployable state.

Databricks Repos

Azure Databricks Repos provides a graphical Git client and APIs. This enables standard Git activities such as cloning repositories, pushing and pulling, branch management, and visual comparison between different commits.

Within the Databricks Repos, code developed for different data-related projects can follow the best practices using Git for version control, collaboration, and CI/CD.

Possibilities with Databricks Repos

Databricks Repos comes with all the functionalities of Git:

• Remotely clone, push to, and retrieve the Git repository.
• Developers can create, work, and maintain feature branches before merging to the main branch. In this way, it resolves the problem of conflicts between multiple branches.
• Provide the ability to create, edit, and modify notebooks, including IPYNB notebooks.

Databricks Supported Git Provider

Azure Databricks supports the following providers:

• GitHub and GitHub AE
• Azure DevOps
• Bitbucket Cloud
• Bitbucket Server
• AWS CodeCommit

We will use the GitHub provider for this article; subsequent articles will explain the other providers.

Configure Git Integration for Databricks Workspace

Get username and personal token account.

Step 1: Personal Token Account. To get the personal access token, log in to your GitHub.com account. On your GitHub homepage, click your profile icon at the top right corner and select Settings .

Step 2: Generate Token. In your settings environment, at the left pane, scroll to the bottom and select Developer Settings . This should open another window.

In the Developer Settings window, click on the Personal access tokens and select Tokens(classic) . This should open a new pane where you are expected to Generate a new token.

Note: You may be prompted to authenticate your login credentials at this stage. For this article, I used the GitHub mobile version for the authentication.

Step 3: Setting New Personal Access Token. In the new window, fill in the following information:

• Note: Provide a name to identify your token easily.
• Expiration: Choose a timeframe. This is a tradeoff between convenience and security. The longer the expiration days, the more risks you may encounter if it falls into the wrong hands.
• Repo: Check the repo box.

Scroll to the bottom and select Generate token .

In the new window, copy the generated personal access token and paste it to a private and secure place, as you will not see it again.

Integrate Workspace with GitHub

Now that we have generated our personal access token, we need to integrate Databricks workspace with GitHub.

Use the following steps to integrate GitHub to the Databricks workspace:

Step 1: Link Account. To link an account in the Databricks workspace, from your workspace, click User Settings at the top right corner and select Linked accounts .

Step 2: Git Provider and Activate. For the next step, fill in the following configuration:

• Git Provider: We will use the GitHub provider for this.
• Link: Select the Personal access token.
• Git provider username or email: Use the same email or username as your GitHub account.
• Token: Paste the generated token from the GitHub account.

Now, click Save to fully integrate GitHub with Databricks workspace.

Create GitHub Repository

GitHub Repository is a central storage for code, documents, and other related project assets. It usually serves as a hub for developers to collaborate, keep track of changes, and control code versions. Each Databricks repo is marked as a GitHub repository.

Step 1: Add Repo. To add a new Repo, click Add Repo and fill in the information in the image below. We will be using a private repo as it will be for organizational use, and we do not want such a repository to be in public view.

Step 2: Copy Repo Link. Click the Code icon in the just created repo, copy the URL (HTTPS) link, and head back to your Databricks workspace.

Add Databricks Repo

In your Databricks workspace, click Repos and create a new Repo.

In the new window, fill in the Repo link (HTTPS) you copied from GitHub and click Create Repo . This will create an underlying repo in your Databricks workspace.

Create Branch

In standard practice, it is best to create a development branch where code is developed before moving it to the main branch . Click the main icon. This will open another window.

In the new window, click Create Branch , name it Dev, and switch to the Dev branch. Click Create .

Create Notebook

Before creating a Notebook in Databricks workspace, create a Folder to house your different notebooks.

There are three ways to create notebooks in the Databricks Repo folder: creating a new notebook, importing a notebook, or cloning an existing notebook . Let's try cloning an existing Repo from our Databricks workspace for this article.

Clone Existing Notebook. To clone an existing notebook to the Dev Repo environment, navigate to the notebook you want to use, click on the three dots, and clone to the Repo directory.

You can rename the clone notebook and then click Clone .

Commit & Push

Commit and Push are two key features in the version control system in GitHub.

• Commit is used to save a snapshot of your code change in your local repository. This enables you to monitor your development over time and time travel, if necessary, to a previous version.
• Push is used to send commit to the GitHub repository branch. This allows other collaborators with access to the repository to see your modifications.

To commit and push your code, click the Dev icon (image below). This will take you to another window.

In the new window, you will see some changes. Click Commit & Push . This will take the code to the Dev branch.

Compare & Pull Request

This GitHub feature allows users to compare changes with the other branches before being requested to merge with the main branch.

To perform this function, head to your GitHub.com site. Locate the repo we created earlier. Click on the Compare & pull request tab. This should take you to another window where you will perform the pull request function.

In the new window, we are comparing the Dev branch and the main branch. Click Create Pull request.

Merge Pull Request

Now that we have successfully created a Pull request, we need to merge it to the main branch by clicking Merge Pull Request. Add a Comment if needed.

After successfully merging the notebook with the main branch, head back to your Databricks Repo and switch to the main branch. You will notice the notebook has been added to the main branch.

This article taught us how to generate a personal token in GitHub and integrate it with the Databricks workspace. We also discussed the importance of GitHub and developer best practices for moving the codebase from the development to the production stage. In our next article, we will discuss Databricks workflow and how to integrate our different GitHub Repo to create a complete ETL pipeline.

• Delta Live Tables
• Introduction to Databricks Workflows
• Managing Azure Databricks with the Azure Cloud Shell Command Line Interface
• What are Lake houses in Microsoft Fabric?

About the author

Comments For This Article

Related Content

Azure Databricks Version Control for Notebooks

Create a Python Wheel File to Package and Distribute Custom Code

Establish Secure Connections to Azure SQL using Service Principal Authentication with PySpark Code

Creating a Modern Data Production Pipeline using Azure Databricks and Azure Data Factory

Performance Tuning Apache Spark with Z-Ordering and Data Skipping in Azure Databricks

Advanced Schema Evolution using Databricks Auto Loader

Azure Databricks Tables - Delta Lake, Hive Metastore, TempViews, Managed, External

Free Learning Guides

Learn Power BI

What is SQL Server?

Download Links

Become a DBA

What is SSIS?

Related Categories

Apache Spark

Azure Data Factory

Azure Databricks

Azure Integration Services

Azure Synapse Analytics

Microsoft Fabric

Development

Date Functions

System Functions

JOIN Tables

SQL Server Management Studio

Database Administration

Performance

Performance Tuning

Locking and Blocking

Data Analytics \ ETL

Integration Services

Popular Articles

Date and Time Conversions Using SQL Server

Format SQL Server Dates with FORMAT Function

SQL Server CROSS APPLY and OUTER APPLY

SQL Server Cursor Example

SQL CASE Statement in Where Clause to Filter Based on a Condition or Expression

DROP TABLE IF EXISTS Examples for SQL Server

SQL NOT IN Operator

SQL Convert Date to YYYYMMDD

Rolling up multiple rows into a single row and column for SQL Server data

Format numbers in SQL Server

Script to retrieve SQL Server database backup history and no backups

Resolving could not open a connection to SQL Server errors

How to install SQL Server 2022 step by step

SQL Server PIVOT and UNPIVOT Examples

How to monitor backup and restore progress in SQL Server

An Introduction to SQL Triggers

List SQL Server Login and User Permissions with fn_my_permissions

SQL Server Management Studio Dark Mode

Using MERGE in SQL Server to insert, update and delete at the same time

SQL Server Loop through Table Rows without Cursor

Navigation Menu

Search code, repositories, users, issues, pull requests..., provide feedback.

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly.

To see all available qualifiers, see our documentation .

Databricks Industry Solutions

• 337 followers
• https://databricks.com/solutions/accelerators

Databricks Solution Accelerators are fully functional notebooks that tackle the most common and high-impact use cases that you face every day. Databricks customers utilize Solution Accelerators as a starting-point for new data use-cases and product development. Solution Accelerators are vetted and built by industry experts at Databricks.

By Industry

• Accelerators for Financial Services
• Accelerators for Healthcare and Life Sciences
• Accelerators for Retail and Consumer Goods
• Accelerators for Media and Entertainment
• Accelerators for Manufacturing
• Accelerators for Public Sector
• Accelerators for Technology and Software

Getting started

Although specific solutions can be downloaded as .dbc archives from our websites, we recommend cloning these repositories onto your databricks environment. Not only will you get access to latest code, but you will be part of a community of experts driving industry best practices and re-usable solutions, influencing our respective industries.

To start using a solution accelerator in Databricks simply follow these steps:

• Clone solution accelerator repository in Databricks using Databricks Repos
• Attach the RUNME notebook to any cluster and execute the notebook via Run-All. A multi-step-job describing the accelerator pipeline will be created, and the link will be provided. The job configuration is written in the RUNME notebook in json format.
• Execute the multi-step-job to see how the pipeline runs.
• You might want to modify the samples in the solution accelerator to your need, collaborate with other users and run the code samples against your own data. To do so start by changing the Git remote of your repository to your organization’s repository vs using our samples repository (learn more). You can now commit and push code, collaborate with other user’s via Git and follow your organization’s processes for code development.

The cost associated with running the accelerator is the user's responsibility.

Project support

Please note the code in this project is provided for your exploration only, and are not formally supported by Databricks with Service Level Agreements (SLAs). They are provided AS-IS and we do not make any guarantees of any kind. Please do not submit a support ticket relating to any issues arising from the use of these projects. The source in this project is provided subject to the Databricks License . All included or referenced third party libraries are subject to the licenses set forth below.

Any issues discovered through the use of this project should be filed as GitHub Issues on the Repo. They will be reviewed as time permits, but there are no formal SLAs for support.

Popular repositories Loading

Security Analysis Tool (SAT) analyzes customer's Databricks account and workspace security configurations and provides recommendations that help them follow Databrick's security best practices. Whe…

Python 76 38

Driving a Large Language Model Revolution in Customer Service and Support

Python 67 22

In this solution, we offer a novel approach to sustainable finance by combining NLP techniques and news analytics to extract key strategic ESG initiatives and learn companies' commitments to corpor…

Python 47 27

Build a question answering system based on a given collection of documents with open-source LLMs

Python 44 22

Low effort linking and easy de-duplication. Databricks ARC provides a simple, automated, lakehouse integrated entity resolution solution for intra and inter data linking.

Python 43 18

Bootstrap your large scale forecasting solution on Databricks with Many Models Forecasting (MMF) Project.

Python 34 11

Repositories

Facilitates simple large scale processing of HLS Medical images, documents, zip files. Previously at https://github.com/dmoore247/pixels

Security Analysis Tool (SAT) analyzes customer's Databricks account and workspace security configurations and provides recommendations that help them follow Databrick's security best practices. When a customer runs SAT, it will compare their workspace configurations against a set of security best practices and delivers a report.

Used for reading & writing X12 messages

Radiology LLM Labeling

Accelerator for customer incentive investment using causal inference techniques

Release process for solution accelerators

This organization has no public members. You must be a member to see who’s a part of this organization.

Top languages

Most used topics.