“Data Science in Microsoft Fabric”

  Microsoft Fabric offers Data Science experiences to empower users to complete comprehensive data science workflows for the purpose of data enrichment and business insights. One can engage in a wide range of activities across the entire data science process, all the way from data exploration, preparation and cleansing to experimentation, modeling, model scoring and serving of predictive insights to BI reports.

Users of Microsoft Fabric can access a Data Science Home page where they can discover and access various relevant resources. For instance, they can create machine learning Experiments, Models and Notebooks, as well as import existing Notebooks from the Data Science Home page.

You may be familiar with how a typical data science process works. As a well-recognized process, most machine learning projects follow it.

At a high level, the process involves these steps:

1. Problem formulation and ideation

2. Data discovery and pre-processing

3. Experimentation and modeling

4. Enrich and operationalize.

5. Gain Insights

· Problem formulation and ideation

Ø Data Science users work on the same platform as business users and analysts. Data sharing and collaboration becomes more seamless across different roles as a result. Analysts can easily share Power BI reports and datasets with data science practitioners. The ease of collaboration across roles in Microsoft Fabric makes hand-offs during the problem formulation phase much easier.

· Data discovery and pre-processing

Ø Microsoft Fabric users can interact with data in OneLake using the Lakehouse item. Lakehouse easily attaches to a Notebook to browse and interact with data.

Ø Users can easily read data from a Lakehouse directly into a Pandas data frame. For exploration, this makes seamless data reads from OneLake possible.

Ø A powerful set of tools is available for data ingestion and data orchestration pipelines with data integration pipelines - a natively integrated part of Microsoft Fabric. Easy-to-build data pipelines can access and transform the data into a format that machine learning can consume.

· Data exploration

Ø An important part of the machine learning process is to understand data through exploration and visualization.

Ø Depending on the data storage location, Microsoft Fabric offers a set of different tools to explore and prepare the data for analytics and machine learning. Notebooks become one of the quickest ways to get started with data exploration.

· Apache Spark and Python for data preparation

Ø Microsoft Fabric offers capabilities to transform, prepare, and explore your data at scale. With Spark, users can leverage PySpark/Python, Scala, and SparkR/SparklyR tools for data pre-processing at scale. Powerful open-source visualization libraries can enhance the data exploration experience to help better understand the data.

· Data Wrangler for seamless data cleansing

Ø The Microsoft Fabric Notebook experience added a feature to use Data Wrangler, a code tool that prepares data and generates Python code. This experience makes it easy to accelerate tedious and mundane tasks.

Ø For example, data cleansing and build repeatability and automation through generated code.

· Experimentation and ML modeling

Ø With tools like PySpark/Python, SparklyR/R, notebooks can handle machine learning model training.

Ø ML algorithms and libraries can help train machine learning models. Library management tools can install these libraries and algorithms. Users have therefore the option to leverage a large variety of popular machine learning libraries to complete their ML model training in Microsoft Fabric.

Ø MLflow experiments and runs can track the ML model training. Microsoft Fabric offers a built-in MLflow experience with which users can interact, to log experiments and models.

ü In Conclusion, the Microsoft Fabric Data Science capabilities from the perspective of the data science process. For each step in the data science process.