Databricks Updated Databricks-Machine-Learning-Professional Exam Questions and Answers by omari

Jensen-Shannon (JS) distance is a symmetric measure of the distance between two probability distributions. It is based on the Kullback-Leibler (KL) divergence, but unlike KL, it does not depend on which distribution is considered the reference and which is considered the comparison. JS distance is also bounded between 0 and 1, which makes it easier to interpret and compare across different features.

Kolmogorov-Smirnov (KS) test is a non-parametric test that compares the cumulative distribution functions (CDFs) of two samples. It calculates the maximum difference between the CDFs and compares it to a critical value to determine if the samples are from the same distribution. KS test is sensitive to the shape, location, and scale of the distributions, but it can also be too sensitive for large datasets, where small changes can result in significant differences.

JS distance is more robust than KS test for numeric feature drift detection because it can handle large datasets without being too sensitive to minor fluctuations. It can also capture changes in the shape and spread of the distributions, not just the location. JS distance does not require any manual threshold or cutoff determinations, unlike KS test, which requires choosing a significance level and a critical value. JS distance can also be generalized to multivariate cases, while KS test is limited to univariate cases. References:

• Which test is the best? We compared 5 methods to detect data drift on large datasets
• The Jensen-Shannon Divergence: A Measure of Distance Between Probability Distributions
• Kullback–Leibler vs Kolmogorov-Smirnov distance

Streaming with Spark as a model deployment strategy means applying a machine learning model to data streams that are processed incrementally and continuously by Spark Structured Streaming. Spark Structured Streaming is a scalable and fault-tolerant stream processing engine that enables complex analytics on live data streams using the Dataset/DataFrame API1. Spark Structured Streaming supports various sources and sinks for streaming data, such as Kafka, Kinesis, TCP sockets, Delta tables, etc2. Spark Structured Streaming also supports various types of operations on streaming data, such as aggregations, windowing, joins, and stateful transformations3. To deploy a machine learning model on streaming data, you can use the MLflow model registry to managethe model lifecycle and versioning4. You can also use the MLflow model serving feature to serve the model as a REST API endpoint that can be invoked by Spark Structured Streaming5. Alternatively, you can use the UDF (user-defined function) feature to apply the model to streaming data within Spark Structured Streaming6.

The inference of incrementally processed records as soon as trigger is hit describes the streaming with Spark as a model deployment strategy. A trigger defines when the results of a streaming query should be written to the output sink. A trigger can be based on a processing time interval, a data size limit, or a continuous mode that writes the results as soon as they are available. The trigger ensures that the streaming query is executed incrementally and continuously, and the model inference is applied to the latest available data. The other options are incorrect because:

• Option A: The inference of batch processed records as soon as a trigger is hit does not describe streaming with Spark, but rather batch processing with Spark. Batch processing means applying a machine learning model to a finite set of data that is processed as a single job. Batch processing does not require a trigger, as the results are written to the output sink when the job is completed.
• Option B: The inference of all types of records in real-time does not describe streaming with Spark, but rather a generic definition of real-time processing. Real-time processing means applying a machine learning model to data streams that are processed as soon as they arrive, with minimal latency. Real-time processing does not necessarily use Spark Structured Streaming, as there are other frameworks and tools that can support it, such as Apache Flink, Apache Storm, etc.
• Option C: The inference of batch processed records as soon as a Spark job is run does not describe streaming with Spark, but rather batch processing with Spark. Batch processing means applying a machine learning model to a finite set of data that is processed as a single job. Batch processing does not depend on a Spark job, as the model inference can be done outside of Spark, such as using a REST API endpoint, a command-line tool, etc.
• Option E: The inference of incrementally processed records as soon as a Spark job is run does not describe streaming with Spark, but rather a contradiction. Incrementally processed records imply streaming processing, while a Spark job implies batch processing. Streaming processing and batch processing are different paradigms of data processing, and cannot be mixed in this way. References: Structured Streaming Programming Guide, Input Sources and Output Sinks, Operations on streaming DataFrames/Datasets, MLflow Model Registry, MLflow Model Serving, Apply machine learning models, [Triggers], [Trigger Types], [Batch Processing], [Real-time Processing], [Real-time Data Processing Frameworks], [Deploy machine learning models], [Batch vs Streaming Processing]

The Feature Store Client fs provides a method called create_table that can be used to create and populate a Feature Store table from a Spark DataFrame. The create_table method takes the following parameters:

• name: The name of the feature table to create. It must include the database name, such as ‘recommender_system.new_table’.
• primary_keys: The name or list of names of the columns that uniquely identify each row in the feature table. For example, ‘customer_id’ or [‘customer_id’, ‘date’].
• df: The Spark DataFrame that contains the data to populate the feature table. It must have the same schema as the feature table, and include the primary key columns.
• description: An optional string that describes the feature table.
• partition_columns: An optional list of column names to partition the feature table by. For example, [‘date’, ‘region’].
• online_store: An optional boolean flag that indicates whether to publish the feature table to an online store. The default value is False.
• online_store_options: An optional dictionary that specifies the configuration options for the online store, such as table name, database name, and write mode. For example, {‘table_name’: ‘new_table_online’, ‘database_name’: ‘recommender_system’, ‘write_mode’: ‘overwrite’}.

The create_table method creates a Delta table with the specified name, primary keys, partition columns, and description, and writes the data from the DataFrame to the table. If online_store is True, it also publishes the feature table to an online store with the specified options. The create_table method returns a FeatureTable object that represents the feature table12

The other code blocks are incorrect because:

• B. The create_table method requires the df parameter to populate the feature table with data.
• C. The write method of the DataFrame does not have a mode called ‘fs’ or a path parameter. To write a DataFrame to a Delta table, the mode should be one of ‘append’, ‘overwrite’, ‘ignore’, or ‘error’, and the format should be ‘delta’. To write a DataFrame to a Feature Store table, the create_table or write_table method of the Feature Store Client should be used.
• D. The create_table method does not have a function parameter. To use a Python function to compute the features, the function should be called first and the output DataFrame should be passed to the df parameter.
• E. The write method of the DataFrame does not have a mode called ‘feature’ or a path parameter. To write a DataFrame to a Feature Store table, the create_table or write_table method of the Feature Store Client should be used.

References:

• Feature Store Python API reference - Databricks
• Work with features in Workspace Feature Store - Databricks

MLflow Autologging is a feature that automatically logs parameters, metrics, and artifacts for machine learning frameworks, such as Hyperopt, without requiring any code changes. When using MLflow Autologging with Hyperopt, MLflow will automatically create a parent run for the hyperparameter tuning process and a child run for each unique combination of hyperparameter values. This way, the user can easily compare and visualize the results of different hyperparameter settings in the MLflow UI. MLflow Autologging also supports nested runs, which means that if the user starts a manual run before calling fmin, the parent run will be nested under the manual run. This allows the user to organize and group related runs together. References: MLflow Autologging, Hyperparameter Tuning with MLflow, Apache Spark MLlib and Hyperopt, Using MLFlow with Hyperopt for Automated Machine Learning, Parallelize hyperparameter tuning with scikit-learn and MLflow, [Optimizing Model Performance with MLflow and Hyperopt: From Hyperparameter Tuning to Serving]