Databricks Updated Databricks-Certified-Associate-Developer-for-Apache-Spark-3.0 Exam Questions and Answers by nia

Explanation

Correct code block:

def add_2_if_geq_3(x):

if x is None:

return x

elif x >= 3:

return x+2

return x

add_2_if_geq_3_udf = udf(add_2_if_geq_3)

transactionsDf.withColumn("predErrorAdded", add_2_if_geq_3_udf(col("predError"))).show()

Instead of withColumnRenamed, you should use the withColumn operator.

The udf() method does not declare a return type.

It is fine that the udf() method does not declare a return type, this is not a required argument. However, the default return type is StringType. This may not be the ideal return type for numeric,

nullable data – but the code will run without specified return type nevertheless.

The Python function is unable to handle null values, resulting in the code block crashing on execution.

The Python function is able to handle null values, this is what the statement if x is None does.

UDFs are only available through the SQL API, but not in the Python API as shown in the code block.

No, they are available through the Python API. The code in the code block that concerns UDFs is correct.

Instead of col("predError"), the actual DataFrame with the column needs to be passed, like so transactionsDf.predError.

You may choose to use the transactionsDf.predError syntax, but the col("predError") syntax is fine.

Explanation

Output of correct code block:

+-------------+-------+---------+----+

|transactionId|storeId|productId| f|

+-------------+-------+---------+----+

| 1| 25| 1|null|

| 2| 2| 2|null|

| 3| 25| 3|null|

+-------------+-------+---------+----+

To solve this question, you should be fmailiar with the drop() API. The order of column names does not matter – in this QUESTION NO: the order differs in some answers just to confuse you. Also,

drop() does not take a list. The *cols operator in the documentation means that all arguments passed to drop() are interpreted as column names.

More info: pyspark.sql.DataFrame.drop — PySpark 3.1.2 documentation

Static notebook | Dynamic notebook: See test 2, QUESTION NO: 36 (Databricks import instructions)

Explanation

The challenge in this QUESTION NO: comes from there being an array variable in the schema. In addition, you should know how to pass a schema to the DataFrameReader that is invoked by

spark.read.

The correct way to define an array of strings in a schema is through ArrayType(StringType()). A schema can be passed to the DataFrameReader by simply appending schema(structType) to the

read() operator. Alternatively, you can also define a schema as a string. For example, for the schema of itemsDf, the following string would make sense: itemId integer, attributes array,

supplier string.

A thing to keep in mind is that in schema definitions, you always need to instantiate the types, like so: StringType(). Just using StringType does not work in pySpark and will fail.

Another concern with schemas is whether columns should be nullable, so allowed to have null values. In the case at hand, this is not a concern however, since the QUESTION NO: just asks for a

"valid"

schema. Both non-nullable and nullable column schemas would be valid here, since no null value appears in the DataFrame sample.

More info: Learning Spark, 2nd Edition, Chapter 3

Static notebook | Dynamic notebook: See test 3, QUESTION NO: 19 (Databricks import instructions)

An RDD consists of a single partition.

Quite the opposite: Spark partitions RDDs and distributes the partitions across multiple nodes.