Google Updated Professional-Data-Engineer Exam Questions and Answers by alara

To implement for-user crypto-deletion and ensure that customer data stored in BigQuery is encrypted, using native Google Cloud features, the best approach is to use Customer-Managed Encryption Keys (CMEK) with Cloud Key Management Service (KMS). Here’s why:

Customer-Managed Encryption Keys (CMEK):

CMEK allows you to manage your own encryption keys using Cloud KMS. These keys provide additional control over data access and encryption management.

Associating a CMEK with a BigQuery table ensures that data is encrypted with a key you manage.

For-User Crypto-Deletion:

For-user crypto-deletion can be achieved by disabling or destroying the CMEK. Once the key is disabled or destroyed, the data encrypted with that key cannot be decrypted, effectively rendering it unreadable.

Native Integration:

Using CMEK with BigQuery is a native feature, avoiding the need for custom encryption solutions. This simplifies the management and implementation of encryption and decryption processes.

Steps to Implement:

Create a CMEK in Cloud KMS:

Set up a new customer-managed encryption key in Cloud KMS.

Associate the CMEK with BigQuery Tables:

When creating a new table in BigQuery, specify the CMEK to be used for encryption.

This can be done through the BigQuery console, CLI, or API.

Reference Links:

BigQuery and CMEK

Cloud KMS Documentation

Encrypting Data in BigQuery

 By setting the topic retention policy to 30 days, you can ensure that any new subscriber can access the messages thatwere published to the topic within the last 30 days1. This feature allows youto replay previously acknowledged messages or initialize new subscribers with historical data2. You can configure the topic retention policy by using the Cloud Console, the gcloud command-line tool, or the Pub/Sub API1.

Option A is not efficient, as it requires creating a new topic and duplicating the data for each new subscriber, which would increase the storage costs and complexity. Option C is not effective, as it only affects the unacknowledged messages in a subscription, and does not allow new subscribersto access older data3. Option D is not feasible, as it depends on the source system’s ability and willingness to re-push the data, and it may cause data duplication or inconsistency. References:

1: Create a topic | Cloud Pub/Sub Documentation | Google Cloud

2: Replay and purge messages with seek | Cloud Pub/Sub Documentation | Google Cloud

3: When is a PubSub Subscription considered to be inactive?

The key requirements are:

Migrate Spark 3 batch job.

Minimally change the job (reads from GCS, writes to BQ – standard for Spark on GCP).

Job optimized for Spark (specific executor vCPU/memory).

Ability to choose similar executor settings.

Minimize installation and management effort.

Dataproc Serverless (Option A)is designed for these use cases.

Spark 3 Support:Dataproc Serverless supports various Spark runtimes, including Spark 3.

Minimal Changes:Spark jobs reading from GCS and writing to BigQuery (using the Spark-BigQuery connector) are standard. Minimal code changes are generally needed.

Customizable Resources:Dataproc Serverless allows you to specify resources for the driver and executors, including vCPU and memory. You can configure these to match your optimized on-premises settings (e.g., 8 vCPU, 16 GB memory per executor, though specific available configurations should be checked).

Minimal Installation and Management:This is the core benefit of "serverless." You don't need to provision, manage, or scale clusters. You submit your batch job, and Google Cloud handles the underlying infrastructure. This significantly reduces operational overhead.

Let's analyze why other options are less suitable:

B (Compute Engine VM):You would need to manually install Spark, configure it, manage dependencies, and manage the VM itself. This is high management effort.

C (Google Kubernetes Engine cluster):While you can run Spark on GKE (e.g., using Spark on Kubernetes operator), it involves managing the GKE cluster, Spark deployment configurations, Docker images, etc. This is also significant management effort, more than Dataproc Serverless.

D (Dataproc cluster):A standard Dataproc cluster provides more control than serverless but also requires more management (creating, scaling, and managing the cluster lifecycle). Dataproc Serverless is specifically designed to minimize this management for batch jobs. Given the "minimize installation and management effort" requirement, serverless is preferred over a managed cluster if it meets the job's needs.