Associate-Developer-Apache-Spark最も有効な質問と解答で勉強

Which of the following code blocks uses a schema fileSchema to read a parquet file at location filePath into a DataFrame?

• A. spark.read.schema(fileSchema).format("parquet").load(filePath)
• B. spark.read().schema(fileSchema).parquet(filePath)
• C. spark.read.schema(fileSchema).open(filePath)
• D. spark.read().schema(fileSchema).format(parquet).load(filePath)
• E. spark.read.schema("fileSchema").format("parquet").load(filePath)

Which of the following code blocks reads the parquet file stored at filePath into DataFrame itemsDf, using a valid schema for the sample of itemsDf shown below?
Sample of itemsDf:
1.+------+-----------------------------+-------------------+
2.|itemId|attributes |supplier |
3.+------+-----------------------------+-------------------+
4.|1 |[blue, winter, cozy] |Sports Company Inc.|
5.|2 |[red, summer, fresh, cooling]|YetiX |
6.|3 |[green, summer, travel] |Sports Company Inc.|
7.+------+-----------------------------+-------------------+

• A. 1.itemsDfSchema = StructType([
  2. StructField("itemId", IntegerType()),
  3. StructField("attributes", ArrayType([StringType()])),
  4. StructField("supplier", StringType())])
  5.
  6.itemsDf = spark.read(schema=itemsDfSchema).parquet(filePath)
• B. 1.itemsDfSchema = StructType([
  2. StructField("itemId", IntegerType),
  3. StructField("attributes", ArrayType(StringType)),
  4. StructField("supplier", StringType)])
  5.
  6.itemsDf = spark.read.schema(itemsDfSchema).parquet(filePath)
• C. 1.itemsDfSchema = StructType([
  2. StructField("itemId", IntegerType()),
  3. StructField("attributes", ArrayType(StringType())),
  4. StructField("supplier", StringType())])
  5.
  6.itemsDf = spark.read.schema(itemsDfSchema).parquet(filePath)
• D. 1.itemsDf = spark.read.schema('itemId integer, attributes <string>, supplier string').parquet(filePath)
• E. 1.itemsDfSchema = StructType([
  2. StructField("itemId", IntegerType()),
  3. StructField("attributes", StringType()),
  4. StructField("supplier", StringType())])
  5.
  6.itemsDf = spark.read.schema(itemsDfSchema).parquet(filePath)

Which of the following statements about Spark's configuration properties is incorrect?

• A. The maximum number of tasks that an executor can process at the same time is controlled by the spark.executor.cores property.
• B. The default number of partitions returned from certain transformations can be controlled by the spark.default.parallelism property.
• C. The default number of partitions to use when shuffling data for joins or aggregations is 300.
• D. The maximum number of tasks that an executor can process at the same time is controlled by the spark.task.cpus property.
• E. The default value for spark.sql.autoBroadcastJoinThreshold is 10MB.

The code block displayed below contains at least one error. The code block should return a DataFrame with only one column, result. That column should include all values in column value from DataFrame transactionsDf raised to the power of 5, and a null value for rows in which there is no value in column value. Find the error(s).
Code block:
1.from pyspark.sql.functions import udf
2.from pyspark.sql import types as T
3.
4.transactionsDf.createOrReplaceTempView('transactions')
5.
6.def pow_5(x):
7. return x**5
8.
9.spark.udf.register(pow_5, 'power_5_udf', T.LongType())
10.spark.sql('SELECT power_5_udf(value) FROM transactions')

• A. The returned DataFrame includes multiple columns instead of just one column.
• B. The pow_5 method is unable to handle empty values in column value, the name of the column in the returned DataFrame is not result, and the SparkSession cannot access the transactionsDf DataFrame.
• C. The pow_5 method is unable to handle empty values in column value, the UDF function is not registered properly with the Spark driver, and the name of the column in the returned DataFrame is not result.
• D. The pow_5 method is unable to handle empty values in column value, the name of the column in the returned DataFrame is not result, and Spark driver does not call the UDF function appropriately.
• E. The pow_5 method is unable to handle empty values in column value and the name of the column in the returned DataFrame is not result.

In which order should the code blocks shown below be run in order to assign articlesDf a DataFrame that lists all items in column attributes ordered by the number of times these items occur, from most to least often?
Sample of DataFrame articlesDf:
1.+------+-----------------------------+-------------------+
2.|itemId|attributes |supplier |
3.+------+-----------------------------+-------------------+
4.|1 |[blue, winter, cozy] |Sports Company Inc.|
5.|2 |[red, summer, fresh, cooling]|YetiX |
6.|3 |[green, summer, travel] |Sports Company Inc.|
7.+------+-----------------------------+-------------------+

• A. 1. articlesDf = articlesDf.groupby("col")
  2. articlesDf = articlesDf.select(explode(col("attributes")))
  3. articlesDf = articlesDf.orderBy("count").select("col")
  4. articlesDf = articlesDf.sort("count",ascending=False).select("col")
  5. articlesDf = articlesDf.groupby("col").count()
• B. 2, 5, 4
• C. 2, 3, 4
• D. 5, 2
• E. 4, 5
• F. 2, 5, 3