我正在使用 Spark 2.0 和 Python API。
我有一个数据框,其中有一列类型为 DateType()。我想在包含最近星期一的数据框中添加一列。
我可以这样做:
reg_schema = pyspark.sql.types.StructType([
pyspark.sql.types.StructField('AccountCreationDate', pyspark.sql.types.DateType(), True),
pyspark.sql.types.StructField('UserId', pyspark.sql.types.LongType(), True)
])
reg = spark.read.schema(reg_schema).option('header', True).csv(path_to_file)
reg = reg.withColumn('monday',
pyspark.sql.functions.when(pyspark.sql.functions.date_format(reg.AccountCreationDate,'E') == 'Mon',
reg.AccountCreationDate).otherwise(
pyspark.sql.functions.when(pyspark.sql.functions.date_format(reg.AccountCreationDate,'E') == 'Tue',
pyspark.sql.functions.date_sub(reg.AccountCreationDate, 1)).otherwise(
pyspark.sql.functions.when(pyspark.sql.functions.date_format(reg.AccountCreationDate, 'E') == 'Wed',
pyspark.sql.functions.date_sub(reg.AccountCreationDate, 2)).otherwise(
pyspark.sql.functions.when(pyspark.sql.functions.date_format(reg.AccountCreationDate, 'E') == 'Thu',
pyspark.sql.functions.date_sub(reg.AccountCreationDate, 3)).otherwise(
pyspark.sql.functions.when(pyspark.sql.functions.date_format(reg.AccountCreationDate, 'E') == 'Fri',
pyspark.sql.functions.date_sub(reg.AccountCreationDate, 4)).otherwise(
pyspark.sql.functions.when(pyspark.sql.functions.date_format(reg.AccountCreationDate, 'E') == 'Sat',
pyspark.sql.functions.date_sub(reg.AccountCreationDate, 5)).otherwise(
pyspark.sql.functions.when(pyspark.sql.functions.date_format(reg.AccountCreationDate, 'E') == 'Sun',
pyspark.sql.functions.date_sub(reg.AccountCreationDate, 6))
)))))))
然而,这似乎是很多本应相当简单的代码。有没有更简洁的方法来做到这一点?