我有一个数据框(df
)由50多个列和不同类型的数据类型组成,例如
df3.printSchema()
CtpJobId: string (nullable = true)
|-- TransformJobStateId: string (nullable = true)
|-- LastError: string (nullable = true)
|-- PriorityDate: string (nullable = true)
|-- QueuedTime: string (nullable = true)
|-- AccurateAsOf: string (nullable = true)
|-- SentToDevice: string (nullable = true)
|-- StartedAtDevice: string (nullable = true)
|-- ProcessStart: string (nullable = true)
|-- LastProgressAt: string (nullable = true)
|-- ProcessEnd: string (nullable = true)
|-- ClipFirstFrameNumber: string (nullable = true)
|-- ClipLastFrameNumber: double (nullable = true)
|-- SourceNamedLocation: string (nullable = true)
|-- TargetId: string (nullable = true)
|-- TargetNamedLocation: string (nullable = true)
|-- TargetDirectory: string (nullable = true)
|-- TargetFilename: string (nullable = true)
|-- Description: string (nullable = true)
|-- AssignedDeviceId: string (nullable = true)
|-- DeviceResourceId: string (nullable = true)
|-- DeviceName: string (nullable = true)
|-- srcDropFrame: string (nullable = true)
|-- srcDuration: double (nullable = true)
|-- srcFrameRate: double (nullable = true)
|-- srcHeight: double (nullable = true)
|-- srcMediaFormat: string (nullable = true)
|-- srcWidth: double (nullable = true)
现在我希望所有一种类型的列都可以一次性更改,例如
timestamp_type = [
'PriorityDate', 'QueuedTime', 'AccurateAsOf', 'SentToDevice',
'StartedAtDevice', 'ProcessStart', 'LastProgressAt', 'ProcessEnd'
]
integer_type = [
'ClipFirstFrameNumber', 'ClipLastFrameNumber', 'TargetId', 'srcHeight',
'srcMediaFormat', 'srcWidth'
]
我知道如何一一做,就像我现在所做的那样。
df3 = df3.withColumn("PriorityDate", df3["PriorityDate"].cast(TimestampType()))
df3 = df3.withColumn("QueuedTime", df3["QueuedTime"].cast(TimestampType()))
df3 = df3.withColumn("AccurateAsOf", df3["AccurateAsOf"].cast(TimestampType())
df3= df3.withColumn("srcMediaFormat", df3["srcMediaFormat"].cast(IntegerType()))
df3= df3.withColumn("DeviceResourceId", df3["DeviceResourceId"].cast(IntegerType()))
df3= df3.withColumn("AssignedDeviceId", df3["AssignedDeviceId"].cast(IntegerType()))
但这看起来很难看,而且我很容易错过任何我想更改的列。有什么方法可以编写任何函数来处理要更改的相同类型的列列表。这样我就可以轻松实现convert_data_type并传递这些列名称。
提前致谢