从数据帧中提取值会因 .在 Spark 的列名中

这是我现有的数据框

+------------------+-------------------------+-----------+---------------+-------------------------+---------------------------+------------------------+--------------------------+---------------+-----------+----------------+-----------------+----------------------+--------------------------+-----------+--------------------+-----------+--------------------------------------------------------------------------------------------+-----------------------+------------------+-----------------------------+-----------------------+----------------------------------+
|DataPartition     |TimeStamp                |_lineItemId|_organizationId|fl:FinancialConceptGlobal|fl:FinancialConceptGlobalId|fl:FinancialConceptLocal|fl:FinancialConceptLocalId|fl:InstrumentId|fl:IsCredit|fl:IsDimensional|fl:IsRangeAllowed|fl:IsSegmentedByOrigin|fl:SegmentGroupDescription|fl:Segments|fl:StatementTypeCode|FFAction|!||LineItemName                                                                                |LineItemName.languageId|LocalLanguageLabel|LocalLanguageLabel.languageId|SegmentChildDescription|SegmentChildDescription.languageId|
+------------------+-------------------------+-----------+---------------+-------------------------+---------------------------+------------------------+--------------------------+---------------+-----------+----------------+-----------------+----------------------+--------------------------+-----------+--------------------+-----------+--------------------------------------------------------------------------------------------+-----------------------+------------------+-----------------------------+-----------------------+----------------------------------+
|SelfSourcedPrivate|2017-11-02T10:23:59+00:00|3          |4298009288     |XTOT                     |3016350                    |null                    |null                      |null           |true       |false           |false            |false                 |null                      |null       |BAL                 |I|!|       |Total Assets                                                                                |505074                 |null              |null                         |null                   |null                              |

这是上面数据框的架构

root
 |-- DataPartition: string (nullable = true)
 |-- TimeStamp: string (nullable = true)
 |-- _lineItemId: long (nullable = true)
 |-- _organizationId: long (nullable = true)
 |-- fl:FinancialConceptGlobal: string (nullable = true)
 |-- fl:FinancialConceptGlobalId: long (nullable = true)
 |-- fl:FinancialConceptLocal: string (nullable = true)
 |-- fl:FinancialConceptLocalId: long (nullable = true)
 |-- fl:InstrumentId: long (nullable = true)
 |-- fl:IsCredit: boolean (nullable = true)
 |-- fl:IsDimensional: boolean (nullable = true)
 |-- fl:IsRangeAllowed: boolean (nullable = true)
 |-- fl:IsSegmentedByOrigin: boolean (nullable = true)
 |-- fl:SegmentGroupDescription: string (nullable = true)
 |-- fl:Segments: struct (nullable = true)
 |    |-- fl:SegmentSequence: struct (nullable = true)
 |    |    |-- _VALUE: long (nullable = true)
 |    |    |-- _segmentId: long (nullable = true)
 |-- fl:StatementTypeCode: string (nullable = true)
 |-- FFAction|!|: string (nullable = true)
 |-- LineItemName: string (nullable = true)
 |-- LineItemName.languageId: long (nullable = true)
 |-- LocalLanguageLabel: string (nullable = true)
 |-- LocalLanguageLabel.languageId: long (nullable = true)
 |-- SegmentChildDescription: string (nullable = true)
 |-- SegmentChildDescription.languageId: long (nullable = true)

我想使用下面的代码重命名数据框的标题列。

 val temp = dfTypeNew.select(dfTypeNew.columns.filter(x => !x.equals("fl:Segments")).map(x => col(x).as(x.replace("_", "LineItem_").replace("fl:", ""))): _*)

当我这样做时，我收到以下错误

线程“main”org.apache.spark.sql.AnalysisException 中出现异常： 无法从 LineItemName#368 中提取值：需要结构类型，但得到了 细绳;

当我重命名我的列时没有.我能够提取

错误的存在是因为(.)点用于访问struct场地 要读取具有列名的字段，请使用反引号，如下所示

  val df = Seq(
    ("a","b","c"),
    ("a","b","c")
  ).toDF("x", "y", "z.z")

  df.select("x", "`z.z`").show(false)

Output

+---+---+
|a  |c.c|
+---+---+
|a  |c  |
|a  |c  |
+---+---+

希望这可以帮助！

拉梅什编辑

@Anupam，你所要做的就是使用 Shankar 在你的代码中建议的上述技术

val temp = dfTypeNew.select(dfTypeNew.columns.filter(x => !x.equals("fl:Segments")).map(x => col(s"`${x}`").as(x.replace("_", "LineItem_").replace("fl:", ""))): _*)

就这样。