在您的情况下,数组的表示存在问题。正确的语法是:
{ "metadata": {},
"name": "marks",
"nullable": true, "type": {"containsNull": true, "elementType": "long", "type": "array" } }
.
为了从 json 检索模式,您可以编写下一个 pyspark 片段:
jsonData = """{
"table1": [{
"first_name": "john",
"last_name": "doe",
"subjects": ["maths", "science"],
"marks": [90, 67],
"dept": "abc"
},
{
"first_name": "dan",
"last_name": "steyn",
"subjects": ["maths", "science"],
"marks": [90, 67],
"dept": "abc"
},
{
"first_name": "rose",
"last_name": "wayne",
"subjects": ["maths", "science"],
"marks": [90, 67],
"dept": "abc"
},
{
"first_name": "nat",
"last_name": "lee",
"subjects": ["maths", "science"],
"marks": [90, 67],
"dept": "abc"
},
{
"first_name": "jim",
"last_name": "lim",
"subjects": ["maths", "science"],
"marks": [90, 67],
"dept": "abc"
}
]
}"""
df = spark.read.json(sc.parallelize([jsonData]))
df.schema.json()
这应该输出:
{
"fields": [{
"metadata": {},
"name": "table1",
"nullable": true,
"type": {
"containsNull": true,
"elementType": {
"fields": [{
"metadata": {},
"name": "dept",
"nullable": true,
"type": "string"
}, {
"metadata": {},
"name": "first_name",
"nullable": true,
"type": "string"
}, {
"metadata": {},
"name": "last_name",
"nullable": true,
"type": "string"
}, {
"metadata": {},
"name": "marks",
"nullable": true,
"type": {
"containsNull": true,
"elementType": "long",
"type": "array"
}
}, {
"metadata": {},
"name": "subjects",
"nullable": true,
"type": {
"containsNull": true,
"elementType": "string",
"type": "array"
}
}],
"type": "struct"
},
"type": "array"
}
}],
"type": "struct"
}
或者,您可以使用df.schema.simpleString()
这将返回一个相对简单的模式格式:
struct<table1:array<struct<dept:string,first_name:string,last_name:string,marks:array<bigint>,subjects:array<string>>>>
最后,您可以将上面的模式存储到一个文件中,并稍后使用以下命令加载它:
import json
new_schema = StructType.fromJson(json.loads(schema_json))
正如你已经做的那样。Remember您也可以针对任何 json 数据动态实现所描述的过程。