Python 抓取 JSON 并将其转换为 CSV

2024-01-24

我对 python 很陌生，这可能是一个可怕的问题，但事情是这样的：

我正在尝试从特定网站抓取数据并将该数据转换为 CSV。我已经设法获得一个可用的“scraper”并将文件保存在我的计算机上。然而，当我尝试再次将 JSON 加载到 python 中时，我收到了各种错误。我相信其中一个错误是由于我在一个文件中添加多个 JSON 数组而引起的json.load()只能加载单个 JSON（据我所知）。

长话短说：谁能指导我正确的方向，如何抓取这些数据并获得一个可读性好的 CSV？

请在下面找到我的代码。

import urllib.request
import urllib.parse
import datetime
import time
import json
import csv


def grabData():
    startdate = datetime.date(2015,2,1)
    startdate += datetime.timedelta(days=1)
    datalist = []
    for i in range(2):

        unixtime = time.mktime(startdate.timetuple())
        print(unixtime)

        url = 'https://api.*************/34.700869,135.503755,'+str(int(unixtime))+'?units=si'
        print(url)
        webURL = urllib.request.urlopen(url)
        data = webURL.read()
        print(data)
        encoding = webURL.info().get_content_charset('utf-8')
        json.loads(data.decode(encoding))
        datalist.append(data.decode())

        startdate += datetime.timedelta(days=1)


    text = datalist
    saveFile = open('osakatest.json','a')
    saveFile.write(str(text))
    saveFile.close()


grabData()

这是我得到的输出的片段。

[{"latitude":34.700869,"longitude":135.503755,"timezone":"Asia/Tokyo","offset":9,"currently":{"time":1422831600,"summary":"Clear","icon":"clear-day","precipType":"rain","temperature":3.51,"apparentTemperature":1.37,"dewPoint":-2.53,"humidity":0.65,"windSpeed":2.24,"windBearing":294,"visibility":11.76,"pressure":1028.66},"hourly":{"summary":"Clear throughout the day.","icon":"clear-day","data":[{"time":1422802800,"summary":"Clear","icon":"clear-night","precipType":"rain","temperature":2.94,"apparentTemperature":1.07,"dewPoint":-2.09,"humidity":0.7,"windSpeed":1.91,"windBearing":295,"visibility":14.52,"pressure":1029.11},{"time":1422806400,"summary":"Clear","icon":"clear-night","precipType":"rain","temperature":2.88,"apparentTemperature":0.65,"dewPoint":-2.12,"humidity":0.7,"windSpeed":2.23,"windBearing":292,"visibility":14.52,"pressure":1028.7},{"time":1422810000,"summary":"Clear","icon":"clear-night","precipType":"rain","temperature":2.88,"apparentTemperature":0.78,"dewPoint":-2.17,"humidity":0.69,"windSpeed":2.11,"windBearing":270,"visibility":14.16,"pressure":1028.52},{"time":1422813600,"summary":"Partly Cloudy","icon":"partly-cloudy-night","precipType":"rain","temperature":2.84,"apparentTemperature":0.47,"dewPoint":-2.12,"humidity":0.7,"windSpeed":2.36,"windBearing":282,"visibility":15.77,"cloudCover":0.38,"pressure":1028.25},{"time":1422817200,"summary":"Clear","icon":"clear-night","precipType":"rain","temperature":2.84,"apparentTemperature":0.65,"dewPoint":-2.13,"humidity":0.7,"windSpeed":2.19,"windBearing":285,"visibility":14.52,"pressure":1028.16},{"time":1422820800,"summary":"Clear","icon":"clear-night","precipType":"rain","temperature":2.88,"apparentTemperature":0.56,"dewPoint":-1.72,"humidity":0.72,"windSpeed":2.32,"windBearing":301,"visibility":12.41,"pressure":1028.21},{"time":1422824400,"summary":"Mostly Cloudy","icon":"partly-cloudy-night","precipType":"rain","temperature":2.84,"apparentTemperature":0.55,"dewPoint":-1.82,"humidity":0.71,"windSpeed":2.28,"windBearing":295,"visibility":14.95,"cloudCover":0.75,"pressure":1028.25},{"time":1422828000,"summary":"Clear","icon":"clear-night","precipType":"rain","temperature":2.97,"apparentTemperature":0.69,"dewPoint":-2.1,"humidity":0.69,"windSpeed":2.29,"windBearing":304,"visibility":12.41,"pressure":1028.55},{"time":1422831600,"summary":"Clear","icon":"clear-day","precipType":"rain","temperature":3.51,"apparentTemperature":1.37,"dewPoint":-2.53,"humidity":0.65,"windSpeed":2.24,"windBearing":294,"visibility":11.76,"pressure":1028.66},{"time":1422835200,"summary":"Mostly Cloudy","icon":"partly-cloudy-day","precipType":"rain","temperature":4.28,"apparentTemperature":2.64,"dewPoint":-3.14,"humidity":0.58,"windSpeed":1.89,"windBearing":285,"visibility":14.63,"cloudCover":0.74,"pressure":1028.55},{"time":1422838800,"summary":"Clear","icon":"clear-day","precipType":"rain","temperature":4.95,"apparentTemperature":2.61,"dewPoint":-3.67,"humidity":0.54,"windSpeed":2.79,"windBearing":301,"visibility":11.76,"pressure":1028.43},{"time":1422842400,"summary":"Clear","icon":"clear-day","precipType":"rain","temperature":6.13,"apparentTemperature":3.48,"dewPoint":-3.35,"humidity":0.51,"windSpeed":3.6,"windBearing":316,"visibility":11.76,"pressure":1027.97},{"time":1422846000,"summary":"Mostly Cloudy","icon":"partly-cloudy-day","precipType":"rain","temperature":5.99,"apparentTemperature":3.08,"dewPoint":-3.91,"humidity":0.49,"windSpeed":3.99,"windBearing":320,"visibility":14.45,"cloudCover":0.75,"pressure":1027.18},{"time":1422849600,"summary":"Clear","icon":"clear-day","precipType":"rain","temperature":6.01,"apparentTemperature":3.01,"dewPoint":-3.74,"humidity":0.5,"windSpeed":4.17,"windBearing":329,"visibility":11.76,"pressure":1026.79},{"time":1422853200,"summary":"Clear","icon":"clear-day","precipType":"rain","temperature":5.94,"apparentTemperature":3.4,"dewPoint":-3.33,"humidity":0.51,"windSpeed":3.34,"windBearing":343,"visibility":11.76,"pressure":1026.69},{"time":1422856800,"summary":"Overcast","icon":"cloudy","precipType":"rain","temperature":5.78,"apparentTemperature":2.98,"dewPoint":-3.41,"humidity":0.52,"windSpeed":3.72,"windBearing":342,"visibility":14.71,"cloudCover":0.96,"pressure":1026.77},{"time":1422860400,"summary":"Clear","icon":"clear-day","precipType":"rain","temperature":5.74,"apparentTemperature":3.42,"dewPoint":-3.07,"humidity":0.53,"windSpeed":2.97,"windBearing":337,"visibility":11.72,"pressure":1026.87},{"time":1422864000,"summary":"Clear","icon":"clear-day","precipType":"rain","temperature":5.33,"apparentTemperature":2.86,"dewPoint":-2.67,"humidity":0.56,"windSpeed":3.07,"windBearing":329,"visibility":11.76,"pressure":1027.16},{"time":1422867600,"summary":"Mostly Cloudy","icon":"partly-cloudy-night","precipType":"rain","temperature":4.43,"apparentTemperature":1.56,"dewPoint":-1.98,"humidity":0.63,"windSpeed":3.37,"windBearing":323,"visibility":14.45,"cloudCover":0.75,"pressure":1027.26},{"time":1422871200,"summary":"Clear","icon":"clear-night","precipType":"rain","temperature":4.36,"apparentTemperature":1.63,"dewPoint":-1.97,"humidity":0.63,"windSpeed":3.14,"windBearing":315,"visibility":11.76,"pressure":1027.55},{"time":1422874800,"summary":"Clear","icon":"clear-night","precipType":"rain","temperature":4.25,"apparentTemperature":1.71,"dewPoint":-1.68,"humidity":0.65,"windSpeed":2.87,"windBearing":300,"visibility":12.41,"pressure":1027.77},{"time":1422878400,"summary":"Mostly Cloudy","icon":"partly-cloudy-night","precipType":"rain","temperature":4.27,"apparentTemperature":1.95,"dewPoint":-1.74,"humidity":0.65,"windSpeed":2.59,"windBearing":313,"visibility":15.8,"cloudCover":0.76,"pressure":1027.98},{"time":1422882000,"summary":"Clear","icon":"clear-night","precipType":"rain","temperature":4.25,"apparentTemperature":1.78,"dewPoint":-1.73,"humidity":0.65,"windSpeed":2.77,"windBearing":297,"visibility":14.52,"pressure":1028.02},{"time":1422885600,"summary":"Clear","icon":"clear-night","precipType":"rain","temperature":4.24,"apparentTemperature":2.03,"dewPoint":-1.72,"humidity":0.65,"windSpeed":2.46,"windBearing":296,"visibility":14.52,"pressure":1028.12}]},"daily":{"data":[{"time":1422802800,"summary":"Clear throughout the day.","icon":"clear-day","sunriseTime":1422827846,"sunsetTime":1422865697,"moonPhase":0.44,"precipType":"rain","temperatureMin":2.84,"temperatureMinTime":1422824400,"temperatureMax":6.13,"temperatureMaxTime":1422842400,"apparentTemperatureMin":0.47,"apparentTemperatureMinTime":1422813600,"apparentTemperatureMax":3.48,"apparentTemperatureMaxTime":1422842400,"dewPoint":-2.49,"humidity":0.62,"windSpeed":2.62,"windBearing":310,"visibility":13.45,"cloudCover":0.73,"pressure":1027.9}]},"flags":{"sources":["isd"],"isd-stations":["476490-99999","476510-99999","476630-99999","476840-99999","477500-99999","477590-99999","477610-99999","477690-99999","477700-99999","477710-99999","477720-99999","477740-99999","477760-99999","477770-99999","477790-99999","477800-99999"],"units":"si"}}]

理想的 CSV 输出：

time,summary,icon,precipType,temperature,apparentTemperature,dewPoint,humidity,windSpeed,windBearing,visibility,pressure,cloudCover
1422802800,Clear,clear-night,rain,2.94,1.07,-2.09,0.7,1.91,295,14.52,1029.11,
1422806400,Clear,clear-night,rain,2.88,0.65,-2.12,0.7,2.23,292,14.52,1028.7,
1422810000,Clear,clear-night,rain,2.88,0.78,-2.17,0.69,2.11,270,14.16,1028.52,
1422813600,Partly Cloudy,partly-cloudy-night,rain,2.84,0.47,-2.12,0.7,2.36,282,15.77,1028.25,0.38

我假设你完成了解析网站的部分并且你得到了最终的结果list其中包含一个python dict。我将在我的解决方案中仅使用最终输出。

所以，你可以尝试这些代码。他们会解析你当前的list并返回一个csv.

PS:

处理一个dict像你的一样并用我当前的代码解析它可能不会按照你在问题中给出的顺序输出输出。

否则，您可以使用和修改这些代码，直到它们满足您的需要。

但是，使用当前的解决方案，您可以获得正确的csv。如果您在下面的代码中发现任何错误或无法理解的内容，请随时发表评论。

解决方案1：使用pandas:

import pandas as pd 
from os import getcwd

a = [{"latitude":34.700869,"longitude":135.503755,"timezone":"Asia/Tokyo","offset":9,"currently":{"time":1422831600,"summary":"Clear","icon":"clear-day","precipType":"rain","temperature":3.51,"apparentTemperature":1.37,"dewPoint":-2.53,"humidity":0.65,"windSpeed":2.24,"windBearing":294,"visibility":11.76,"pressure":1028.66},"hourly":{"summary":"Clear throughout the day.","icon":"clear-day","data":[{"time":1422802800,"summary":"Clear","icon":"clear-night","precipType":"rain","temperature":2.94,"apparentTemperature":1.07,"dewPoint":-2.09,"humidity":0.7,"windSpeed":1.91,"windBearing":295,"visibility":14.52,"pressure":1029.11},{"time":1422806400,"summary":"Clear","icon":"clear-night","precipType":"rain","temperature":2.88,"apparentTemperature":0.65,"dewPoint":-2.12,"humidity":0.7,"windSpeed":2.23,"windBearing":292,"visibility":14.52,"pressure":1028.7},{"time":1422810000,"summary":"Clear","icon":"clear-night","precipType":"rain","temperature":2.88,"apparentTemperature":0.78,"dewPoint":-2.17,"humidity":0.69,"windSpeed":2.11,"windBearing":270,"visibility":14.16,"pressure":1028.52},{"time":1422813600,"summary":"Partly Cloudy","icon":"partly-cloudy-night","precipType":"rain","temperature":2.84,"apparentTemperature":0.47,"dewPoint":-2.12,"humidity":0.7,"windSpeed":2.36,"windBearing":282,"visibility":15.77,"cloudCover":0.38,"pressure":1028.25},{"time":1422817200,"summary":"Clear","icon":"clear-night","precipType":"rain","temperature":2.84,"apparentTemperature":0.65,"dewPoint":-2.13,"humidity":0.7,"windSpeed":2.19,"windBearing":285,"visibility":14.52,"pressure":1028.16},{"time":1422820800,"summary":"Clear","icon":"clear-night","precipType":"rain","temperature":2.88,"apparentTemperature":0.56,"dewPoint":-1.72,"humidity":0.72,"windSpeed":2.32,"windBearing":301,"visibility":12.41,"pressure":1028.21},{"time":1422824400,"summary":"Mostly Cloudy","icon":"partly-cloudy-night","precipType":"rain","temperature":2.84,"apparentTemperature":0.55,"dewPoint":-1.82,"humidity":0.71,"windSpeed":2.28,"windBearing":295,"visibility":14.95,"cloudCover":0.75,"pressure":1028.25},{"time":1422828000,"summary":"Clear","icon":"clear-night","precipType":"rain","temperature":2.97,"apparentTemperature":0.69,"dewPoint":-2.1,"humidity":0.69,"windSpeed":2.29,"windBearing":304,"visibility":12.41,"pressure":1028.55},{"time":1422831600,"summary":"Clear","icon":"clear-day","precipType":"rain","temperature":3.51,"apparentTemperature":1.37,"dewPoint":-2.53,"humidity":0.65,"windSpeed":2.24,"windBearing":294,"visibility":11.76,"pressure":1028.66},{"time":1422835200,"summary":"Mostly Cloudy","icon":"partly-cloudy-day","precipType":"rain","temperature":4.28,"apparentTemperature":2.64,"dewPoint":-3.14,"humidity":0.58,"windSpeed":1.89,"windBearing":285,"visibility":14.63,"cloudCover":0.74,"pressure":1028.55},{"time":1422838800,"summary":"Clear","icon":"clear-day","precipType":"rain","temperature":4.95,"apparentTemperature":2.61,"dewPoint":-3.67,"humidity":0.54,"windSpeed":2.79,"windBearing":301,"visibility":11.76,"pressure":1028.43},{"time":1422842400,"summary":"Clear","icon":"clear-day","precipType":"rain","temperature":6.13,"apparentTemperature":3.48,"dewPoint":-3.35,"humidity":0.51,"windSpeed":3.6,"windBearing":316,"visibility":11.76,"pressure":1027.97},{"time":1422846000,"summary":"Mostly Cloudy","icon":"partly-cloudy-day","precipType":"rain","temperature":5.99,"apparentTemperature":3.08,"dewPoint":-3.91,"humidity":0.49,"windSpeed":3.99,"windBearing":320,"visibility":14.45,"cloudCover":0.75,"pressure":1027.18},{"time":1422849600,"summary":"Clear","icon":"clear-day","precipType":"rain","temperature":6.01,"apparentTemperature":3.01,"dewPoint":-3.74,"humidity":0.5,"windSpeed":4.17,"windBearing":329,"visibility":11.76,"pressure":1026.79},{"time":1422853200,"summary":"Clear","icon":"clear-day","precipType":"rain","temperature":5.94,"apparentTemperature":3.4,"dewPoint":-3.33,"humidity":0.51,"windSpeed":3.34,"windBearing":343,"visibility":11.76,"pressure":1026.69},{"time":1422856800,"summary":"Overcast","icon":"cloudy","precipType":"rain","temperature":5.78,"apparentTemperature":2.98,"dewPoint":-3.41,"humidity":0.52,"windSpeed":3.72,"windBearing":342,"visibility":14.71,"cloudCover":0.96,"pressure":1026.77},{"time":1422860400,"summary":"Clear","icon":"clear-day","precipType":"rain","temperature":5.74,"apparentTemperature":3.42,"dewPoint":-3.07,"humidity":0.53,"windSpeed":2.97,"windBearing":337,"visibility":11.72,"pressure":1026.87},{"time":1422864000,"summary":"Clear","icon":"clear-day","precipType":"rain","temperature":5.33,"apparentTemperature":2.86,"dewPoint":-2.67,"humidity":0.56,"windSpeed":3.07,"windBearing":329,"visibility":11.76,"pressure":1027.16},{"time":1422867600,"summary":"Mostly Cloudy","icon":"partly-cloudy-night","precipType":"rain","temperature":4.43,"apparentTemperature":1.56,"dewPoint":-1.98,"humidity":0.63,"windSpeed":3.37,"windBearing":323,"visibility":14.45,"cloudCover":0.75,"pressure":1027.26},{"time":1422871200,"summary":"Clear","icon":"clear-night","precipType":"rain","temperature":4.36,"apparentTemperature":1.63,"dewPoint":-1.97,"humidity":0.63,"windSpeed":3.14,"windBearing":315,"visibility":11.76,"pressure":1027.55},{"time":1422874800,"summary":"Clear","icon":"clear-night","precipType":"rain","temperature":4.25,"apparentTemperature":1.71,"dewPoint":-1.68,"humidity":0.65,"windSpeed":2.87,"windBearing":300,"visibility":12.41,"pressure":1027.77},{"time":1422878400,"summary":"Mostly Cloudy","icon":"partly-cloudy-night","precipType":"rain","temperature":4.27,"apparentTemperature":1.95,"dewPoint":-1.74,"humidity":0.65,"windSpeed":2.59,"windBearing":313,"visibility":15.8,"cloudCover":0.76,"pressure":1027.98},{"time":1422882000,"summary":"Clear","icon":"clear-night","precipType":"rain","temperature":4.25,"apparentTemperature":1.78,"dewPoint":-1.73,"humidity":0.65,"windSpeed":2.77,"windBearing":297,"visibility":14.52,"pressure":1028.02},{"time":1422885600,"summary":"Clear","icon":"clear-night","precipType":"rain","temperature":4.24,"apparentTemperature":2.03,"dewPoint":-1.72,"humidity":0.65,"windSpeed":2.46,"windBearing":296,"visibility":14.52,"pressure":1028.12}]},"daily":{"data":[{"time":1422802800,"summary":"Clear throughout the day.","icon":"clear-day","sunriseTime":1422827846,"sunsetTime":1422865697,"moonPhase":0.44,"precipType":"rain","temperatureMin":2.84,"temperatureMinTime":1422824400,"temperatureMax":6.13,"temperatureMaxTime":1422842400,"apparentTemperatureMin":0.47,"apparentTemperatureMinTime":1422813600,"apparentTemperatureMax":3.48,"apparentTemperatureMaxTime":1422842400,"dewPoint":-2.49,"humidity":0.62,"windSpeed":2.62,"windBearing":310,"visibility":13.45,"cloudCover":0.73,"pressure":1027.9}]},"flags":{"sources":["isd"],"isd-stations":["476490-99999","476510-99999","476630-99999","476840-99999","477500-99999","477590-99999","477610-99999","477690-99999","477700-99999","477710-99999","477720-99999","477740-99999","477760-99999","477770-99999","477790-99999","477800-99999"],"units":"si"}}]

WORKING_DIRECTORY = getcwd()

def to_csv(a, key, path = WORKING_DIRECTORY, file_name = "new_file.csv"):
    full_path = path + '/' + file_name
    # Only those two keys are supported for this piece of code
    # More work and debugging for the other keys is needed
    if key == "hourly" or key == "daily":
        pd.DataFrame.from_dict(a[key]["data"]).to_csv(full_path)
    else:
        raise Exception("Those keys are not supported yet!")

for k in a:
    to_csv(k, "hourly", file_name="hourly.csv")
    to_csv(k, "daily", file_name="daily.csv" )

解决方案2：手动完成工作：（这不是一个好方法，但您可以了解如何在不使用外部模块的情况下面对此类问题。）

# This code will use OrderedDict 
# In order to maintain the order of the dicts
from collections import OrderedDict

a = [{"latitude":34.700869,"longitude":135.503755,"timezone":"Asia/Tokyo","offset":9,"currently":{"time":1422831600,"summary":"Clear","icon":"clear-day","precipType":"rain","temperature":3.51,"apparentTemperature":1.37,"dewPoint":-2.53,"humidity":0.65,"windSpeed":2.24,"windBearing":294,"visibility":11.76,"pressure":1028.66},"hourly":{"summary":"Clear throughout the day.","icon":"clear-day","data":[{"time":1422802800,"summary":"Clear","icon":"clear-night","precipType":"rain","temperature":2.94,"apparentTemperature":1.07,"dewPoint":-2.09,"humidity":0.7,"windSpeed":1.91,"windBearing":295,"visibility":14.52,"pressure":1029.11},{"time":1422806400,"summary":"Clear","icon":"clear-night","precipType":"rain","temperature":2.88,"apparentTemperature":0.65,"dewPoint":-2.12,"humidity":0.7,"windSpeed":2.23,"windBearing":292,"visibility":14.52,"pressure":1028.7},{"time":1422810000,"summary":"Clear","icon":"clear-night","precipType":"rain","temperature":2.88,"apparentTemperature":0.78,"dewPoint":-2.17,"humidity":0.69,"windSpeed":2.11,"windBearing":270,"visibility":14.16,"pressure":1028.52},{"time":1422813600,"summary":"Partly Cloudy","icon":"partly-cloudy-night","precipType":"rain","temperature":2.84,"apparentTemperature":0.47,"dewPoint":-2.12,"humidity":0.7,"windSpeed":2.36,"windBearing":282,"visibility":15.77,"cloudCover":0.38,"pressure":1028.25},{"time":1422817200,"summary":"Clear","icon":"clear-night","precipType":"rain","temperature":2.84,"apparentTemperature":0.65,"dewPoint":-2.13,"humidity":0.7,"windSpeed":2.19,"windBearing":285,"visibility":14.52,"pressure":1028.16},{"time":1422820800,"summary":"Clear","icon":"clear-night","precipType":"rain","temperature":2.88,"apparentTemperature":0.56,"dewPoint":-1.72,"humidity":0.72,"windSpeed":2.32,"windBearing":301,"visibility":12.41,"pressure":1028.21},{"time":1422824400,"summary":"Mostly Cloudy","icon":"partly-cloudy-night","precipType":"rain","temperature":2.84,"apparentTemperature":0.55,"dewPoint":-1.82,"humidity":0.71,"windSpeed":2.28,"windBearing":295,"visibility":14.95,"cloudCover":0.75,"pressure":1028.25},{"time":1422828000,"summary":"Clear","icon":"clear-night","precipType":"rain","temperature":2.97,"apparentTemperature":0.69,"dewPoint":-2.1,"humidity":0.69,"windSpeed":2.29,"windBearing":304,"visibility":12.41,"pressure":1028.55},{"time":1422831600,"summary":"Clear","icon":"clear-day","precipType":"rain","temperature":3.51,"apparentTemperature":1.37,"dewPoint":-2.53,"humidity":0.65,"windSpeed":2.24,"windBearing":294,"visibility":11.76,"pressure":1028.66},{"time":1422835200,"summary":"Mostly Cloudy","icon":"partly-cloudy-day","precipType":"rain","temperature":4.28,"apparentTemperature":2.64,"dewPoint":-3.14,"humidity":0.58,"windSpeed":1.89,"windBearing":285,"visibility":14.63,"cloudCover":0.74,"pressure":1028.55},{"time":1422838800,"summary":"Clear","icon":"clear-day","precipType":"rain","temperature":4.95,"apparentTemperature":2.61,"dewPoint":-3.67,"humidity":0.54,"windSpeed":2.79,"windBearing":301,"visibility":11.76,"pressure":1028.43},{"time":1422842400,"summary":"Clear","icon":"clear-day","precipType":"rain","temperature":6.13,"apparentTemperature":3.48,"dewPoint":-3.35,"humidity":0.51,"windSpeed":3.6,"windBearing":316,"visibility":11.76,"pressure":1027.97},{"time":1422846000,"summary":"Mostly Cloudy","icon":"partly-cloudy-day","precipType":"rain","temperature":5.99,"apparentTemperature":3.08,"dewPoint":-3.91,"humidity":0.49,"windSpeed":3.99,"windBearing":320,"visibility":14.45,"cloudCover":0.75,"pressure":1027.18},{"time":1422849600,"summary":"Clear","icon":"clear-day","precipType":"rain","temperature":6.01,"apparentTemperature":3.01,"dewPoint":-3.74,"humidity":0.5,"windSpeed":4.17,"windBearing":329,"visibility":11.76,"pressure":1026.79},{"time":1422853200,"summary":"Clear","icon":"clear-day","precipType":"rain","temperature":5.94,"apparentTemperature":3.4,"dewPoint":-3.33,"humidity":0.51,"windSpeed":3.34,"windBearing":343,"visibility":11.76,"pressure":1026.69},{"time":1422856800,"summary":"Overcast","icon":"cloudy","precipType":"rain","temperature":5.78,"apparentTemperature":2.98,"dewPoint":-3.41,"humidity":0.52,"windSpeed":3.72,"windBearing":342,"visibility":14.71,"cloudCover":0.96,"pressure":1026.77},{"time":1422860400,"summary":"Clear","icon":"clear-day","precipType":"rain","temperature":5.74,"apparentTemperature":3.42,"dewPoint":-3.07,"humidity":0.53,"windSpeed":2.97,"windBearing":337,"visibility":11.72,"pressure":1026.87},{"time":1422864000,"summary":"Clear","icon":"clear-day","precipType":"rain","temperature":5.33,"apparentTemperature":2.86,"dewPoint":-2.67,"humidity":0.56,"windSpeed":3.07,"windBearing":329,"visibility":11.76,"pressure":1027.16},{"time":1422867600,"summary":"Mostly Cloudy","icon":"partly-cloudy-night","precipType":"rain","temperature":4.43,"apparentTemperature":1.56,"dewPoint":-1.98,"humidity":0.63,"windSpeed":3.37,"windBearing":323,"visibility":14.45,"cloudCover":0.75,"pressure":1027.26},{"time":1422871200,"summary":"Clear","icon":"clear-night","precipType":"rain","temperature":4.36,"apparentTemperature":1.63,"dewPoint":-1.97,"humidity":0.63,"windSpeed":3.14,"windBearing":315,"visibility":11.76,"pressure":1027.55},{"time":1422874800,"summary":"Clear","icon":"clear-night","precipType":"rain","temperature":4.25,"apparentTemperature":1.71,"dewPoint":-1.68,"humidity":0.65,"windSpeed":2.87,"windBearing":300,"visibility":12.41,"pressure":1027.77},{"time":1422878400,"summary":"Mostly Cloudy","icon":"partly-cloudy-night","precipType":"rain","temperature":4.27,"apparentTemperature":1.95,"dewPoint":-1.74,"humidity":0.65,"windSpeed":2.59,"windBearing":313,"visibility":15.8,"cloudCover":0.76,"pressure":1027.98},{"time":1422882000,"summary":"Clear","icon":"clear-night","precipType":"rain","temperature":4.25,"apparentTemperature":1.78,"dewPoint":-1.73,"humidity":0.65,"windSpeed":2.77,"windBearing":297,"visibility":14.52,"pressure":1028.02},{"time":1422885600,"summary":"Clear","icon":"clear-night","precipType":"rain","temperature":4.24,"apparentTemperature":2.03,"dewPoint":-1.72,"humidity":0.65,"windSpeed":2.46,"windBearing":296,"visibility":14.52,"pressure":1028.12}]},"daily":{"data":[{"time":1422802800,"summary":"Clear throughout the day.","icon":"clear-day","sunriseTime":1422827846,"sunsetTime":1422865697,"moonPhase":0.44,"precipType":"rain","temperatureMin":2.84,"temperatureMinTime":1422824400,"temperatureMax":6.13,"temperatureMaxTime":1422842400,"apparentTemperatureMin":0.47,"apparentTemperatureMinTime":1422813600,"apparentTemperatureMax":3.48,"apparentTemperatureMaxTime":1422842400,"dewPoint":-2.49,"humidity":0.62,"windSpeed":2.62,"windBearing":310,"visibility":13.45,"cloudCover":0.73,"pressure":1027.9}]},"flags":{"sources":["isd"],"isd-stations":["476490-99999","476510-99999","476630-99999","476840-99999","477500-99999","477590-99999","477610-99999","477690-99999","477700-99999","477710-99999","477720-99999","477740-99999","477760-99999","477770-99999","477790-99999","477800-99999"],"units":"si"}}]


def get_data(a, key):

    # This part of code for the complex parts of the dict
    # For: the key 'hourly' and 'daily'
    data, final = [], ''
    if key == 'daily' or key == 'hourly':
        data = [OrderedDict(k) for k in a[key]["data"]]
        # Note here that your JSON don't have the same numbers of keys in their inside dicts
        # This is why this code fetch the max dict's items
        # For example: The key 'cloudCover' isn't present in all the 
        # dicts with key 'hourly' of the original JSON
        max_topic = max(map(lambda x: x.keys(), data), key=lambda x: len(x))
        topic = ','.join(max_topic) + '\n'
        for k in data:
            for j in max_topic:
                try:
                    final += str(k[j]) +', '
                except KeyError:
                    # if the keys isn't found add an empty string
                    final += ' ,' 
            final += '\n'

    # This part for the key 'flags'
    elif key == 'flags':
        data = a[key]
        topic = "{0},{1},{2}\n".format("isd-stations", "sources", "units")
        sub_final = ',{0},{1}\n'.format(''.join(a[key]['sources']), a[key]['units'])
        final = sub_final + ',\n'.join(a[key]["isd-stations"])

    elif key == 'currently':
        data = OrderedDict(a[key])
        topic = ','.join(a[key].keys()) + '\n'
        final = ','.join(map(str, a[key].values()))

    # This part for the other keys like: 'offset'
    # If there will be another key
    # This code will throw an exception 'KeyError'
    else:
        topic = key + '\n'
        final = str(a[key]) + '\n'

    return topic, final

def write_to_csv(a, key, file_name="new_file.csv"):
    topic, data = get_data(a, key)
    with open(file_name, 'a') as f:
        f.write("{0}{1}".format(topic, ''.join(data)))
        # An empty line used if you will run this code multiple times
        # and you'll append your data in a single file
        # This will seperate your formated data for better visualisation
        f.write("\n")


for k in a:
    write_to_csv(k, "hourly", "hourly.csv")
    write_to_csv(k, "daily", "daily.csv")
    write_to_csv(k, "flags", "flags.csv")

Output:

每小时.csv

temperature,windBearing,cloudCover,icon,precipType,summary,humidity,dewPoint,time,pressure,visibility,windSpeed,apparentTemperature
2.94, 295,  ,clear-night, rain, Clear, 0.7, -2.09, 1422802800, 1029.11, 14.52, 1.91, 1.07, 
2.88, 292,  ,clear-night, rain, Clear, 0.7, -2.12, 1422806400, 1028.7, 14.52, 2.23, 0.65, 
2.88, 270,  ,clear-night, rain, Clear, 0.69, -2.17, 1422810000, 1028.52, 14.16, 2.11, 0.78, 
2.84, 282, 0.38, partly-cloudy-night, rain, Partly Cloudy, 0.7, -2.12, 1422813600, 1028.25, 15.77, 2.36, 0.47, 
...

每日.csv

windBearing,apparentTemperatureMax,summary,apparentTemperatureMin,apparentTemperatureMaxTime,temperatureMaxTime,apparentTemperatureMinTime,windSpeed,pressure,temperatureMin,sunsetTime,icon,cloudCover,moonPhase,time,temperatureMinTime,visibility,dewPoint,sunriseTime,precipType,humidity,temperatureMax
310, 3.48, Clear throughout the day., 0.47, 1422842400, 1422842400, 1422813600, 2.62, 1027.9, 2.84, 1422865697, clear-day, 0.73, 0.44, 1422802800, 1422824400, 13.45, -2.49, 1422827846, rain, 0.62, 6.13,
...

标志.csv

isd-stations,sources,units
,isd,si
476490-99999,
476510-99999,
476630-99999,
476840-99999,
...

本文内容由网友自发贡献，版权归原作者所有，本站不承担相应法律责任。如您发现有涉嫌抄袭侵权的内容，请联系:hwhale#tublm.com(使用前将#替换为@)

python

json

csv

Python 抓取 JSON 并将其转换为 CSV 的相关文章

将html数据解析成python列表进行操作

我正在尝试读取 html 网站并提取其数据例如我想查看公司过去 5 年的 EPS 每股收益基本上我可以读入它并且可以使用 BeautifulSoup 或 html2text 创建一个巨大的文本块然后我想搜索该文件我一直在使用
Python getstatusoutput 替换不返回完整输出

我发现了这个很棒的替代品getstatusoutput Python 2 中的函数在 Unix 和 Windows 上同样有效不过我觉得这个方法有问题output被构建它只返回输出的最后一行但我不明白为什么任何帮助都是极好的 def
将 python2.7 与 Emacs 24.3 和 python-mode.el 一起使用

我是 Emacs 新手我正在尝试设置我的 python 环境到目前为止我已经了解到在 python 缓冲区中使用 python mode el C c C c将当前缓冲区的内容加载到交互式 python shell 中显然使用了什么
来自 iPhone/iPad 的 json Web 服务

有人可以帮助我解决如何从 iphone 或 ipad 使用 json Web 服务的问题吗这里我的要求是使用 API 密钥实现 json webservice 如果可能的话发布一些教程或示例链接谢谢规范的 JSON 处理库是here
在Python中连接反斜杠

我是 python 新手所以如果这听起来很简单请原谅我我想加入一些变量来生成一条路径像这样 AAAABBBBCCCC 2 2014 04 2014 04 01 csv Id TypeOfMachine year month year
全局传递 xhr onload 函数的值

在我正在创建的应用程序中我有以下 XMLHttpRequest 并且我正在尝试传递结果data在的里面xhr onload 到在同一父函数中创建的数组中 var url http api soundcloud com resolve j
如何将张量流模型部署到azure ml工作台

我在用Azure ML Workbench执行二元分类到目前为止一切正常我有很好的准确性我想将模型部署为用于推理的 Web 服务我真的不知道从哪里开始 azure 提供了这个doc https learn microsoft co
为什么 PyYAML 花费这么多时间来解析 YAML 文件？

我正在解析一个大约 6500 行的 YAML 文件格式如下 foo1 bar1 blah name john age 123 metadata whatever1 whatever whatever2 whatever stuff thi
从Python中的字典列表中查找特定值

我的字典列表中有以下数据 data I versicolor 0 Sepal Length 7 9 I setosa 0 I virginica 1 I versicolor 0 I setosa 1 I virginica 0 Sepal
Python，将函数的输出重定向到文件中

我正在尝试将函数的输出存储到Python中的文件中我想做的是这样的 def test print This is a Test file open Log a file write test file close 但是当我这样做时我收到
jq：将对象数组转换为对象

我收到了来自curl的回复格式如下 list value 1 id 12 value 15 id 13 value 4 id 14 给定 id 之间的映射如下所示 12 newId1 13 newId2 14 newId3 我想做这个
如何使用shell脚本从json字符串中grep特定字段值[重复]

这个问题在这里已经有答案了下面是文件中可用的 JSON 字符串我需要从中提取值status在 shell 脚本中预期输出 status success 响应 json eventDate null dateProccessed nul
Cython 和类的构造函数

我对 Cython 使用默认构造函数有疑问我的 C 类 Node 如下 Node h class Node public Node std cerr lt lt calling no arg constructor lt lt std e
import matplotlib.pyplot 给出 AttributeError: 'NoneType' 对象没有属性 'is_interactive'

我尝试在 Pycharm 控制台中导入 matplotlib pyplt import matplotlib pyplot as plt 然后作为回报我得到 Traceback most recent call last File D Pr
Elasticsearch：预期的字段名称，但得到了 START_OBJECT

我一直在尝试运行以下查询但每次运行时都会收到以下错误 nested ElasticsearchParseException Expected field name but got START OBJECT field value fact
根据列 value_counts 过滤数据框（pandas）

我是第一次尝试熊猫我有一个包含两列的数据框 user id and string 每个 user id 可能有多个字符串因此会多次出现在数据帧中我想从中导出另一个数据框一个只有那些user ids列出至少有 2 个或更多string
为什么 Pickle 协议 4 中的 Pickle 文件是协议 3 中的两倍，而速度却没有任何提升？

我正在测试 Python 3 4 我注意到 pickle 模块有一个新协议因此我对 2 个协议进行了基准测试 def test1 pickle3 open pickle3 wb for i in range 1000000 pickle
python import inside函数隐藏现有变量

我在我正在处理的多子模块项目中遇到了一个奇怪的 UnboundLocalError 分配之前引用的局部变量问题并将其精简为这个片段使用标准库中的日志记录模块 import logging def foo logging info fo
Pandas 每周计算重复值

我有一个Dataframe包含按周分组的日期和 ID df date id 2022 02 07 1 3 5 4 2022 02 14 2 1 3 2022 02 21 9 10 1 2022 05 16 我想计算每周有多少 id 与上周重
cv2.VideoWriter：请求一个元组作为 Size 参数，然后拒绝它

我正在使用 OpenCV 4 0 和 Python 3 7 创建延时视频构造 VideoWriter 对象时文档表示 Size 参数应该是一个元组当我给它一个元组时它拒绝它当我尝试用其他东西替换它时它不会接受它因为它说参数不是

随机推荐

为什么 Android Lint 在显式使用 Locale.US 时会警告 String.format 使用默认区域设置？

我最初打电话String format这边走 return String format s f f anotherString doubleA doubleB 这使得 Android Lint 生成此警告隐式使用默认区域设置是错误的常见来
在 MongoDB 中，如何索引数组中的嵌入对象字段？

多键的 mongodb 文档给出了查询数组中嵌入对象字段的示例 http www mongodb org display DOCS Multikeys http www mongodb org display DOCS Multikeys
可垂直滚动的 GroupBox

我正在使用创建一个应用程序Windows 申请表我基本上是 C 和 Visual Studio 的新手从昨天开始就一直在使用它到目前为止我已经成功创建了一个简单的表单如屏幕截图所示 My Form 背景细节我正在使用框中提供的
通过NotificationListenerService读取Google地图通知

是否可以读取用户移动时定期显示的 Google 地图导航通知例如定期更新距下一回合的剩余距离我尝试使用NotificationListenerService来实现此目的但是当我尝试使用getNotifications getExtr
使用react.js 在单页应用程序中导航

我正在使用 React 和 Backbone 构建一个单页面应用程序并试图找出处理应用程序中内容之间导航的最佳实践我的应用程序将有一个侧边栏其中包含链接例如照片和设置侧边栏始终存在因此单击设置后我希望在不重新加载整个
TemporaryKey.pfx 出现问题我没有密码

我的一个客户有一个问题他有一位 Microsoft CRM 开发人员但他离开了现在该项目仍未完成我已经下载了自定义应用程序的源代码并且我不想继续开发我尝试过 ClickOne 部署它会弹出一个与 myapp Temporary
ASP.NET Identity 2.1 将 PK 更改为 int 错误

我已经按照中所述更改了 PKhttp www asp net identity overview extensibility change primary key for users in aspnet identity http www
如何标记扩展宏（local :dir ）？

我知道我的标题令人困惑因为tokenize命令被指定为一个字符串我有许多文件夹其中包含大量独立的命名不当的 Excel 文件其中大部分是从网站上抓取的手动选择不方便需要依赖Stata扩展宏功能local dir阅读它们我的
如何像 Pinterest/Evernote 一样从 UICollectionView 过渡到 UIViewController

我有一个 UICollectionView 当选择一个项目时我希望它能够全屏显示动画因此它会从单元格大小过渡到全屏并成为 UIViewController Pinterest 和 Evernote 都有这种行为点击单元格会将单元格转换
jQuery 性能 - 按数据属性还是按类选择？

哪个更快为什么选择 div 满足插件需求 div data something or div something 我倾向于前者因为它更干净基于这个问题 https stackoverflow com questions 61795
Jquery - event.target 和 this 关键字之间的区别？

有什么区别event target and this 假设我有 test click function e thisEventOb e target this this alert thisEventObj alert this 我知道警报
您可以将 HTML 图像标签的 src 属性设置为控制器方法吗？

我知道你可以做到这一点 img src http some svg on the web 但是如果我有一个带有注释的控制器方法怎么办 ResponseBody RequestMapping value getSVG public Respo
pygtk导入gtk错误

我下载了中描述的所有内容 pygtk http www pygtk org 用于安装一切都很顺利直到我尝试打字 import gtk 它抛出了一个ImportError如下 from gtk import gtk ImportError
如果进程被终止，写入 write() 的数据是否会刷新到磁盘？

我正在使用一个程序qdbm http sourceforge net projects qdbm 维护键值存储 qdbm 链接到程序中在某些情况下该进程会在 qdbm 数据库中放入一个值然后通过调用外部 init 脚本通过 syst
Phonegap垂直滚动

我正在为 ios 构建一个phonegap 应用程序并且遇到垂直滚动问题即使没有任何内容网络视图中似乎也会有一些像素滚动这会影响我在 html 中内置的绝对定位的导航栏和选项卡栏这是我的 html 页面没有内容但我仍然得到图像
Promise 重试，直到使用 Typescript 成功/失败

我的移动应用程序会连续将多个文件上传到服务器这些文件通常来自连接强度有问题的偏远地区为此我想尝试几次发送文件我还想在失败时继续尝试下一个导出结束时显示所有错误消息即已上传 10 个文件 3 个失败但是我无法弄清楚带有承诺的
两行Xerces程序中的异常

下面的代码给了我一个例外XMLFormatTarget行但如果我更改字符串 C test xml to test xml 效果很好 test cpp Defines the entry point for the console appl
将引号之间的逗号替换为空格

您好我的表中每一行都有如下数据 0150566115 HEALTH 401K IC ON ICON HEALTH 401K 1 08 21 2014 我想要的是删除双引号之间的每个逗号然后用逗号分割字符串的其余部分我不想检查双引号
std::sort 与本地类型比较

下面的例子 file mysort cc include
Python 抓取 JSON 并将其转换为 CSV

我对 python 很陌生这可能是一个可怕的问题但事情是这样的我正在尝试从特定网站抓取数据并将该数据转换为 CSV 我已经设法获得一个可用的 scraper 并将文件保存在我的计算机上然而当我尝试再次将 JSON 加载到 pyth

Python 抓取 JSON 并将其转换为 CSV

Python 抓取 JSON 并将其转换为 CSV 的相关文章

随机推荐

热门标签