(编辑:如果您无法从响应中获取“Content-length”标头,则您所访问的 Web 服务可能不会返回该标头。在这种情况下,无法知道响应正文的长度而不完全读取它。您可以在以下示例中通过删除在响应中设置 Content-length 标头的行来模拟这一点。)
标准围棋net/http
包可以很好地处理大型响应。这是一个独立的示例来演示:
// Start a mock HTTP server that returns 2GB of data in the response. Make a
// HTTP request to this server and print the amount of data read from the
// response.
package main
import (
"fmt"
"io"
"log"
"net/http"
"strings"
"time"
)
const oneMB = 1024 * 1024
const oneGB = 1024 * oneMB
const responseSize = 2 * oneGB
const serverAddr = "localhost:9999"
func startServer() {
// Mock HTTP server that always returns 2GB of data
go http.ListenAndServe(serverAddr, http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) {
w.Header().Set("Content-length", fmt.Sprintf("%d", responseSize))
// 1MB buffer that'll be copied multiple times to the response
buf := []byte(strings.Repeat("x", oneMB))
for i := 0; i < responseSize/len(buf); i++ {
if _, err := w.Write(buf); err != nil {
log.Fatal("Failed to write to response. Error: ", err.Error())
}
}
}))
// Some grace period for the server to start
time.Sleep(100 * time.Millisecond)
}
func main() {
startServer()
// HTTP client
req, err := http.NewRequest("GET", "http://"+serverAddr, nil)
if err != nil {
log.Fatal("Error creating HTTP request: ", err.Error())
}
client := http.Client{}
resp, err := client.Do(req)
if err != nil {
log.Fatal("Error making HTTP request: ", err.Error())
}
// Read the response header
fmt.Println("Response: Content-length:", resp.Header.Get("Content-length"))
bytesRead := 0
buf := make([]byte, oneMB)
// Read the response body
for {
n, err := resp.Body.Read(buf)
bytesRead += n
if err == io.EOF {
break
}
if err != nil {
log.Fatal("Error reading HTTP response: ", err.Error())
}
}
fmt.Println("Response: Read", bytesRead, "bytes")
}
如果响应太大,您不会想读取内存中的整个响应。将其写入临时文件,然后进行处理。
相反,如果您正在寻找在网络不太可靠时可靠地执行此操作的选项,请查找“HTTP 范围请求”,您可以使用它来恢复部分下载的数据。