使用 awk getline bash 从指定时间范围内的日志文件中提取数据

我正在搜索解析日志文件并在此链接中找到了我需要的内容从日志文件中提取指定时间范围内的数据 https://stackoverflow.com/questions/7575267/extract-data-from-log-file-in-specified-range-of-time

但最有用的答案（由@Kent发布）：

# this variable you could customize, important is convert to seconds. 
# e.g 5days=$((5*24*3600))
x=$((5*60))   #here we take 5 mins as example

# this line get the timestamp in seconds of last line of your logfile
last=$(tail -n1 logFile|awk -F'[][]' '{ gsub(/\//," ",$2); sub(/:/," ",$2); "date +%s -d \""$2"\""|getline d; print d;}' )

#this awk will give you lines you needs:
awk -F'[][]' -v last=$last -v x=$x '{ gsub(/\//," ",$2); sub(/:/," ",$2); "date +%s -d \""$2"\""|getline d; if (last-d<=x)print $0 }' logFile 

我认为错误在于"date +%s -d .... part

给出以下错误：

sh: -c: line 0: unexpected EOF while looking for matching `"'
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `"'
sh: -c: line 1: syntax error: unexpected end of file

在我在这里提问之前，我花了很多时间尝试解决，但没有找到任何解决方案。

该脚本将被调用crontab获取最后 1 分钟的日志行并计算一分钟内列出某个 ip 的次数，以便我可以检测它是否是攻击。这是另一项任务，希望专家能够帮助在同一问题中提供所需的代码。（我认为它可以在两行内解决）。

问题可能只是您没有引用 shell 变量。看：

$ foo='ab cd'

$ awk -v bar="$foo" 'BEGIN{print bar}'
ab cd

$ awk -v bar=$foo 'BEGIN{print bar}'
awk: fatal: cannot open file `BEGIN{print bar}' for reading (No such file or directory)

是的，我知道这是一个不同的错误消息 - 当您将 shell 变量不加引号时会发生什么，具体取决于变量的值、目录的内容等，其中一些非常糟糕，例如删除中的每个文件你的文件系统。

因此，引用您的变量：

-v last="$last" -v x="$x"

然后看看问题是否仍然存在。

顺便说一句，这是如何使用 GNU awk 和输入文件来真正解决您的原始问题http://pastebin.com/BXmS4zLn http://pastebin.com/BXmS4zLn:

$ cat tst.awk
BEGIN {
    ARGV[ARGC++] = ARGV[ARGC-1]

    mths = "JanFebMarAprMayJunJulAugSepOctNovDec"

    if (days)  { hours = days * 24  }
    if (hours) { mins  = hours * 60 }
    if (mins)  { secs  = mins * 60  }
    deltaSecs = secs
}

NR==FNR {
    nr2secs[NR] = mktime($6" "(match(mths,$5)+2)/3" "$4" "gensub(/:/," ","g",$7))
    next
}

nr2secs[FNR] >= (nr2secs[NR-FNR] - deltaSecs)

$ awk -v hours=1 -f tst.awk file
157.55.34.99 - -  06 Sep 2013 09:13:10 +0300  "GET /index.php HTTP/1.1" 200 16977 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
85.163.134.149 - -  06 Sep 2013 09:50:23 +0300  "GET /wap/wapicons/mnrwap.jpg HTTP/1.1" 200 1217 "http://mydomain.com/main.php" "Mozilla/5.0 (Linux; U; Android 4.1.2; en-gb; GT-I9082 Build/JZO54K) AppleWebKit/534.30 (KHTML, like Gecko) Version/4.0 Mobile Safari/534.30"
83.113.48.218 - -  06 Sep 2013 10:13:07 +0300  "GET /english/nicons/word.gif HTTP/1.1" 200 803 "http://mydomain.com/french/details.php?eid=127928&cid=18&fromval=1&frid=18" "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.0; Trident/5.0)"

$ gawk -v mins=60 -f tst.awk file
157.55.34.99 - -  06 Sep 2013 09:13:10 +0300  "GET /index.php HTTP/1.1" 200 16977 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
85.163.134.149 - -  06 Sep 2013 09:50:23 +0300  "GET /wap/wapicons/mnrwap.jpg HTTP/1.1" 200 1217 "http://mydomain.com/main.php" "Mozilla/5.0 (Linux; U; Android 4.1.2; en-gb; GT-I9082 Build/JZO54K) AppleWebKit/534.30 (KHTML, like Gecko) Version/4.0 Mobile Safari/534.30"
83.113.48.218 - -  06 Sep 2013 10:13:07 +0300  "GET /english/nicons/word.gif HTTP/1.1" 200 803 "http://mydomain.com/french/details.php?eid=127928&cid=18&fromval=1&frid=18" "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.0; Trident/5.0)"

$ gawk -v mins=20 -f tst.awk file
83.113.48.218 - -  06 Sep 2013 10:13:07 +0300  "GET /english/nicons/word.gif HTTP/1.1" 200 803 "http://mydomain.com/french/details.php?eid=127928&cid=18&fromval=1&frid=18" "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.0; Trident/5.0)"

您可以指定 days= 或 hours= 或 mins= 或 secs= 变量，它会做正确的事情。

如果您只需要一个脚本来获取您的问题所指出的最后 1 分钟的日志行（现在？），并且希望看到一个单行代码来执行此操作：

$ gawk 'NR==FNR {nr2secs[++nr] = mktime($6" "(match("JanFebMarAprMayJunJulAugSepOctNovDec",$5)+2)/3" "$4" "gensub(/:/," ","g",$7)); next} nr2secs[FNR] >= (nr2secs[nr] - 60)' file file
83.113.48.218 - -  06 Sep 2013 10:13:07 +0300  "GET /english/nicons/word.gif HTTP/1.1" 200 803 "http://mydomain.com/french/details.php?eid=127928&cid=18&fromval=1&frid=18" "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.0; Trident/5.0)"