据我所知,PLY 没有实现推送解析器接口,而这是使用 bison 最容易解决这个问题的方式。但是,注入您自己的词法分析器包装器非常容易,它可以处理 dedent 令牌队列。
最小的词法分析器实现需要实现token()
返回一个对象的方法type
and value
属性。 (如果您的解析器使用它,您也需要它,但我不会在这里担心这一点。)
现在,我们假设底层(PLY 生成的)词法分析器生成NEWLINE
其值是换行符后面的前导空格长度的标记。如果某些行不参与 INDENT/DEDENT 算法,则NEWLINE
应抑制这些行;我们在这里不考虑这种情况。一个简单的词法分析器函数示例(仅适用于空格,不适用于制表符)可能是:
# This function doesn't handle tabs. Beware!
def t_NEWLINE(self, t):
r'\n(?:\s*(?:[#].*)?\n)*\s*'
t.value = len(t.value) - 1 - t.value.rfind('\n')
return t
现在我们用一个处理缩进的包装器来包装 PLY 生成的词法分析器:
# WARNING:
# This code hasn't been tested much and it also may be inefficient
# and/or inexact. It doesn't do python-style tab handling. Etc. etc.
from collections import namedtuple, deque
# These are the tokens. We only generate one of each here. If
# we used lineno or didn't trust the parser to not mess with the
# token, we could generate a new one each time.
IndentToken = namedtuple('Token', 'type value')
dedent = IndentToken('DEDENT', None)
indent = IndentToken('INDENT', None)
newline= IndentToken('NEWLINE', None)
class IndentWrapper(object):
def __init__(self, lexer):
"""Create a new wrapper given the lexer which is being wrapped"""
self.lexer = lexer
self.indent_stack = [0]
# A queue is overkill for this case, but it's simple.
self.token_queue = deque()
# This is just in case the ply-generated lexer cannot be called again
# after it returns None.
self.eof_reached = False
def token(self):
"""Return the next token, or None if end of input has been reached"""
# Do we have any queued tokens?
if self.token_queue:
return self.token_queue.popleft()
# Are we done?
if self.eof_reached:
return None
# Get a token
t = self.lexer.token()
if t is None:
# At end of input, we might need to send some dedents
self.eof_reached = True
if len(self.indent_stack) > 1:
t = dedent
for i in range(len(self.indent_stack) - 1):
self.token_queue.append(dedent)
self.indent_stack = [0]
elif t.type == "NEWLINE":
# The NEWLINE token includes the amount of leading whitespace.
# Fabricate indent or dedents as/if necessary and queue them.
if t.value > self.indent_stack[-1]:
self.indent_stack.append(t.value)
self.token_queue.append(indent)
else:
while t.value < self.indent_stack[-1]:
self.indent_stack.pop()
self.token_queue.append(dedent)
if t.value != self.indent_stack[-1]:
raise IndentError # Or however you indicate errors
return t