如何仅选择语料库术语的子集以在 tm 中创建 TermDocumentMatrix

2023-12-08

我有一个巨大的语料库,我只对我预先知道的少数术语的出现感兴趣。有没有办法使用以下方法从语料库创建术语文档矩阵tm包,其中只使用和包含我预先指定的术语?

我知道我可以对语料库的结果 TermDocumentMatrix 进行子集化,但由于内存大小限制,我想避免从构建完整的术语文档矩阵开始。


您可以通过构建自定义转换函数来修改语料库以仅保留所需的术语。请参阅小插图为tm package以及对的帮助content_transformer函数以获取更多信息:

library(tm)

# Create a corpus from the text listed below
corp = VCorpus(VectorSource(doc))

# Custom function to keep only the terms in "pattern" and remove everything else
(f <- content_transformer(function(x, pattern) 
  regmatches(x, gregexpr(pattern, x, perl=TRUE, ignore.case=TRUE))))

(仅供参考,上面的第二行代码改编自这个答案.)

# The pattern we'll search for
keep = "sleep|dream|die"

# Run the transformation function using the pattern above
tm_map(corp, f, keep)[[1]]

这是运行转换函数的结果:

<<PlainTextDocument (metadata: 7)>>
  c("die", "sleep", "sleep", "die", "sleep", "sleep", "Dream")

这是我用来创建语料库的原始文本:

doc = "To be, or not to be, that is the question—
Whether 'tis Nobler in the mind to suffer
The Slings and Arrows of outrageous Fortune,
Or to take Arms against a Sea of troubles,
And by opposing, end them? To die, to sleep—
No more; and by a sleep, to say we end
The Heart-ache, and the thousand Natural shocks
That Flesh is heir to? 'Tis a consummation
Devoutly to be wished. To die, to sleep,
To sleep, perchance to Dream; Aye, there's the rub"
本文内容由网友自发贡献,版权归原作者所有,本站不承担相应法律责任。如您发现有涉嫌抄袭侵权的内容,请联系:hwhale#tublm.com(使用前将#替换为@)

如何仅选择语料库术语的子集以在 tm 中创建 TermDocumentMatrix 的相关文章

随机推荐