python - How to improve speed with Stanford NLP Tagger and NLTK

Question

Is there any way to use the Standford Tagger in a more performant fashion?

Each call to NLTK's wrapper starts a new java instance per analyzed string which is very very slow especially when a larger foreign language model is used...

http://www.nltk.org/api/nltk.tag.html#module-nltk.tag.stanford

Answer 1

Found the solution. It is possible to run the POS Tagger in servlet mode and then connect to it via HTTP. Perfect.

http://nlp.stanford.edu/software/pos-tagger-faq.shtml#d

example

start server in background

nohup java -mx1000m -cp /var/stanford-postagger-full-2014-01-04/stanford-postagger.jar edu.stanford.nlp.tagger.maxent.MaxentTaggerServer -model /var/stanford-postagger-full-2014-01-04/models/german-dewac.tagger -port 2020 >& /dev/null &

adjust firewall to limit access to port 2020 from localhost only

iptables -A INPUT -p tcp -s localhost --dport 2020 -j ACCEPT
iptables -A INPUT -p tcp --dport 2020 -j DROP

test it with wget

wget http://localhost:2020/?die welt ist sch?n

shutdown server

pkill -f stanford

restore iptable settings

iptables -D INPUT -p tcp -s localhost --dport 2020 -j ACCEPT
iptables -D INPUT -p tcp --dport 2020 -j DROP