robots. Txt parser Java

• Java

I want to know how to parse robots in Java txt.

Do you have a code?

Solution

Heritrix is an open source web crawler written in Java Through their Javadoc, I see that they have a utility class robotstxt for parsing robots Txt file

The content of this article comes from the network collection of netizens. It is used as a learning reference. The copyright belongs to the original author.
THE END
分享
二维码
< <上一篇
下一篇>>