parsing robots.txt file using c++

Question

Hello, is there is any library to check robots.txt or else how can i right it in c++ with boost regex please explain with some examples....

Answer 1

Check out the examples in the Boost Regex library. If you edit your question to give a better idea of what exactly you are looking for in your robots.txt file, someone can help you with the Regex syntax.

For example, if you are trying to find the names of all User-agents in the file, you could use an expression like this.

boost::regex expression("^User-agent:\s*(.*)");