Is it possible to create custom Parsers in Boost.Spirit?

Question

I was trying to create a custom Parser class in Boost.Spirit (2.3), but it didn't work out. The code is:

template <class Iter>
class crule : public boost::spirit::qi::parser<crule<Iter> >
{
  rule<Iter> r_;
public:
  crule(const rule<Iter>& r) : r_(r) {}
  template <class T>
  crule(const T& t) : r_(t) {}
  template<class Ctx, class Skip>
  bool parse(Iter& f, const Iter& l, Ctx& context, Skip& skip, typename rule<Iter>::template attribute<Ctx, Iter>::type& attr) const {
    return r_.parse(f, l, context, skip, attr);
  }
  template <class Ctx>
  boost::spirit::info what(Ctx& context) const {
    return r_.what(context);
  }
  template <class Context, class It>
  struct attribute {
    typedef typename rule<Iter>::template attribute<Context, It>::type type;
  };
};

and although I have (at least I think I have) fulfilled all the requirements, I get errors when I try to use this class in a parsing expression:

shell_grammar.h:134: error: no match for 'operator!' in '!shell_grammar<Iter>::token(boost::spirit::qi::rule<Iter, boost::fusion::unused_type, boost::fusion::unused_type, boost::fusion::unused_type>) [with Iter = __gnu_cxx::__normal_iterator<const char*, std::basic_string<char, std::char_traits<char>, std::allocator<char> > >](boost::spirit::qi::rule<__gnu_cxx::__normal_iterator<const char*, std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, boost::fusion::unused_type, boost::fusion::unused_type, boost::fusion::unused_type>(((const boost::spirit::qi::rule<__gnu_cxx::__normal_iterator<const char*, std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, boost::fusion::unused_type, boost::fusion::unused_type, boost::fusion::unused_type>&)((const boost::spirit::qi::rule<__gnu_cxx::__normal_iterator<const char*, std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, boost::fusion::unused_type, boost::fusion::unused_type, boost::fusion::unused_type>*)(&((shell_grammar<__gnu_cxx::__normal_iterator<const char*, std::basic_string<char, std::char_traits<char>, std::allocator<char> > > >*)this)->shell_grammar<__gnu_cxx::__normal_iterator<const char*, std::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::reserved_words)))))'

shell_grammar.h:134: note: candidates are: operator!(bool) <built-in>

I tried to look at the implementation of other parsets (eg. not_predicate), but can't figure out what is the difference that makes it work.

Motvation

The reason I do it is related to this question. I want to parse POSIX shell language, which has peculiar lexical rules. Particularly, the "skipper parser" has to be applied even in lexemes, but it has to be different from the "phrase level" skipper parser. Which is what the lexeme directive can't do, and skip doesn't pre-skip (AFAIK), which is what I need, too. So I want to create a function

something token(std::string);

that would return a rule matching the token. One way is creating my own rule wrapper that would serve as a terminal (since rule alone cannot be used for its reference semantics), another would be creating a new parser (that would be a nonterminal in proto), and implement shell's token parsing in it.

Answer 1

It is quite possible, but I have found it to be as much work (and harder to debug) than just writing my own lexers and recursive descent parsers by hand. Even fairly small Spirit grammars can take me weeks of wrestling with the compiler.

This error message you got shows the kind of problems you run into. Any time you get an error, its an error from some template instantiation down deep in the bowels of Spirit, with many further layers of template instatiations added in to confuse matters. In order to have any hope of deciphering the error messages, you pretty much have to understand the code for the entire facility.

I hate to be critical, because Spirit is a worthy effort. I did my Master's thesis on implementing an object-oriented compiler-generator, so I'm a fan of the concept. I really wanted to like it, but Spirit is just too hard for anyone but serious C++ experts to use.

To compare with what can be done, take a look at the Ada OpenToken project. Spirit is probably more flexible, but compile errors are much more sensible in OpenToken, and a glance through the version history on that page shows a very large percentage of their effort has been put into helping users debug errors.

Answer 2

The code you provided looks ok (at least as far as the interface of the actual parser is concerned). But in order to integrate a custom parser with Spirit you need to do more work. Spirit's website has an example for a custom parser component explaining all required steps here.

It looks to me as if you were unnecessarily trying to do things the hard way. But I don't fully understand what you're trying to achieve, so I might be wrong. If you explained your use case I'm sure we could come up with a simpler solution.

Answer 3

BTW this is what I came up to:

You need to register the class as a literal in boost::proto like this:

template <class T>
struct crulexx : public boost::proto::literal<rule<T> >
{
  template <class U>
  crulexx(const U& u) : boost::proto::literal<rule<T> >(rule<T>(u)) {}
};

It works for me in this test. However, I got segfaults in other piece of code using it, which I will have to debug.