Hey, I've built a project over the course of a year. I've put in a lot of hours into it, and it has come out great. A bunch of people use it, and a bunch of people write plugins for it. However, I have since moved on to different languages, different styles, the codebase is dirty and hackish, and I'm not sure I want to continue working in the framework I built it in.
When do you know that you should shut down a project and move on?
When you don't want to work on it anymore. Have you thought about open sourcing it so the community of users could continue to support it if they wanted?
Are you certain you won't gain more valuable experience/knowledge from working on this project? Why not reimplement it in the language/platform of your choice thereby making it a learning experience?
If you are sure you are ready to leave the project, then you can do so, but you should consider your users first. They still find it useful; they have contributed hours of work to write the plugins.
So, your first imperative is to make sure that those who want to use what they find useful can still do so, and can still enhance it. So, ask for a volunteer to take over the project, or a group. Provide them with the source, and your candid assessment of where the flaws are and what needs to be done to rectify them (rewrite).
If you simply close down the project, you will make a lot of people unhappy with you who are currently happy with you - that would be silly. It's easy enough to make enemies without going out of your way to alienate people who will otherwise respect your decision.
Once you have lost inspiration and a project starts to get boring and you are to the point you can't get much or anymore experience from the project it's time to move on. If a project's codebase is too screwed up to fix then it's time to move on. If people use it and people write plugins for it, I'd personally make the project open source and turn it over to the community to continue otherwise just end it.
Find a new maintainer since it's already open source. Ask for a volunteer. If they like it they will surely jump at the opportunity to maintain it.
I would say that if you are going to spend more time rewriting and trying to work around your current framework then starting from scratch with a better design and plan, then it's a good time to move on.
I think it really depends on your motivation for the project though. If it's purely financial then you can do a straight cost analysis. If it was for a hobby or learning experience, you may want to move on even if it would be easier to stay with the current project.