I'm interested in knowing any software engineering techniques developers might use when first starting an Open Source project. I would imagine that conventional processes would not be applicable, as many assume interaction with other developers, which might not be available when a project is first starting out.
What facets of the plethora of software engineering methodologies out there do open source developers use?
The processes don't actually change. You still have use cases, you create an architecture, you design, you have a backlog, you do sprints, you code, you test, you deliver, you do configuration management and quality assurance.
The only difference is that the meetings are shorter. And if you have disagreements, you have to change your dosage.
Perhaps Producing Open Source Software by Karl Fogel may be a good starting point.
Not only does the book talk about the technical aspects of starting and maintaining an open source project, it also goes into what the social aspect of OSS project, such as how to effectively communicate with other people. After all, it may be communication that actually makes or breaks a OSS project.
It's also available for reading online as well.
Honestly, even with a single developer, it's just as important to set up a project website (wiki, bug tracker, roadmap/timeline, homepage) use source control, create design documents, comment and document your code, etc. For one, it's good (I'd go so far as to say "right") programming practice. Secondly, as soon as the project grows to > 1 developer, you'll find yourself having to set up all the same processes and procedures, only now they'll be interrupting your ability to work on new code. Far better to have them in place from the beginning.
Of particular importance is documentation, both of design and implementation. After all, that's how new developers will get started with maintaining and extending your code. The better the docs, the easier it will be for them and the better chance your project has of taking off and attracting a development team. Similarly, using source control and bug trackers will help you keep proper track of your development and give you revision history and rollback capability for mistakes. Further, when new developers come on board, they will benefit greatly from the commit logs in understanding what has been done in the past - and why.
Perhaps you could also claim that there's some "methodology" involved when it comes to bootstrapping an open source project. Most successful small open projects grow according to the snowball effect.
In practice: keep your head cold and don't bother to "open source" until you have at least some assets to share. You'll only be wasting your time on setting up the infrastructure if your source code repository is empty.
The other way around is usually more effective IMHO: start with a repository. Hack away solo and use it to "pre-bake" your project. Then, after a while, go publicize your intents and switch to a public repository.
In the other case, when you can't create anything tangible yet - starting with a blog can help you get accumulate ideas, motivation and attention.