Is it possible for a router to obtain the MAC address of nearby devices that have not connected to it? I want to write an app that notifies me somehow (email or something) when a specific MAC address comes within range of a router that I have (the app is much more specific than that so I cannot use prebuilt tools but if they are open source they could help). The router will be connected to the internet and the device is an iPhone so it is constantly looking for WiFi access points so I feel like all the ingredients are there but I am not sure if what I am asking is possible. If it is possible (even if only by rewriting the router OS) what will it take? C is as low level as I am hoping to go...
Depends on the router. There are routers out there which have completely open operating systems, such as the Linksys WRT54GL, for which the OpenWRT, DD-WRT and Tomato operating systems exist. You could (for example) develop a simple bash script to interpret the log files and email you.
You experience with other routers may vary. Many have log files which are accessible via http or telnet, so you may be able to interpret this externally even if you can't run your own code on the router.
I am guessing that by router you actually mean a wifi access point (which may be integrated into a home router) and you are talking about wifi connections. Then no, it is not possible. The access point only knows the MAC address when the device at least tried to connect to its network. The list of nearby networks that the device shows however does not require any form of connection to any of the networks. Instead, they listen for the beacons that the access points send out.
A Linux-based router that you can program, which usually means one that runs OpenWRT or DD-WRT can do this; you can put the router in monitor mode, and have tcpdump or tshark listen for that particular MAC address. If you want, you could write your own capture application that would use libpcap to listen. If you can live with only scanning the active channel, the router could be providing an access point at the same time.
You could also listen to the logs of a standard router, but the issue there is that the phone would have to at least attempt association before the router would log anything; with a linux router, you can just listen for that MAC address emitting any packet at all.
My favourite Linux-based routers come from Ubiquiti, because their RouterStation Pro is built to run OpenWRT, so there is no hacking and complete hardware documentation.