Design for wire between server and access point and RF to endpoint device

Answer 1

There are small TCP/IP stacks, for example LwIP.

Answer 2

You didn't mention the amount of data to be communicated, or bandwidth considerations?

A 3rd party TCPIP stack targeted at the 8051 would simplify all the networking issues with connecting 100 units. You probably will still end up with a proprietary protocol that sits on top of the tcpip stack but then it is just simple point-to-point communication between the server and each end point.

Answer 3

I find the diagram confusing:

If this isn't its own network and the server to AP link is running on your internal company network, there isn't really an option, there must be a TCP/IP stack on the AP.

If this is its own isolated network then what is the router for?

If this is, in fact, its own isolated network then you are right, there really isn't a need for the Ethernet connectivity at all. The overhead you will see on the wireless is huge, your no overhead ideal data rate is 250kbit/sec, running ZigBee on 802.15.4 @ 2.4ghz point to point your real data throughout is usually around 20kbit/sec. A custom protocol should be able to obtain lower overhead but this would need to be defined.

If I were designing this I would choose a SoC for the AP that had on board 802.15.4 and CAN (Controller Area Network). Depending on size and data rate just get a PCI CAN card for the server and connect it up, use something like DeviceNet as your protocol layer for server to AP communications. This can be expanded by using CAN switches and repeaters. CAN is used all the time in industrial automation, a little googling can find you example of tens of thousands of nodes used in some manufacturing plants.