I have several vehicles that send a data to server each minute. The server should be listening and decode the data to store in the database. There will be thousands of entries per minute. What is the best approach to solve that problem?
views:
521answers:
3
A:
How long do you expect each operation to take? From what you're saying it seems like you can just write the data straight to the db after processing, so you don't have to synchronize your threads at all (The db should have that taken care of for you).
CookieOfFortune
2009-03-31 20:19:21
I just need to decode the data (fast operation) and save to DB... It can be to only 1000 vehicle or to 10000...
Paul
2009-03-31 20:23:50
It sounds like your transmission rate is pretty slow (GPRS) compared to your processing time, so why even multithread at all? You'll just be spending most of the time waiting for more data. In any case, your reciever should have a buffer already that should be emptied pretty quickly.
CookieOfFortune
2009-03-31 21:18:22
+6
A:
My personal favorite, WCF or WebService farm pumps the data to a Microsoft Message Queue (MSMQ) and have a application server (1 or more) convert the data and put it into the DB.
As you get deeper (if you ever need to), you can use the features of MSMQ to handle timeouts, load buffering, 'dead-letters', server failures, whatever. Consider this article.
On the web facing side of this, because it is stateless and thin you can easily scale out this layer without thinking about complex load balancing. You can use DNS load balancing to start and then move to a better solution when you need it.
As a further note, by using MSMQ, you can also see how far 'behind' the system is by looking at how many messages are in the queue. If that number is near 0, then you good. If that number keeps rising non-stop, you need more performance (add another application server).
JasonRShaver
2009-03-31 20:22:44
Thanks for help... I never heard about that MSMQ ... The performance is good? And Dont I need to implement a multithreading system?
Paul
2009-03-31 20:33:45
This is really what MSMQ was built for. The performance is very good. I have a system using it that the communication overhead of the above is about 25ms (separate machines each) from request to web to MSMQ to application to DB.
JasonRShaver
2009-03-31 20:37:26
As for multithreaded, not required, but can help performance a bit. You can create a object that reads off the MSMQ to do processing and then instantiate a copy on one thread per CPU if you wanted.
JasonRShaver
2009-03-31 20:39:25
I´m reading MSMQ tutorial, and It said:"Message queuing must be installed on both the sending (client) and receiving machine (server)."How can it works that way? My vehile device is a simple hardware with a GPRS device to send the data...
Paul
2009-03-31 20:54:49
Have the device send the data (via WCF, SMS, Web Services, whatever) to server somewhere that puts it into MSMQ. This is also AMAZINGLY safer security wise as your DB can be non-public facing.
JasonRShaver
2009-04-01 02:11:44
This structure is possible?CAR(Gprs device) --> WebService --> MSMQIs it possible to call a webservice function from a device(gprs) ?And to receive I am supposed to create a application server(?) ... Do you have any sample?Thanks
Paul
2009-04-01 20:52:27
Yup, your structure is correct. If you want it to work without needing GPRS, you can use SMS as well. And for the application server, you would just need to listen to the queue, process the data and put it in the database, all inside a loop. As for a sample... I don't really have one of those. =(
JasonRShaver
2009-04-02 17:12:56
Ok thanks! Just one more question...What about if I dont use MSMQ ? Is it really necessary? If I receive the data with my listener(tcp) and put directly on DB Instead of adding in MSMQ? Thanks!
Paul
2009-04-02 17:17:46
Yea, that is fine. the MSMQ is only if the processing takes a long time so the car does not have to wait, but if that process is fast enough, then there is no reason to complicate things =)
JasonRShaver
2009-04-03 02:40:20
+1
A:
We're doing exactly what Jason says, except using a direct TCP/UDP socket listener with a custom payload for higher performance.
RichardM
2009-03-31 23:01:02
Aye, and with UDP you can get performance at an unheard of level if you don't care about missing a call every once in a while. That is why alot of MMORPG games use custom UDP protocols.
JasonRShaver
2009-04-01 02:13:24