file upload issue

views:

131

answers:

+2 Q:

file upload issue

Hello everyone,

I am implementing a similar scenario like Youtube for my web site -- uploading video and so that all people could watch. My question is, I think we need to encode the video after upload before other people could play remotely, but encode process is very slow (I tried with Microsoft Expression Encoder and also mplayer, encode a 20 mins video takes about 10 mins). To my surprise, even if we upload a very big video to Youtube, seems other people could select and play immediately, looks like encoding process takes almost zero time.

Does anyone have any ideas please? How did Youtube do encode process so fast magically? Or Youtube totally skip the encode process -- but if skip encode process, how could Youtube ensure all video uploaded is playable (i.e. videos follow some parameter, like bit rate, picture size or something)?

thanks in advance, George

+3 A:

Don't forget that YouTube is owned by Google, so having 450,000 servers available may help !

(a slightly facetious, answer, I confess. And obviously YouTube won't have access to all of those. But the point remains that they'll have phenomenal amounts of resource to throw at this issue)

Brian Agnew 2009-04-04 13:20:18

@Brian, even though encode video should take some time. I do not think youtube will divide my 20M video into hundreds of pieces, distribute to hundreds of machines, encode on each individual machine, and combine the altogether?

George2 2009-04-04 13:24:58

(continued) maybe Youtube does not encode at all? I did not find such information. Any ideas or comments?

George2 2009-04-04 13:25:36

I confess I don't know how YouTube would make use of such resources, but Google are experts at the subdivision and parallelisation of work, so I'd be surprised if some parallelisation doesn't occur. Complete guesswork, however

Brian Agnew 2009-04-04 13:27:27

@george2: Why wouldn't they break it up into 100's or 1000's of pieces? That's what multicore, parallel processing is all about -- lots of small concurrent transactions.

S.Lott 2009-04-04 13:28:46

@Brian, could you show me some reference documents about how parallelisation could improve video encode process please? I tried mplayer and expression encoder, and I watched my two CPUs are both almost 100% occupied, I think parallelisation is already used, correct? (continue)

George2 2009-04-04 13:33:42

(continued) so I want to know where is the room or idea to improve?

George2 2009-04-04 13:34:17

@S.Lott, I tried mplayer and expression encoder, and I watched my two CPUs are both almost 100% occupied, I think multi-core is already used to maximum, correct? I want to know where is the room or idea to improve, do you have any good ideas?

George2 2009-04-04 13:34:50

Sorry - I can't advise further. I know very little in this area. I would advise a lot of Googling, or hopefully someone here can offer more insights. Best of luck!

Brian Agnew 2009-04-04 13:41:55

@Brian, it is ok. Let us see if others have good advice. It is appreciated if you could share some points in the future when you have good ideas. :-)

George2 2009-04-04 14:04:30

+1 A:

Before doing anything else, get a better encoder. There are lots of encoders available. Some are open source, some are proprietary. Apple has some very, very nice encoders that are fast, but expensive.

Look at http://www.ffmpegx.com/ for the state of the art in open-source encoder technology. They integrate a diverse set of technologies.

S.Lott 2009-04-04 13:30:33

@S.Lott, I have tried with ffmpegx, I think mplayer is using it internally. On my computer encode 20 mins video takes 10 mins. So, I think Google is not using something similar. Are you sure whether Youtube will encode video or not encode at all (just play what is uploaded)?

George2 2009-04-04 14:03:41

+1 A:

Also, hardware encoder probably is used by youtube.

And, is it possible playing while encoding only if encode is fast than play it.

arsane 2009-04-04 14:09:51

@arsane, 1. any reference to H/W encoder which is faster? 2. "is it possible playing while encoding only if encode is fast then play it" -- I have no experience, could you show me some reference which shows how to play a video while it is still in encoding process?

George2 2009-04-04 14:15:07

(continued) I also want to learn some guidelines about when we should encode and when we should not encode. Actually, I do not see any benefits of encoding -- without encoding, the uploaded video could still be played. :-)

George2 2009-04-04 14:15:58

For the H/W encoder, you can have a google of it. But I have no idea which H/W encoder is the best.like:http://www.streetwise.com.au/product_info.php?products_id=7061 can encode to youtube vedio.For the second play/encode way, I'm not an encoding expert, but I think it should be possible.

arsane 2009-04-04 14:21:30

I think encoding benifits following things:1. bandwidth saving.2. client browser support.(I can only play flash vedio in firefox under linux)...

arsane 2009-04-04 14:31:07

@arsane, I studied the encoder H/W, I am very interested how fast it is while comparing with software based encoder. But I did not find any number. Do you have any documents or have such number? Thanks.

George2 2009-04-05 12:21:57

@arsane, for the encoding while play scenario, it is appreciated if you could share some documents or codes when you have found good materials about this topic, I am very interested in this solution and thinks it is very cool.

George2 2009-04-05 12:22:54

+1 A:

How did Youtube do encode process so fast magically?

Without any particular insider knowledge, I would expect (1) chopping the input video into shorter segments to send to multiple encoders, and (2) encoding the stream at the same time as you are uploading it, rather than waiting until you've uploaded the entire file to start.

Or Youtube totally skip the encode process

No, everything must be converted to FLV for playback.

bobince 2009-04-04 15:14:11

@bobince, 1. are there any reference code or documents about how to implement encoding at the same time when uploading? I am very interested to learn more about this cool idea.

George2 2009-04-05 12:16:40

(continue) 2. do you have any documents or samples about how to divide video into small parts, encode them and re-assemble them? I am very confused about how to divide and how tore-assemble, because I think we have to parse and understand internal video format in order to divide/reassembly?

George2 2009-04-05 12:18:08

You'd have to handle request body parsing yourself instead of leaving it up to the form parser in whatever higher-level framework/language you're using. Then you'd use something like ffmpeg's libavformat to demux the streams and split on independent packets where the format allows it.

bobince 2009-04-05 14:34:28

@bobince, are you answering my question No. 1 or question No. 2?

George2 2009-04-07 09:31:37

(2). I'm not aware of any public code for (1), but it would be highly platform-dependent. You'd have to circumvent the usual form processing handler to get access to the file upload in progress.

bobince 2009-04-07 09:47:16

ansaurus

tags:

views:

answers:

file upload issue

related questions