Multiple H.264 video streams in one RTP session

Question

I would like to dynamically switch the video source in a streaming video application. However, the different video sources have unique image dimensions. I can generate individual SDP files for each video source, but I would like to combine them into a single SDP file so that the viewing client could automatically resize the display window as the video source changed. Here are two example SDP files:

640x480.sdp:

v=0
o=VideoServer 305419896 9876543210 IN IP4 192.168.0.2
s=VideoStream640x480
t=0 0
c=IN IP4 192.168.0.2
m=video 8000/2 RTP/AVP 96
a=rtpmap:96 H264/90000
a=fmtp:96 packetization-mode=0; profile-level-id=4D4033; sprop-parameter-sets=Z01AM5ZkBQHtCAAAAwAIAAADAYR4wZU=,aO48gJ==
a=control:trackID=1

960x480.sdp:

v=0
o=VideoServer 305419896 9876543210 IN IP4 192.168.0.2
s=VideoStream960x480
t=0 0
c=IN IP4 192.168.0.2
m=video 8000/2 RTP/AVP 96
a=rtpmap:96 H264/90000
a=fmtp:96 packetization-mode=0; profile-level-id=4D4033; sprop-parameter-sets=J01AM5WwPA9sBAIA,KO4G8gA=
a=control:trackID=1

How can these individual files be combined into a single SDP file?

Answer 1

I've gone over the RFC (RFC2327 - SDP: Session Description Protocol) and it appears you can just concatenate the two SDP documents. The document states explicitly:

When SDP is conveyed by SAP, only one session description is allowed per packet. When SDP is conveyed by other means, many SDP session descriptions may be concatenated together (the `v=' line indicating the start of a session description terminates the previous description).

Answer 2

I think it depends on your decoder. If it supports parameters change inside the stream, then if you can tell the encoder to put the corresponding header when changing resolution, your decoder should automatically switch.

What is your question exactly ? Is it : How can I change resolution without stopping / restarting the stream ?

I don't Think you can tell in advance to a decoder, here are the potential resolution that you will see with some sdp magic. Either your decoder is able to understand H264 parameter change, and then you are fine, or you have to stop restart the whole thing, and then dynamic sdp is sufficient.

I know that for example, VLC is able to detect MP4 encoding change (for example moving from variable bit rate to constant bit rate), but will crash if you change resolution The only thing you can do with sdp is to combine different media description, for example with different dynamic payload type and different control-id attribute.

Answer 3

I agree with last post, but since I have no reputation, I'll pass along my suggestion:

Video size information should be conveyed in the headers of h.264.

Ensure that your video includes the video size with each Intra frame. Ensure the decoder/client application checks the stream for the video size and adjusts. Ensure that the video switch is made on Intra frame boundaries.

Answer 4

The parameters in your two sdp examples are very close - the stream name and the sprop-parameter-sets differ. I assume you don't care about the stream name. If you need separate sprop-parameter-sets and the clients support the standard well you can use separate dynamic payload types for each resolution and have a single SDP as follows:

    v=0
    o=VideoServer 305419896 9876543210 IN IP4 192.168.0.2
    s=VideoStream640x480
    t=0 0
    c=IN IP4 192.168.0.2
    m=video 8000/2 RTP/AVP 96 97
    a=rtpmap:96 H264/90000
    a=fmtp:96 packetization-mode=0; profile-level-id=4D4033; sprop-parameter-sets=Z01AM5ZkBQHtCAAAAwAIAAADAYR4wZU=,aO48gJ==
    a=rtpmap:97 H264/90000
    a=fmtp:97 packetization-mode=0; profile-level-id=4D4033; sprop-parameter-sets=J01AM5WwPA9sBAIA,KO4G8gA=
    a=control:trackID=1

Similar to other answers if you don't actually need the different stream names or the different sprop-parameter-sets you should be able to use your first SDP and switch format mid stream. I don't know the actual payload of H.264 or your particular decoder well enough to ensure that this will work in your applications but it is very common in videoconferencing applications to allow dynamically switching between resolutions without signaling a change or requiring a separate dynamic payload type.

Although you can concatenate two SDP documents as mentioned in another answer I don't think it will help in this case. H.264 decoders can only work with a single sprop-parameter-sets parameter at a time I believe. Since both SDPs would have the same payload type, source port, etc. the receiver would not know when to use which sprop-parameter-sets parameter. UPDATE: Note some implementations get their sprops inband and not from the SDP (or only initially from the SDP). If that applies in your environment the SDP sprop-parameter-sets can be updated inband

References:

RFC 3984 RTP Payload Format for H.264 Video RFC 4566 SDP: Session Description Protocol

[Sorry for not giving the full cite - feel free to correct]

Answer 5

You can either do the dynamic payload change or the in-stream parameter set change, or SIP re-INVITE.

Payload changes have a problem that if you don't control the encoder and decoder you need to make sure the other end accepts both payloads, and that they'll switch payloads correctly (and fast enough for you - there's no requirement on that).

in-stream changes have a problem if the parameter-set packets are lost. You can use a different set of parameter sets (switch from parameter-set 1 to 2 when you change) to avoid mis-decoding - if the sets are lost, you should just get a frozen or blank picture. I'd advise retransmitting them a few times (not in too-quick succession).

SIP re-INVITE is out-of-band and handshaked, and thus safe, but adds delay to any switch and may glitch depending on the receiver, and could be rejected.

(Note: I'm an author of RFC 3984bis, the update to RFC 3984)