Choosing the Right IP Cores for Low-Latency Video Streaming

50 %
50 %
Information about Choosing the Right IP Cores for Low-Latency Video Streaming

Published on December 13, 2016

Author: castcores

Source: slideshare.net

1. Choosing the Right IP Cores for Low-Latency Video Streaming Nikos Zervas CEO CAST, Inc. December 2016 www.cast-inc.com

2. Slide 2 Low Latency Video Streaming IP Introduction   Low latency is important when there is real-time interaction with the video content   Gaming, Video Conferencing, Automotive, Industrial, Medical, Defense   How low is low latency?   For human interaction sub-100ms is very low latency   Machine interaction may require even less

3. Slide 3 Low Latency Video Streaming IP Introduction   End-to-end or glass-to-glass latency is the amount of time between the instant a frame is captured to the instant the same frame is displayed   Latency is introduced by temporal storage (buffering)   Latency is measured in time or buffering (storage) units

4. Slide 4 Low Latency Video Streaming IP Latency comes from buffers Mission: Eliminate Excessive Buffering   Buffers are hidden within IP cores, and IP cores may impose external buffers Focus on what really counts:   Pixel buffers do not really count   2000 pixels: 0.03 msec for 1080p30   Video Line buffers count   100 lines: 3.01 msec for 1080p30   Frame buffers count even more   1 frame: 33.33 msec for 30fps One frame 33.33 ms One line 30.86 μs One pixel 16.08 ns Latency For 1080p30 Video

5. Slide 5 Low Latency Video Streaming IP Choosing the Right IP Core Throughput is irrelevant: Two cores with same throughput may have totally different latency Input/output data-ordering may impose external buffers for reordering Choose “line-based” vs “frame-cased” image processing algorithms Some cores do not produce any output before they process a full frame, and some do not start processing unless a full frame is available at their input. TIP

6. Slide 6 Low Latency Video Streaming IP Glass-to-Glass Latency Why do I even need a BRAB and a DSB? Processing Stage Buffering Latency (for 1080p30) Capture post-processing (e.g. bayer filter, 3A, ISP) Few lines (e.g.32) < 1 msec Video Compression (JPEG, MPEG-1/2/4 or H.264) 8 or 16 lines for conversion from raster scan Few thousand of pixels on the encoder pipeline 0.5 msec <0.05 msec Bit-Rate Averaging Buffer A number of frames (e.g. more than 30) to sub-frame (e.g. 1/2 frame) from 1 sec to 16 msec Network Processing (e.g. RTP/UDP/IP encapsulation) Few Kbytes <1 msec Decoder Stream Buffer A number of frames (e.g. more than 30) to sub-frame (e.g. 1/2 frame) from 1 sec to 16 msec Video Decompression (JPEG, MPEG-1/2/4 or H.264) 8 or 16 lines for conversion from raster scan Few thousand of pixels on the decoder pipeline 0.5 msec <0.05 msec Display pre-processing (e.g. Scaling, Chroma Resampling) Few lines (e.g. 32) <1 msec

7. Slide 7 Low Latency Video Streaming IP Bit-Rate Control and Latency   Bit rate is not natively constant nor predictable   Bit rate control averages the bit rate to a constant value 0 5 10 15 20 25 0 100 200 300 400 500 600 700 800 900 1000 Mbps &me (msec) 0 5 10 15 20 25 0 100 200 300 400 500 600 700 800 900 1000 Mbps &me (msec) 1st Averaging Period 2nd Averaging Period 3rd Averaging Period

8. Slide 8 Low Latency Video Streaming IP BRAB & DSB ! ! ! !   Bit-Rate Averaging Buffer (BRAB) on the encoder side “normalizes” encoder bit rate   Can be reduced to a zero-latency FIFO, if channel can “absorb” bit rate picks   Decoder Stream Buffer (DSB) buffers one averaging period worth of stream, before decoding is initiated, to avoid video playback interruption BRAB

9. Slide 9 Low Latency Video Streaming IP BRAB & DSB ! ! ! !ü  BRAB Latency = DSB Latency = Averaging Period BRAB 0 5 10 15 20 25 0 100 200 300 400 500 600 700 800 900 1000 Mbps &me (msec) 1st Averaging Period 2nd Averaging Period 3rd Averaging Period

10. Slide 10 Low Latency Video Streaming IP Rate Control is What Matters   All rate control algorithms adjust the compression-level based on the remaining bit rate budget in the averaging period (=buffer occupancy) ü  Accurate prediction of future content’s compressed size is infeasible, but some algorithms make better predictions by:   Assessing the “complexity” of the remaining content   Getting “trained” based on history ü  The frequency of compression-level adjustments varies §  More frequent adjustment allows to quickly adopt to content changes and correct inaccuracies of predictions

11. Slide 11 Low Latency Video Streaming IP The Role of Rate Control Rate Control defines Latency but also Quality   Rate Control is not described in standards Same compression algorithm, bit rate and averaging period– Different Encoders

12. Slide 12 Low Latency Video Streaming IP Choosing the Right IP Core Focus on encoder’s rate control attributes, and evaluate latency and quality (IP core vendor should be able to help you) Test streams do not always stress rate control. Rate control algorithms may overflow or underflow BRAB & DSB, unless they are able to regulate compression levels multiple times within the averaging period Regardless of rate control attributes, software codecs are not suitable for ultra-low latency TIP

13. Slide 13 Low Latency Video Streaming IP Choosing the Right H.264 IP Core Choose an H.264 encoder IP core supporting Intra-Refresh Intra-Refresh allows construction of an intra-coded reference over a number of frames. It eliminates I-frames and the related bit-rate “peaks” TIP 0 5 10 15 20 25 30 35 0 200 400 600 800 1000 Mbps &me (msec) 0 5 10 15 20 25 30 35 0 200 400 600 800 1000 Mbps &me (msec) IPP coding Intra-Refresh coding

14. Slide 14 Low Latency Video Streaming IP JPEG Choosing the Right JPEG IP Core A JPEG encoder IP core is not suitable for sub- frame latencies, because the JPEG standard only allows Quantization changes at frame boundaries Some JPEG encoders can regulate bit-rate also by truncating code-bits from Huffman-coded blocks. The stream remains complaint, and allows intra- frame bit-rate “corrections”

15. Slide 15 Low Latency Video Streaming IP Compression Cores from CAST PLUS: ü  Hardware RTP, UDP/IP stacks ü  Complete Video-Over-IP plaTorms ü  ExperVse to help you architect your system and select the right algorithms Compression Standard Suggested Min Averaging Period Compression Levels Motion JPEG 1/4 Frame Moderate Intra-only H.264 1/4 Frame High H.264 w Intra Refresh 1/4 Frame Very High H.264 w/o Intra Refresh 3-4 Framse Very High

Add a comment