2016 Streaming Media West - How to improve video quality of Service with an Effective AB testing Workflow

100 %
0 %
Information about 2016 Streaming Media West - How to improve video quality of Service with...

Published on November 1, 2016

Author: EricaBeavers

Source: slideshare.net

1. How to improve quality of service with player-side iterations and an effective AB testing workflow Streaming Media West – Track D Tuesday, November 1, 2016 2:45 to 3:30 pm

2. ● Creators of peer-accelerated video delivery for large-scale OTT ● Pioneers in hybrid video delivery systems to accompany exponential growth ● Experts in HTML5 video through integrations into open-source & proprietary players Infinite scale, limitless delivery. Streamroot: Who are we?

3. I. How to build an AB Testing workflow When and what to test, analytics pipeline, AB testing setup II. Dailymotion use case: ABR iterations for quick, dramatic QoS improvements ABR iterations to hls.js to cut rebuffering III. Streamroot use case: using effective AB testing to improve our product Streamroot workflow and example Presentation Outline Infinite scale, limitless delivery.

4. Infinite scale, limitless delivery. AB Testing is Data-Driven Development. …here applied to Media engines & QoS. I. Building an effective AB testing workflow

5. Infinite scale, limitless delivery. I. Building an effective AB testing workflow AB testing makes sense when: You can’t test locally: distributed and unmanaged networks (ABR, P2P). Test subjective hypothesis, magic numbers. You deploy a completely new feature that breaks your old logic.

6. Infinite scale, limitless delivery. I. Building an effective AB testing workflow Before starting: Define your metrics Determine an appropriate sample size and testing timeframe Define your variables Define your populations

7. Infinite scale, limitless delivery. For testing, you need: The right data A DB to store it A BI/analytics tool I. Building an effective AB testing workflow

8. Infinite scale, limitless delivery. I. How to build: Data pipeline General workflow

9. Infinite scale, limitless delivery. Streamroot workflow I. How to build: Data Pipeline General workflow

10. Many options based on your requirements: open-source, proprietary, Cloud... Infinite scale, limitless delivery. I. How to build: Data Pipeline What to chose from Message broker / stream processor Storage + possible query/visualization BI/viz tools Advanced analytics Kafka (Kinesis) Druid Caravel R RabbitMQ InfluxDB Grafana Matlab ActiveMQ Cassandra (DynamoDB) Tableau PostGresSQL QlikView ElasticSearch Kibana

11. Infinite scale, limitless delivery. I. How to build: AB Testing workflow General setup

12. Client AB testing methods: 1) Dynamic Config Injection 2) Different builds with a reverse proxy Infinite scale, limitless delivery. I. How to build: AB Testing workflow General setup

13. Infinite scale, limitless delivery. I. How to build: AB Testing workflow Config injection

14. Infinite scale, limitless delivery. I. How to build: AB Testing workflow Config injection

15. Infinite scale, limitless delivery. I. How to build: AB Testing workflow Dynamic config injection PROs Simple & easy splits Separate - fast to deploy & rollback Scalable CONs Limited scope Complications in code Need to handle scaling & high availability WARNING Async implementation!

16. Infinite scale, limitless delivery. I. How to build: AB Testing workflow Different builds with reverse proxy

17. Infinite scale, limitless delivery. PROs Serve very different (major) versions of player Perfect for rolling out new features smoothly (10% then 30% then...) CONs Good CI/CD necessary to roll out new versions quickly Heavy stack to maintain and scale WARNING Make sure the file request is not slower! Single point of failure! I. How to build: AB Testing workflow Different builds with reverse proxy

18. 1. AB test when it makes sense. 1. Choose the right metrics. 1. Build a pipeline with tools that are adapted to your use case. 1. Use the right method depending on what you’re testing. Infinite scale, limitless delivery. I. How to build: AB Testing workflow Tips & sum up

19. Infinite scale, limitless delivery. II. Dailymotion Some numbers: world’s 2nd largest video hosting platform 100 million video views each day 63% Desktop 31% Mobile 8% Tablet 1% SmartTV 69% HTML5 31% Flash

20. Infinite scale, limitless delivery. II. Dailymotion ABR iterations for quick, dramatic QoS improvements I should have rehearsed this more… When’s lunch?

21. Adaptive bitrate algorithms can have a huge impact on QoS. They are easy to tweak, but improvements depend on your specific use case. One size does not fit all. Infinite scale, limitless delivery. II. Dailymotion ABR iterations for quick, dramatic QoS improvements

22. Infinite scale, limitless delivery. II. Dailymotion Improving hls.js’s ABR logic

23. Infinite scale, limitless delivery. II. Dailymotion Improving hls.js’s ABR logic Introduce history parameter to bandwidth estimation.

24. Infinite scale, limitless delivery. II. Dailymotion Improving hls.js’s ABR logic Average out the values.

25. https://github.com/dailymotion/hls.js/blob/master/src/controller/abr-controller.js EWMA: Infinite scale, limitless delivery. II. Dailymotion Improving hls.js’s ABR logic Shaka Player:

26. ABR now based on two bandwidth moving averages: • Fast → adapt down quickly • Slow → adapt up more slowly Bandwidth estimation = min(fast, slow) Infinite scale, limitless delivery. II. Dailymotion Improving hls.js’s ABR logic

27. Control over media engine Control over events QoS events AB testing & customization ABR algos, config, etc. Infinite scale, limitless delivery. CDN Director Analytics aggregation and visualization II. Dailymotion AB testing workflow: total control over their stack

28. 67.4% native 70.3% hls.js 78.6% flashls Infinite scale, limitless delivery. II. Dailymotion Comparing the results - live Rebuffering, percentile rank

29. native flashls hls.js,s=9,f=3hls.js,s=0,f=0hls.js,s=6,f=1 Infinite scale, limitless delivery. II. Dailymotion Comparing the results - live Rebuffering, percentile rank

30. hls.js, s=0, f=0 hls.js, s=9, f=3 Infinite scale, limitless delivery. II. Dailymotion Number of level switches - live

31. With the right workflow, big improvement in less than 2 weeks of testing. Numbers are use-case specific (segment length, network, etc.). Lots of other options to test → a continuous effort to improve QoS! Infinite scale, limitless delivery. II. Dailymotion Conclusions

32. Infinite scale, limitless delivery. III. Streamroot Peer-accelerated delivery

33. ...We have a very distributed network! AB testing on: - matching devices to get the best connections - Transfer algorithms and protocol improvements - QoS impact and improvements Infinite scale, limitless delivery. III. Streamroot Why are we using AB testing

34. Infinite scale, limitless delivery. III. Streamroot The pipeline GET build-a.js Pipeline AB testing File Distribution

35. AB Test on AskSegment Infinite scale, limitless delivery. III. Streamroot The test case

36. Infinite scale, limitless delivery. III. Streamroot Results AskSegment Timeout Samples P2P % Longer 15,772,542 67.41% Shorter 16,152,624 67.43% No effect?

37. Infinite scale, limitless delivery. III. Streamroot Results AskSegment Timeout Type Samples P2P % Longer Live 9,004,140 66.43% Shorter Live 9,340,504 68.02% Longer VOD 6,768,403 68.72% Shorter VOD 6,812,120 66.61% Live: +1.59% for shorter timeout VOD: +2.11% for longer timeout

38. - Understand and know your data - Ab testing is sometimes the only way to prove your hypothesis - You will be wrong sometimes: Make sure you can rollback quickly!! - The most difficult is to start! Once it is set up: be relentless! Continuous AB testing FTW Infinite scale, limitless delivery. III. Streamroot Takeaways

39. Infinite scale, limitless delivery. Questions?

40. Nikolay Rodionov, Co-Founder and CPO, nikolay@streamroot.io Erica Beavers, Head of Partnerships, erica@streamroot.io Infinite scale, limitless delivery. Contact us!

Add a comment