- Author:
Hongzi Mao,Mohammad Alizadeh(MIT Mohammad lab) bitrate adaptation, video streaming, reinforcement learning
摘要：
客户端视频播放器通常会应用adaptive bitrate(ABR) algorithms(自适应码率调整算法 来优化用户的quality of experience(QoE)(用户体验)。 尽管有很多算法被提出，state-of-the-art算法们的局限性仍然存在: 它们只是基于简化的或者不准确的环境模型来设定固定的控制规则。结果就是当网络环境变化或者是QoE的目标发生变化时，现有的策略往往达不到理想的效果。为了解决这些问题，作者提出了Pensieve，一种通过强化学习来自动生成ABR算法的系统。通过学习到的算法为将来的video chunk选择合适bitrate，从而优化QoE metrics。Pensieve训练的神经网络模型根据客户端视频播放器提供的观测数据作出预测。 实验表明，Pensieve能适应各种各样的环境以及不同的QoE metrics，取得了比state-of-the-art scheme高12%-25%的平均QoE。此外，Pensieve也很好地适应了未曾遇到过的网络情况，表明它generalizes well.
引入：
相关工作有ABR算法，它根据观察到的网络的带宽和playback的buffer的占据量，来做出bitrate的决定。它的目标是最大化QoE矩阵。然而选择正确的bitrate是比较困难的，基于以下四种情况：
- 网络的带宽经常变动
- QoE需求(high bitrate，minimal rebuffering,smoothness)
- bitrate决定的级联效应
- ABR决定的coarse-grained特性
ABR算法通常有两种做决策的手段——
1。网络带宽(基于速率的算法);
2。playback buffersize(基于buffer的机制);
另一个state-of-art算法MPC，解决QoE最优问题，做决策基于horizon的未来块。因为是直接最优化”QoE”,所以效果优于fixed heuristics的方法，然而，MPC的好坏依赖于动态系统的精确建模。MPC对于未来网络带宽的错误预测十分敏感。
在我们这篇PAPER中提出的是Pensieve,自动学习ABR算法，并且使用现代化的RL技术学习控制策略。Reward和QoE指标息息相关，用的是A3C算法。Pensieve使用的模拟是大型网络跟踪语料库。
Pensieve对于未见过的网络情况和视频属性也能达到比较好的效果。
专有名词：
Adaptive bitrate streamingis a technique used in streaming multimedia over computer networks. While in the past most video or audio streaming technologies utilized streaming protocols such as RTP with RTSP, today’s adaptive streaming technologies are almost exclusively based on HTTP[1] and designed to work efficiently over large distributed HTTP networks such as the Internet.It works by detecting a user’s bandwidth and CPU capacity in real time and adjusting the quality of the media stream accordingly. [2] It requires the use of an encoder which can encode a single source media (video or audio) at multiple bit rates. The player client[3] switches between streaming the different encodings depending on available resources.[4] “The result: very little buffering, fast start time and a good experience for both high-end and low-end connections.”[5]
More specifically, and as the implementations in use today are, adaptive bitrate streaming is a method of video streaming over HTTP where the source content is encoded at multiple bit rates, then each of the different bit rate streams are segmented into small multi-second parts. The streaming client is made aware of the available streams at differing bit rates, and segments of the streams by a manifest file. When starting, the client requests the segments from the lowest bit rate stream. If the client finds the download speed is greater than the bit rate of the segment downloaded, then it will request the next higher bit rate segments. Later, if the client finds the download speed for a segment is lower than the bit rate for the segment, and therefore the network throughput has deteriorated, then it will request a lower bit rate segment. The segment size can vary depending on the particular implementation, but they are typically between two (2) and ten (10) seconds.