Cross-Origin Track

Can somebody please explain to me why <script>, <img>, <audio>, and <video> will happily display contents cross-origin, but <track> is freaking sacred and needs to follow much stricter rules?

This is madness!

If I can play a video from server X, why can’t I display captions from that server as well?