This specification defines the timing object. The timing object is a local object that may be used by Web clients to ensure precisely timed operation as well as flexible timing control. If multiple timing-sensitive components take direction from the same timing object, their behaviour will be precisely aligned in time (synchronized). Crucially, this is also the case in distributed settings. A central motivation for the timing object is that it may be connected to an online timing resource. This way, the local timing object is a gateway to precisely timed operations, both in single-device and multi-device scenarios.

This timing object is not to be confused with concepts proposed by Navigation Timing, an initiative targeting precise measurement of time consumption in Web browsers.

The specification is intended for discussion within the Multi-Device Timing Community Group. Its content does not yet represent the consensus of the Community Group.

This specification is incomplete. Some procedures are either missing or described as similar to procedures defined in [[!HTML5]]. The main goal of this draft is to propose a technical solution, show how it could be integrated in [[!HTML5]] and gather feedback from interested parties before dwelving into details.

Introduction

Timing mechanisms allow operations to be executed at the correct time. The Web already has several mechanisms supporting timed operations, including setTimeout, setInterval, as well as media frameworks and animation frameworks for timed media presentation. Unfortunately, these mechanisms are limited in scope as they only apply to timed operations within a single web page. The W3C Multi-device Timing Community Group has been formed with the goal of extending the Web with native support for precisely timed operation across Web pages hosted by different devices, as well as providing a unifying timing model to which all frameworks for timed operation may integrate. The timing object is the central concept in this initiative.

Synchronized playback of multi-device linear media is an important use-case for multi-device timing. However, multi-device timing has wide utility for a variety of timing related challenges, including distributed media control and remote control, distributed synchronization, distributed time-shifting, distributed time-ordering and distributed capture/recording (time-stamping). This way, multi-device timing has applicability in many application domains, including TV and radio (broadcast and IP-based), secondary device applications, online education, VoD services, social and collaborative media, data visualization, music etc.

For a more detailed list of use cases that this specification enables, see .

Linear composition

Linear composition or temporal composition is simply the idea that complex linear media can be built from simpler, independent linear components. Web support for linear composition would imply that classical benefits of composition, i.e. flexibility, reusability, extensibility and mashup-ability, apply to the construction of Web-based linear media. For example, in the single-device scenario, imagine a linear presentation made from HTML5 video, some timed meta-information, a SMIL component, a Web Animation, a map with timed georeferenced data, and a timed Twitter widget. Or, in the multi-device scenario, imagine the same components distributed or duplicated across multiple devices in a room or within a social group. In both cases linear composition requires temporal interoperability among heterogeneous linear components, available through precisely coordinated, timed playback.

Unfortunately, Web support for linear composition is not optimal. Each media framework tends to define its own custom timing control mechanisms, making it difficult to achieve linear composition in practice. The central purpose of the timing object is thus to simplify interoperability by providing a common basis for timed operation and control in linear media.

For a more detailed introduction to linear composition on the Web, see [[LINEARCOMPOSITION]].

Design goals

The design of the timing object has two main goals:

  1. provide a unifying API for timed operation and temporal control that timed components can integrate with. This would include timing sensitive Web applications in general, as well as existing frameworks for timed media — such as media elements in [[HTML5]], Web Animations [[WEB-ANIMATIONS-1]], Web Audio API [[WEBAUDIO]] and SMIL Timing [[SMIL3]]. The API must support highly precise timing and be expressive enough to support control primitives appropriate for a wide range of media types and applications.
  2. encapsulate complexity of distributed time synchronization, thereby allowing integrating frameworks to be included in precisely timed multi-device operation. In particular, by providing the same API for local and online timing resources, timed components may be used in single-device and multi-device scenarios, without modification.

Fig. 1 illustrates these functions, including an online service to provide multi-device timing. In this example, a timing object is instantiated on each of the three devices, and each instance is connected to a single, shared online timing resource. On each device, the timing object acts as the director for independent, timing-sensitive User Interface (UI) components. This way, a video will aim to present video frames in accordance with the timing resource. Similarly, a timing-sensitive Twitter widget might replay timestamped Tweets. If the timing object pauses, all the connected components are notified and react accordingly. Note that the different UI components may go about their business in complete isolation. There is no need for components to communicate, except indirectly through the timing object. This loose coupling is the key to linear composability.

The image illustrates the concept of an online timing resource how it used to synchronize HTTPTimingObjects across three different Web clients.
Timing objects on three devices connected to the same online timing resource

Furthermore, when timing objects are connected to an online timing resource, they merely act as local representatives. For example, when requested to pause, the timing object will simply forward the request to the online timing resource. As the online object pauses, notifications will be multicast to all connected clients. The timing object will only update its internal state and notify connected UI components when it receives this notification. For online timing objects, clients can expect update latency of about one round trip time (RTT), whereas local timing resources should have no update latency. Apart from this, online and local timing resources should be hard to distinguish. Note also that the argument made for the independence of timing-sensitive components in the single-device scenario now applies to the multi-device scenario as well.

This specification describes the integration of timing objects with HTML5 Media elements and text tracks. Integration with other frameworks for timed operation, such as Web Animation, SMIL, and WebAudio will be covered separately.

The timing object

The timing object is a very simple object, essentially an advanced stop watch (Fig. 2). If started, its value changes predictably in time, until at some point later, it is paused, or perhaps reset. It may be queried for its value at any time. For example, it should take exactly 2.0 seconds for the value to advance from 3.0 to 5.0 when the velocity is 1.0. Such deterministic behavior is required for reliable distributed synchronization. In terms of implementation, the timing object is a fairly thin wrapping around a monotonic system clock (integration with online timing resources adds complexity). The precision of the timing object is limited by the underlying system clock.

The timing object is essentially an advanced stop watch.
The timing object is essentially an advanced stop watch.

The timing object is more expressive than a traditional stop watch. It supports any velocity or acceleration, and may jump (instantaneously) to any position on the timeline (Fig. 3). In fact, a timing object essentially implements linear motion along a unidimensional axis. State vectors [[MSV]], based on the classical equations of linear motion under constant acceleration, are used to represent that motion. At any point in time, position, velocity or acceleration may be requested to change. Querying the timing object reveals not only its current position but also its velocity and acceleration at that moment. The expressiveness of this model implies that a wide variety of control primitives can be supported. For example, discrete jumps on the timeline may be used to control a slide show, or velocity may correspond to playbackRate for the control of continuous media. Acceleration may be for instance be required by animation frameworks.

The timing object is essentially and advanced stop watch.
The timing object visualized as a cursor moving along a time-axis.

Programming with timing objects

Timing objects are resources used by a Web application, and the programmer may define as many as required. What purposes they serve in the application is up to the programmer. If the application needs a shared, multi-device clock, simply starting a timing object (and never stopping it) might be sufficient. If the clock value should represent milliseconds, set the velocity to 1000 (advances the timing object with 1000 milliseconds per second). If the timing object represents media offset, specify the playback position, the velocity, and perhaps a media duration. For videos where offset is measured in seconds or frames, set the velocity accordingly. Or, for musical applications it may be practical to let the timing object represent beats per second. Note also that the timing object may represent time-changes with any kind of floating-point variable. For example, if you have data that is organized according to, say height above sea level, you may want to animate how this data changes as you move vertically. In this case the timing object could represent meters or feet above sea level, and positive and negative velocities would allow you to move gradually both upwards and downwards.

In general, the timing object is particularly useful when you have a variable that needs to change predictably in time.

Connecting the timing object

The proposed timing model is one where timed components are connected to timing objects, thereby accepting the timing object as director of timed operation. In other words, the timing object is the master, and the timed component is the slave. However, the connection is not entirely one-way. Timed components may also enforce control over the timing object by issuing update requests. This allows interactive components to implement timing control on behalf of end users. For instance, a UI element implementing media controls may both be a slave of a timing object by slavishly visualizing media offset, while at the same time allow the end user to pause the presentation by clicking a the pause button. Crucially, control enforced on a timing object applies equally on all connected components.

Flexibility is another feature with this timing model. Since the timing object is an independent object, timed components and timing objects may be connected and disconnected dynamically. For example, in the context of live media presentation, flexible switching may be enabled between a timing object (representing the public, live-head of an experience), and a different timing object (representing private, personal, time-shifted experience). Another example would be switching to a different timing object in order to join a friend in a co-viewing session.

To support this timing model, a clear separation between timing object and consuming components is required. Specifically, a pattern emerges where timed components are given a reference to a timing object through a timingsrc property. On the other hand the timing object is kept usage agnostic and clean of references to consuming components.

Timed components that need to connect with the timing object broadly fall into two categories: components which already have an internal timing model, and components that do not. Within the first category we specifically have media players and existing media frameworks. For such components connection with the timing object mainly implies time-alignment of the internal timing model with the external timing object. In particular, this document discusses integration with media elements, see .

Components within the second category do not provide their own timing model, but may be useful for presenting timed data. UI elements such as map and canvas fall into this category. Custom components such as plotting libraries, media controls, subtitle viewers, Twitter widgets etc., may also need to connect with a timing object (in order to present data consistently in time). In fact, an entire Web page may be considered a timed component, where replacement of both data and presentational elements is directed by a timing object. All these timed components have a single challenge in common, the need to present or execute timed data at the correct time, i.e. time-aligned with the timing object. This document addresses this common need by connecting the timing object with a text track, see .

Finally, in order to support precisely timed operation in multi-device applications, the timing object must be connected with an online timing resource. One could imagine future standardization of a protocol between timing object and the online timing service. Currently though, the decision is rather to leave space open for innovation among timing providers. The TimingProvider interface is introduced, see . The idea is that third-parties may provide different implementations of timing provider objects (in JavaScript), to which timing objects may then connect. As a result, the connection to a timing provider object may be provided by the inclusion of a third-party JavaScript library.

Media capture and playback

Two very common uses of the timing object include media capture and playback. For media playback, the timing object may play the role of director or controls. Multiple independent components, possibly on different devices, may be responsible for timed presentation of timed media. If all components take direction from a single timing object, playback is ensured to be timing-consistent.

For media capture, the timing object may represent a shared clock for timestamping various types of captured media in multi-device scenarios. As long as timestamps are sampled from a shared clock and included in the captured media, time-shifted playback is possible. In live capturing scenarios, the timing object should likely just run continuously like a clock, or perhaps be adjusted slightly from time to time to stay in sync with some real-world clock source, e.g. a trusted source for epoch time. On the other hand, media capture in a studio setting may be a much more of an iterative process, recording one track or sample at a time. In such scenarios, the timing object might have to be paused and rewound for each take. The timing object will then play the dual role of playback director for previously captured media, as well as production clock for current capture.

Reference point in capture and playback

The concept of reference point refers to the exact time-alignment of timing object and media in capture and playback. Disagreement about the details of this alignment may be a source of (small) synchronization errors. For this reason, a common definition of reference points for time alignment in capture and playback is helpful.

For media capture, media timestamps should reflect the point in time when external, physical signals are received by sensors, i.e. ideally when photons hit the lense or when sound waves hit the microphone. As timestamping activity is delayed relative to this ideal reference point, timestamps picked from the timing object must be adjusted (upstream delay subtracted). This implies that capturing components must know (or figure out) their upstream latency (if significant). If all components are able to do this correctly, variation in processing delay will not produce synchronization errors.

For media playback, media timestamps should reflect the point in time when timed media fragments take physical effect. I.e., ideally when pixels are modified on the screen, or when the loudspeaker modifies its vibration frequency. If playback commands are subject to non-negligible delay before they take effect, the playback component must schedule commands earlier. This implies that playback component must know (or figure out) their downstream delay. If all components are able to do this correctly, variation in processing delay will not produce synchronization errors.

This definition of reference points is in accordance with definitions used in related standardization work, for instance see [[DVB-CSS]] / HbbTV2.0 CSS clause 5.7.2.

Strictly speaking, the timing object is usage-agnostic and can not mandate a specific definition for reference point. However, as interoperability of timing-sensitive components (e.g. capture and playback) is a major motivation for the timing object, the above definitions for reference point are recommended.

Naming

With regards to the naming of the timing object, there were multiple options available, e.g. Clock, WebClock, MediaClock or SharedClock. For a descriptive name, we argue that clock (or variations thereof) might be a bit limiting. A clock (in particular a system clock) is mainly a read-only resource (from the perspective of the programmer). Stop-watch would better indicate the interactive nature of the timing object, however, this too would fail to indicate the role of the timing object as controller or director in media. From a purely descriptive point of view, deterministic motion is perhaps the most precise name, matching the implementation and also indicating that linear media is made from two basic constituents - media content and media motion. On the other hand, the term motion is (still) unfamiliar and does not indicate any specific uses. For these reasons it was decided to name the object after its main purpose which is correct timing and temporal control. Something like TemporalControllerObject or TimingDirectorObject would perhaps be most correct. However, as these terms are a bit clunky TimingObject was selected as an acceptable compromise.

Requirements phrased in the imperative as part of algorithms (such as "strip any leading space characters" or "return false and terminate these steps") are to be interpreted with the meaning of the key word ("must", "should", "may", etc.) used in introducing the algorithm.

Conformance requirements phrased as algorithms or specific steps may be implemented in any manner, so long as the end result is equivalent. (In particular, the algorithms defined in this specification are intended to be easy to follow, and not intended to be performant).

This specification defines conformance criteria for two classes of products:

Terminology

The following terms, procedures and interfaces are defined in [[!HTML5]]:

The following interfaces are defined in [[!DOM]]:

The EventHandler interface represents a callback used for event handlers as defined in [[!HTML5]].

The internal clock is based on the precise, high-resolution, and monotonic clock defined in [[!HR-TIME]], with values representing seconds, not milliseconds. In essence, the internal clock is defined as performance.now()/1000.0. All time values refered to by the timing object specification are expressed in seconds. This includes timestamps in state vectors as well as skew estimates communicated from timing provider objects to timing objects.

A timing resource is an object. This object essentially implements the abstraction of a linear motion in real time. For example, imagine a point moving along an axis. At a specific moment in time, this point has a specific position on the axis, a specific velocity and a specific acceleration. These three properties (all floating point values) define the state of the timing resource at that moment in time.

A timing resource may either be an internal timing resource, an object under the control of a user agent, or an external timing resource, an object under the control of a timing resource provider. Both types exist locally in the user agent, but the external timing resource may be a proxy object for an online timing resource, i.e. an object hosted remotely by an online service.

A user agent may typically offer native implementations of timing objects, especially when the protocol needed is not available to Web applications. For instance, the clock synchronization mechanism defined in DVB for companion screens and streams [[DVB-CSS]] uses a UDP-based protocol. A user agent could perhaps expose a DVBCSSTimingObject interface to allow Web applications to connect to a companion screen that supports this protocol.

This document provides interface definitions using the [[!WEBIDL]] standard.

Examples

This section contains a few minimal code examples that show how a timing object may be used.

Query

      // Create timing object at position 2
      // (note velocity and acceleration default to 0)
      var to = new TimingObject({ position: 2.0 });

      // Read and report current value to the console
      // (renders "pos:2 vel:0, acc:0" in this example)
      var vector = to.query();
      console.log("pos:" + vector.position +
        " vel:" + vector.velocity +
        " acc:" + vector.acceleration);
      

Update

      // Pointers to "play" and "pause" buttons in the DOM
      var playbutton, pausebutton;

      // Start/Stop timing object when buttons are clicked
      var to = new TimingObject();
      playbutton.onclick = function () {
        to.update({ velocity: 1.0 });
      };
      pausebutton.onclick = function () {
        to.update({ velocity: 0.0 });
      };
      

Polling

By querying the timing object repeatedly we can refresh time sensitive UI and demonstrate that the value of the timing object is changing predictably in time.

      var elem; // DOM Element
      var to = new TimingObject();
      setInterval(function () {
        elem.innerHTML = to.query().position;
      }, 100);
      

Alternatively, use the built-in timeupdate event (fixed frequency).

      var elem; // DOM Element
      var to = new TimingObject();
      var to.addEventListener("timeupdate", function (e) {
        elem.innerHTML = to.query().position;
      });
      

Update Event

Update operations cause abrupt changes to the timing object. Register a handler to be notified.

      var elem; // DOM Element
      var to = new TimingObject();
      to.addEventListener("update", function (e) {
        var vector = to.query();
        if (vector.velocity === 0.0 && vector.acceleration === 0.0) {
          elem.innerHTML = "I'm not moving!";
        } else {
          elem.innerHTML = "I'm moving!";
        }
      });

      // Make timing object move and then stop it
      to.update({ velocity: 5, acceleration: 1 });
      to.update({ velocity: 0, acceleration: 0 });
      

Connect to a media element

A media element connected to a timing object will provide media playback precisely time-aligned with that timing object. In essence, the media element becomes a slave to the timing object. Pausing the timing object implies pausing the media element, etc.

      var video; // Pointer to a video element in the DOM
      var to = new TimingObject();
      video.timingsrc = to; // Video now directed by timing object

      // Start timing object (video seeks and starts by implication)
      to.update({ position: 5.0, velocity: 1.0 });
      

Connect to a timing text track

A timing text track connected to a timing object will provide enter/exit events for timed data, precisely time-aligned with that timing object. In essence, the timing text track becomes a slave to the timing object. Pausing the timing object implies pausing the timing text track, etc.

      var elem; // DOM element
      var to = new TimingObject();

      // Create timed data
      var track = new TimingTextTrack("metadata");
      var cue = new DataCue(12.783, 13.612, "dog bark");
      track.addCue(cue);
      cue.addEventListener("enter", function () {
        elem.innerHTML = "Dog is barking";
      });
      cue.addEventListener("exit", function () {
        elem.innerHTML = "Dog is silent";
      });
      track.timingsrc = to; // track directed by timing object
      to.update({ velocity: 1.0 }); // start playing timed data
      

Connect to a timing provider

A timing object may take its input from a timing provider object provided by some third-party provider (or developed by the app developer). Timing provider objects will typically act as proxies to online timing resource, thus enabling cross-device synchronization.

      // TimingProvider object
      // (constructor and parameters are provider-specific)
      var tp;

      // Create timing object associated with timing provider source
      var to = new TimingObject(tp);
      var to.addEventListener("timeupdate", function (e) {
        var vector = to.query();
        console.log("pos:" + vector.position +
          " vel:" + vector.velocity +
          " acc:" + vector.acceleration);
      });
      

Timing Object

A timing object is an object that exposes a timing resource represented by a state vector to a Web application.

A timing object can have a timing provider source, which is a TimingProvider object that encapsulates the logic needed to represent an external timing resource. The timing provider source is initialized once and for all when the timing object is created.

If the timing object is associated with a timing provider object, the timing object has an internal clock skew. This represents the current estimate of the skew between the internal clock and the clock employed by the timing resource provider. The internal clock skew is provided by the timing provider object, and defined so that clocktiming provider === clockuser agent + skew, where clockuser agent is the internal clock of the timing object, expressed in seconds. internal clock skew is also expressed in seconds. This equation is used by the user agent to convert timestamps between the timelines of the two clocks. The user agent must ensure monotonic behavior by, if necessary, applying skew changes gradually (see ).

A timing object has an internal vector that represents the initial conditions of the current motion. The internal vector is used by the query() operation to calculate a snapshot state vector of the current motion. If the timing object is associated with a timing provider object, the internal vector is provided by the timing resource provider and therefore includes a timestamp value from the timeline of the timing resource provider.

A timing object has a state property that describes the operational state of the timing object. If the timing object is associated with a timing provider object, the operational state will reflect the state of the timing provider object, and by implication the underlying communication link.

A timing object can have a start position and an end position that together define the range of positions for the timing object. Internally, the timing object must enforce this range restriction by updating its internal vector if a range violation occurs (at the correct moment). The timing object will maintain a current range timeout for this purpose, pointing to a scheduled timeout when set.

A timing object implements the following interface:

        [ Constructor (
           optional TimingStateVectorUpdate vector,
           optional unrestricted double startPosition,
           optional unrestricted double endPosition),
         Constructor (TimingProvider provider)]
        interface TimingObject : EventTarget {
          readonly    attribute TimingProvider timingProviderSource;
          readonly    attribute TimingObjectState readyState;
          readonly    attribute unrestricted double startPosition;
          readonly    attribute unrestricted double endPosition;
                      attribute EventHandler      onreadystatechange;
                      attribute EventHandler      onchange;
                      attribute EventHandler      ontimeupdate;
                      attribute EventHandler      onerror;
          TimingStateVector query ();
          Promise           update (TimingStateVectorUpdate newVector);
        };
      

The startPosition and endPosition attributes MUST respectively be set to the start position and end position that the state vector may take, if any, or -Infinity and Infinity otherwise.

The readyState attribute MUST be set to the current state of the timing object. If the timing object represents an internal timing resource, this will typically always be connecting or open. The readyState attribute may take other values when the timing object represents an external timing resource.

The timingProviderSource attribute MUST be set to the timing provider source of the timing object, if any, or null otherwise.

Create a new timing object

When the TimingObject constructor that takes an optional vector, a start position and an end position is invoked, the user agent MUST run the following steps:

  1. Let timing be a newly created TimingObject.
  2. Let t be a fresh timestamp read from the internal clock, defining the time of creation.
  3. Let vectorInit be the four-tuple (p, v, a, t), where (p, v, a) are from the constructors first argument, or (0,0,0) if no argument is provided.
  4. Let timing's internal vector be a newly constructed TimingStateVector that represents vectorInit.
  5. Let timing's timing provider source be null.
  6. Let timing's start position be the constructor's second argument or -Infinity if not given.
  7. Let timing's end position be the constructor's third argument or Infinity if not given.
  8. If timing's range does not cover the position of the internal vector:
    1. Let timing's internal vector's position be start position or end position, whichever is closest
    2. Let timing's internal vector's velocity and acceleration be 0.0 if the direction of the motion would make the position leave the range immediately.
    3. Set the internal timeout of timing.
  9. Let timing's state be open.
  10. Queue a task to fire a simple event named readystatechange at timing.
  11. Return timing.

When the TimingObject constructor that takes a TimingProvider is invoked, the user agent MUST run the following steps:

  1. Let provider be the constructor's argument.
  2. Let timing be a newly created TimingObject whose timing provider source is provider.
  3. Let timing's start position be provider's startPosition property.
  4. Let timing's end position be provider's endPosition property.
  5. Let timing's state be provider's readyState property.
  6. Let timing's internal vector be provider's vector property.
  7. Let timing's internal skew be provider's skew property.
  8. Observe provider.
  9. When an update to the readyState property is observed, run the following substeps:
    1. If the transition from the current state of timing to the new value is in the list of allowed state transitions, let timing's state be the new value. Otherwise, let timing's state be closed and stop observing provider.
    2. If the error property of provider is not null, queue a task to fire a simple event named error at timing.
    3. Queue a task to fire a simple event named readystatechange at timing.
  10. When an update to the vector property is observed, run the following substeps:
    1. Let newVector be the new value of provider's vector property.
    2. Process vector change with newVector as parameter.
  11. When an update to the skew property is observed, run the following substeps:
    1. Let newSkew be the new value of provider's skew property.
    2. Process skew change with newSkew as parameter.
  12. Return timing.

When a user agent is required to observe an object and run specific steps when an update to a property is detected, it MUST start to monitor the changes made to that object in the background and run the given steps as soon as the requested change is observed.

When a user agent is required to stop observing an object, it MUST stop any monitoring that was running on that object in the background.

This observing mechanism is meant to emulate the Object.observe method that is being proposed in EcmaScript 7 to simplify the TimingProvider interface and requirements set on a timing resource provider, as well as to work around the fact that TimingProvider cannot inherit EventTarget. The Multi-Device Timing Community Group welcomes feedback as to whether this approach is doable in practice.

Process a query operation

The query method returns a snapshot derived from the evaluation of internal vector against the current timestamp of the internal clock. When the method is invoked on a timing object timing, the user agent MUST run the following steps:

  1. Let timing be that TimingObject.
  2. If the state of timing is not open, throw new InvalidStateError and abort all further steps.
  3. Let tu be the reading of the internal clock, defining the processing time of the operation.
  4. Let (pint, vint, aint, tint) represent the internal vector of timing.
  5. If the timing provider source of timing is null, let d be tu-tint.
  6. If the timing provider source of timing is not null, let d be translate_u2p(tu)-tint.
  7. Let p be pint + vint * d + 1/2 * aint * d2
  8. Let v be vint + aint * d
  9. Let a be aint
  10. Let result be newly created TimingStateVector from four-tuple (p, v, a, tu)
  11. Range violation is detected. If timing's range does not cover p, run the following substeps:
    1. Let start be timing's startPosition property.
    2. Let end be timing's endPosition property.
    3. Let pmodified be either start or end, whichever is closest to p.
    4. Let vector be newly created TimingStateVectorUpdate that from three-tuple (pmodified, 0, 0).
    5. Let timing's internal vector be vector.
    6. Clear the current range timeout of timing.
    7. Queue a task to fire a simple event named change at the timing.
    8. Let result be result from invoking query method on timing (recursively).
  12. Return result.

Process an update operation

The update method sends a request for the timing object to update its current motion based on the provided state vector, or, if the timing object is associated with a timing provider object, the request is sent to an external timing resource via the timing provider source. In both cases, a Promise that the update was taken into account is returned. The method supports null or undefined values for the provided state vector attributes. This provides a simple mechanism for tying movements together. The idea is to allow one aspect of the movement to be updated while preserving the others. For instance, {position:null, velocity:value, acceleration:null} means update the velocity to the given value while preserving the current position and acceleration. If all attributes of the state vector are null or undefined, the method is a noop and returns immediately.

When the update method is invoked on a timing object timing, the user agent MUST run the following steps:

  1. Let timing be that TimingObject.
  2. If the state of timing is not open, return a new Promise, reject the promise with InvalidStateError and abort all further steps.
  3. Let newVector be the method's first parameter.
  4. If the timing provider source of timing is not null, run the following substeps, and abort the remaining steps:
    1. Let promiseprovider be the result of calling update() on the timing provider source object with newVector as parameter. If the call throws an exception, return a new Promise, reject the promise with the exception and abort all remaining steps. Similarly, if the call does not return a Promise, return a new Promise, reject the promise with a TypeError and abort all remaining steps.
    2. Return promiseprovider.
  5. Let nowVector be result from invoking query method on timing.
  6. Let intVector be timing's' internal vector.
  7. Let (pnew, vnew, anew) be the three-tuple represented by newVector.
  8. Let (pnow, vnow, anow, tnow) be the four-tuple represented nowVector.
  9. Let (pint, vint, aint, tint) be the four-tuple represented intVector.
  10. Let p be pnew, or pnow if pnew is null or undefined.
  11. Let v be vnew, or vnow if vnew is null or undefined.
  12. Let a be anew, or anow if anew is null or undefined.
  13. Avoid range violation on update. Run the following substeps:
    1. If timing's range does not cover p throw a new IllegalValueError and abort any further steps.
    2. Let start be timing's startPosition property.
    3. If p === start && (v < 0|| (v === 0 && a < 0)) throw a new IllegalValueError and abort any further steps.
    4. Let end be timing's endPosition property.
    5. If p === end && (v > 0 || (v === 0 && a > 0)) throw a new IllegalValueError and abort any further steps.
  14. Let vector be a newly created TimingStateVector that represents the four-tuple (p, v, a, tnow).
  15. Set the internal vector of timing to vector.
  16. Set the internal timeout of timing.
  17. Queue a task to fire a simple event named change at timing.
  18. Resolve promise.

Set the internal timeout

If timing object's properties startPosition and endPosition are specified, e.g. [0,123], the timing object must schedule a future update operation on itself, to ensure that the range is not violated.

When the user agent is required to set the internal timeout of a timing object, it must run the following steps:

  1. Let timing be that TimingObject.
  2. If timing's current range timeout is not null, cancel the timeout and let timing's current range timeout be null.
  3. Let startPos be timing's startPosition.
  4. Let endPos be timing's endPosition.
  5. Given the current motion, let endpoint be the first of startPos or endPos to be violated first, if any, and let t be the time when this will occur according to the internal clock.
  6. If endpoint is null or Infinity or -Infinity, abort all further steps.
  7. Let vector be the TimingStateVector represented by four-tuple (endpoint, 0, 0, t).
  8. Let timing's current range timeout be a newly created timeout to execute an update operation on timing with vector as parameter, at the moment when the internal clock reaches t.

Cover a position

When the user agent is required to evaluate whether a range composed of possibly infinite lower and upper bounds covers a provided value, the user agent MUST run the following steps:

  1. Let value be the position to evaluate.
  2. If value is smaller than the range's lower bound, return false, and abort these steps.
  3. If value is greater than the range's upper bound, return false, and abort these steps.
  4. Return true.

Process Vector Change

When the user agent is required to process vector change on a timing object, it MUST run the following steps:

  1. Let timing be that TimingObject.
  2. Let newVector be the new vector.
  3. Let timing's internal vector be newVector.
  4. Set the current range timeout to a newly created timeout to execute an update operation on timing with newVector as parameter, at the moment when the internal clock reaches t.
  5. Queue a task to fire a simple event named change at timing.

Process Skew Change

When the user agent is required to process skew change on a timing object, it MUST run the following steps:

  1. Let timing be that TimingObject.
  2. Let newSkew be the new skew.
  3. Let oldSkew be timing's internal clock skew.
  4. Let timing's internal clock skew be newSkew.
  5. Update variables used by calculate_skew_adjustment() based on newSkew and oldSkew

Translate timestamp from user agent to timing resource provider timeline

tp = translate_u2p(tu) translates a timestamp from the timeline of the user agent (u) to the timeline of the timing resource provider (p).

When the translate_u2p method is invoked, the user agent MUST run the following steps:

  1. Let tu be the first parameter.
  2. Let skew be the value of timing provider source's skew property.
  3. Let effectiveskew be the result of skew - calculate_skew_adjustment()
  4. Return tu + effectiveskew.

Translate timestamp from timing resource provider to user agent timeline

tu = translate_p2u(tp) translates a timestamp from the timeline of the timing resource provider (p) to the timeline of the user agent (u).

When the translate_p2u method is invoked, the user agent MUST run the following steps:

  1. Let tp be the first parameter.
  2. Let skew be the value of timing provider source's skew property.
  3. Let effectiveskew be the result of skew - calculate_skew_adjustment()
  4. Return tp - effectiveskew.

Calculate skew adjustment

A user agent MUST implement skew changes reported by a timing provider object gradually, in order to ensure the monotonicity of timestamp values reported by the query method of a timing object. For example, if the skew estimate increases by 2 ms at one point, one might want to apply this change at a certain rate, for instance 1 ms change per 10ms, so that, after 20ms, the 2ms change is fully applied. To achieve this, the user agent may update the internal clock skew immediately on skew change, and then define a linear adjustment function that is subtracted from the skew. In the above example the function will subtract the full 2ms initially, 1ms after 10ms, and then 0ms after 20ms. New skew changes must be integrated into the adjustment function.

The internal calculate_skew_adjustment() method implements this logic. If the method always returns 0, skew changes are fully applied as soon as they are received (i.e. not gradually applied). The logic of this method involves some bookkeeping, i.e. how much of the skew changed is already applied at a given point, and how much remains. New skew changes require this information to be re-evaluated. Note that this function is only invoked as part of a query operation. The actual logic of this method is left unspecified.

In practice, it is not clear that monotonic behavior at the millisecond scale is required by any of the existing use cases. Monotonic behavior is still included as requirement on the assumption that it may enable future use cases, and that is does not add much complexity.

Connection states

The connection with the timing resource that the timing object represents may take the following states:

connecting
The timing object is attempting to establish a connection with the timing resource it represents. This is the initial state when a new TimingObject is created. This state is also used when a TimingProvider object is associated with the TimingObject.
open
The connection with the timing resource is established and communication is possible.
closing
The procedure to close down the connection with the timing resource has started.
closed
The connection with the timing resource has been closed or could not be established.

The allowed state transitions are:

Event handlers

The following are the event handlers (and their corresponding event handler event types) that MUST be supported as attributes by a TimingObject object. All events use the Event interface.

Event handler Event handler event type Fired
onchange change The internal vector is changed. Fired after the update() method has returned, or when an update is received from the external timing resource.
onreadystatechange readystatechange The readyState attribute changed.
ontimeupdate timeupdate Fires periodically with fixed frequency 5Hz, except when timing object is paused. This is intended as a shorthand alternative for setting up a polling-loop with setInterval, or as an emulation of the pulse-based timing model that programmers are currently used to.
onerror error Fired when an operation on the timing object cannot complete for some reason, (e.g. because of a loss of connection with the timing resource).

Timing Provider Object

A timing provider object is an object exposed by a timing resource provider that encapsulates the logic necessary to associate a timing object with an external timing resource.

The concept is introduced in order to leave space open for innovation among online timing providers. The purpose is to decouple the user agent from any particular timing resource provider: third-parties may provide different implementations of timing provider objects in JavaScript, to which timing objects may be connected. In particular, this means that the protocols and logic used to identify, create or destroy an external timing resource, synchronize clocks and propagate state vectors to connected user agents, are up to the timing resource provider. Similarly, a timing resource provider may require Web applications or users to authenticate themselves before they grant them access to a particular timing resource.

In practice, a Web application willing to use a particular timing resource provider needs to load the corresponding JavaScript library provided by this timing resource provider, create a timing provider object and pass that object to the timing object constructor.

The TimingProvider interface allows the timing object to exercise control over the timing provider object, as well as be notified of state changes as soon as they occur, in particular state vectors and clock skew estimates are exchanged across this interface.

The skew specifies how timestamps may be translated between the timeline of the internal clock and that used by the timing resource provider. In the case of online timing resources, the skew is a floating estimate (due to variance in network latency and relative clock drift over time). The query operation on the timing object should always be resolved based on a fresh skew estimate received from the timing provider object. Skew is expressed in seconds and defined by the following equation (where clockuser agent is the internal clock):

clocktiming provider === clockuser agent + skew

A timing provider object implements the following interface:

        callback interface TimingProvider {
          readonly    attribute TimingStateVector vector;
          readonly    attribute unrestricted double startPosition;
          readonly    attribute unrestricted double endPosition;
          readonly    attribute DOMString         readyState;
          readonly    attribute unrestricted double skew;
          Promise update (TimingStateVectorUpdate newVector);
        };
      

The vector attribute MUST be set to the state vector that represents the initial conditions of the current motion.

The startPosition attribute MUST be set to the lower bound that constrains the position of the state vector, if any.

The endPosition attribute MUST be set to the upper bound that constrains the position of the state vector, if any.

The readyState attribute MUST be set to the current state of the connection with the timing resource.

The skew attribute MUST be set to the current estimate of the skew.

Process an update operation

The update method sends a request to the timing resource to have it update the external timing resource, based on the provided state vector, and returns a Promise that the update was processed by the timing resource provider (see the update method of TimingObject objects for more details). When the method is invoked on a timing provider object provider, the timing resource provider MUST run the following steps:

  1. Let promise be a new Promise.
  2. Return promise and run the following steps in parallel.
  3. Run whatever logic is necessary to pass the update request to the external timing resource that provider represents.
  4. If this logic succeeds, resolve promise. Otherwise reject promise with an error.

Report new conditions of the current motion

When the timing resource provider needs to report an update to the state vector (position, velocity or acceleration) of the external timing resource that a TimingProvider object encapsulates, it MUST set its vector property to a new TimingStateVector that represents the new conditions.

Report a state change

When the timing resource provider needs to report a change of connection state with the external timing resource that a TimingProvider object encapsulates, it MUST set its readyState property to the new value.

Report a skew change

When the timing resource provider needs to report a change of the skew estimate with the external timing resource that a TimingProvider object encapsulates, it MUST set its skew property to the new value.

Report an error

When the timing resource provider needs to report an unrecoverable error for a TimingProvider object, it MUST set its error property to the error and set its readyState property to closed.

This will effectively make any TimingObject object that has this TimingProvider object as timing provider source enter a final closed state.

State Vector

A state vector represents the classical four-tuple (position, velocity, acceleration, timestamp) associated with the mathematical description of linear motion under constant acceleration. A state vector is used to represent the motion of a timing resource.

In particular, the internal vector of a timing object is a state vector, the query() operation returns a state vector and the update() operation takes a state vector as parameter.

A State vector implements the following interface:

        dictionary TimingStateVectorUpdate {
          double position;
          double velocity;
          double acceleration;
        };

        dictionary TimingStateVectorInit : TimingStateVectorUpdate {
          double timestamp;
        };

        [ Constructor (TimingStateVectorInit vectorDict)]
        interface TimingStateVector {
          readonly    attribute double position;
          readonly    attribute double velocity;
          readonly    attribute double acceleration;
          readonly    attribute double timestamp;
        };
      

The position attribute MUST be set to the position of the state vector on its unidimensional axis. The position unit is usage specific. The position may for example represent a point in time in seconds, a height in meters, a slide number in a slide show or something else entirely.

The velocity attribute MUST be set to the velocity of the state vector on its unidimensional axis. The velocity represents the rate of change of the position, measured in position units per second.

The acceleration attribute MUST be set to the acceleration of the state vector on its unidimensional axis. The acceleration represents the rate of change of the velocity, measured in position units per second squared.

The timestamp attribute MUST be set to the moment in time when position, velocity and acceleration were|are|will be valid. The timestamp is from the timeline of the user agent defined by the internal clock, represented in seconds.

Create a new state vector

The TimingStateVector constructor is only intended to be used by the timing resource provider to set the vector attribute of a TimingProvider object.

When the TimingStateVector constructor is called, the user agent MUST run the following steps:

  1. Let vector be a newly created TimingStateVector.
  2. Let (pinit, vinit, ainit, tinit) be the four-tuple represented by the constructor's first argument.
  3. Initialize vector so that it represents the four-tuple (pinit, vinit, ainit, tinit).
  4. Return vector.

Media elements and the timing object

This section specifies an extension of media elements that allows them to use a timing object as a timing source.

If this specification progresses along the standardisation track, this section should be merged in a future version of [[!HTML5]].

Introduction

The essential function of the media element is to provide playback of video and audio data. Playback is typically controlled by end-user interaction (e.g. play, pause, seekTo) as well as the availability of data (e.g. live streaming, buffering). In particular, if the buffer underflows during playback, playback is halted until data becomes available. We call this the default playback mode.

This section specifies timed playback mode as an alternative mode of operation for the media element. Timed playback mode implies that media playback is directed exclusively by an external timing object. The goal is to allow presentation of video and audio frames at the correct time (millisecond scale), relative to a timing object. In other words, in timed playback mode the function of the media element is to time-align its internal media player with a timing object. This way, media elements may take part in precisely coordinated multi-media presentations, both in single-device and multi-device scenarios. In particular, as errors in time-alignment should be well below 10 ms, use-cases such as echo-less Web radio (multiple devices in a room), multi-track music (on multiple devices) or synchronization with musical instruments should be supported.

Timed playback mode and default playback mode are different in a few important respects:

In timed playback mode there is no obligation to present all the media content. If the timing object starts playback before the media element is ready, then a small segment of media will simply not be presented. The media element must always focus on time-aligning its presentation with the timing object. This is equally true on load, after timing object change events, or in the event of missing data. If correct data is not available at any given point in time, the media element simply defaults to presenting a black screen, perhaps decorated with graphics illustrating the advancing offset of the timing object as well as the appropriate error message.

The media element never halts the timing object in order to wait for data or other internal processing. Instead, end-users may choose to pause the presentation manually, or navigate back to see missing segments. Crucially, media controls (e.g. interactive buttons and progress bar) associated with the media element must be applied exclusively to the timing object. (The internal media player of the media element will be affected indirectly as a response to change events emitted by the timing object). This way, manually pausing a media element for buffering affects not only the media element but any other media component connected to the same timing object.

To achieve precisely time-aligned playback, a media element must continuously monitor and adjust its internal media player. Even if the internal media player is perfectly time-aligned with the timing object at one point in time, media players may slowly drift away from the timing object over time. To counter this, the media element must monitor their time-alignment periodically, and adjust as the error grows too large. Using variable playback rate internally is an effective way of implementing adjustments gradually and precisely.

Media elements take direction from a timing object, but playback capabilities are likely limited when it comes to high velocities, backwards playback or acceleration (supported by the timing object). Still, media elements must accept any state that the timing object supports, so throwing exceptions in these circumstances is not acceptable. Instead, media elements are free to fall back to black screen or graphical illustrations whenever the timing object specifies motions that are not supported by the internal media player.

Procedure: Set the current timing object for media element.

From the perspective of a media element, a TimingObject plays a role similar to a MediaController.

A media element in timed playback mode has a current timing source which is a TimingObject. The timing object imposes the media clock that the media element uses.

          partial interface HTMLMediaElement {
            attribute TimingObject timingsrc;
          };
        

The timingsrc attribute on a media element MUST, on getting, return the current timing source, if any, or null otherwise. On setting, the user agent MUST set the current timing object for the media element.

When the user agent is to set the current timing object for the media element, it MUST run the following steps:

  1. Let m be the media element in question.
  2. Let old controller be m's current media controller, it if currently has one, and null otherwise.
  3. Let m have no current media controller, if it currently has one.
  4. Remove the element's mediagroup content attribute, if any.
  5. Let m have no current timing source, it if currently has one.
  6. If the provided timing object is null, then jump to the update controller step below.
  7. Let m's current timing source be the providedtiming object.
  8. Let new timing be m's current timing source.
  9. check error in time-alignment.
  10. Update controller: If old controller is not null and still has one or more slaved media elements, then report the controller state for old controller.

When a media element has a current timing source, its currentTime and playbackRate properties reflect the current state of the internal media player, not the timing object. This implies backwards compatibility for applications that uses the media element as timing source. Note that if the velocity of the timing object is 1.0, the effective playback rate of the media element might be slightly less or slightly larger than 1.0. An internal playback rate of 1.0 generally does NOT imply that playback offset advances with the exact speed of the system clock. In addition, media elements may adjust for error in time-alignment gradually by adjusting their internal playback rate every now and then.

Procedure: Set the current media controller for media element

The procedures defined in [[!HTML5]] that update the current media controller should be updated to include the following step before the current media controller is set:

  1. Let m have no current timing source, if it currently has one.

Procedure: Set the current position for media element

The procedure defined in [[!HTML5]] that prevents setting currentTime when the media element has a current media controller should be updated to also prevent setting the attribute when the media element has a current timing source.

An alternative to preventing the setting of currentTime would be to map such a request onto the current timing source. Following this approach, other media control commands such as play and pause could be mapped similarly.

The playbackRate attribute has no effect when the media element has a current timing source. Note also that playbackRate attribute does not trivially map to the current definition of a timing object. This is because playbackRate is effectively a persistent wish for a specific playbackRate, rather than the actual playbackRate at any time. If needed, the timing object may easily be extended with this functionality by application code.

Procedure: Check error in time-alignment for media element

The user agent must check error in time-alignment between timing object and the internal player of the media element, and then make steps to adapt if error are too big. Checking the error involves the following steps:

  1. Let m be that media element.
  2. Let timing be that timing source.
  3. Let max be the maximum error that is tolerated.
  4. Let ideal be the result of timing.query().position.
  5. Let actual be the currentTime of the internal media player.
  6. Let error be (ideal - actual).
  7. If (error > max) start adjust time-alignment with given error
  8. .

Procedure: Time-align media element with timing object

Below is a rough explanation of how time-alignment may be adjusted. Details are left unspecified. However, a reference implementation of time-alignment, MediaSync [[MEDIASYNC]], is provided by the Multi-device Timing Community Group. This is a pure JavaScript implementation where time-alignment is implemented using the public API of the media element. This demonstrates that time-alignment at the millisecond scale is achievable, even without access to the internals of the media element. The assumption is that time-alignment may be even more precise and effective if supported by media elements internally.

If the error in time-alignment is very large (e.g. after a timing object update event) adjustments may require one or multiple seekTo(X) operations on the internal media player. The expected time-consumption of seekTo() operations should likely be taken into consideration when calculating X. If the internal media player (media format) supports variable playbackRate, this may be an effective way of implementing smaller adjustments gradually and more precisely. In both cases, adjusting the time-alignment is likely a multi-step process that goes on until the error is acceptible or cannot be further reduced.

Timed data and the timing object

This section specifies how a timing object may be used to direct timed presentation/execution of timed data.

Introduction

In this document, the process of translating a timed script or timed data into timed execution is broadly referred to as sequencing. Sequencing functionality is already provided by existing media frameworks, for example text tracks integrated with media elements, audio sample scheduling within the Web Audio API [[WEBAUDIO]], or timegraph traversal within SMIL Timing [[SMIL3]].

Similarly, the ability to connect sequencing logic (e.g. a text track) to a timing object would be very useful. It would allow timed data to be time-aligned (synchronized) with any other timed component, provided only that they are directed by the same timing object. Crucially, this would also be true in the multi-device scenario, as distributed timing objects may connect to and be directed by the same online timing source. Below we list important design goals for a general purpose sequencing mechanism for the Web.

In short, the sequencing mechanism should be made available as a generic programming concept/tool. This would presumably benefit both individual programmers as well as developers of advanced media frameworks. This specification extends the concept of text track to achieve this result so that application developers that already use text tracks driven by media elements may easily start using timing text tracks driven by a timing object.

An open-source JavaScript implementation of a sequencing mechanism is available [[SEQUENCER]] to allow developers to quickly start exploring the benefits of sequencing driven by a timing object. Please note that interfaces used in that implementation are not entirely aligned with those defined in this specification.

Timing text track

A timing text track is a text track that is directed by a timing object. The timing text track includes sequencing logic for the set of text track cues that composes the text track. As such, a timing text track has a current sequencer which is the timing object.

          [ Constructor (TextTrackKind kind, optional DOMString label = "", optional DOMString language = "")]
          interface TimingTextTrack : TextTrack {
            attribute TimingObject timingsrc;
          };
        

The timingsrc attribute MUST, on getting, return the current sequencer of the timing text track, if any, or null otherwise. On setting, the user agent MUST associate the track with a timing object.

As opposed to a text track, a timing text track is not and cannot be associated with a media element.

Is inheriting from TextTrack as proposed above the right approach? An alternative approach would be to extend TextTrack as done for the HTMLMediaElement. The problem with this appraoch is that TextTrack does not expose a constructor, so the only way to create a timing text track would be to add an addTextTrack method to the timing object (similar to the one on HTMLMediaElement). However, this is undesireable, as we would like to maintain a clean separation between timing object and objects that take direction from it. For instance, a clear separation ensures the flexibility to dynamically switch timing object for the timing text track.

Note also that the TextTrack design bear witness of close integration with media elements. For example, as the media element must provide UI support for text track visualization, it requires meta information such as track type (kind) and language. However, this meta information may quickly become irrelevant in a context where programmers are making custom timed presentations from application-specific data formats. Furthermore, as the programming of timed presentation is made very easy and flexible using the timing object combined with a general purpose sequencing mechanism, programmers will be much less dependent on the limited UI support for subtitles and captions, that is currently provided by media elements. In this perspective it seems odd to insist that all timing text tracks must be of a specific kind and declare a language.

A media element acts as sequencer for a list of text tracks. It is not clear that timing text tracks similarly need to be managed within a container. If such a container object is deemed necessary (e.g. for consistency with existing standards), such a container object must be defined in this document, say a Timing Track. The sequencing logic (track marches on) would then be provided by that container object rather than by timing text tracks individually.

There is a tradeoff with respect to how much we want timing text tracks to divert from regular text tracks. Keeping it true to its origins may be awkward in some respects. Taking it too far away may be confusing, possibly suggesting that a clean sheets approach with a new sequencer object might be preferable. Note also that programmers may use regular text tracks with a media element, a then let the media element be directed by a timing object.

Procedure: Create a new timing text track

When the TimingTextTrack constructor is invoked, the user agent MUST run the following steps:

  1. Create a new TimingTextTrack object.
  2. Create a new timing text track corresponding to the new object, and sets its text track kind to kind, its text track label to label, its text track language to language, its text track readiness state to the text track loaded state, its text track mode to the text track hidden mode, and its text track list of cues to an empty list.
  3. Return the new TimingTextTrack object.

Procedure: Associate timing text track with a timing object

When the user agent is required to associate the track with a timing object, it MUST run the following steps:

  1. Let track be the timing text track in question.
  2. Let track's current sequencer be the new value.
  3. Run the track marches on algorithm of track.
        var elem; // DOM element that renders subtitles
        var subtitles = [ // Timed data
          {text: "Zed's dead baby", start: 12.2, end: 17.4},
          {text: "Zed's dead", start: 19.7, end: 22.5}
        ];
        var track = new TimingTextTrack("metadata");
        subtitles.forEach(function (subtitle) {
          var cue = new DataCue(subtitle.start, subtitle.end, subtitle.text));
          track.addCue(cue);
        });
        var timing = new TimingObject(); 
        track.timingsrc = timing; // Associate timed data with timing object

        // React to cue changes (assuming only one active subtitle)
        track.addEventListener("cuechange", function () {
          var cue = (track.activeCues.length > 0) ? track.activeCues[0] : null;
          elem.innerHTML = cue ? cue.data : "";
        });
        timing.update({ velocity: 1.0 }); // Start playback
        

Procedure: Track marches on

The track marches on procedure evaluates the state of the timing text track at a specific point in time. The procedure may cause events to be emitted, and it may also cause a timeout to be created for the next execution of the procedure. The user agent must run this procedure whenever:

The specification does not detail the exact implementation of this procedure, except to note that the track marches on steps for the timing text track are essentially the same steps as the time marches on steps for a media element, replacing the current playback position by the position property of the timing object as returned by a call to query().position at the time of evaluation, and noting that timing text tracks do not have rules for updating the text track rendering, since the timing text track is UI-independent. Alternatively, the implementation of the timing text track may be based on the sequencer reference implementation.

Implementation guidelines multi-device synchronization

Distributed synchronization is a main use case that a user agent or a timing resource provider may enable. Timing objects running on different devices will be synchronized if they are associated with the same online timing resource. To achieve precise and reliable synchronization across the Internet, the implementation needs to address two distinct issues: the synchronization of state vectors and the synchronization of clocks. A brief introduction to a specific solution is given below. Details are available in [[MSV]].

The guideline discusses communication between an online timing resource and a timing object. In reality though, these two entities do not communicate directly, but indirectly through a mediating timing provider object running on the user agent. In the perspective of the timing object the timing provider object is a proxy object for the online timing resource.

State vector synchronization

If an online timing resource is updated, effects must apply equally to all connected timing objects as quickly as possible, including the timing object on which the update request might have been issued. This has implications for the processing of update requests. Local timing objects should simply forward the request, i.e. a new state vector), across the network to the online timing resource, where the request will be processed. Effects (i.e. the resulting state vector) will be multicast by the online timing resource to all connected timing objects, finally triggering a change event on these objects. Queries are always resolved locally, using the last state vector received from the online timing resource.

Note that this strategy does not guarantee that timing objects emit change events exactly at the same time. It is possible to achieve this by adding additional delay (thereby masking the differences in network latency). This would be a trade-off against the responsiveness of the user experience.

Clock synchronization

Ideally, if multiple timing objects (e.g. on different devices) connected to the same online timing resource are queried at the exact same moment, they should ideally return the same state vector (same position, velocity and acceleration). This requires that state vectors are synchronized, and that the clock skew between online timing resource is zero (synchronized clocks) or known (so that it can be compensated).

As synchronized system clocks is not a valid assumption in the Web environment, it follows that clock skew estimation must be resolved as part of the communication between timing objects and the online timing resource. To do this, timing objects (or timing provider objects) should maintain a software clock that is continuously synchronized with the clock used by the online timing resource. This may be achieved through periodic exchange with the online timing resource to evaluate the clock skew, taking into account the round-trip time (RTT) of these exchanges to improve the measurements. Using this evaluation, timing objects can tranform state vectors into the timeframe of their own local clock, i.e. the internal clock.

Clock synchronization can be very fast. Stable estimates may be reached within fractions of a second. Implementations may for instance use an open Web sockets [[WEBSOCKETS]] connection to minimize the latency between timing provider objects and the online timing resource. If implemented correctly on both ends, this approach provides a basis to achieve < 10ms media synchronization across the Internet.

Use cases and requirements

Precise timing has wide applicability in both single-device and multi-device applications. Here we identify some common use cases.

Social Viewing and Media Control

Allow people to enjoy the same content at the same time. Alice and Bob would like to watch the next episode together, even if Alice is on a train and Bob stays at home. If Alice pauses the video while briefly speaking with the conductor, Bob's video pauses too. Alice and Bob may always trust the other to see the exact same thing, making it very easy for them to maintain a conversation, for instance by using a chat service or the phone. It would also be possible for Alice and Bob to split temporarily. Alice would see Bob moving along the progress line without her, and Bob would see Alice staying behind. When Alice is ready to resume viewing a bit later, she may continue on her own, play doublespeed or skip to catch up with Bob, or convice Bob to wait for her or jump back to see the segment again with her.

This use case is based on UC2-3 Identical Media Stream Synchronization, a use case defined by the W3C Web and TV Interest Group

This use case requires the management and sharing of multiple timing objects, at least one for Alice and one for Bob. Precision requirements for video synchronization are quite coarse as the use case is described. However, Alice and Bob might also choose to use this application in circumstances where they are sitting next to eachother, or they might hook their devices on to projectors in order to show a movie to a larger audience. Sub-framerate precision is required in order to avoid echo from device speakers.

Broadcasting - Time-consistent, Live Web Productions

The BBC provides live coverage of popular sport events such as the FIFA World Cup. Modern Web presentations of this kind tend to include many independent media streams. For example, there might be multiple video streams for different camera angles, visualizations of players on the field, player statistics, and there might be commentary from studio as well as viewers. Earlier, Bob was annoyed that user comments could sometimes ruin the experience by shouting "GOAL!" 20 seconds before the goal appeared in the video stream. However, after BBC started timeshifting live Web content to match distribution latency in their video backend, this problem seems to be a thing of the past.

This requires figuring out the estimated latency of the slowest distribution mechanism, and then to define a timing resource from which all parts of a presentation takes direction. In effect, early spoilers will be delayed (buffered) by the Web browsers and applied to UI components at the correct moment in time.

Broadcasting - Time-shifted, Live Web productions

A commercial media provider offers live Web content supplementing live broadcast of F1 races. Sometimes, as F1 races are held in different time zones, the provider will re-broadcast nightly races in the morning. As a keen F1 enthusiast, Alice is discontent that the Web content can not be time-shifted along with the re-broadcast of the race. So is Bob, the media provider's head of marketing, as he would like to sell time sensitive, Web-based ads for F1 races, and have the ads be presented correctly for both live and time-shifted broadcasts, as well as later on demand viewers.

This requires timestamping of live Web production so that is may be time-shifted correctly and aligned with a time-shifted broadcast.

Broadcasting - Timed, Web-based Secondary Device

Alice opens the BBC Sport Web page on her iPad during the olympic games. The backend service consults Alice's profile and learns that she is already watching figure skating from the iPlayer on her laptop computer. As a result, the BBC Sports Web page is personalized with an offer to load BBC's Web-based extra material for figure skating. Alice accepts, and the iPad goes on to present stats, images, infographics, commentary and alternative camera angles, all precisely timed with the iPlayer. Alice notices that other viewers have already highlighted a Chinese athlete performing a bit earlier. She clicks to see it, and both the iPlayer (on the laptop) and the timed Web presentation (on the iPad) immediately skip back. After having seen the athlete's performance, now with added commentary provided by excited viewers, Alice contributes by giving it a thumbs up, and then clicks the back-to-live button. Both the laptop and the iPad snap back to presenting the live action.

This use case is based on UC2-4 Related Media Stream Synchronization, a use case defined by the W3C Web and TV Interest Group

This use case requires managing personal timing resources for individual viewers, as well as a common timing resource representing the live clock. Both iPlayer and Web-based secondary device offerings must take direction from online timing resources, and allow dynamic switching of timing objects based on user input. Presenting alternative camera angles as part of secondary device offerings requires precise synchronization.

Online Education - Timed, Multi-device Web Presentations

Alice teaches an online course for international students from across the globe. Every week she goes through a new Web-based slide set. Just before the session, she makes two links available for her students on the class Web page, links for the primary and the secondary view of the presentation. When the time is there she connects on an audio link and requests the first slide to be presented. Slides are presented on her view and on all the views of her connected students, at exactly the same time. This way, Alice can be sure that what she says on the audio link is always backed up with the correct illustrations. In effect, Alice remotely controls the Web browsers of all her students. Alice also has included a video in one of the slides. She plays the video for all the students on the primary view, while presenting some related bullet points on the secondary view. Alice pauses the video at a certain point in order to highlight an important aspect. Beforehand she has prepared some bookmarks in the video, allowing her to effectively skip to the interesting parts. At some point one of the students has a question related to the movie. Alice temporarily gives the student access to control the video, and the student rewinds it to explain the origin of his question. Afterwards, Alice withdraws the controls from the student and continues. The entire event is recorded and timestamped, allowing students that missed the class to replay the audio with and the slide presentation precisely as it was given. This review also provides synchronized presentations of timed comments provided by students as well as text-based search capabilities allowing efficient navigation within the presentation.

Multi-device slide show navigation requires a single shared timing resource, where position represents slide number. Videos require additional timing resources. Finally, timestamping of audio, slide navigation, video navigation, students commentary etc., require an addition timing resource as shared clock. Later individual replay might also require new timing resources.

Multi-Screen Data Visualization

Bob is heading a research project responsible for collecting and presenting a variety of data series related to climate changes in the Arctic. A Web-based approach to visualization makes for a very dynamic and extensible visualization system. Multiple screens may be used to present and align very different timed visualizations, making it easier to detect temporal patterns and possible correlations. The different visualizations may be navigated in unison. For instance, Bob may slow down the presentation speed for the heavy melting days to co-present this with measurements of salinity levels in the ocean water. Bob may also search his data to find the day with the highest temperature. By clicking on the result, all screens immediately display the relevant data for that day. Alice is recognized as an international expert in this area. Bob shares the link with Alice in an email, they may co-view the presentation even though they are stationed in different continents, and Alice may explore the visualization herself before giving her opinion of Bob's finding.

A shared timing resource is required to support playback of timed data streams on multiple screens. The utility is not limited to Web-based visualization. For instance, native visualization frameworks (e.g. 3D) may also interact with multi-device timing resources, thus enabling very different visualization systems to cooperate in presenting a common data model.

Seamless Workflows

Bob is watching a Netflix show using Chromecast on the TV when Alice interrupts and demands access to the living room big screen for herself and her friends. Bob reluctantly agrees to finish the show in his bedroom. Though, before he gets up he opens Netflix on his iPad. Netflix, well aware that Bob is already actively watching something, defaults to presenting the exact same thing, in perfect synchrony with the TV. Ensured that the transition was smooth, Bob leaves for his bedroom while staring at his iPad.

The use case requires online timing resources and content resources to be tied to login information. So, when an already logged in user enters a VoD service with a second device, the default action could be to resolve which show is already playing and what timing resource is being used, and then to set up the new view with the exact same resources.

Alternative Soundtracks

Alice's Portuguese mother has a hearing deficiency, so as the two of them sit down to watch a movie together, Alice hooks her smart phone up with the Portuguese audio track and lets her mother adjust the sound volume. Any issues with lip-sync should be due to the dubbing, not the audio synchronization.

This use case is based on UC2-6 Clean Audio, a use case defined by the W3C Web and TV Interest Group

This requires lip-synch precision between video and audio.

Timing Interoperability between Independent Providers

Bob watches soccer. Unfortunately, the clueless commentator is constantly pissing him off as he clearly favours the other team. Bob turns to a dedicated Web radio channel for an alternative commentary. The new commentator is better, but he is out of sync. Bob chooses the SyncWith option in the radio channel Web page and selects his TV provider. After confirming his credentials, the radio channel Web page is able to connect to the same timing resource used by the TV provider, and the Web radio is immediately in sync with his TV.

This use case requires media providers to give access to timing resources associated with a given user, also when that user is requesting access from an external service. Synchronization of audio and video requires lip-synch precision.

Linear Content Adaptation

Bob is watching cricket, but he and Alice soon need to get in the car to start the long drive to visit Bob's parents. He would like to keep following the cricket game on his smart phone, but does not have the bandwidth nor the money to enjoy HD content. Instead, Bob selects the timed HTML option. The audio goes on undisturbed, but the HD video is immediately terminated and replaced with a light-weight, timed animation of the cricket field. Alice is behind the wheels, and Bob may continue to enjoy the cricket match through audio and full support from all the HTML-based extra information.

Shared timing resources allow smooth and dynamic switching from HD to timed-based HTML.

Distributed Capture and Replayability

Web browsers are increasingly useful for capturing user generated data. The trivial example is basic interaction, where users manually post comments or upload media. More interestingly, browser integration with peripheral devices such as Web-cameras and microphones allow Web browsers to become input devices for live, distributed media productions. Also, mobil devices come with a range of built-in sensors. All this imput data may become even more valuable if it is timestamped according to the same clock. The timing objects, with support for multi-device synchronization enables precise timestamping of media capture for distributed phenomena. Furthermore, the same data may be replayed time-shifted in multi-screen presentations using the timing object as playback director.

Shared timing resources enables timed capture and preservation of timing relations in the distributed scenario.

Distributed Music Production and Playback

Musical productions are naturally distributed when multiple instruments play together. In the domain of music orchestration, the MIDI protocol has been uses as a carrier of tempo and rythm among electronic instruments. However, as MIDI signals are broadcast over physical (and wireless) links, higher latency reduces the synchronization. For this reason, the use of MIDI orchestration is typically limited to in-house setups. In contrast, the timing object with support for online synchronization supports millisecond precision globally, opening up for innovative distributed musical experiences. It also allows the involvement of visual components and devices that are not instruments. Support for MIDI instruments can still be achieved by synchronization of MIDI controllers using the timing object.

Shared timing resources closes the gap between the world of musical orchestration and multi-device orchestration in the Web.

Ad-insertion and Time-sensitive Ads

Seamless switching between video content and inserted segments of Ads is identified as an important use case for commercial interests. Media Task Force Ad Insertion Use Cases. Support for precisely timed playback of video elements is the basis for seamless switching. The timing object object may define a common timeline for both main content and ads, and timed text track support may be used to schedule videos in and out at the correct time, possibly with cross-fading implememted in css. The timing text track may even be used to implement pre-loading of videos, so that they are ready to play content just before a video switch. Using a timing object as director of media playback like this additionally opens up the possibility of presenting time-sensitive ads at the right time, possibly on a secondary device.

The timing object enables seamless, dynamic switching between different sources of timed media, and opens up for timed advertisements on multiple devices.