Bring back requestViewportScaling?

[klausw]

Initial versions of the WebXR spec had included a requestViewportScaling API that allows applications to use a subset of the overall framebuffer for rendering.

Issue #617 had requested deferring it to simplify the initial WebXR API, but it sounded as if people were generally not opposed to the API as such and would be open to bringing it back at a later time.

In the meantime, we've gotten feedback that performance scaling can be tricky especially for smartphone AR applications, where render costs can rise dramatically when people move close to complex AR objects. For example, uses aggressive autoscaling in non-XR canvas mode, and the developers would be very interested in having similar functionality available in WebXR AR mode.

Can we revisit this to see if it would make sense to reintroduce this API?

For reference, the removal was in #631 .

[klausw]

/agenda

[thetuvix]

As a note, HoloLens 1 originally had only a latent request model here (setting HolographicCamera.ViewportScaleFactor), where you'd ask for a particular scale and be given in future frames some similar scale you must use, perhaps rounded to the nearest 4 pixels or such. Since then, our native API moved to support arbitrary viewport overriding while you're already in a frame (HolographicCameraPose.OverrideViewport) and OpenXR now requires that runtimes support arbitrary an imageRect during layer submission.

Allowing apps to render to an arbitrary subset of the framebuffer and simply tell the UA what portion they're using that frame seems like a reasonable design to me if no other vendors will hit gotchas there.

[Manishearth]

Somewhat relevant: In my implementation of FPO views in Servo I downscale them by a constant for performance. It would be interesting if this could be exposed and overrideable through such an API, though I don't think the current proposal is for a per-view thing.

[Manishearth]

@klausw @toji can you jot down what the conclusion on this was from the call?

[toji]

Marking as future because while this did see some interest on the call and so we should continue looking into it, it's something that can be brought in additively later with no complications. I'll try and capture some more of the call discussion in another comment.

[klausw]

In summary, the rough consensus seemed to be that people seemed generally OK with a mechanism for apps to request use of under-sized viewports. There didn't seem to be specific concerns that implementations would be unable to accommodate this, but it seems safer to provide a mechanism for UAs to decline the request or to adjust the resulting value if needed. Providing GPU metrics to applications would make this more useful, but it seems worth experimenting with viewport scaling even in the absence of that.

Meeting minutes: https://www.w3.org/2020/06/30-immersive-web-minutes.html

Background: the current framebufferScaleFactor is set when creating the XRWebGLLayer, and cannot be changed for an existing layer. It's possible to change the scale at runtime by creating a new XRWebGLLayer and setting it as a new base layer via XRSession's updateRenderState, but this generally involves reallocations in the graphics pipeline. Applications should expect dropped frames when changing this, and it's not suitable as a per-frame dynamic scaling mechanism.
TODO: add a note to the spec to state this explicitly?

The framebuffer is carved up into views, for example one view per eye for a typical VR display. Currently, this viewport allocation is fully controlled by the UA.

In the previous WebVR API, applications had full control over viewport allocation and could request using only subsets of the framebuffer. Some implementations (Hololens?) had concerns that full application control could be inefficient or problematic. UA control could enforce constraints such as aligning to a preferred pixel grid.

According to Microsoft, dynamic viewport scaling hadn't seen much uptake by developers, including in corresponding native APIs, where developers often chose a fixed size and kept that for the duration of the experience. One concern was that it was difficult for applications to do smart scaling if they don't have any metrics about GPU utilization or time budgets.

Currently, developers targeting phone AR, for example , have expressed strong interest in dynamic scaling. In phone AR, even a rudimentary approach that accepts dropped frames seemed useful, since dropping frames isn't nearly as impactful as for a VR experience.

According to @thetuvix, OpenXR supports full viewport control, so implementations based on that should easily be able to do arbitrary viewports. Developers were inappropriately using framebuffer scaling, and viewport scaling would be much more efficient. It would be nice if there were a "keep me at framerate" flag in engines, but that seems rare, but it would be useful to have a way to prototype that.

The viewport scale mechanism should take effect on the same frame where it was requested. If a UA can't do this directly, they could use workarounds such as adjusting reprojection to compensate, but this must not cause visible dropped frames (especially if it would cause black flashes or similar) to avoid user discomfort if apps change the scale frequently.

The UA could provide a recommended viewport scale for each frame, for example based on a heuristic using UA-internal metrics. Applications could opt in to dynamic scaling by using that as-is, or apply clamping or other adjustments as needed, i.e. a lower bound to ensure small text remains readable. UAs could also potentially do automatic viewport scaling by default with an opt-out mechanism, but this risks surprising applications.

[klausw]

One more thing - I think there was also interest in applying a custom viewport scale to third-person views. For this use case, the requested and/or recommended viewport size should be a per-view property, not per-frame, so that the views can be scaled independently.

[klausw]

Just to make the proposal a bit more concrete, how would something like this sound?

• add a new XRWebGLLayer.getScaledViewport(XRView, scale) method, with 0 < scale <= 1.0. (Viewports can only be scaled down, not up.) UAs are free to clamp or modify this value, including ignoring the scaling entirely.

• add a readonly recommendedViewportScale property to XRView where UAs can optionally provide a suggested value.

Alternatively, assuming people are OK with overloaded methods in the API, this could be done by adding a new optional scale parameter to the existing getViewport method, treating an undefined scale as 1.0 for compatibility with current behavior.

In code, based on example 5 from https://immersive-web.github.io/webxr/#xrviewport-interface :

  for (xrView of viewer.views) {
    let xrViewport = xrWebGLLayer.getViewport(xrView);
    gl.viewport(xrViewport.x, xrViewport.y, xrViewport.width, xrViewport.height);

Scaled viewports:

let xrViewport = xrWebGLLayer.getScaledViewport(xrView, xrView.recommendedViewportScale || 1.0);

// Variant: overload getViewport with an extra parameter
let xrViewport = xrWebGLLayer.getViewport(xrView, xrView.recommendedViewportScale || 1.0);

// Variant: an undefined value is treated as 1.0
let xrViewport = xrWebGLLayer.getViewport(xrView, xrView.recommendedViewportScale);

Of course, applications could use their own logic to calculate a scale factor from scratch, or base it on clamping/modifying the UA-provided recommended scale factor.

Allowing the UA's flexibility to ignore or modify the requested scale factor is helpful for several reasons. It allows the UA to align the viewports to pixel boundaries where this improves efficiency, and avoids the need for complicated rectangle area allocation tracking. For example, if an application for some reason calls getScaledViewport multiple times for one view in the same frame with different scales, or with inconsistent per-eye values, the UA would be free to use its preferred value and ignore values inconsistent with its current viewport allocations.

[elalish]

This sounds great to me; I like the overload version a little better, but it's not a strong opinion. Out of curiosity, in what circumstances would recommendedViewportScale != 1.0?

In any case, this scaling is critical for smooth frame rates, especially since I've observed frame rates dropping by more than 4x just by walking up to an AR model. The shader only runs on the blocks of pixels the rendering covers, which is usually not all that much, but when you get close, it can quickly become a disaster. In <model-viewer> we can keep the frame rate up using viewport scaling on our 3D view simply by targeting a frame rate slightly lower than the max. It is working well in practice, and once this API is available in WebXR, we'll do the same there.

[thetuvix]

I'm generally comfortable with keeping the UA in the loop to allow runtimes to give back a slightly tweaked viewport (or even ignore the request and just return the original viewport) if needed.

One gotcha with the particular design proposed above is that it gives side effects to the call to getViewport. Today, you can call getViewport(view) however often you like within the code you run for a given frame and it just gives you that view's viewport - the reason it's a function is purely to index the available viewports by the view parameter. With this change, each getScaledViewport call would get you a scaled viewport, but what then chooses the official viewport that the UA will read from at the end of the frame? If it's simply the latest call to getViewport in a given frame, that seems both subtle and fragile if multiple components expect that they can just fetch the viewport as needed without coordination.

It may make more sense to have a requestViewportScale function here to make the active nature of the request more clear. You would then call getViewport(view) as often as needed afterwards to make use of the updated viewport you're rendering into. So long as some early part of the render loop is the only code that calls the request function, different render components of the app can later freely call getViewport as needed without coordinating with one another.

That approach still has the subtlety that the app must call getViewport can change out from under a render component that might not expect it if requestViewportScale is called halfway through the frame. However, that might be acceptable.

Perhaps more importantly, if most platforms just blindly accept any viewport you request, some apps may not actually go through the extra bother of calling getViewport to find out the UA's effective viewport and just run with the exact viewport they requested. If that only breaks on a small number of devices, many apps may mess that up, and so those devices may feel pressured to just allow arbitrary viewport requests. If we take another look and are comfortable just requiring all UAs on all platforms to allow any arbitrary viewport that fits in the framebuffer, we could just make this setViewport(view).

[klausw]

@elalish wrote:

This sounds great to me; I like the overload version a little better, but it's not a strong opinion. Out of curiosity, in what circumstances would recommendedViewportScale != 1.0?

The recommended viewport scale is intended as a mechanism for the UA to suggest a scale factor based on internal metrics or heuristics, with the goal that an application using that scale factor should get an acceptable compromise between resolution and framerate. It would stay at 1.0 if the application's rendering is keeping up with the target framerate at full viewport size, and would only drop lower if the rendering time exceeds the frame time budget.

Ideally, the UA's recommended viewport scale would remain at 1.0 if the frame rate is low due to reasons other than GPU rendering time, i.e. if the application spends an excessive amount of CPU time in JS processing, where reducing the viewport scale would reduce the visual quality without any performance benefits.