Warping an image with an optical flow

class Warper

__init__(

param: WarpParameter,

img_size: Tuple[int, int],

)

Instantiate a warper module, parameterized by param.

Parameters:

• param (WarpParameter) – Warping parameters (filter length, fractional accuracy).

• img_size (Tuple[int, int]) – [Height, Width].

coords_grid(h: int, w: int) → Tensor

Return a [1, 2, H, W] tensor, where the 1st feature gives the column index and the 2nd the row index. For instance:

Parameters:

• h (int) – height of the grid

• w (int) – width of the grids

Returns:

Tensor giving the column and row indices.

Return type:

Tensor

get_coeffs(s: Tensor) → Tensor

Generate interpolation coefficients from the fractional displacement s in [0, 1[.

Parameters:

s (Tensor) – Fractional displacement for each pixel, shape is [1, 1, H, W].

Returns:

Corresponding interpolation coefficients for each pixel,

shape is [1, filter_size, H, W].

Return type:

Tensor

interpolate_1d(neighbors: Tensor, fractional_flow: Tensor) → Tensor

Performs the interpolations of neighboring integer values to get the value located at fractional_flow.

Parameters:

• neighbors (Tensor) – [B, filter_size, C, H, W]. We compute B x C x H x W interpolations in parallel. Each of them has filter_size neighbors.

• fractional_flow (Tensor) – [1, 1, H, W]. Fractional flow for each warping. All channels share the same fractional_flow, hence C=1 here. All batches share the same fractional_flow, hence B=1 here. This is used to shift 2d blocks of pixels with B=filter_size.

Returns:

[B, C, H, W] the interpolated neighbors

Return type:

Tensor

forward(x: Tensor, flow: Tensor) → Tensor

Warp a [1, C, H, W] tensor using a [1, 2, H, W] optical flow. The optical flow is expressed in absolute pixel i.e. an horizontal motion of -3 means that the pixel output pixel at [i, j] is equal to x[i - 3, j].

y = warp(x, flow) –> y[i, j] = x[i + vertical flow, j + horizontal flow]

where vertical flow is flow[:, 0, :, :] (row-wise) and horizontal flow is flow[:, 1, :, :] (column wise).

As such, the value in flow describes the motion from y (the warping result) to x (the reference.)

Parameters:

• x (Tensor) – Tensor to be warped, shape is [1, C, H, W]

• flow (Tensor) – Motion to warp x: shape is [1, 2, H, W].

Returns:

Warped tensor [1, C, H, W]

Return type:

Tensor

class WarpParameter

Dataclass storing the parameters of the motion compensation warping

pretty_string() → str

Return a pretty string presenting the WarpParameter.

Returns:

Pretty string ready to be printed.

Return type:

str

vanilla_warp_fn(x: Tensor, flow: Tensor, mode: str = 'bicubic') → Tensor

Motion compensation (warping) of a tensor [B, C, H, W] with a 2-d displacement [B, 2, H, W]. This function does not allows for longer filters than 4 taps (bicubic) and does not quantize the flows to a given subpixel accuracy in eval mode.

Some code in this function is inspired from https://github.com/microsoft/DCVC/blob/main/DCVC-FM/src/models/block_mc.py License: MIT

Parameters:

• x (Tensor) – Tensor to be motion compensated [B, C, H, W].

• flow (Tensor) – Displacement [B, C, H, W]. flow[:, 0, :, :] corresponds to the horizontal displacement. flow[:, 1, :, :] is the vertical displacement.

• mode (str)

Returns:

Motion compensated tensor [B, C, H, W].

Return type:

Tensor