Coding Structure¶

class CodingStructure[source]¶

Dataclass representing the organization of the video i.e. which frames are coded using which references.

A few examples:

# A low-delay P configuration
# I0 ---> P1 ---> P2 ---> P3 ---> P4 ---> P5 ---> P6 ---> P7 ---> P8
intra_period=8 p_period=1

# A hierarchical Random Access configuration
# I0 -----------------------------------------------------> P8
# \-------------------------> B4 <-------------------------/
#  \----------> B2 <---------/ \----------> B6 <----------/
#   \--> B1 <--/ \--> B3 <--/   \--> B5 <--/  \--> B7 <--/
intra_period=8 p_period=8

# There is no more prediction from I0 to P8. Instead the GOP in split in
# half so that there is no inter frame with reference further than --p_period

# I0 -----------------------> P4 ------------------------> P8
#  \----------> B2 <---------/ \----------> B6 <----------/
#   \--> B1 <--/ \--> B3 <--/   \--> B5 <--/  \--> B7 <--/
intra_period=8 p_period=4

A coding is composed of a few hyper-parameters and most importantly a list of Frame describing the different frames to code.

Parameters:

  • intra_period (int) – Number of inter frames in the GOP. As such, the first (intra) frame of two successive GOPs would be spaced by intra_period inter frames. Set this to 0 for all intra coding.

  • p_period (int) – Distance to the furthest P prediction in the GOP. Set this to 1 for low-delay P or to intra_period for the usual random access configuration.

  • seq_name (str) – Name of the video. Mainly used for logging purposes. Defaults to "".

frames: List[Frame]¶

All the frames to code, deduced from the GOP type, intra period and P period. Frames are index in display order (i.e. temporal order). frames[0] is the 1st frame, while frames[-1] is the last one.

compute_gop(
intra_period: int,
p_period: int,
) → List[Frame][source]¶

Return a list of frames with one intra followed by intra_period inter frames. The relation between the inter frames is implied by p_period. See examples in the class description.

Parameters:

  • intra_period (int) – Number of inter frames in the GOP.

  • p_period (int) – Distance between I0 and the first P frame or between subsequent P-frames.

Returns:

List describing the frames to code.

Return type:

List[Frame]

pretty_string() → str[source]¶

Return a pretty string formatting the data within the class

Return type:

str

get_number_of_frames() → int[source]¶

Return the number of frames in the coding structure.

Returns:

Number of frames in the coding structure.

Return type:

int

get_max_depth() → int[source]¶

Return the maximum depth of a coding configuration

Returns:

Maximum depth of the coding configuration

Return type:

int

get_all_frames_of_depth(
depth: int,
) → List[Frame][source]¶

Return a list with all the frames for a given depth

Parameters:

depth (int) – Depth for which we want the frames.

Returns:

List of frames with the given depth

Return type:

List[Frame]

get_max_coding_order() → int[source]¶

Return the maximum coding order of a coding configuration

Returns:

Maximum coding order of the coding configuration

Return type:

int

get_frame_from_coding_order(
coding_order: int,
) → Frame | None[source]¶

Return the frame whose coding order is equal to coding_order. Return None if no frame has been found.

Parameters:

coding_order (int) – Coding order for which we want the frame.

Returns:

Frame whose coding order is equal to coding_order.

Return type:

Frame | None

get_max_display_order() → int[source]¶

Return the maximum display order of a coding configuration

Returns:

Maximum display order of the coding configuration

Return type:

int

get_frame_from_display_order(
display_order: int,
) → Frame | None[source]¶

Return the frame whose display order is equal to display_order. Return None if no frame has been found.

Parameters:

display_order (int) – Coding order for which we want the frame.

Returns:

Frame whose coding order is equal to display_order.

Return type:

Frame | None

set_encoded_flag(coding_order: int, flag_value: bool) → None[source]¶

Set the flag self.already_encode of the frame whose coding order is coding_order to the value flag_value.

Parameters:

  • coding_order (int) – Coding order of the frame for which we’ll change the flag

  • flag_value (bool) – Value to be set

Return type:

None

unload_all_decoded_data() → None[source]¶

Remove the data describing the decoded data from the memory. This is used before saving the coding structure. The decoded data can be retrieved by re-inferring the trained model.

Return type:

None

unload_all_original_frames() → None[source]¶

Remove the data describing the original frame from the memory. This is used before saving the coding structure. The original frames can be retrieved by reloading the sequence

Return type:

None

unload_all_references_data() → None[source]¶

Remove the data describing all the references from the memory. This is used before saving the coding structure. The reference data can be retrieved by re-inferring the trained model.

Return type:

None

get_frame_depth_in_gop(idx_frame: int) → int[source]¶

Return the depth of a frame with index <idx_frame> within a hierarchical GOP.

Some notes:

  • idx_frame == 0 always corresponds to an intra frame i.e. depth = 0

  • idx_frame == p_period is the P-frame i.e. depth = 1

  • This should be used separately for the successive chained GOPs.

Parameters:

  • idx_frame (int) – Display order of the frame in the GOP.

  • p_period – P-period. Should be a power of two.

Returns:

Depth of the frame in the GOP.

Return type:

int

class Frame[source]¶

Dataclass representing a frame to be encoded. It contains useful info like the display & coding indices, the indices of its references as well as the data of the decoded references and the original (i.e. uncompressed) frame.

Parameters:

  • coding_order (int) – Frame with coding_order=0 is coded first.

  • display_order (int) – Frame with display_order=0 is displayed first.

  • depth (int) – Depth of the frame in the GOP. 0 for Intra, 1 for P-frame, 2 or more for B-frames. Roughly corresponds to the notion of temporal layers in conventional codecs. Defaults to 0.

  • seq_name (str) – Name of the video. Mainly used for logging purposes. Defaults to "".

  • data (Optional[FrameData]) – Data of the uncompressed image to be coded. Defaults to None.

  • already_encoded (bool) – True if the frame has already been coded by the VideoEncoder. Defaults to False

  • index_references (List[int]) – Index of the frame(s) used as references, in display_order. Leave empty when no reference are available i.e. for I-frame. Defaults to [].

  • ref_data (List[FrameData]) – The actual data describing the decoded references. Leave empty when no reference are available i.e. for I-frame. Defaults to [].

frame_type: Literal['I', 'P', 'B']¶

Automatically set from the number of entry in self.index_references.

set_frame_data(
data: Tensor | DictTensorYUV,
frame_data_type: Literal['rgb', 'yuv420', 'yuv444'],
bitdepth: Literal[8, 9, 10, 11, 12, 13, 14, 15, 16],
) → None[source]¶

Set the data representing the frame i.e. create the FrameData object describing the actual frame.

Parameters:

  • data (Tensor | DictTensorYUV) – RGB or YUV value of the frame.

  • frame_data_type (Literal['rgb', 'yuv420', 'yuv444']) – Data type.

  • bitdepth (Literal[8, 9, 10, 11, 12, 13, 14, 15, 16]) – Bitdepth.

Return type:

None

set_decoded_data(decoded_data: FrameData) → None[source]¶

Set the data representing the decoded frame.

Parameters:

  • refs_data – Data of the reference(s)

  • decoded_data (FrameData)

Return type:

None

set_refs_data(refs_data: List[FrameData]) → None[source]¶

Set the data representing the reference(s).

Parameters:

refs_data (List[FrameData]) – Data of the reference(s)

Return type:

None

upsample_reference_to_444() → None[source]¶

Upsample the references from 420 to 444 in place. Do nothing if this is already the case.

Return type:

None

to_device(device: Literal['cpu', 'cuda:0']) → None[source]¶

Push the data attribute to the relevant device in place.

Parameters:

device (Literal['cpu', 'cuda:0']) – The device on which the model should run.

Return type:

None

class FrameData[source]¶

FrameData is a dataclass storing the actual pixel values of a frame and a few additional information about its size, bitdepth of color space.

Parameters:

  • bitdepth (POSSIBLE_BITDEPTH) – Bitdepth, should be in [8, 9, 10, 11, 12, 13, 14, 15, 16].

  • frame_data_type (FRAME_DATA_TYPE) – Data type, either "rgb", "yuv420", "yuv444".

  • data (Union[Tensor, DictTensorYUV]) – The actual RGB or YUV data

img_size: Tuple[int, int]¶

Height & width of the video \((H, W)\)

n_pixels: int¶

Number of pixels \(H \times W\)

to_device(device: Literal['cpu', 'cuda:0']) → None[source]¶

Push the data attribute to the relevant device in place.

Parameters:

device (Literal['cpu', 'cuda:0']) – The device on which the model should run.

Return type:

None

to_string() → str[source]¶

Pretty string describing the frame data.

Return type:

str