mmagic.models.editors.pconv

Package Contents

Classes

MaskConvModule	Mask convolution module.
PartialConv2d	Implementation for partial convolution.
PConvDecoder	Decoder with partial conv.
PConvEncoder	Encoder with partial conv.
PConvEncoderDecoder	Encoder-Decoder with partial conv module.
PConvInpaintor	Inpaintor for Partial Convolution method.

class mmagic.models.editors.pconv.MaskConvModule(*args, **kwargs)

Bases: mmcv.cnn.ConvModule

Mask convolution module.

This is a simple wrapper for mask convolution like: ‘partial conv’. Convolutions in this module always need a mask as extra input.

Parameters

• in_channels (int) – Same as nn.Conv2d.

• out_channels (int) – Same as nn.Conv2d.

• kernel_size (int or tuple[int]) – Same as nn.Conv2d.

• stride (int or tuple[int]) – Same as nn.Conv2d.

• padding (int or tuple[int]) – Same as nn.Conv2d.

• dilation (int or tuple[int]) – Same as nn.Conv2d.

• groups (int) – Same as nn.Conv2d.

• bias (bool or str) – If specified as auto, it will be decided by the norm_cfg. Bias will be set as True if norm_cfg is None, otherwise False.

• conv_cfg (dict) – Config dict for convolution layer.

• norm_cfg (dict) – Config dict for normalization layer.

• act_cfg (dict) – Config dict for activation layer, “relu” by default.

• inplace (bool) – Whether to use inplace mode for activation.

• with_spectral_norm (bool) – Whether use spectral norm in conv module.

• padding_mode (str) – If the padding_mode has not been supported by current Conv2d in Pytorch, we will use our own padding layer instead. Currently, we support [‘zeros’, ‘circular’] with official implementation and [‘reflect’] with our own implementation. Default: ‘zeros’.

• order (tuple[str]) – The order of conv/norm/activation layers. It is a sequence of “conv”, “norm” and “act”. Examples are (“conv”, “norm”, “act”) and (“act”, “conv”, “norm”).

supported_conv_list = ['PConv']

forward(x, mask=None, activate=True, norm=True, return_mask=True)

Forward function for partial conv2d.

Parameters

• x (torch.Tensor) – Tensor with shape of (n, c, h, w).

• mask (torch.Tensor) – Tensor with shape of (n, c, h, w) or (n, 1, h, w). If mask is not given, the function will work as standard conv2d. Default: None.

• activate (bool) – Whether use activation layer.

• norm (bool) – Whether use norm layer.

• return_mask (bool) – If True and mask is not None, the updated mask will be returned. Default: True.

Returns

Result Tensor or 2-tuple of

Tensor: Results after partial conv.

Tensor: Updated mask will be returned if mask is given and return_mask is True.

Return type

Tensor or tuple

class mmagic.models.editors.pconv.PartialConv2d(*args, multi_channel=False, eps=1e-08, **kwargs)

Bases: torch.nn.Conv2d

Implementation for partial convolution.

Image Inpainting for Irregular Holes Using Partial Convolutions [https://arxiv.org/abs/1804.07723]

Parameters

• multi_channel (bool) – If True, the mask is multi-channel. Otherwise, the mask is single-channel.

• eps (float) – Need to be changed for mixed precision training. For mixed precision training, you need change 1e-8 to 1e-6.

forward(input, mask=None, return_mask=True)

Forward function for partial conv2d.

Parameters

• input (torch.Tensor) – Tensor with shape of (n, c, h, w).

• mask (torch.Tensor) – Tensor with shape of (n, c, h, w) or (n, 1, h, w). If mask is not given, the function will work as standard conv2d. Default: None.

• return_mask (bool) – If True and mask is not None, the updated mask will be returned. Default: True.

Returns

Results after partial conv. torch.Tensor : Updated mask will be returned if mask is given and return_mask is True.

Return type

torch.Tensor

class mmagic.models.editors.pconv.PConvDecoder(num_layers=7, interpolation='nearest', conv_cfg=dict(type='PConv', multi_channel=True), norm_cfg=dict(type='BN'))

Bases: mmengine.model.BaseModule

Decoder with partial conv.

About the details for this architecture, pls see: Image Inpainting for Irregular Holes Using Partial Convolutions

Parameters

• num_layers (int) – The number of convolutional layers. Default: 7.

• interpolation (str) – The upsample mode. Default: ‘nearest’.

• conv_cfg (dict) – Config for convolution module. Default: {‘type’: ‘PConv’, ‘multi_channel’: True}.

• norm_cfg (dict) – Config for norm layer. Default: {‘type’: ‘BN’}.

forward(input_dict)

Forward Function.

Parameters

input_dict (dict | torch.Tensor) – Input dict with middle features or torch.Tensor.

Returns

Output tensor with shape of (n, c, h, w).

Return type

torch.Tensor

class mmagic.models.editors.pconv.PConvEncoder(in_channels=3, num_layers=7, conv_cfg=dict(type='PConv', multi_channel=True), norm_cfg=dict(type='BN', requires_grad=True), norm_eval=False)

Bases: mmengine.model.BaseModule

Encoder with partial conv.

About the details for this architecture, pls see: Image Inpainting for Irregular Holes Using Partial Convolutions

Parameters

• in_channels (int) – The number of input channels. Default: 3.

• num_layers (int) – The number of convolutional layers. Default: 7.

• conv_cfg (dict) – Config for convolution module. Default: {‘type’: ‘PConv’, ‘multi_channel’: True}.

• norm_cfg (dict) – Config for norm layer. Default: {‘type’: ‘BN’}.

• norm_eval (bool) – Whether to set norm layers to eval mode, namely, freeze running stats (mean and var). Note: Effective on Batch Norm and its variants only. Default: False.

train(mode=True)

Set BatchNorm modules in the model to evaluation mode.

forward(x, mask)

Forward function for partial conv encoder.

Parameters

• x (torch.Tensor) – Masked image with shape (n, c, h, w).

• mask (torch.Tensor) – Mask tensor with shape (n, c, h, w).

Returns

Contains the results and middle level features in this module. hidden_feats contain the middle feature maps and hidden_masks store updated masks.

Return type

dict

class mmagic.models.editors.pconv.PConvEncoderDecoder(encoder, decoder)

Bases: mmengine.model.BaseModule

Encoder-Decoder with partial conv module.

Parameters

• encoder (dict) – Config of the encoder.

• decoder (dict) – Config of the decoder.

forward(x, mask_in)

Forward Function.

Parameters

• x (torch.Tensor) – Input tensor with shape of (n, c, h, w).

• mask_in (torch.Tensor) – Input tensor with shape of (n, c, h, w).

Returns

Output tensor with shape of (n, c, h’, w’).

Return type

torch.Tensor

class mmagic.models.editors.pconv.PConvInpaintor(data_preprocessor: Union[dict, mmengine.config.Config], encdec: dict, disc: Optional[dict] = None, loss_gan: Optional[dict] = None, loss_gp: Optional[dict] = None, loss_disc_shift: Optional[dict] = None, loss_composed_percep: Optional[dict] = None, loss_out_percep: bool = False, loss_l1_hole: Optional[dict] = None, loss_l1_valid: Optional[dict] = None, loss_tv: Optional[dict] = None, train_cfg: Optional[dict] = None, test_cfg: Optional[dict] = None, init_cfg: Optional[dict] = None)

Bases: mmagic.models.base_models.OneStageInpaintor

Inpaintor for Partial Convolution method.

This inpaintor is implemented according to the paper: Image inpainting for irregular holes using partial convolutions

forward_tensor(inputs, data_samples)

Forward function in tensor mode.

Parameters

• inputs (torch.Tensor) – Input tensor.

• data_sample (dict) – Dict contains data sample.

Returns

Dict contains output results.

Return type

dict

train_step(data: List[dict], optim_wrapper)

Train step function.

In this function, the inpaintor will finish the train step following the pipeline:

1. get fake res/image

2. optimize discriminator (if have)

3. optimize generator

If self.train_cfg.disc_step > 1, the train step will contain multiple iterations for optimizing discriminator with different input data and only one iteration for optimizing generator after disc_step iterations for discriminator.

Parameters

• data (List[dict]) – Batch of data as input.

• optim_wrapper (dict[torch.optim.Optimizer]) – Dict with optimizers for generator and discriminator (if have).

Returns

Dict with loss, information for logger, the number of samples and results for visualization.

Return type

dict