cvpods.utils package

cvpods.utils.benchmark.benchmark module

cvpods.utils.benchmark.benchmark.timeit(num_iters: int = - 1, warmup_iters: int = 0)[source]

This is intened to be used as a decorator to time any function.

Parameters

  • num_iters (int) – number of iterations used to compute the average time (sec) required to run the function. If negative, the number of iterations is determined dynamically by running the function a few times to make sure the estimate is stable.

  • warmup_iters (int) – number of iterations used to warm up the function. This is useful for functions that exhibit poor performance during the first few times they run (due to caches, autotuning, etc).

Returns

Dict[str, float] –

dictionary of the aggregated timing estimates.
”iterations”: number of iterations used to compute the estimated

time.

”mean”: averate time (sec) used to run the function. “median”: median time (sec) used to run the function. “min”: minimal time (sec) used to run the function. “max”: maximal time (sec) used to run the function. “stddev”: standard deviation of the time (sec) used to run the

function.

cvpods.utils.benchmark.benchmark.benchmark(func, bm_name: str, kwargs_list: List[Dict], *, num_iters: int = - 1, warmup_iters: int = 0) → None[source]

Benchmark the input function and print out the results.

Parameters

  • func (callable) – a closure that returns a function for benchmarking, where initialization can be done before the function to benchmark.

  • bm_name (str) – name of the benchmark to print out, e.g. “BM_UPDATE”.

  • kwargs_list (list) – a list of argument dict to pass to the function. The intput function will be timed separately for each argument dict.

  • num_iters (int) – number of iterations to run. Defaults to run until 0.5s.

  • warmup_iters (int) – number of iterations used to warm up the function.

Outputs:

For each argument dict, print out the time (in microseconds) required to run the function along with the number of iterations used to get the timing estimate. Example output:

BM_UPDATE_100 820 914 610 BM_UPDATE_1000 7655 8709 66 BM_UPDATE_10000 78062 81748 7 ——————————————————————-

cvpods.utils.benchmark.timer module

class cvpods.utils.benchmark.timer.Timer[source]

Bases: object

A timer which computes the time elapsed since the start/reset of the timer.

reset()[source]

Reset the timer.

pause()[source]

Pause the timer.

is_paused() → bool[source]
Returns

bool – whether the timer is currently paused

resume()[source]

Resume the timer.

seconds() → float[source]
Returns

(float) –

the total number of seconds since the start/reset of the

timer, excluding the time when the timer is paused.

cvpods.utils.distributed.comm module

This file contains primitives for multi-gpu communication. This is useful when doing distributed training.

cvpods.utils.distributed.comm.get_host_ip()[source]
cvpods.utils.distributed.comm.get_world_size() → int[source]
cvpods.utils.distributed.comm.get_rank() → int[source]
cvpods.utils.distributed.comm.get_local_rank() → int[source]
Returns

The rank of the current process within the local (per-machine) process group.

cvpods.utils.distributed.comm.get_local_size() → int[source]
Returns

The size of the per-machine process group, i.e. the number of processes per machine.

cvpods.utils.distributed.comm.is_main_process() → bool[source]
cvpods.utils.distributed.comm.synchronize()[source]

Helper function to synchronize (barrier) among all processes when using distributed training

cvpods.utils.distributed.comm.all_gather(data, group=None)[source]

Run all_gather on arbitrary picklable data (not necessarily tensors).

Parameters

  • data – any picklable object

  • group – a torch process group. By default, will use a group which contains all ranks on gloo backend.

Returns

list[data] – list of data gathered from each rank

cvpods.utils.distributed.comm.gather(data, dst=0, group=None)[source]

Run gather on arbitrary picklable data (not necessarily tensors).

Parameters

  • data – any picklable object

  • dst (int) – destination rank

  • group – a torch process group. By default, will use a group which contains all ranks on gloo backend.

Returns

list[data] –

on dst, a list of data gathered from each rank. Otherwise,

an empty list.

cvpods.utils.distributed.comm.all_reduce(data, op='sum')[source]
cvpods.utils.distributed.comm.shared_random_seed()[source]
Returns

int –

a random number that is the same across all workers.

If workers need a shared RNG, they can use this shared seed to create one.

All workers must call this function, otherwise it will deadlock.

cvpods.utils.distributed.comm.reduce_dict(input_dict, average=True)[source]

Reduce the values in the dictionary from all processes so that process with rank 0 has the reduced results.

Parameters

  • input_dict (dict) – inputs to be reduced. All the values must be scalar CUDA Tensor.

  • average (bool) – whether to do average or sum

Returns

a dict with the same keys as input_dict, after reduction.

cvpods.utils.env.collect_env module

cvpods.utils.env.collect_env.collect_env_info()[source]

cvpods.utils.env.env module

cvpods.utils.env.env.seed_all_rng(seed=None)[source]

Set the random seed for the RNG in torch, numpy and python.

Parameters

seed (int) – if None, will use a strong random seed.

Returns

seed (int) – used seed value.

cvpods.utils.env.env.setup_environment()[source]

Perform environment setup work. The default setup is a no-op, but this function allows the user to specify a Python source file or a module in the $cvpods_ENV_MODULE environment variable, that performs custom setup work that may be necessary to their computing environment.

cvpods.utils.env.env.setup_custom_environment(custom_module)[source]

Load custom environment setup by importing a Python source file or a module, and run the setup function.

cvpods.utils.file.download module

cvpods.utils.file.download.download(url: str, dir: str, *, filename: Optional[str] = None, progress: bool = True) → str[source]
Download a file from a given URL to a directory. If file exists, will not

overwrite the existing file.

Parameters

  • url (str) –

  • dir (str) – the directory to download the file

  • filename (str or None) – the basename to save the file. Will use the name in the URL if not given.

  • progress (bool) – whether to use tqdm to draw a progress bar.

Returns

str – the path to the downloaded file or the existing one.

cvpods.utils.file.file_io module

cvpods.utils.file.file_io.get_cache_dir(cache_dir: Optional[str] = None) → str[source]

Returns a default directory to cache static files (usually downloaded from Internet), if None is provided.

Parameters

  • cache_dir (None or str) – if not None, will be returned as is. If None, returns the default cache directory as:

  • $CVPODS_CACHE, if set (1)) –

  • otherwise ~/.torch/cvpods_cache (2)) –

cvpods.utils.file.file_io.file_lock(path: str)[source]

A file lock. Once entered, it is guaranteed that no one else holds the same lock. Others trying to enter the lock will block for 30 minutes and raise an exception.

This is useful to make sure workers don’t cache files to the same location.

Parameters

path (str) – a path to be locked. This function will create a lock named path + “.lock”

Examples:

>>> filename = "/path/to/file"
>>> with file_lock(filename):
        if not os.path.isfile(filename):
            do_create_file()
class cvpods.utils.file.file_io.PathHandler[source]

Bases: object

PathHandler is a base class that defines common I/O functionality for a URI protocol. It routes I/O for a generic URI which may look like “protocol://*” or a canonical filepath “/foo/bar/baz”.

class cvpods.utils.file.file_io.PathManager[source]

Bases: object

A class for users to open generic paths or translate generic paths to file names.

static open(path: str, mode: str = 'r') → IO[Any][source]

Open a stream to a URI, similar to the built-in open.

Parameters

path (str) – A URI supported by this PathHandler

Returns

file – a file-like object.

static copy(src_path: str, dst_path: str, overwrite: bool = False) → bool[source]

Copies a source path to a destination path.

Parameters

  • src_path (str) – A URI supported by this PathHandler

  • dst_path (str) – A URI supported by this PathHandler

  • overwrite (bool) – Bool flag for forcing overwrite of existing file

Returns

status (bool) – True on success

static get_local_path(path: str) → str[source]

Get a filepath which is compatible with native Python I/O such as open and os.path.

If URI points to a remote resource, this function may download and cache the resource to local disk.

Parameters

path (str) – A URI supported by this PathHandler

Returns

local_path (str) – a file path which exists on the local file system

static exists(path: str) → bool[source]

Checks if there is a resource at the given URI.

Parameters

path (str) – A URI supported by this PathHandler

Returns

bool – true if the path exists

static isfile(path: str) → bool[source]

Checks if there the resource at the given URI is a file.

Parameters

path (str) – A URI supported by this PathHandler

Returns

bool – true if the path is a file

static isdir(path: str) → bool[source]

Checks if the resource at the given URI is a directory.

Parameters

path (str) – A URI supported by this PathHandler

Returns

bool – true if the path is a directory

static ls(path: str) → List[str][source]

List the contents of the directory at the provided URI.

Parameters

path (str) – A URI supported by this PathHandler

Returns

List[str] – list of contents in given path

static mkdirs(path: str) → None[source]

Recursive directory creation function. Like mkdir(), but makes all intermediate-level directories needed to contain the leaf directory. Similar to the native os.makedirs.

Parameters

path (str) – A URI supported by this PathHandler

static rm(path: str) → None[source]

Remove the file (not directory) at the provided URI.

Parameters

path (str) – A URI supported by this PathHandler

static stat(path: str)[source]

get status of the file at the provided URI.

Parameters

path (str) – A URI supported by this PathHandler

static upload(local: str, remote: str)[source]

Upload the local file (not directory) to the specified remote URI.

Parameters

  • local (str) – path of the local file to be uploaded.

  • remote (str) – the remote s3uri.

static register_handler(handler: cvpods.utils.file.file_io.PathHandler) → None[source]

Register a path handler associated with handler._get_supported_prefixes URI prefixes.

Parameters

handler (PathHandler) –

cvpods.utils.imports module

cvpods.utils.imports.dynamic_import(config_name, config_path)[source]

Dynamic import a project.

Parameters

  • config_name (str) – module name

  • config_path (str) – the dir that contains the .py with this module.

Examples::
>>> root = "/path/to/your/retinanet/"
>>> project = root + "retinanet.res50.fpn.coco.800size.1x.mrcnn_sigmoid"
>>> cfg = dynamic_import("config", project).config
>>> net = dynamic_import("net", project)

cvpods.utils.memory module

cvpods.utils.memory.retry_if_cuda_oom(func)[source]

Makes a function retry itself after encountering pytorch’s CUDA OOM error. It will first retry after calling torch.cuda.empty_cache().

If that still fails, it will then retry by trying to convert inputs to CPUs. In this case, it expects the function to dispatch to CPU implementation. The return values may become CPU tensors as well and it’s user’s responsibility to convert it back to CUDA tensor if needed.

Parameters

func – a stateless callable that takes tensor-like objects as arguments

Returns

a callable which retries func if OOM is encountered.

Examples:

output = retry_if_cuda_oom(some_torch_function)(input1, input2)
# output may be on CPU even if inputs are on GPU

Note

  1. When converting inputs to CPU, it will only look at each argument and check if it has .device and .to for conversion. Nested structures of tensors are not supported.

  2. Since the function might be called more than once, it has to be stateless.

cvpods.utils.visualizer.colormap module

An awesome colormap for really neat visualizations. Copied from Detectron, and removed gray colors.

cvpods.utils.visualizer.colormap.colormap(rgb=False, maximum=255)[source]
Parameters

  • rgb (bool) – whether to return RGB colors or BGR colors.

  • maximum (int) – either 255 or 1

Returns

ndarray – a float32 array of Nx3 colors, in range [0, 255] or [0, 1]

cvpods.utils.visualizer.colormap.random_color(rgb=False, maximum=255)[source]
Parameters

  • rgb (bool) – whether to return RGB colors or BGR colors.

  • maximum (int) – either 255 or 1

Returns

ndarray – a vector of 3 numbers

cvpods.utils.visualizer.video_visualizer module

class cvpods.utils.visualizer.video_visualizer.VideoVisualizer(metadata, instance_mode=<ColorMode.IMAGE: 0>)[source]

Bases: object

__init__(metadata, instance_mode=<ColorMode.IMAGE: 0>)[source]
Parameters

metadata (MetadataCatalog) – image metadata.

draw_instance_predictions(frame, predictions)[source]

Draw instance-level prediction results on an image.

Parameters

  • frame (ndarray) – an RGB image of shape (H, W, C), in the range [0, 255].

  • predictions (Instances) – the output of an instance detection/segmentation model. Following fields will be used to draw: “pred_boxes”, “pred_classes”, “scores”, “pred_masks” (or “pred_masks_rle”).

Returns

output (VisImage) – image object with visualizations.

draw_sem_seg(frame, sem_seg, area_threshold=None)[source]
Parameters

  • sem_seg (ndarray or Tensor) – semantic segmentation of shape (H, W), each value is the integer label.

  • area_threshold (Optional[int]) – only draw segmentations larger than the threshold

draw_panoptic_seg_predictions(frame, panoptic_seg, segments_info, area_threshold=None, alpha=0.5)[source]

cvpods.utils.visualizer.visualizer module

class cvpods.utils.visualizer.visualizer.ColorMode[source]

Bases: enum.Enum

Enum of different color modes to use for instance visualizations.

IMAGE

Picks a random color for every instance and overlay segmentations with low opacity.

SEGMENTATION

Let instances of the same category have similar colors, and overlay them with high opacity. This provides more attention on the quality of segmentation.

IMAGE_BW

same as IMAGE, but convert all areas without masks to gray-scale. Only available for drawing per-instance mask predictions.

IMAGE = 0
SEGMENTATION = 1
IMAGE_BW = 2
class cvpods.utils.visualizer.visualizer.VisImage(img, scale=1.0)[source]

Bases: object

__init__(img, scale=1.0)[source]
Parameters

  • img (ndarray) – an RGB image of shape (H, W, 3).

  • scale (float) – scale the input image

save(filepath)[source]
Parameters

filepath (str) – a string that contains the absolute path, including the file name, where the visualized image will be saved.

get_image()[source]
Returns

ndarray –

the visualized image of shape (H, W, 3) (RGB) in uint8 type.

The shape is scaled w.r.t the input image using the given scale argument.

class cvpods.utils.visualizer.visualizer.Visualizer(img_rgb, metadata, scale=1.0, instance_mode=<ColorMode.IMAGE: 0>)[source]

Bases: object

__init__(img_rgb, metadata, scale=1.0, instance_mode=<ColorMode.IMAGE: 0>)[source]
Parameters

  • img_rgb – a numpy array of shape (H, W, C), where H and W correspond to the height and width of the image respectively. C is the number of color channels. The image is required to be in RGB format since that is a requirement of the Matplotlib library. The image is also expected to be in the range [0, 255].

  • metadata (MetadataCatalog) – image metadata.

draw_instance_predictions(predictions)[source]

Draw instance-level prediction results on an image.

Parameters

predictions (Instances) – the output of an instance detection/segmentation model. Following fields will be used to draw: “pred_boxes”, “pred_classes”, “scores”, “pred_masks” (or “pred_masks_rle”).

Returns

output (VisImage) – image object with visualizations.

draw_sem_seg(sem_seg, area_threshold=None, alpha=0.8)[source]

Draw semantic segmentation predictions/labels.

Parameters

  • sem_seg (Tensor or ndarray) – the segmentation of shape (H, W).

  • area_threshold (int) – segments with less than area_threshold are not drawn.

  • alpha (float) – the larger it is, the more opaque the segmentations are.

Returns

output (VisImage) – image object with visualizations.

draw_panoptic_seg_predictions(panoptic_seg, segments_info, area_threshold=None, alpha=0.7)[source]

Draw panoptic prediction results on an image.

Parameters

  • panoptic_seg (Tensor) – of shape (height, width) where the values are ids for each segment.

  • segments_info (list[dict]) – Describe each segment in panoptic_seg. Each dict contains keys “id”, “category_id”, “isthing”.

  • area_threshold (int) – stuff segments with less than area_threshold are not drawn.

Returns

output (VisImage) – image object with visualizations.

draw_dataset_dict(dic)[source]

Draw annotations/segmentaions in cvpods Dataset format.

Parameters

dic (dict) – annotation/segmentation data of one image, in cvpods Dataset format.

Returns

output (VisImage) – image object with visualizations.

overlay_instances(*, boxes=None, labels=None, masks=None, keypoints=None, assigned_colors=None, alpha=0.5)[source]
Parameters

  • boxes (Boxes, RotatedBoxes or ndarray) – either a Boxes, or an Nx4 numpy array of XYXY_ABS format for the N objects in a single image, or a RotatedBoxes, or an Nx5 numpy array of (x_center, y_center, width, height, angle_degrees) format for the N objects in a single image,

  • labels (list[str]) – the text to be displayed for each instance.

  • masks (masks-like object) –

    Supported types are:

    • structures.masks.PolygonMasks, structures.masks.BitMasks.

    • list[list[ndarray]]: contains the segmentation masks for all objects in one image.

      The first level of the list corresponds to individual instances. The second level to all the polygon that compose the instance, and the third level to the polygon coordinates. The third level should have the format of [x0, y0, x1, y1, …, xn, yn] (n >= 3).

    • list[ndarray]: each ndarray is a binary mask of shape (H, W).

    • list[dict]: each dict is a COCO-style RLE.

  • keypoints (Keypoint or array like) – an array-like object of shape (N, K, 3), where the N is the number of instances and K is the number of keypoints. The last dimension corresponds to (x, y, visibility or score).

  • assigned_colors (list[matplotlib.colors]) – a list of colors, where each color corresponds to each mask or box in the image. Refer to ‘matplotlib.colors’ for full list of formats that the colors are accepted in.

Returns

output (VisImage) – image object with visualizations.

overlay_rotated_instances(boxes=None, labels=None, assigned_colors=None)[source]
Parameters

  • boxes (ndarray) – an Nx5 numpy array of (x_center, y_center, width, height, angle_degrees) format for the N objects in a single image.

  • labels (list[str]) – the text to be displayed for each instance.

  • assigned_colors (list[matplotlib.colors]) – a list of colors, where each color corresponds to each mask or box in the image. Refer to ‘matplotlib.colors’ for full list of formats that the colors are accepted in.

Returns

output (VisImage) – image object with visualizations.

draw_and_connect_keypoints(keypoints)[source]

Draws keypoints of an instance and follows the rules for keypoint connections to draw lines between appropriate keypoints. This follows color heuristics for line color.

Parameters

keypoints (Tensor) – a tensor of shape (K, 3), where K is the number of keypoints and the last dimension corresponds to (x, y, probability).

Returns

output (VisImage) – image object with visualizations.

draw_text(text, position, *, font_size=None, color='g', horizontal_alignment='center', rotation=0)[source]
Parameters

  • text (str) – class label

  • position (tuple) – a tuple of the x and y coordinates to place text on image.

  • font_size (int, optional) – font of the text. If not provided, a font size proportional to the image width is calculated and used.

  • color – color of the text. Refer to matplotlib.colors for full list of formats that are accepted.

  • horizontal_alignment (str) – see matplotlib.text.Text

  • rotation – rotation angle in degrees CCW

Returns

output (VisImage) – image object with text drawn.

draw_box(box_coord, alpha=0.5, edge_color='g', line_style='-')[source]
Parameters

  • box_coord (tuple) – a tuple containing x0, y0, x1, y1 coordinates, where x0 and y0 are the coordinates of the image’s top left corner. x1 and y1 are the coordinates of the image’s bottom right corner.

  • alpha (float) – blending efficient. Smaller values lead to more transparent masks.

  • edge_color – color of the outline of the box. Refer to matplotlib.colors for full list of formats that are accepted.

  • line_style (string) – the string to use to create the outline of the boxes.

Returns

output (VisImage) – image object with box drawn.

draw_rotated_box_with_label(rotated_box, alpha=0.5, edge_color='g', line_style='-', label=None)[source]
Parameters

  • rotated_box (tuple) – a tuple containing (cnt_x, cnt_y, w, h, angle), where cnt_x and cnt_y are the center coordinates of the box. w and h are the width and height of the box. angle represents how many degrees the box is rotated CCW with regard to the 0-degree box.

  • alpha (float) – blending efficient. Smaller values lead to more transparent boxes.

  • edge_color – color of the outline of the box. Refer to matplotlib.colors for full list of formats that are accepted.

  • line_style (string) – the string to use to create the outline of the boxes.

  • label (string) – label for rotated box. It will not be rendered when set to None.

Returns

output (VisImage) – image object with box drawn.

draw_circle(circle_coord, color, radius=3)[source]
Parameters

  • circle_coord (list(int) or tuple(int)) – contains the x and y coordinates of the center of the circle.

  • color – color of the polygon. Refer to matplotlib.colors for a full list of formats that are accepted.

  • radius (int) – radius of the circle.

Returns

output (VisImage) – image object with box drawn.

draw_line(x_data, y_data, color, linestyle='-', linewidth=None, alpha=1.0)[source]
Parameters

  • x_data (list[int]) – a list containing x values of all the points being drawn. Length of list should match the length of y_data.

  • y_data (list[int]) – a list containing y values of all the points being drawn. Length of list should match the length of x_data.

  • color – color of the line. Refer to matplotlib.colors for a full list of formats that are accepted.

  • linestyle – style of the line. Refer to matplotlib.lines.Line2D for a full list of formats that are accepted.

  • linewidth (float or None) – width of the line. When it’s None, a default value will be computed and used.

  • alpha (float) – blending efficient. Smaller values lead to more transparent lines.

Returns

output (VisImage) – image object with line drawn.

draw_binary_mask(binary_mask, color=None, *, edge_color=None, text=None, alpha=0.5, area_threshold=4096)[source]
Parameters

  • binary_mask (ndarray) – numpy array of shape (H, W), where H is the image height and W is the image width. Each value in the array is either a 0 or 1 value of uint8 type.

  • color – color of the mask. Refer to matplotlib.colors for a full list of formats that are accepted. If None, will pick a random color.

  • edge_color – color of the polygon edges. Refer to matplotlib.colors for a full list of formats that are accepted.

  • text (str) – if None, will be drawn in the object’s center of mass.

  • alpha (float) – blending efficient. Smaller values lead to more transparent masks.

  • area_threshold (float) – a connected component small than this will not be shown.

Returns

output (VisImage) – image object with mask drawn.

draw_polygon(segment, color, edge_color=None, alpha=0.5)[source]
Parameters

  • segment – numpy array of shape Nx2, containing all the points in the polygon.

  • color – color of the polygon. Refer to matplotlib.colors for a full list of formats that are accepted.

  • edge_color – color of the polygon edges. Refer to matplotlib.colors for a full list of formats that are accepted. If not provided, a darker shade of the polygon color will be used instead.

  • alpha (float) – blending efficient. Smaller values lead to more transparent masks.

Returns

output (VisImage) – image object with polygon drawn.

get_output()[source]
Returns

output (VisImage) – the image output containing the visualizations added to the image.

cvpods.utils.file.serialize module

class cvpods.utils.file.serialize.PicklableWrapper(obj)[source]

Bases: object

Wrap an object to make it more picklable, note that it uses heavy weight serialization libraries that are slower than pickle. It’s best to use it only on closures (which are usually not picklable).

This is a simplified version of https://github.com/joblib/joblib/blob/master/joblib/externals/loky/cloudpickle_wrapper.py

cvpods.utils.dump.events module

cvpods.utils.dump.events.get_event_storage()[source]
Returns

The EventStorage object that’s currently being used. Throws an error if no :class`EventStorage` is currently enabled.

class cvpods.utils.dump.events.EventWriter[source]

Bases: object

Base class for writers that obtain events from EventStorage and process them.

write()[source]
close()[source]
class cvpods.utils.dump.events.JSONWriter(json_file, window_size=20)[source]

Bases: cvpods.utils.dump.events.EventWriter

Write scalars to a json file.

It saves scalars as one json per line (instead of a big json) for easy parsing.

Examples parsing such a json file:

$ cat metrics.json | jq -s '.[0:2]'
[
  {
    "data_time": 0.008433341979980469,
    "iteration": 20,
    "loss": 1.9228371381759644,
    "loss_box_reg": 0.050025828182697296,
    "loss_classifier": 0.5316952466964722,
    "loss_mask": 0.7236229181289673,
    "loss_rpn_box": 0.0856662318110466,
    "loss_rpn_cls": 0.48198649287223816,
    "lr": 0.007173333333333333,
    "time": 0.25401854515075684
  },
  {
    "data_time": 0.007216215133666992,
    "iteration": 40,
    "loss": 1.282649278640747,
    "loss_box_reg": 0.06222952902317047,
    "loss_classifier": 0.30682939291000366,
    "loss_mask": 0.6970193982124329,
    "loss_rpn_box": 0.038663312792778015,
    "loss_rpn_cls": 0.1471673548221588,
    "lr": 0.007706666666666667,
    "time": 0.2490077018737793
  }
]

$ cat metrics.json | jq '.loss_mask'
0.7126231789588928
0.689423680305481
0.6776131987571716
...
__init__(json_file, window_size=20)[source]
Parameters

  • json_file (str) – path to the json file. New data will be appended if the file exists.

  • window_size (int) – the window size of median smoothing for the scalars whose smoothing_hint are True.

write()[source]
close()[source]
class cvpods.utils.dump.events.TensorboardXWriter(log_dir: str, window_size: int = 20, **kwargs)[source]

Bases: cvpods.utils.dump.events.EventWriter

Write all scalars to a tensorboard file.

__init__(log_dir: str, window_size: int = 20, **kwargs)[source]
Parameters

  • log_dir (str) – the directory to save the output events

  • window_size (int) – the scalars will be median-smoothed by this window size

  • kwargs – other arguments passed to torch.utils.tensorboard.SummaryWriter(…)

write()[source]
close()[source]
class cvpods.utils.dump.events.CommonMetricPrinter(max_iter, window_size=20, **kwargs)[source]

Bases: cvpods.utils.dump.events.EventWriter

Print common metrics to the terminal, including iteration time, ETA, memory, all losses, and the learning rate.

To print something different, please implement a similar printer by yourself.

__init__(max_iter, window_size=20, **kwargs)[source]
Parameters

max_iter (int) – the maximum number of iterations to train. Used to compute ETA.

write()[source]
class cvpods.utils.dump.events.EventStorage(start_iter=0, window_size=20)[source]

Bases: object

The user-facing class that provides metric storage functionalities.

In the future we may add support for storing / logging other types of data if needed.

__init__(start_iter=0, window_size=20)[source]
Parameters

start_iter (int) – the iteration number to start with

put_image(img_name, img_tensor)[source]

Add an img_tensor to the _vis_data associated with img_name.

Parameters

  • img_name (str) – The name of the image to put into tensorboard.

  • img_tensor (torch.Tensor or numpy.array) – An uint8 or float Tensor of shape [channel, height, width] where channel is 3. The image format should be RGB. The elements in img_tensor can either have values in [0, 1] (float32) or [0, 255] (uint8). The img_tensor will be visualized in tensorboard.

clear_images()[source]

Delete all the stored images for visualization. This should be called after images are written to tensorboard.

put_scalar(name, value, smoothing_hint=True)[source]

Add a scalar value to the HistoryBuffer associated with name.

Parameters

smoothing_hint (bool) –

a ‘hint’ on whether this scalar is noisy and should be smoothed when logged. The hint will be accessible through EventStorage.smoothing_hints(). A writer may ignore the hint and apply custom smoothing rule.

It defaults to True because most scalars we save need to be smoothed to provide any useful signal.

put_scalars(*, smoothing_hint=True, **kwargs)[source]

Put multiple scalars from keyword arguments.

Examples

storage.put_scalars(loss=my_loss, accuracy=my_accuracy, smoothing_hint=True)

history(name)[source]
Returns

HistoryBuffer – the scalar history for name

histories()[source]
Returns

dict[name -> HistoryBuffer] – the HistoryBuffer for all scalars

latest()[source]
Returns

dict[name -> number] – the scalars that’s added in the current iteration.

latest_with_smoothing_hint()[source]

Similar to latest(), but the returned values are either the un-smoothed original latest value, or a median of the given window_size, depend on whether the smoothing_hint is True.

This provides a default behavior that other writers can use.

smoothing_hints()[source]
Returns

dict[name -> bool] –

the user-provided hint on whether the scalar

is noisy and needs smoothing.

step()[source]

User should call this function at the beginning of each iteration, to notify the storage of the start of a new iteration. The storage will then be able to associate the new data with the correct iteration number.

property vis_data
property iter
property iteration
name_scope(name)[source]
Yields

A context within which all the events added to this storage will be prefixed by the name scope.

cvpods.utils.dump.history_buffer module

class cvpods.utils.dump.history_buffer.HistoryBuffer(max_length: int = 1000000)[source]

Bases: object

Track a series of scalar values and provide access to smoothed values over a window or the global average of the series.

__init__(max_length: int = 1000000)[source]
Parameters

max_length – maximal number of values that can be stored in the buffer. When the capacity of the buffer is exhausted, old values will be removed.

update(value: float, iteration: float = None)[source]

Add a new scalar value produced at certain iteration. If the length of the buffer exceeds self._max_length, the oldest element will be removed from the buffer.

latest()[source]

Return the latest scalar value added to the buffer.

median(window_size: int)[source]

Return the median of the latest window_size values in the buffer.

avg(window_size: int)[source]

Return the mean of the latest window_size values in the buffer.

global_avg()[source]

Return the mean of all the elements in the buffer. Note that this includes those getting removed due to limited buffer storage.

values()[source]
Returns

list[(number, iteration)] – content of the current buffer.

cvpods.utils.dump.logger module

cvpods.utils.dump.logger.setup_logger(output=None, distributed_rank=0, *, color=True, name='cvpods', abbrev_name=None)[source]

Initialize the cvpods logger and set its verbosity level to “INFO”.

Parameters

  • output (str) – a file name or a directory to save log. If None, will not save log file. If ends with “.txt” or “.log”, assumed to be a file name. Otherwise, logs will be saved to output/log.txt.

  • name (str) – the root module name of this logger

  • abbrev_name (str) – an abbreviation of the module, to avoid long names in logs. Set to “” to not log the root module in logs. By default, will abbreviate “cvpods” to “c2” and leave other modules unchanged.

Returns

logging.Logger – a logger

cvpods.utils.dump.logger.log_first_n(lvl, msg, n=1, *, name=None, key='caller')[source]

Log only for the first n times.

Parameters

  • lvl (int) – the logging level

  • msg (str) –

  • n (int) –

  • name (str) – name of the logger to use. Will use the caller’s module by default.

  • key (str or tuple[str]) – the string(s) can be one of “caller” or “message”, which defines how to identify duplicated logs. For example, if called with n=1, key=”caller”, this function will only log the first call from the same caller, regardless of the message content. If called with n=1, key=”message”, this function will log the same content only once, even if they are called from different places. If called with n=1, key=(“caller”, “message”), this function will not log only if the same caller has logged the same message before.

cvpods.utils.dump.logger.log_every_n(lvl, msg, n=1, *, name=None)[source]

Log once per n times.

Parameters

  • lvl (int) – the logging level

  • msg (str) –

  • n (int) –

  • name (str) – name of the logger to use. Will use the caller’s module by default.

cvpods.utils.dump.logger.log_every_n_seconds(lvl, msg, n=1, *, name=None)[source]

Log no more than once per n seconds. :param lvl: the logging level :type lvl: int :param msg: :type msg: str :param n: :type n: int :param name: name of the logger to use. Will use the caller’s module by default. :type name: str

cvpods.utils.dump.logger.create_small_table(small_dict)[source]

Create a small table using the keys of small_dict as headers. This is only suitable for small dictionaries.

Parameters

small_dict (dict) – a result dictionary of only a few items.

Returns

str – the table as a string.

cvpods.utils.dump.logger.create_table_with_header(header_dict, headers=['category', 'AP'], min_cols=6)[source]

create a table with given header.

Parameters

  • header_dict (dict) –

  • headers (list) –

  • min_cols (int) –

Returns

str – the table as a string