objwatch.wrappers package

Submodules

Module contents

class objwatch.wrappers.ABCWrapper[source]

Bases: ABC

Abstract base class for function wrappers to extend tracing and logging functionality.

__init__()[source]
abstract wrap_call(func_name: str, frame: FrameType) → str[source]

Process and format the function call information.

Parameters:

  • func_name (str) – Name of the function being called.

  • frame (FrameType) – The current stack frame.

Returns:

Formatted call message.

Return type:

str

abstract wrap_return(func_name: str, result: Any) → str[source]

Process and format the function return information.

Parameters:

  • func_name (str) – Name of the function returning.

  • result (Any) – The result returned by the function.

Returns:

Formatted return message.

Return type:

str

abstract wrap_upd(old_value: Any, current_value: Any) → Tuple[str, str][source]

Process and format the update information of a variable.

Parameters:

  • old_value (Any) – The old value of the variable.

  • current_value (Any) – The new value of the variable.

Returns:

Formatted old and new values.

Return type:

Tuple[str, str]

_extract_args_kwargs(frame: FrameType) → Tuple[List[Any], dict][source]

Extract positional and keyword arguments from the current frame.

Parameters:

frame (FrameType) – The current stack frame.

Returns:

Lists of positional and keyword arguments.

Return type:

Tuple[List[Any], dict]

_format_args_kwargs(args: List[Any], kwargs: dict) → str[source]

Format positional and keyword arguments into a string.

Parameters:

  • args (List[Any]) – List of positional arguments.

  • kwargs (dict) – Dictionary of keyword arguments.

Returns:

Formatted arguments string.

Return type:

str

_format_value(value: Any, is_return: bool = False) → str[source]

Format a value into a string.

Parameters:

  • value (Any) – The value to format.

  • is_return (bool) – Flag indicating if the value is a return value.

Returns:

Formatted value string.

Return type:

str

_format_return(result: Any) → str[source]

Format the return value of a function.

Parameters:

result (Any) – The result returned by the function.

Returns:

Formatted return message.

Return type:

str

_abc_impl = <_abc._abc_data object>
class objwatch.wrappers.BaseWrapper[source]

Bases: ABCWrapper

BaseWrapper implements the ABCWrapper abstract methods to provide basic logging functionality.

wrap_call(func_name: str, frame: FrameType) → str[source]

Format the function call information.

Parameters:

  • func_name (str) – Name of the function being called.

  • frame (FrameType) – The current stack frame.

Returns:

Formatted call message.

Return type:

str

wrap_return(func_name: str, result: Any) → str[source]

Format the function return information.

Parameters:

  • func_name (str) – Name of the function returning.

  • result (Any) – The result returned by the function.

Returns:

Formatted return message.

Return type:

str

wrap_upd(old_value: Any, current_value: Any) → Tuple[str, str][source]

Format the update information of a variable.

Parameters:

  • old_value (Any) – The old value of the variable.

  • current_value (Any) – The new value of the variable.

Returns:

Formatted old and new values.

Return type:

Tuple[str, str]

_abc_impl = <_abc._abc_data object>
class objwatch.wrappers.CPUMemoryWrapper[source]

Bases: ABCWrapper

CPUMemoryWrapper extends ABCWrapper to log memory statistics for the system’s CPU memory usage.

This class gathers memory statistics about the system’s memory usage using the psutil.virtual_memory() method. The following statistics are tracked, expressed in bytes:

Main metrics:

  • total: total physical memory (exclusive swap).

  • available: the memory that can be given instantly to processes without the system going into swap. This is calculated by summing different memory metrics that vary depending on the platform. It is supposed to be used to monitor actual memory usage in a cross platform fashion.

  • percent: the percentage usage calculated as (total - available) / total * 100.

Other metrics:

  • used: memory used, calculated differently depending on the platform and designed for informational purposes only. total - free does not necessarily match used.

  • free: memory not being used at all (zeroed) that is readily available; note that this doesn’t reflect the actual memory available (use available instead). total - used does not necessarily match free.

  • active (UNIX): memory currently in use or very recently used, and so it is in RAM.

  • inactive (UNIX): memory that is marked as not used.

  • buffers (Linux, BSD): cache for things like file system metadata.

  • cached (Linux, BSD): cache for various things.

  • shared (Linux, BSD): memory that may be simultaneously accessed by multiple processes.

  • slab (Linux): in-kernel data structures cache.

  • wired (BSD, macOS): memory that is marked to always stay in RAM. It is never moved to disk.

If you want to capture specific CPU memory metrics, you can configure the mem_types variable before starting the tracking. For example, to capture the total memory, available memory, and memory usage percentage, you can add the following code before instantiation of objwatch wrappers:

`python from objwatch.wrappers import CPUMemoryWrapper CPUMemoryWrapper.mem_types = ["total", "available", "percent"] `

For the latest help and more detailed information, please refer to the official psutil documentation at: https://psutil.readthedocs.io/en/latest/index.html#psutil.virtual_memory

__init__()[source]

Initialize the CPUMemoryWrapper with optional memory types to capture stats.

Parameters:

mem_types (List[str]) – A list of memory types to capture.

mem_types: List[str] = ['total', 'available', 'percent']
_capture_memory() → dict[source]

Capture the current system memory statistics.

Returns:

A dictionary of memory stats from psutil.

Return type:

dict

_format_memory(stats: dict) → str[source]

Format the memory statistics into a string.

Parameters:

stats (dict) – The memory stats to format.

Returns:

A formatted string representing the memory stats.

Return type:

str

wrap_call(func_name: str, frame: FrameType) → str[source]

Wrap the function call to log memory stats before the function is executed.

Parameters:

  • func_name (str) – Name of the function being called.

  • frame (FrameType) – The current stack frame.

Returns:

A string representing the formatted memory stats.

Return type:

str

wrap_return(func_name: str, result: Any) → str[source]

Wrap the function return to log memory stats after the function is executed.

Parameters:

  • func_name (str) – Name of the function returning.

  • result (Any) – The result returned by the function.

Returns:

A string representing the formatted memory stats.

Return type:

str

_abc_impl = <_abc._abc_data object>
wrap_upd(old_value: Any, current_value: Any) → Tuple[str, str][source]

Wrap the update of a variable to log memory stats after the update.

Parameters:

  • old_value (Any) – The old value of the variable.

  • current_value (Any) – The new value of the variable.

Returns:

Formatted old and new values with memory stats.

Return type:

Tuple[str, str]

class objwatch.wrappers.TensorShapeWrapper[source]

Bases: ABCWrapper

TensorShapeWrapper extends ABCWrapper to log the shapes of torch.Tensor objects.

__init__()[source]
wrap_call(func_name: str, frame: FrameType) → str[source]

Format the function call information, including tensor shapes if applicable.

Parameters:

  • func_name (str) – Name of the function being called.

  • frame (FrameType) – The current stack frame.

Returns:

Formatted call message.

Return type:

str

wrap_return(func_name: str, result: Any) → str[source]

Format the function return information, including tensor shapes if applicable.

Parameters:

  • func_name (str) – Name of the function returning.

  • result (Any) – The result returned by the function.

Returns:

Formatted return message.

Return type:

str

wrap_upd(old_value: Any, current_value: Any) → Tuple[str, str][source]

Format the update information of a variable, including tensor shapes if applicable.

Parameters:

  • old_value (Any) – The old value of the variable.

  • current_value (Any) – The new value of the variable.

Returns:

Formatted old and new values.

Return type:

Tuple[str, str]

_format_value(value: Any, is_return: bool = False) → str[source]

Format a value into a string, logging tensor shapes if applicable.

Parameters:

  • value (Any) – The value to format.

  • is_return (bool) – Flag indicating if the value is a return value.

Returns:

Formatted value string.

Return type:

str

_abc_impl = <_abc._abc_data object>
class objwatch.wrappers.TorchMemoryWrapper[source]

Bases: ABCWrapper

TorchMemoryWrapper extends ABCWrapper to log memory statistics from GPU.

This class is designed to gather memory statistics for the GPU memory allocator and track various memory metrics associated with memory allocation, reservation, and freeing. It utilizes the memory_stats function to fetch these statistics and organizes them for further use.

If you want to capture specific GPU memory metrics, you can configure the mem_types variable before starting the tracking. For example, to capture the current and peak allocated memory across all pools, you can add the following code before instantiation of objwatch wrappers:

`python from objwatch.wrappers import TorchMemoryWrapper TorchMemoryWrapper.mem_types = ["allocation.all.current", "allocation.all.peak"] `

mem_types define which statistics should be captured. The available statistics are:

  • "allocation.{all,large_pool,small_pool}.{current,peak,allocated,freed}": number of allocation requests received by the memory allocator.

  • "allocated_bytes.{all,large_pool,small_pool}.{current,peak,allocated,freed}": amount of allocated memory.

  • "segment.{all,large_pool,small_pool}.{current,peak,allocated,freed}": number of reserved segments from cudaMalloc().

  • "reserved_bytes.{all,large_pool,small_pool}.{current,peak,allocated,freed}": amount of reserved memory.

  • "active.{all,large_pool,small_pool}.{current,peak,allocated,freed}": number of active memory blocks.

  • "active_bytes.{all,large_pool,small_pool}.{current,peak,allocated,freed}": amount of active memory.

  • "inactive_split.{all,large_pool,small_pool}.{current,peak,allocated,freed}": number of inactive, non-releasable memory blocks.

  • "inactive_split_bytes.{all,large_pool,small_pool}.{current,peak,allocated,freed}": amount of inactive, non-releasable memory.

For these statistics, values are broken down as follows.

Pool type:

  • all: combined statistics across all memory pools.

  • large_pool: statistics for the large allocation pool (as of October 2019, for size >= 1MB allocations).

  • small_pool: statistics for the small allocation pool (as of October 2019, for size < 1MB allocations).

Metric type:

  • current: current value of this metric.

  • peak: maximum value of this metric.

  • allocated: historical total increase in this metric.

  • freed: historical total decrease in this metric.

In addition to the core statistics, torch also provide some simple event counters:

  • "num_alloc_retries": number of failed cudaMalloc calls that result in a cache flush and retry.

  • "num_ooms": number of out-of-memory errors thrown.

  • "num_sync_all_streams": number of synchronize_and_free_events calls.

  • "num_device_alloc": number of GPU allocation calls. This includes both cuMemMap and cudaMalloc.

  • "num_device_free": number of GPU free calls. This includes both cuMemUnmap and cudaFree.

The caching allocator can be configured via ENV to not split blocks larger than a defined size (see Memory Management section of the Gpu Semantics documentation). This helps avoid memory fragmentation but may have a performance penalty. Additional outputs to assist with tuning and evaluating impact:

  • "max_split_size": blocks above this size will not be split.

  • "oversize_allocations.{current,peak,allocated,freed}": number of over-size allocation requests received by the memory allocator.

  • "oversize_segments.{current,peak,allocated,freed}": number of over-size reserved segments from cudaMalloc().

The caching allocator can be configured via ENV to round memory allocations in order to reduce fragmentation. Sometimes the overhead from rounding can be higher than the fragmentation it helps reduce. The following stat can be used to check if rounding adds too much overhead:

  • "requested_bytes.{all,large_pool,small_pool}.{current,peak,allocated,freed}": memory requested by client code, compare this with allocated_bytes to check if allocation rounding adds too much overhead.

For the latest help and more detailed information, please refer to the official PyTorch documentation at: https://pytorch.org/docs/stable/generated/torch.cuda.memory_stats.html#torch-cuda-memory-stats

__init__()[source]

Initialize the TorchMemoryWrapper with optional memory types to capture stats.

Parameters:

mem_types (List[str]) – A list of memory types to capture.

mem_types: List[str] = ['allocation.all.current', 'allocation.all.peak']
_capture_memory() → dict[source]

Capture the current GPU memory statistics.

Returns:

A dictionary of GPU memory stats.

Return type:

dict

_format_memory(stats: dict) → str[source]

Format the memory statistics into a string.

Parameters:

stats (dict) – The memory stats to format.

Returns:

A formatted string representing the memory stats.

Return type:

str

wrap_call(func_name: str, frame: FrameType) → str[source]

Wrap the function call to log memory stats before the function is executed.

Parameters:

  • func_name (str) – Name of the function being called.

  • frame (FrameType) – The current stack frame.

Returns:

A string representing the formatted memory stats.

Return type:

str

wrap_return(func_name: str, result: Any) → str[source]

Wrap the function return to log memory stats after the function is executed.

Parameters:

  • func_name (str) – Name of the function returning.

  • result (Any) – The result returned by the function.

Returns:

A string representing the formatted memory stats.

Return type:

str

_abc_impl = <_abc._abc_data object>
wrap_upd(old_value: Any, current_value: Any) → Tuple[str, str][source]

Wrap the update of a variable to log memory stats after the update.

Parameters:

  • old_value (Any) – The old value of the variable.

  • current_value (Any) – The new value of the variable.

Returns:

Formatted old and new values with memory stats.

Return type:

Tuple[str, str]