issue with “weights” masks always zero-ed out during training #2751
Description
Is there an existing issue for this?
- I have searched the existing issues
Bug description
Hi, I observe this behavior during DLC training that I have yet to understand. I was wondering if you could clarify.
Before line 162 in this file: <DLC environment>/lib/python3.10/site-packages/deeplabcut/pose_estimation_pytorch/models/target_generators/heatmap_targets.py
weights was an all-one tensors.
# heatmap (batch_size, height, width, num_kpts)
# coords (batch_size, num_kpts, num_individuals, 3)
for b in range(batch_size):
for heatmap_idx, group_keypoints in enumerate(coords[b]):
for keypoint in group_keypoints:
# FIXME: Gradient masking weights should be parameters
if keypoint[-1] == 0:
# full gradient masking
weights[b, heatmap_idx] = 0.0
elif keypoint[-1] == -1:
# full gradient masking
weights[b, heatmap_idx] = 0.0
elif keypoint[-1] > 0:
# keypoint visible
self.update(
heatmap=heatmap[b, :, :, heatmap_idx],
grid=grid,
keypoint=keypoint[..., :2],
locref_map=self.get_locref(locref_map, b, heatmap_idx),
locref_mask=self.get_locref(locref_mask, b, heatmap_idx),
)
But after this code block, weights seems to be all zero-ed out to be an all-zero tensors, leading to difficulty in training the model.
The problem seems to be that the innermost loop is iterating through each row in this group_keypoints 2-D array. For example,
group_keypoints
array([[ 74.45522, 308.99606, 2. ],
[322.70728, 156.14667, 2. ],
[ -1. , -1. , -1. ],
[ -1. , -1. , -1. ],
[ -1. , -1. , -1. ],
[ -1. , -1. , -1. ],
[ -1. , -1. , -1. ],
[ -1. , -1. , -1. ],
[ -1. , -1. , -1. ],
[ -1. , -1. , -1. ]], dtype=float32)
(where its shape is (10, 3) – 10 animals maximum)
Then at least one of the iteration will have keypoint[-1] == -1 and if so, the entire section of weights will be zero-ed out. This doesn’t seem to be the intended behavior of masking. Could you please double check? Thanks.
The DLC version is: Loading DLC 3.0.0rc4...
Operating System
RH9
DeepLabCut version
The DLC version is: Loading DLC 3.0.0rc4...
DeepLabCut mode
multi animal
Device type
gpu
Steps To Reproduce
No response
Relevant log output
No response
Anything else?
No response
Code of Conduct
- I agree to follow this project's Code of Conduct