Skip to content

Connected Components checkpoint file clean up #162

@Saravanan21

Description

@Saravanan21

After running connected components, there is a large amount of temporary memory left in the spark folders - mainly in "blockmgr-*" folders, as well as the last checkpoint (although that's much smaller). The blockmgr folders are deleted when spark shuts down, but checkpoint folder isn't. The spark GC doesn't otherwise seem to clean up the blockmgr, so if I run connected components several times it keeps growing linearly.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions