Skip to content

关于推理速度的问题 #35

Open
@Fdioa

Description

@Fdioa

为什么在推理时,reason-in-documents模块推理的速度会比主模块的慢很多?
下面是qwq_32B_q4在48G显存上推理的速度:
————————————————————————————————————————————————————————————————————————
Processed prompts: 99%|██████████████████████████████████████████▋| 497/500 [39:28<00:11, 3.78s/it, est. speed input: 78.85 toks/s, output: 75.03 toks/s]
Processed prompts: 100%|██████████████████████████████████████████▊| 498/500 [39:33<00:08, 4.30s/it, est. speed input: 78.83 toks/s, output: 75.43 toks/s]
Processed prompts: 100%|██████████████████████████████████████████▉| 499/500 [39:37<00:04, 4.23s/it, est. speed input: 78.85 toks/s, output: 75.95 toks/s]
Processed prompts: 100%|███████████████████████████████████████████| 500/500 [39:42<00:00, 4.25s/it, est. speed input: 78.87 toks/s, output: 76.87 toks/s]
Processed prompts: 100%|███████████████████████████████████████████| 500/500 [39:42<00:00, 4.76s/it, est. speed input: 78.87 toks/s, output: 76.87 toks/s]
Generation completed, processing outputs...
Sequence marked as complete.
—————————————————————————————————————————————————————————————————————————
Processed prompts: 71%|██████████████████████████▏ | 337/477 [12:31:19<5:40:32, 145.95s/it, est. speed input: 122.79 toks/s, output: 4.98 toks/s]
Processed prompts: 71%|██████████████████████████▏ | 338/477 [12:32:45<4:56:14, 127.87s/it, est. speed input: 122.85 toks/s, output: 4.98 toks/s]
Processed prompts: 71%|██████████████████████████▎ | 339/477 [12:34:00<4:17:32, 111.98s/it, est. speed input: 123.06 toks/s, output: 5.00 toks/s]
Processed prompts: 71%|██████████████████████████▎ | 340/477 [12:36:09<4:27:20, 117.08s/it, est. speed input: 123.17 toks/s, output: 5.00 toks/s]
Processed prompts: 71%|██████████████████████████▍ | 341/477 [12:39:59<5:42:18, 151.02s/it, est. speed input: 122.86 toks/s, output: 4.99 toks/s]
Processed prompts: 72%|██████████████████████████▌ | 342/477 [12:41:00<4:38:41, 123.87s/it, est. speed input: 123.09 toks/s, output: 5.00 toks/s]

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions