Assuming the input and output have fixed lengths of p and d respectively, how should I perform profiling and auto-search?
In the profile_one_cycle function within run_llama3.py, how should I set the total_batch_sizes for the prefill and decode phases?
|
total_batch_sizes = [128, 256, 384, 512, 640, 768, 896, 1024, 1152, 1280, 1408, 1536, 1664, 1792, 1920, 2048] |
|
total_batch_sizes = [128, 256, 384, 512, 640] |
When profiling the decode phase, should I set input_length to p and output_length to d ?
|
input_length = 1024 |
|
output_length = 0 |
In new_search.py, how do I determine the values for global_batch_size, decode_batch_size, and seq_len?
|
global_batch_size = 3072 |
|
decode_batch_size = 1280 |
|
# global_batch_size = 2048 |
|
# decode_batch_size = 640 |
|
>>>>>>> origin/master |
|
# global_batch_size = 1024 |
|
# decode_batch_size = 384 |
|
|
|
seq_len = 1024 |
I'm looking forward to your reply very much.
Assuming the input and output have fixed lengths of p and d respectively, how should I perform profiling and auto-search?
In the profile_one_cycle function within run_llama3.py, how should I set the total_batch_sizes for the prefill and decode phases?
Nanoflow/entry/run_llama3.py
Line 199 in f179a90
Nanoflow/entry/run_llama3.py
Line 208 in f179a90
When profiling the decode phase, should I set input_length to p and output_length to d ?
Nanoflow/entry/run_llama3.py
Lines 211 to 212 in f179a90
In new_search.py, how do I determine the values for global_batch_size, decode_batch_size, and seq_len?
Nanoflow/auto_search/new_search.py
Lines 21 to 29 in f179a90
I'm looking forward to your reply very much.