Issue with trying this model using provided example
#1
by
violetxi
- opened
Using the example script resulted in the following error with model = model.to(torch.bfloat16) commented out, but got the following error:
in forward
past_key_values_length = past_key_values.get_usable_length(seq_length)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'DynamicCache' object has no attribute 'get_usable_length'. Did you mean: 'get_seq_length'?
And, when including model = model.to(torch.bfloat16), the following error occurs at this step:
RuntimeError: You can't move a model that has some modules offloaded to cpu or disk.
torch: 2.8.0
transformers: 4.57.1