Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

learn.get_X_preds RAM Memory Spike #692

Open
lesego94 opened this issue Mar 12, 2023 · 4 comments
Open

learn.get_X_preds RAM Memory Spike #692

lesego94 opened this issue Mar 12, 2023 · 4 comments
Labels
fixed? under review Waiting for clarification, confirmation, etc

Comments

@lesego94
Copy link

lesego94 commented Mar 12, 2023 •
edited

My 15 GB GPU Crashes when making predictions. Upon further reading, (link attached) I found that the function get_X_preds does some very inefficient reloading of gigabytes of data to make predictions whereas this could be done one at a time at a fraction of the memory costs. This may have already been fixed in fastai, but the fix hasn't made it to tsai.
Please read the attached forum for details.

https://forums.fast.ai/t/learn-get-preds-memory-inefficiency-quick-fix/84029

Does anyone know how to get around this issue? or how I can load my model in batches perhaps?
@oguiza oguiza added the under review Waiting for clarification, confirmation, etc label Mar 14, 2023
@oguiza
Copy link
Contributor

oguiza commented Mar 14, 2023

Hello @lesego94,
I have resolved another problem (#695) that could potentially be responsible for the GPU memory spike. It would be great if you could install tsai from GitHub using:
pip install git+https://github.com/timeseriesAI/tsai.git

and verify if the problem still persists.

@oguiza oguiza added the fixed? label Mar 14, 2023
@lesego94
Copy link
Author

Hi Oguiza, appreciate you looking into this. It did not work. I realized I made a mistake earlier, the memory spike is occurring in my CPU ram, not GPU.

image

Let me give you some information about what I'm running. Im running the PatchTST model notebook 15_PatchTST_a_new_transformer_for_LTSF.ipynb using my own dataset, with 288,670 total parameters. The spike only occurs when I use:

learn = load_learner('models/patchTST.pt') scaled_preds, *_ = learn.get_X_preds(X[splits[1]])

image

@oguiza
Copy link
Contributor

oguiza commented Mar 14, 2023

Ok, I understand.
Have you pip installed tsai from the gh repo?
Here are a couple of things to test:

  • Try doing just X[splits[1]] (the issue might be this is too big). If that’s the case you could create a loop to cast all predictions.
  • Have you tried reducing the batch size parameter in get_X_preds? You can pass bs=1 to test it.

@oguiza oguiza changed the title learn.get_X_preds GPU Memory Spike learn.get_X_preds RAM Memory Spike Mar 14, 2023
@lesego94
Copy link
Author

Thanks Oguiza, I will try your suggestions and get back to you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
fixed? under review Waiting for clarification, confirmation, etc
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@oguiza @lesego94