python - sklearn, joblib: Loading large model causes memory error

Question

I have trained and saved an sklearn RandomForestClassifier in the following way:

from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import make_classification
X, y = make_classification(n_samples=1000, n_features=4, n_informative=2, n_redundant=0, random_state=0, shuffle=False)
clf = RandomForestClassifier(max_depth=2, random_state=0).fit(X, y)

import joblib
joblib.dump(clf, "D:/mymodel.gz", compress=3)

The final model (as saved on HDD) is about 6 GB large.

When I try to load the model again using:

clf_loaded = joblib.load("D:/mymodel.gz")
pred = clf_loaded.predict(X)

I get a memory error even though my available RAM is just under 60 GB.

It seems that joblib.load() requires a huge amount of RAM in excess of the original file size (6 GB).

Q: What are alternative options to save an sklearn model locally and load it again without consuming massive RAM space?

As far as I know, pickle will not outperform joblib and is per docs not recommended for sklearn models.

I checked some other questions such as this, this, this, and this but could not find a workable solution.

question from:https://stackoverflow.com/questions/66067303/sklearn-joblib-loading-large-model-causes-memory-error

Answer 1

Waitting for answers