I am practicing data labeling models and below is my code. The purpose of this model is to give proper tags based on the reviews. I have successfully finished the model and ROC parts. I just want to know how could I launch this model on the raw dataset and get the output. (For example, if the review only mentioned politics, the number in the politics will be 1 and the rest of the columns will be 0). Thanks in advance :)
train, test= train_test_split(data, test_size=0.2, random_state=42)
list_classes = ["Politics", "Sports", "Business", "Fishion", "Industry", "IT",
"Entertainment", "International", "COVID", "National","War", "Advertisement", "Comic", "Other"]
train_text = train['clean_review']
test_text = test['clean_review']
all_text = pd.concat([train_text, test_text])
word_vectorizer = TfidfVectorizer(
sublinear_tf=True,
strip_accents='unicode',
analyzer='word',
token_pattern=r'w{1,}',
ngram_range=(1, 1),
max_features=10000)
word_vectorizer.fit(train_text)
train_word_features = word_vectorizer.transform(train_text)
test_word_features = word_vectorizer.transform(test_text)
char_vectorizer = TfidfVectorizer(
sublinear_tf=True,
strip_accents='unicode',
analyzer='char',
ngram_range=(2, 6),
max_features=50000)
char_vectorizer.fit(train_text)
train_char_features = char_vectorizer.transform(train_text)
test_char_features = char_vectorizer.transform(test_text)
train_features = hstack([train_char_features, train_word_features])
test_features = hstack([test_char_features, test_word_features])
scores = []
submission = pd.DataFrame.from_dict({'index': test['index']})
for list_classes in list_classes:
train_target = train[list_classes]
classifier = LogisticRegression(C=0.1, solver='sag')
cv_score = np.mean(cross_val_score(classifier, train_features, train_target, cv=3, scoring='roc_auc'))
scores.append(cv_score)
print('CV score for class {} is {}'.format(list_classes, cv_score))
classifier.fit(train_features, train_target)
submission[list_classes] = classifier.predict_proba(test_features)[:, 1]
print('Total CV score is {}'.format(np.mean(scores)))
与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…