python - Data pre-processing steps with different features

Question

question from:https://stackoverflow.com/questions/65864813/data-pre-processing-steps-with-different-features

Answer 1

Your error seems to try to concat arrays and series.

I'm not familiar with pipeline and columntransformer, so I may be mistaken ; it seems though that it doesn't capture the feature names from CountVectorizer, so it won't do any good to have an unlabelled dataframe : maybe you could stick to numpy arrays. If I'm mistaken, it should be easy enough to jump from np.array to dataframe anyway...

So, you could do :

df_train = np.append(
  X_train, #this is an array
  np.array(y_train).reshape(len(y_train),1), #convert the Serie to numpy array of correct shape
  axis=1)
print(df_train)

[[1 0 1 0 0 1 0 1 0 1 1 0 1]
 [0 1 0 1 1 0 1 0 1 1 0 1 1]]

Hope this helps (though as I said, I'm not familiar with these sklearn libraries...)

EDIT

Something more complete and without those pipelines (which I'm not sure are needed anyway) ; it is failing on my computer because of the input dataset, but you may have more success with your complete dataset.

df = pd.DataFrame(
        [["an example of text", 0, 0, 0, 0],
         ["ANOTHER example of text", 1, 1, 0, 1],
         ["What's happening?Let's talk at 5", 1, 0, 1, 1]
        ],
        columns=["Text", "is_it_capital?", "is_it_upper?", "contains_num?", "Label"]
        )

X=df[['Text','is_it_capital?', 'is_it_upper?', 'contains_num?']]
y=df['Label']

X_train, X_test, y_train, y_test  = train_test_split(X, y, test_size=0.25, random_state=40)

cv = CountVectorizer()

X_train = (
        pd.DataFrame(
                cv.fit_transform(X_train['Text']).toarray(),
                columns=cv.get_feature_names(),
                index=X_train.index
                ) #This way you keep the labels/indexes in a dataframe format
        .join(X_train.drop('Text', axis=1)) #add your previous 'get_dummies' columns
        )

X_test = (
        pd.DataFrame(
                cv.transform(X_test['Text']).toarray(),
                columns=cv.get_feature_names(),
                index=X_test.index
                )
        .join(X_test.drop('Text', axis=1))
        )

#Then compute your regression directly :
lr = LogisticRegression()
lr = lr.fit(X_train, y_train)
y_pred = lr.predict(X_test)