|
|
|
|
|
|
|
|
To run the scikit-learn examples make sure you have installed the following library: |
|
|
|
|
|
```bash |
|
|
pip install -U scikit-learn |
|
|
``` |
|
|
|
|
|
The metrics in `evaluate` can be easily integrated with an Scikit-Learn estimator or [pipeline](https://scikit-learn.org/stable/modules/generated/sklearn.pipeline.Pipeline.html |
|
|
|
|
|
However, these metrics require that we generate the predictions from the model. The predictions and labels from the estimators can be passed to `evaluate` mertics to compute the required values. |
|
|
|
|
|
```python |
|
|
import numpy as np |
|
|
np.random.seed(0) |
|
|
import evaluate |
|
|
from sklearn.compose import ColumnTransformer |
|
|
from sklearn.datasets import fetch_openml |
|
|
from sklearn.pipeline import Pipeline |
|
|
from sklearn.impute import SimpleImputer |
|
|
from sklearn.preprocessing import StandardScaler, OneHotEncoder |
|
|
from sklearn.linear_model import LogisticRegression |
|
|
from sklearn.model_selection import train_test_split |
|
|
``` |
|
|
|
|
|
Load data from https://www.openml.org/d/40945: |
|
|
|
|
|
```python |
|
|
X, y = fetch_openml("titanic", version=1, as_frame=True, return_X_y=True) |
|
|
``` |
|
|
|
|
|
Alternatively X and y can be obtained directly from the frame attribute: |
|
|
|
|
|
```python |
|
|
X = titanic.frame.drop('survived', axis=1) |
|
|
y = titanic.frame['survived'] |
|
|
``` |
|
|
|
|
|
We create the preprocessing pipelines for both numeric and categorical data. Note that pclass could either be treated as a categorical or numeric feature. |
|
|
|
|
|
```python |
|
|
numeric_features = ["age", "fare"] |
|
|
numeric_transformer = Pipeline( |
|
|
steps=[("imputer", SimpleImputer(strategy="median")), ("scaler", StandardScaler())] |
|
|
) |
|
|
|
|
|
categorical_features = ["embarked", "sex", "pclass"] |
|
|
categorical_transformer = OneHotEncoder(handle_unknown="ignore") |
|
|
|
|
|
preprocessor = ColumnTransformer( |
|
|
transformers=[ |
|
|
("num", numeric_transformer, numeric_features), |
|
|
("cat", categorical_transformer, categorical_features), |
|
|
] |
|
|
) |
|
|
``` |
|
|
|
|
|
Append classifier to preprocessing pipeline. Now we have a full prediction pipeline. |
|
|
|
|
|
```python |
|
|
clf = Pipeline( |
|
|
steps=[("preprocessor", preprocessor), ("classifier", LogisticRegression())] |
|
|
) |
|
|
|
|
|
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0) |
|
|
|
|
|
clf.fit(X_train, y_train) |
|
|
y_pred = clf.predict(X_test) |
|
|
``` |
|
|
|
|
|
As `Evaluate` metrics use lists as inputs for references and predictions, we need to convert them to Python lists. |
|
|
|
|
|
|
|
|
```python |
|
|
|
|
|
|
|
|
y_test = y_test.tolist() |
|
|
y_pred = y_pred.tolist() |
|
|
|
|
|
|
|
|
|
|
|
accuracy_metric = evaluate.load("accuracy") |
|
|
accuracy = accuracy_metric.compute(references=y_test, predictions=y_pred) |
|
|
print("Accuracy:", accuracy) |
|
|
|
|
|
``` |
|
|
|
|
|
You can use any suitable `evaluate` metric with the estimators as long as they are compatible with the task and predictions. |
|
|
|