Reviewing generated predictions to improve your model performance
Overview
Where an enrichment model has been built with a partially labeled data source, you can improve the model’s performance by reviewing current predictions and submitting additional labels within the Graft UI. By retraining the model with this additional information the updated model can be applied to related Entities updating predictions and their accuracy.
Within Graft we provide Active Learning to support the easy review of generated predictions and management of labels and their application to the existing enrichment model, simplifying your operational processes.
Ahead of starting active learning we recommend reviewing Enrichment Dashboard to become familiar with the UI and available options.
Prerequistes
Active Learning can be used when a user defined (Custom) Enrichment exists. A sample how-to guide for building an enrichment from a partially labeled data source can be found here.
Accessing Active Learning
To access Active Learning either Click on the Enrichments Tab from with the project and Click on the Enrichment you would like to work on OR Click on the Enrichment name within the Entity Dashboard you will be presented with the Enrichment Dashboard
- Click on CORRECT PREDICTIONS.
You will be presented with the following columns...
| Column | Description |
|---|---|
| High ROI | Indicator of a label which Graft believes will maximize the improvement in your model. Column is removed when High ROI Labels only is checked |
| Assigned Label | Drop down of available labels for the data set - default is unlabeled |
| Predicted Label | The predicted Label made by the current Enrichment model |
| Predicted Probability | the confidence score of the predicted label (larger the number the more confident Graft is in it predicted label) |
followed by the remainder of the entity fields.
TABLE FORMATTING TIPS
- Fields can be resized to facilitate easy display and review of the embedded field contents.
- For text fields hovering over the field will display a pop out if the text is too long to be displayed in the column
Filtering
By default the Active Learning UI provides 100 rows which are unlabelled with 5 rows per page. You can use the filter to select any subset of rows by selecting the Column, Operator (like, equal, not equal to) and filter value. This allows the user to target specific rows for review and update based on their preferences. Alternatively Check the High ROI Labels Only box to display those labels which Graft believes will provide the best improvement
For example filtering by the predicted label = Bad shows only those rows which can not be updated as needed
Updating a label
Once you have reviewed the Predicted Label and the embedded field to determine if the prediction is good or not, Graft provides 2 options for setting a label for a single row in the table. Graft maintains a count of changes on the right hand of the screen.
Accepting a prediction
If the prediction Graft has made is correct, Click on the Bullseye icon in the Predicted Label column. The Assigned Label column entry for that row will be updated
Once clicked the Assigned Label field is updated
If you wish to change the label Click on the Assigned Label drop down menu and select another Prediction option or return the label to Unlabeled
Assigned Label column update
From the Assigned Label drop down Select the appropriate label from those available.
Bulk updates
If you wish to make multiple updates to the presented labels Check all the required columns, when 2 or more rows are selected the BULK UPDATE button will become available
- Click on the button
- From the window you are presented with the option to Accept Predictions or select a single value from the available labels in the drop down list.
- Click UPDATE to make your changes
ACCEPTING PREDICTIONS
Accept predictions allows you to quickly set a number of rows to their predicted values, they do not need to be the same label value.
Note Selecting multiple rows with different predicted labels but then selecting a new label value from the drop down list will update all selected rows to that new value regardless of the original predicted label
Saving your label changes
When you have made at least one change the SAVE and DISCARD CHANGES buttons will become available. Until either button is Clicked your changes will remain Staged and not applied to the enrichment. These will remain available if your session is interrupted or you step away from your machine.
CHANGES BEST PRACTICES
We recommend that you periodically SAVE your changes. You are able to delete changes ahead of retraining the enrichment model if needed
- Clicking SAVE will present a summary of the changes and allow you to add a note which is saves along with the changes, which may be helpful to others to understand what progress has been made
- Click SAVE again to store the note and the changes OR Click CANCEL
Once saved the updated rows will be removed from the table and an entry will appear in the Recent Activity tab. Clicking on the Report icon will display the same change summary information as when you originally saved
Deleting label changes
Should you wish to delete a series of changes from earlier work from the Recent Activity tab Click on the Trash Can icon on the right of the appropriate row
POTENTIAL IMPACT ON YOUR ENRICHMENT MODEL
If you delete changes AFTER you have trained your Enrichment model you will be prompted to TRAIN ENRICHMENT again. It is recommended that this is done so that Entities are not using the incorrect version of the Enrichment.
Retraining your enrichment model
When you have saved your changes, you have the option to Retrain your enrichment model.