The Optimize tab allows you to test other model configurations for your App, as part of building your App Graft selects the most appropriate model based on the data type, however, Graft supports access to a number of models and you are able to select other models to try based on speed, performance, accuracy and price (Graft provides access to OpenAI and Cohere models which charge through your personal account rather than Graft) to see which is most appropriate for your particular use case. Depending on your use case you may find that a smaller (and perhaps cheaper) model provides adequate results and is faster.
Once you have made your evaluation you are able to select your preferred model and Graft will reprocess your data using that selection.
Optimize is also helpful when new models are introduced and you wish to compare their performance against the model you currently have in production. Again, Graft facilitates the simple switching of models based on your needs in production without the need to recode your environment
Selecting Models to Test
Once your App has been built you will be presented with the TRY IT screen
- Click on OPTIMIZE to switch to test new models
You will be presented with a list of all models currently supported by Graft for the particular modality you are working with (i.e. if you are searching docs Graft will only present Models for text).
The first model in the list will be the model currently being used by your App and is selected by default (in green)
Each model listed will show a short description followed by indications of Speed, Quality and Price
Graft will automatically check potential model options, you can select/deselect models based on your needs. As the models are selected they move to the top of the model list.
You can select up to 3 further models to compare against the current model (4 Total)
Once you have made your selections click RUN TESTS
Graft will begin processing each of your selected models using a sample of the data.
Note the Optimization tests are based on a 100 record sample of your data, NOT the full data set
As soon as 2 or more models have been processed the screen will update and will allow you to begin testing. Large models can take a few minutes to process a sample dataset, but will return results of recent query when complete.
The Current model is shown in a green tile. Each tile provides summary model information from the prior screen and an estimate of how long processing your entire data set would take and the option to SELECT MODEL (only available once the sample processing is complete)
Testing your models
To check the relative performance of your selected models we need to add in a search query (text or image depending on your use case)
- Enter some test text in the Search bar
- Optional: Enable Rerank
- Click SEARCH
Graft will return results for each model which has completed processing.
Results are shown relative to the current default model showing the search field (in this case product_description) and the ranking 1 thru 10. Each model is then compared showing its ranking of the same search field value. Where the test result is not in the results from a new model "Not in Top 10" is displayed.
In addition to the ranking value the similarity score is provided for those in the top 10 for each model to help understand how close to the test rest the result is. Ie it is possible to be ranked #1 but still be 'not similar' to the test text
Selecting a new Model
Once you have completed your testing you are able to either return to the TRY IT or API tabs retaining the current model or you may click on SELECT MODEL of your chosen model. Graft will then process 100 rows of data and return you to the TRY IT tab.
You are able to return to the Optimize tab as many times as you need to complete testing of any model against the current model.
Once you are happy with your model selection click on SYNC DATA to process your entire data set with your preferred model