WARNING: ADVANCED TUTORIAL
Hyperparameters are adjustable settings that determine the behavior of the language model during training and inference.
Changing these hyperparameters can affect the quality and diversity of the model's output, and can be useful for fine-tuning the model to specific tasks or applications.
The following hyperparameters are adjustable in GPT-3, GPT-3.5, and GPT-4:
Think of it as how "wild" or "tame" you want your AI friend to be when talking.
A higher temperature makes the AI more random and creative, while a lower one makes it more focused and sensible.
Example prompt:
<aside> 💡 "What should I do on a sunny day?"
</aside>
Temperature: 2.0 (high) Hypothetical response: "Ride a giant ice cream cone to the moon and have a picnic with aliens!"
Explanation: The AI comes up with a very random and creative idea because of the high temperature.
Temperature: 0.5 (low) Hypothetical response: "You can go outside and play with your friends, ride a bike, or have a picnic in the park."
Explanation: The AI gives a more focused and sensible answer because of the low temperature.
Accepted value: 0.0001 to infinity
Imagine your AI friend has a list of words it can use to reply. Top K tells it how many words it can pick from the top of that list.
A smaller Top K makes the AI more focused, while a larger Top K lets it be more random.
Example prompt:
<aside> 💡 "What is your favorite animal?"
</aside>
Top K: 5 (low) Hypothetical response: "My favorite animal is a dog."
Explanation: The AI chooses from a smaller set of words, so the answer is more focused and common.
Top K: 50 (high) Hypothetical response: "My favorite animal is a pygmy marmoset."
Explanation: The AI chooses from a larger set of words, making the answer more diverse and less common.
Accepted value: 0 to infinity
Top P is like a filter for your AI friend's words. It only picks words that have a combined chance of being picked that is less than or equal to the Top P value.
A higher Top P allows more variety, while a lower Top P makes the AI more focused.
Example prompt:
<aside> 💡 "What's your favorite thing to do at the park?"
</aside>
Top P: 0.95 (high) Hypothetical response: "I love to feed the ducks, explore hidden paths, and look for unusual insects under rocks!"
Explanation: The AI has more variety in its answer because of the high Top P value.
Top P: 0.5 (low) Hypothetical response: "I like to play on the swings."
Explanation: The AI gives a more focused and common answer due to the low Top P value.
Accepted value: 0 to 1
Imagine your AI friend talks a lot, but you don't want it to say the same thing over and over. Repetition Penalty helps you with that.
A higher penalty makes the AI avoid repeating words, while a lower penalty lets the AI repeat words more often.
Example prompt:
<aside> 💡 "Tell me a story about a cat."
</aside>
Repetition Penalty: 1.0 (low) Hypothetical response: "There was a cat. The cat was fluffy. The cat liked to chase its tail. The cat was happy."
Explanation: The AI repeats the word "cat" often because of the low repetition penalty.
Repetition Penalty: 2.0 (high) Hypothetical response: "Once upon a time, a fluffy feline lived in a cozy little house. This furry creature enjoyed chasing its tail and lived a joyful life."
Explanation: The AI avoids repeating the word "cat" and uses different words to describe it because of the high repetition penalty.
Accepted value: -infinity to infinity