Crafting Your Own Dataset For Instruction Fine Tuning

Fine-Tuning Llama2 in Google Colab: A Step-by-Step Guide (Part 2)

Crafting Your Own Dataset for Instruction Fine-Tuning

Loading Dataset Model and Tokenizer

To begin, let's load the necessary dataset model and tokenizer. In this example, we'll use the guanaco-llama2-1k dataset from the Hugging Face hub, which contains 1000 samples processed for compatibility with Llama 2.

 import transformers from datasets import load_dataset  # Load the dataset dataset = load_dataset("guanaco/llama2-1k")  # Load the tokenizer tokenizer = transformers.AutoTokenizer.from_pretrained("guanaco/llama2-1k") 

How to Custom-Create Your Own Dataset

If you prefer to create your own custom dataset for instruction fine-tuning, you can follow these steps:

1. Gather your own text data with instructions.
2. Preprocess the data by cleaning and tokenizing it.
3. Organize the preprocessed data into a format compatible with Hugging Face datasets.
4. Create a Hugging Face dataset object and upload it to the hub.

Lengths lenxinput_ids for x in tokenized_train_dataset lengths.

To determine the lengths of the input IDs for the tokenized training dataset, you can use the following code:

 import numpy as np from transformers import AutoTokenizer  # Tokenize the training dataset tokenized_train_dataset = tokenizer(train_dataset["text"])  # Get the lengths of the input IDs lengths = [len(x["input_ids"]) for x in tokenized_train_dataset]  # Print the lengths print(lengths)