LLM [2] Preparing the data
We clean the text: headers, language, blank lines, joining with <|endoftext|>.
We open the archive we downloaded in the previous post. The books we will train the model on are in the folder: gutenberg/data/raw/PG*_raw.txt
- 51,667 files
- 20 GB
- Metadata: metadata/metadata.csv
Code for this post: https://github.com/format37/gpt
I put config.yaml in the project root. Put your own paths in it. Mine are:
prepare_dataset:
metadata_folder: /home/alex/projects/gutenberg/metadata/metadata.csv
data_dir: /home/alex/projects/gutenberg/data/raw
target_dir: /home/alex/projects/gutenberg/data/targetOpen any PG*_raw.txt file.
The Project Gutenberg EBook of Journal des Goncourt (Deuxième série,
deuxième volume), by Edmond de Goncourt
This eBook is for the use of anyone anywhere at no cost and with
almost no restrictions whatsoever. You may copy it, give it away or
re-use it under the terms of the Project Gutenberg License included
with this eBook or online at www.gutenberg.org
Title: Journal des Goncourt (Deuxième série, deuxième volume)
Mémoires de la vie littéraire
Author: Edmond de Goncourt
Release Date: December 30, 2005 [EBook #17420]
Language: French
Character set encoding: UTF-8
*** START OF THIS PROJECT GUTENBERG EBOOK JOURNAL DES GONCOURT ***
Produced by Carlo Traverso, Mireille Harmelin and the
Online Distributed Proofreading Team of Europe. This file
was produced from images generously made available by the
Bibliothèque nationale de France (BnF/Gallica) at
http://gallica.bnf.fr.
JOURNAL DES GONCOURT
_Mémoires de la Vie Littéraire_
DEUXIÈME SÉRIE--DEUXIÈME VOLUME--TOME CINQUIÈME
1872-1877
PARIS, BIBLIOTHÈQUE-CHARPENTIER, 11, RUE DE GRENELLE.
1891
PRÉFACE
(_Réponse à Monsieur Renan_)
Monsieur Renan me faisait l'honneur de me dire, il y a des années, qu'une
lettre fausse avait été publiée par LE FIGARO, comme émanant de lui, et
que son dédain de l'imprimé était tel, qu'il n'avait pas réclamé.
Le monsieur Renan de l'année dernière, est vraiment bien changé.
A propos de vieilles conversations de 1870, rapportées dans mon Journal:
voici la lettre, que le _Petit Lannionnais_ publiait de l'auteur de la VIE
DE JÉSUS-CHRIST.
Paris, 26 novembre, 1890.
Ah! mon cher cousin, que je vous sais gré de vous indigner pour moi,
en ce temps de mensonge...One file is one book. The header contains the license. Then come the title and the paragraphs, separated by blank lines and three asterisks. We want the text to look the way it will look in a chat: paragraphs separated by a blank line.
Selecting the files we need:
- English texts
- Files that actually exist
from tqdm import tqdm
import os
import yaml
import pandas as pd
from cleanup import strip_headers
def file_list(config):
target_dir = config["target_dir"]
if not os.path.exists(target_dir):
os.makedirs(target_dir)
# Select only english texts
df = pd.read_csv(config["metadata_folder"])
print('Initial size:', len(df))
df = df[(df.language=="['en']") & (df.type=='Text')]
print('English texts:', len(df))
# Only existing files
files = os.listdir(config["data_dir"])
downloaded = [file.split("_")[0] for file in files]
df = df[df.id.isin(downloaded)]
print('Existing:', len(df))
# Compose file path
pathes = []
for name in df.id:
path = os.path.join(config['data_dir'], f"{name}_raw.txt")
pathes.append(path)
return pathesTo strip the license we need cleanup.py - make sure it is in your data/ folder. It originally comes from c-w/gutenberg - I removed the 100-line limit on the search for the *** END OF marker, so the ending is cut off from short books too.
- Collapse every run of blank lines into a single blank line
def collapse_blank_lines(text):
result = []
for line in text.split("\n"):
if line.strip():
# Has text in a line
result.append(line)
elif result and result[-1] != "":
# First empty string after the text
result.append("")
# Skip other empty strings
return "\n".join(result)- Join the books with the <|endoftext|> token, so the model knows that a document has ended and the next text is unrelated to it
- Write files of up to 500 MB, so we can train in parts
def combine_files(files, config):
max_size_mb = 500
separator = "<|endoftext|>"
target_dir = config['target_dir']
if not os.path.exists(target_dir):
os.makedirs(target_dir)
current_content = []
current_size = 0
file_counter = 1
for file_path in tqdm(files):
try:
with open(file_path, "r", encoding="utf-8") as file:
content = file.read()
except UnicodeDecodeError:
# Not UTF-8 (UTF-16, cp1252): skip the book
tqdm.write(f"Skipping {file_path}: not UTF-8")
continue
# Remove extra \ufeff garbage
content = content.replace("\ufeff", "")
# Remove license
content = strip_headers(content)
# Collapse blank lines
content = collapse_blank_lines(content)
# Write each 500 mb
estimated_size = len(content.encode("utf-8"))
if current_size + estimated_size > max_size_mb * 1024 * 1024:
target_file_path = os.path.join(
target_dir,
f"combined_{file_counter}.txt"
)
with open(target_file_path, "w", encoding="utf-8") as target_file:
target_file.write(separator.join(current_content))
file_counter += 1
current_content = [content]
current_size = estimated_size
else:
current_content.append(content)
current_size += estimated_size
# Write tail file
if current_content:
target_file_path = os.path.join(target_dir, f"combined_{file_counter}.txt")
with open(target_file_path, "w", encoding="utf-8") as target_file:
target_file.write(separator.join(current_content))
print(f"{file_counter} file(s) saved in {target_dir}")
def main():
with open("config.yaml") as f:
config = yaml.safe_load(f)["prepare_dataset"]
files = file_list(config)
combine_files(files, config)
if __name__=='__main__':
main()Run it:
python data/prepare_dataset.pyOn my machine it took about 23 minutes. I deliberately skipped parallelization and deduplication, to keep things simple where they would not matter much for the final quality of GPT-2 small.
Deduplication removes near-identical books. For example, the same book can come out in different editions, and both editions can end up in the dataset. Strictly speaking they differ character by character, but the text is almost identical. Counting by matching title + author, about 0.5% of our dataset are duplicates - the model will see a few Shakespeare plays 4 times instead of 2.
During the run you will see about 66 messages like
Skipping /home/alex/projects/gutenberg/data/raw/PG10511_raw.txt: not UTF-8This is fine. A few books are dropped to avoid encoding problems.
Let’s look at the result:
The Magna Carta
Contents
The Text of Magna Carta
Magna Carta 1215
The text of THE MAGNA CARTA
A note from Michael Hart, preparer of the 0.1 version.
This file contains a number of versions of the Magna Carta, some of
which were a little mangled in transit. I am sure our volunteers will
find and correct errors I didn’t catch, and that version 0.2 - 1.0 will
have significant improvements, as well as at least one more version in
Latin.
Version 1.0 may contain a dozen different versions.
The Text of Magna Carta
JOHN, by the grace of God King of England, Lord of Ireland, Duke of
Normandy and Aquitaine, and Count of Anjou, to his archbishops,
bishops, abbots, earls, barons, justices, foresters, sheriffs,
stewards, servants, and to all his officials and loyal subjects,
Greeting.
KNOW THAT BEFORE GOD, for the health of our soul and those of our
ancestors and heirs, to the honour of God, the exaltation of the holy
Church, and the better ordering...
...on the fifteenth day of June,
in the seventeenth year of our reign.
<|endoftext|> SENECA
APOCOLOCYNTOSIS
WITH AN ENGLISH TRANSLATION BY
W.H.D. ROUSE, M.A. LITT. D.
MCMXX
INTRODUCTION
This piece is ascribed to Seneca by ancientNote that the special token <|endoftext|> separates the books.
Summary:
- 39,250 books
- 30 files of up to 500 MB
- 14.3 GB in total
On to the next part - the tokenizer.