This is just a small fun project to test out continuous pretraining. I wanted to see if I could make a small model output stories in the style of one of my favorite authors, Lord Dunsany.
It's a rather small dataset, 52 short stories from "Gods of Pegana" and "Time and the Gods". I did not include "Book of Wonders", at least not this time.

This gemma2 model was trained 2x faster with Unsloth and Huggingface's TRL library.

Cherry-picked, 1 out of 3. Input in bold

It was such, that in the deep desert, where few men dared journey with their caravans, there was one man who had not journeyed with them. And this man's name was the Ptah-hotep, and he had been a prophet of the gods before the other prophets had appeared.

And it had once been the will of Thoth that Ptah-hotep should be a prophet when all the other prophets were gone—but after some time he had forgotten, and did not desire to have him any longer among his prophets. But no sooner had the other prophets been sent away than the voice of Thoth said: “Ptah-hotep is still a prophet.”

And so Ptah-hotep went upon his journeys into the world, for Ptah-hotep knew how to read writing, though the other prophets could not read nor write.

And as Ptah-hotep traveled over the face of the earth he found many strange things, but mostly old things. And Ptah-hotep wrote down what he saw, and told other men about them. Only he never forgot who was his master. For every day he carried with him two things, and made sure that they should always accompany him as long as he lived; those things were his two scrolls, which held his writings about the world.

One scroll he wrote upon, and it was called The Scroll that tells of All Things; and in it are the names of all the hills and mountains, and of the valleys and plains, and of the forests, and of the rivers and streams.

But only the gods know how this scroll came into being. It may be that Ptah-hotep first wrote it out by himself. It may be that Ptah-hotep stole it from the hand of Thoth, or borrowed it from the scribes whom the gods have appointed to keep sacred record of such things as Ptah-hotep has written concerning the land.

In The Scroll that tells of All Things are the names of all the cities and towns that lie on the face of the earth, and of all the villages, and of the caves where there is shelter for man. But there are no names in that scroll of the huts wherein the people sleep, nor of the little houses where children play together, nor of the walls and gates where none go, nor even of the tombs where their bones are laid away. Nor does Ptah-hotep write anything about the ships that sail upon the sea, though he knows the names of the captains of those ships, and of the sailors who fight aboard them, and of the great captain of the ship that sails over the dark waters toward the land of the setting sun.

And so also with the plants and the animals that live on the earth, and with all other things that spring up from the ground, or fly through the air, or swim beneath the waves. They have no names except in The Scroll that tells of All Things. And it may be that when Ptah-hotep dies they will forget the names of men, but not the names of the hills and mountains, and of the rivers and streams, nor the names of the trees, nor the names of the flowers, nor the names of the grass. For some years after his death perhaps the name of Man shall yet be remembered; but after a time the name of Ptah-hotep shall be forgotten, and only The Scroll that tells of All Things shall be remembered, and afterward there shall be nothing else to tell of the days of men.