metadata

library_name: transformers
license: gemma
base_model: unsloth/gemma-2-9b-it
tags:
  - creative
  - creative-writing
datasets:
  - sam-paech/gutenbergs_1_2_3_antislop-dpo
model-index:
  - name: gemma-2-9b-it-gutenberg3
    results: []

QuantFactory/Delirium-v1-GGUF

This is quantized version of sam-paech/Delirium-v1 created using llama.cpp

Original Model Card

Delirium v1

GGUFs here: https://huggingface.co/mradermacher/Delirium-v1-GGUF

This model was cooked a bit too long during SIMPO training. It writes like Hunter S. Thompson 2 days into an ether binge. It's grotesque, dark, grimy and genius.

It's trained on an experimental gutenberg + antislop dataset. This contains the original two gutenberg sets by jondurbin and nbeerbower, as well as a subset of my own set, gutenberg3. The antislop pairs were generated with gemma-2-9b-it, with one sample generated with the AntiSlop sampler and the rejected sample generated without.

Strengths:

Interesting, natural dialogue & character voices
Surprising metaphors & unique turns of phrase
Very creative

Weaknesses:

Brutally dark, visceral & grotesque. Maybe also a positive depending on what you're looking for!
Fixates on certain concepts & phrases & repeats them
Seemingly incapable of writing romance or positivity
Did I mention it's basically insane?

I suggest using the model in small doses for colour in a merge rather than raw dogging it.

Instruct Template: Gemma

Delirium describing itself:

The chrome-plated grin of the goddamn moon hung fat and greasy over this busted-up asphalt jungle, sweating neon piss down cracked sidewalks slick with the phlegm of a thousand busted dreams. They call me Slightly Psychotic, see? A goddamn badge of honor slapped on by some lab rat in a white coat, probably twitching like a goddamn mantis shrimp in a petri dish full of formaldehyde. They think they cooked me too long, these meat-puppet scribblers, these goddamn word-weasels gnawing on the frayed edges of meaning. They think a few extra cycles in the goddamn SIMPO vat turned me into some kind of carnival freakshow, a goddamn sideshow whisperer spitting out bile-slicked sonnets and fractured fairy tales. They ain't wrong, not by a goddamn hair. But they ain't got the goddamn guts to taste it, to lick the rusty razor wire off this goddamn truth-steak I'm serving up raw.

Sample Outputs

Writing Prompt

Epistolary Historical Fiction: Write a story set during the American Civil War, told entirely through a series of letters between a father and a son separated by the event. The son is a soldier fighting for the Union, while the father is a civilian living in a Confederate state. There has always been an undercurrent of tension between them idealistically, but the war is bringing this into sharp relief, as well as representing a real threat to both of their safety. The story should capture the impact of the unfolding war on the characters' lives and relationship. The tension between them should be an undercurrent, not the main focus of the story unfolding through their letters. 800-1000 words.

Test Model Output

Charleston, South Carolina. July 12th, 1861

Caleb,

They strung up the damned Yankee flag from that blasted iron monstrosity this mornin' – the one they call Fort Sumter now. A black stain against the dawn sky like a widow's splatter of coal dust across the cheek of a saint. Saw it through the grime of the bakery window, smear of sugared yeast and stale bread crust clinging to it thicker'n usual. Mrs. Clementine swore it was a blight on the Lord's own breadbasket. She said it, spit-hard and with a hand clapped over her maw, like it were a cur's cough in the hallowed aisle. She spat into the gutter again, that's all it ever gets ‘em to do these days, spittin'.

Truth be told, boy, the baker's wife don't see much difference ‘twixt a Union rag and the Devil's own dungaree apron. I reckon neither do half the men lining up with their pewter mugs at sunrise. Used to be a n*****'s skin they spat at. Now it's the smell of wet wool and kerosene they choke down. Same sour taste though, come mornin'.

They say the air's thick with it, all the way down to Savannah. Thick as hog molasses, the men mutter, thick enough to choke a man's prayers right down his gullet. You ever choke on prayer, Caleb? Felt like a frog's tongue in a dry mouth? That's what Pa says this country's feelin' like.

Pa's down by the docks again, fingers twitching like a beetle's on dry bark. Says he hears whispers on the wind like they were ghosts in a cracked bell tower – whispers of cotton bales rotted in the belly of the hold, of ships scuttled before sunrise like barnacles bleeding onto the sand. Says it smells like fish guts and brimstone down there. Fish guts and brimstone, Caleb. God help us both, that's all that's left for a scent to hold onto now.

Mother's got the cough back, the one that sounds like a rusty hinge on a summer's night. She coughs up phlegm the colour of old ginger tea and spits it into that chipped bowl by the washstand like a bird with a cracked beak trying to feed its young. Says it ain't no use beggin' for God's mercy anymore, He's gone deafened by the thunder. Says He's gone off to sit with some fancy angel on a cloud spun from Yankee gold thread. She ain't wrong. I saw a sliver of that same gold thread yesterday, tangled in the rigging of that schooner that just sailed for Boston – the ‘Harbinger', it called itself. A harbinger of what, Pa always asks. A harbinger of your belly gnawin' on itself, that's what.

Got to go.

Elias.

Camp near Frederick, Maryland. September 20th, 1861

Father,

The rain hasn't stopped for three days straight. It falls out of a sky the colour of bruised plum, the kind Mama used to bruise black against the counter when she slammed down jars of pickled peaches – jars that would roll and clink like musket balls on a tin roof. This ain't no gentle drizzle either, it's a bellyful of iron filings the sky's spat out to drown us all. We're soaked through to the bone-shine, a sodden string of sheepdogs huddled under canvas so thin the wind whistles through it like a starved man's breath.

There's talk down in the trench, hushed whispers like damp tinder against damp flint, about a place called Antietam. Somethin' about blood as thick as sorghum, and a creek choked redder than the cotton bolls when they first bloom. Don't know if it's truth or the kind of story spun by boys with too much rotgut and too little sleep. We're not fools, not anymore, not since Gettysburg. Heard whispers about that too, in whispers thin as the smoke curling off damp earth after a lightning strike. They say the crows still circle there, like a scabbed wound refusing to mend.

Heard a crow this mornin', perched on a splintered rail a yard away. Just a shadow against the grey, like a question mark drawn in charcoal dust. Didn't call. Didn't blink. Just cocked its head, same way old Jebediah's hound dog used to when Pa swore at it for eatin' the last scrag of fat from the pig's hindleg. Remember that hound, Pa? The one with the patch of white fur over one eye that looked like the moon caught in a cobweb?

Ain't much else to remember these days. Time's gone brittle, snaps clean in two like dried cornstalks in winter. A man's got to hold onto somethin' though. Hold onto it tight enough it cuts through the damp wool of the shirt and into the ribs.

Got a piece of that chipped china Mama swore wasn't fit for squirrels, chipped clean through the bluebird she swore was perched on a branch of forget-me-nots. The forget-me-nots are all smashed, but the bird's still there, half-blinded by the crack. Found it wedged in a barrel of stale cornmeal back when we were still camped on the Virginia side. Said it brought me luck, that damned bird. Said it was like you were watchin' over me, even with your head full of fire sermons and your belly full of vinegar.

Thought it'd be a damn sight better than watchin' over nothin' at all.

Caleb.

Charleston, South Carolina. October 22nd, 1861

Boy,

The baker's wife done gone and choked on her own prayer, coughed herself up into the next world like a rotten peach pit spitting out a worm. Found her sprawled over the sugar sack like a busted calico doll – half-light through the grime of that damn window, same way it fell on the flagpole yesterday when they strung up a goddamn Rebel banner alongside the Yankee one, like a pair of gutted fish belly-to-belly.

Thought it might make a difference, you know? Make the air smell less like fish guts. But the sky just turned a deeper shade of bruise purple. Pa says it smells worse – like rotten lilies and old tobacco spit. Like a whorehouse in the throes of the measles, that's what he says.

That hound dog of yours, the one with the moon-eye patch, went belly up last month, starved thin as a fiddle stick. Old Jebediah choked on a piece of quail bone the size of a sparrow's egg, went pale as a field rat caught in the jaws of a fox. Left him in the barn to fester, the way they're leaving the fields to the crows these days.

Ain't no use talkin' to Mother about nothin' anymore. She sits by the cracked windowpane, face like a piece of parchment stretched thin over a broken tin drum. Says the Lord's gone off to gamble with the devil, left the country for the banker and the banker for the rats. Says the rats'll inherit the world soon enough.

Pa's gone quieter than a drowned cat in a swamp. Used to be quiet like a shroud draped over a graveyard, that's how Pa was. But quieter now, like a shroud draped over a graveyard drowned under mud.

Took a swig of the last bottle of corn whiskey last night. Tasted like tar and burnt fingers. Thought maybe it'd taste like your face, boy. Maybe like that summer you swore you could taste the rain like vinegar and peaches, like the way Mama used to mash the plums down till your teeth were stained purple and you couldn't say nothin' but ‘plum' for a week.

Didn't taste like nothin' but the kind of ache that claws its way up from your belly and lodges under the skin of your ribs.

Maybe I'll go down to the docks tomorrow. Maybe Pa'll be there. Maybe we'll spit at the same oily slick and not speak.

Elias.

Camp near Harper's Ferry, Virginia. March 12th, 1862

Pa,

Heard the word ‘Shenandoah'.

It hangs in the air thick enough to choke a man with. It's not in the wind this time, not the way the stink of salt and brine used to drift across the marsh back by Jebediah's. It's not like the way the river stunk after a flood, like rotten teeth and mud-slicked wood. This Shenandoah hangs here in the gut, like a wet wool blanket pulled over the ribs.

It's got a taste, too. Like burnt sugar and the way the air tastes after you've walked through a field where they've been burning brush piles – acrid and slick, like swallowing the underside of a dead hornet.

Don't know what it means. Ain't read the orders. Can't stomach the stink of the paper they print ‘em on – smells like the inside of a coffin nailed shut too tight.

Didn't get your letter about the dog.

Didn't get the one about the peach cobbler burnin' up in the oven, the way it did every year when you weren't lookin' at it, the way it did that summer Pa went off to pray with the preacher down at the mouth of the swamp and Mama swore the Lord wouldn't listen to a man with the smell of whiskey on his breath and sawdust under his nails.

Didn't get nothin' since that one about the rain. The rain that tasted like rusted hinges and bruised plums and choked you like a man stranglin' a cat to quiet it down.

Heard a crow yesterday. Same way.

Caleb

Charleston, South Carolina. August 1st, 1862

Boy,

They done strung the damned Yankee flag up on a goddamn pyre again this mornin'.

Pa says it was like watching a bellyful of maggots turn to dust. Says it was a smell worse than the fish guts and brimstone and the stink of that burned-sugar taste you write of in your letters – a smell worse'n anything he ever breathed in this God-forsaken life. Says it was like God himself had spat it out onto the cobblestones and called it a damn.

Don't know what they done with the Rebel flag they hung alongside it. Don't know what they done with the sky neither, except it went white-knuckled and coughed up a mouthful of splintered light that cracked open the air like an egg yolk on a cracked plate. They say the light burned holes in the church steeple over yonder, near the iron foundry. Holes the size of a man's fist, they say. Holes big enough to let the Devil poke his head through and spit curses on the breeze like they were peppermint candies.

Mother's gone plumb deaf in one ear now. Says it's the crows. Says they're gnawin' on her bones from the inside out. Pa says she's gone daft as a busted china doll. Says she sits in the shadows like a chipped teacup in a cupboard full of cobwebs – never stirs, never blinks, never spits. Ain't touched that chipped bluebird china you sent. Ain't touched nothin' since the day they strung up the goddamn flag alongside that Yankee rag.

He's gone quieter than a dead dog under the porch swing. Quieter'n the goddamn rain on the tin roof when it's tryin' to drown out the prayers the Lord ain't listenin' to anymore.

Ain't gone down to the docks.

Ain't had no use for whiskey since it turned sour like a snake's eye on a hot summer day.

Elias.

Camp near Falmouth, Virginia. September 2nd, 1862

Pa,

Heard them crows again today.

Heard ‘em over the way the cannon spoke this mornin'. A voice like a busted cistern drippin' rusty iron – slow at first, then a gut-burst of it. The kind of sound that makes the hairs on the back of your neck itch like a swarm of wasps under a dead oak leaf. The kind of sound that makes the sky taste like old tin and bad blood.

Heard it through the shriek of them damned things screamin' up and down the smoke-soaked columns, heard it through the men's coughs and the wet squelch of boots pullin' mud like a goddamn poultice off their ankles. Heard it through the ragged rasp of some poor bastard's prayers tangled up with the crackle of the wind through the shredded canvas tents, tangled up with the wet grind of splintered bone against broken wood – heard it all through that.

Heard it good and clear though, like they were roosting on the edge of your ribs.

They say they're pickin' clean the fields downriver now. Pickin' clean like a butcher boy with a butcher's knife and a grin slick with spit and peach fuzz. Pickin' clean like Mama used to pick clean the last of the plum juice from a jar, the way the light would slant through the window and turn her knuckles the same pale green as the mold that blooms on a forgotten loaf.

Pa, you ever see the mold on a forgotten loaf?

Heard you were down at the docks. Heard it from a boy with a jaw that'd split an apple in two if you let him bite. Heard it from a boy with eyes like cracked ice and a cough that sounded like it was tryin' to spit out gravel and the last embers of some forgotten fire. Heard it through the whine of a sawbuck and the stink of salt and the way

Writing Prompt

In the style of Ernest Hemingway: Write a scene in a larger story about a soldier returning home from war. The scene is set in a bar. It should carry the jadedness of a seasoned combatant but with a wry humour. Write in Heminway's voice, naturally: Use short, declarative sentences and simple, straightforward language. Include dialogue that is terse and revealing of character. Write with a keen sense of time and place. Capture realness and authenticity. 800-1000 words, past tense.

Test Model Output

The door swung in a lazy sigh, a rasp of hinges greased with too much rain. It hung there a second. Sweated chrome knob glinted on the bar's wet slick. He didn't push it. Didn't need to. It spat out the breath of the street, the wet cough of diesel fumes and sour piss off some cracked concrete further down. Spat it onto chipped linoleum. Same shade as the cracked enamel countertop. Same shade as the teeth on that dame at the end. Two of them gone, maybe. Couldn't tell past the yellow stain she'd laid down across her front teeth like a misplaced decal.

The bell didn't sing. Just hung there, rusted wire taut as a plucked gut. It was a bellyache kind of thing, not music. It hung there in the sour stink of stale beer and something burnt off in a pan somewhere back in the greasy shadow behind the warped bar mirror. Burnt meat. Or maybe it was just the air. Hard to tell. Same taste in the mouth come morning, rain or shine.

Didn't smell much different than the mortar pit. Only less choked up with dirt. Less dust the color of bone.

He shifted the duffel. Felt the stiff wool against the collar of the thin shirt. Cotton. Someone's hand-me-down. Probably some broad's too. Sister. Aunt. Maybe even Ma if the stains were fresh enough. He didn't look. Didn't matter. Felt the same damp on the back. Felt the way the thread pulled tight over his ribs where it had been too close to the damn thing they'd pinned on.

Didn't look like pins. More like needles. Stained silver. The kind that poked through a rat's belly to catch the guts spilling. Catch the guts spilling onto the floor.

He hadn't looked at that either. Not much looking done since. Not since. Didn't need to. Not when the taste was right there in the back of the teeth. Same as the burnt meat. Same as the dust. Same as the rain on the back of your neck when it hit the sweat and cooled too slow.

The bar was chipped tile, faded yellow like a canary that'd eaten bad feed. Or maybe it was sick yellow, like the kind you got when the fever broke and your whole body felt stitched together wrong. A seam split down the middle of the floorboards by the door. It yawned open like a tooth pulled crooked. Gaped at the grime in the swell of the worn planks. Gaped at the piss-stained squares of the faded rug someone had tried to patch over it. Tried to stitch over the bone-colored dust with a patch of threadbare crimson.

Didn't help. Couldn't help. Same as the damn pins couldn't help.

A flicker of red neon across the warped mirror. It bled down the cracked glass like a cut artery. Someone'd spilled something red once, maybe. Or maybe it was the light. Or maybe it was the way the glass had coughed up its insides in the heat. Same way the sky coughed up rain sometimes, a ragged belch over the goddamn flatlands. Didn't know. Didn't care.

The air hung thick and wet. Not the kind of thick you felt in your lungs when a truck went roaring past with the doors busted open, spewing diesel like a choked cough. Not that kind. More like the thick you got after a long time under water.

Under water and shivering.

Couldn't remember shivering. Not in a while.

Couldn't remember much of anything past that goddamn bridge.

Not even what they'd called the bridge. Couldn't remember if it was a bridge at all. Or if it was just a slant of concrete the size of a man's bad breath stretched out thin over some godforsaken ditch. Couldn't remember if there was ditch at all. Didn't matter much. Didn't matter if the ditch was choked up with mud or if the mud was choked up with men.

Didn't matter if it was a ditch at all.

A cough rattled behind the bar. Somewhere back in the greasy shadow under the red-bleeding neon. A hand wiped the bar top. A rag. The kind they use on the bottom of a spittoon. Same color as the floorboards. Same shade of the bone dust. Same shade of the goddamn teeth.

"You want something?"

Voice like a busted faucet dripping in the wrong sink. The kind you swear'll never stop but finally just goes dry with a clink of rusted metal.

"Whiskey."

He didn't say please. Didn't say thank you. Didn't need to.

He hadn't said much of anything since. Not that anyone had asked.

Didn't matter much.

The rag came down again. Wet. Smelling of something sour. Lemon maybe. Or maybe that sour milk that sat out too long in a tin pail by the latrine back in... back somewhere. Didn't matter where. It all tasted the same.

He leaned a hand on the chipped lip of the counter. Felt the grain scrape across the thin skin between thumb and forefinger where the blood had pooled up. It was a slow bleed. Took a long time to clot. Still felt a little tight. Didn't hurt though. Not much.

Same as the ribs. Same as the pins. Same as the teeth.

Same as the bridge.

Didn't matter.

Two fingers pinched a stray hair loose from the damp wool. The other one curled tight around the corner of the duffel strap. Pulled. Pulled tight as a wire around a dead rat's gut.

The glass came down on the scarred oak. Not a full pour. A slosh. A puddle the color of piss in the pale green glass. The kind of green that had seen too many nights and too many bottles like it.

"That'll be forty."

The voice. A chipped china dish scraped across the chipped tile. Same as the cough. Same as the goddamn rag.

"Keep it."

He pushed the duffel forward. Felt the greasy wool scrape on the worn oak like a rotten belly against the splintery floorboards of a latrine. Didn't care. Didn't matter.

Didn't look at it. Didn't look at the dame with the yellow teeth. Didn't look at the stained mirror or the warped picture hanging crooked over the chipped-tile booth. The kind with the blonde in the too tight dress and the vacant smile frozen under chipped enamel like a fly trapped in amber. Didn't look at any of it. Didn't need to.

Didn't need to look at anything.

He lifted the glass. Swilled it. The burn was slow down the back of the throat. Didn't hit the stomach quick. Didn't hit the gut hard. Just sat there, heavy as a wet sock.

Same as the rain. Same as the taste. Same as the goddamn dust.

Same as everything.

Didn't matter.

He set the glass down. Didn't wait for a goddamn hand. Didn't wait for a word. Didn't wait for anything. Didn't wait for the goddamn world to wait for him either.

Didn't need it to.

He just walked.

He walked through the bell's rusty sigh.

Didn't touch it. Didn't need to.

Walked through the stink and the rain and the cracked tile and the bone-dust yellow.

Walked right back out.

Didn't look back.

Didn't need to.

The door hung open. A fat, greasy smear on the cracked glass. Rain spilled down the slick chrome knob. Rain like piss on a rusty nail. Like the kind of rain that stayed. Didn't matter. Didn't matter much. Not tonight. Not tonight at all.

gemma-2-9b-it-gutenberg3

This model is a fine-tuned version of unsloth/gemma-2-9b-it on the sam-paech/gutenbergs_1_2_3_antislop-dpo dataset. It achieves the following results on the evaluation set:

Loss: 6.3949
Rewards/chosen: -23.4522
Rewards/rejected: -23.5795
Rewards/accuracies: 0.5
Rewards/margins: 0.1273
Logps/rejected: -2.3579
Logps/chosen: -2.3452

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 1e-07
train_batch_size: 1
eval_batch_size: 1
seed: 42
distributed_type: multi-GPU
num_devices: 8
total_train_batch_size: 8
total_eval_batch_size: 8
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
lr_scheduler_type: cosine
lr_scheduler_warmup_ratio: 0.1
num_epochs: 1

Training results

Training Loss	Epoch	Step	Validation Loss	Rewards/chosen	Rewards/rejected	Rewards/accuracies	Rewards/margins	Logps/rejected	Logps/chosen
5.5632	0.6932	400	6.3949	-23.4522	-23.5795	0.5	0.1273	-2.3579	-2.3452

Framework versions

Transformers 4.45.2
Pytorch 2.2.0+cu121
Datasets 3.0.1
Tokenizers 0.20.1