Pygmalion-Metharme-7b-4bit-TopScore

Maintainer: Monero

Total Score

45

Last updated 9/6/2024
Text-to-Text

📈

PropertyValue
Run this modelRun on HuggingFace
API specView on HuggingFace
Github linkNo Github link provided
Paper linkNo paper link provided

Create account to get full access

or

If you already have an account, we'll log you in

Metharme 7B 4bit

An instruction-tuned LLaMA biased towards fiction writing and conversation.

Model Details

Converted from the XORs weights from PygmalionAI's release https://huggingface.co/PygmalionAI/metharme-7b

Quantized for KoboldAI use using https://github.com/0cc4m/GPTQ-for-LLaMa

I created several dozen quantized variations of this model and believe this variation to be "best."

Model:

Wikitext2

Ptb-New

C4-New

Metharme 7b - 16bit

5.7208476066589355

41.61103439331055

7.512405872344971

Metharme-AO-TS-Trits-damp0.1

6.172733783721924

45.19890213012695

7.872506141662598

Other benchmark scores at the bottom of readme.

Metharme 7B is an instruct model based on Meta's LLaMA-7B.

This is an experiment to try and get a model that is usable for conversation, roleplaying and storywriting, but which can be guided using natural language like other instruct models. See the prompting section below for examples.

It was trained by doing supervised fine-tuning over a mixture of regular instruction data alongside roleplay, fictional stories and conversations with synthetically generated instructions attached.

Prompting

The current model version has been trained on prompts using three different roles, which are denoted by the following tokens: <|system|>, <|user|> and <|model|>.

The <|system|> prompt can be used to inject out-of-channel information behind the scenes, while the <|user|> prompt should be used to indicate user input. The <|model|> token should then be used to indicate that the model should generate a response. These tokens can happen multiple times and be chained up to form a conversation history.

Prompting example

Here's a concrete example. Let's say you have the following system prompt:

This is a text adventure game. Describe the scenario to the user and give him three options to pick from on each turn.

And the user has typed in the following message:

Start!

To perform inference against the model, the prompt should be formatted like this:

<|system|>This is a text adventure game. Describe the scenario to the user and give him three options to pick from on each turn.<|user|>Start!<|model|>

The model might generate something like:

You are standing in front of an old, abandoned house. The windows are boarded up, and there's no sign of life around it. As you approach, you notice a strange feeling emanating from within. Suddenly, you hear a voice calling out to you... 'Come inside!'

  • Go inside the house.
  • Ignore the call and move away.
  • Run as fast as you can.

It will automatically emit an end-of-text token (</s>) when it judges that the response is complete.

Pretend then that the user has replied with go inside. The next prompt should then be:

<|system|>This is a text adventure game. Describe the scenario to the user and give him three options to pick from on each turn.<|user|>Start!<|model|>You are standing in front of an old, abandoned house. The windows are boarded up, and there's no sign of life around it. As you approach, you notice a strange feeling emanating from within. Suddenly, you hear a voice calling out to you... 'Come inside!'

- Go inside the house.
- Ignore the call and move away.
- Run as fast as you can.<|user|>go inside<|model|>

Which might generate something like:

You enter the house, and immediately, a wave of unsettling energy washes over you. Your senses seem heightened, but your mind feels foggy. A figure appears before you - a tall man with piercing eyes. He speaks to you in a language you don't understand.

  • Ask him what he wants.
  • Attack him.
  • Run away.

Same process applies. Usually, it is best to do a sliding window over the user and model turns, but keep the system prompt fixed at the start of the context window.

Limitations and biases

The intended use-case for this model is fictional writing for entertainment purposes. Any other sort of usage is out of scope.

As such, it was not fine-tuned to be safe and harmless: the base model

and
this fine-tune have been trained on data known to contain profanity and texts that are lewd or otherwise offensive. It may produce socially unacceptable or undesirable text, even if the prompt itself does not include anything explicitly offensive. Outputs might often be factually wrong or misleading.

Click to Expand Benchmarks of different quantized variations

The lower the number, the better the score. Benchmarks Sorted by C4-New score

GPTQ Variation:

Wikitext2

Ptb-New

C4-New

Metharme-7b-16bit

5.7208476066589355

41.61103439331055

7.512405872344971

Metharme-ao-ts-trits-damp0.1

6.172733783721924

45.19890213012695

7.872506141662598

Metharme-ao-trits-damp0.1

6.163661956787109

46.50249099731445

7.877425193786621

Metharme-ao-ts-damp0.1

6.184001445770264

46.17180633544922

7.880400657653809

Metharme-7b-4bit-ao-damp0.1

6.220707893371582

47.82929611206055

7.884565353393555

Metharme-ao-ts-trits

6.310682773590088

46.4483757019043

7.898126602172852

Metharme-7b-4bit-ao

6.281311511993408

46.79158401489258

7.906069755554199

Metharme-ao-trits

6.283935546875

46.57590103149414

7.907411575317383

Metharme-ao-ts

6.329496383666992

46.88129806518555

7.910323143005371

Metharme-ao-ts-sym-trits-damp0.1

6.232576370239258

48.081459045410156

7.95023250579834

Metharme-ao-ts-sym-damp0.1

6.210323333740234

47.66789245605469

7.952476978302002

Metharme-ao-sym-trits-damp0.1

6.329384803771973

48.06882858276367

7.959168910980225

Metharme-ao-ts-sym-trits

6.471063137054443

49.650611877441406

7.969552040100098

Metharme-ao-ts-sym

6.460526943206787

47.190460205078125

7.9732160568237305

Metharme-ao-sym-trits

6.390106678009033

48.15375900268555

7.9804582595825195

Metharme-7b-4bit-ao-sym

6.477842807769775

48.53507614135742

7.993765354156494

Metharme-ao-sym

6.477842807769775

48.53507614135742

7.993765354156494

Metharme-7b-4bit-ts-32g-trits

6.632943153381348

47.973228454589844

8.013848304748535

Metharme-7b-4bit-ts-32g-sym-damp0.1

6.274552822113037

47.35737228393555

8.06270980834961

Metharme-7b-4bit-ts-32g-sym-trits-damp0.1

6.266031265258789

47.346702575683594

8.068148612976074

Metharme-7b-4bit-32g-damp0.1

6.107605934143066

47.91380310058594

8.068695068359375

Metharme-7b-4bit-32g-trits-damp0.1

6.128157138824463

48.04175567626953

8.0708646774292

Metharme-7b-4bit-ts-32g-damp0.1

6.219024658203125

45.834869384765625

8.071272850036621

Metharme-7b-4bit-32g-trits

7.017086029052734

45.04129409790039

8.074845314025879

Metharme-7b-4bit-32g-sym-trits-damp0.1

6.109438896179199

47.35737228393555

8.075060844421387

Metharme-7b-4bit-ts-32g-trits-damp0.1

6.118431568145752

45.67333221435547

8.077078819274902

Metharme-7b-4bit-32g

6.902080535888672

50.237754821777344

8.081602096557617

Metharme-7b-4bit-ts-32g

6.424218654632568

48.48588943481445

8.089512825012207

Metharme-7b-4bit-ts-32g-sym-trits

6.82415771484375

48.82029724121094

8.090987205505371

Metharme-7b-4bit-32g-sym-damp0.1

6.566899299621582

48.0670166015625

8.095841407775879

Metharme-7b-4bit-ts-128g-trits-damp0.1

6.289113521575928

46.06787109375

8.122251510620117

Metharme-7b-4bit-32g-sym

6.518134117126465

49.66925811767578

8.13516616821289

Metharme-7b-4bit-32g-sym-trits

6.206963539123535

46.88833999633789

8.13610553741455

Metharme-7b-4bit-128g-damp0.1

6.242006301879883

45.30938720703125

8.14249324798584

Metharme-7b-4bit-ts-32g-sym

6.387663841247559

48.07244110107422

8.173730850219727

Metharme-7b-4bit-128g-trits-damp0.1

6.262309551239014

47.80055618286133

8.192194938659668

Metharme-7b-4bit-128g

10.206376075744629

49.00401306152344

8.198845863342285

Metharme-7b-4bit-ts-128g-damp0.1

6.17774772644043

46.47630310058594

8.20170783996582

Metharme-7b-4bit-ts-128g-sym-trits-damp0.1

6.225503921508789

53.12746047973633

8.240595817565918

Metharme-7b-4bit-128g-trits

8.68796443939209

49.73833465576172

8.2406587600708

Metharme-7b-4bit-ts-128g-sym-damp0.1

6.584965705871582

55.20026397705078

8.268644332885742

Metharme-7b-4bit-ts-128g-trits

7.350858688354492

44.25314712524414

8.274221420288086

Metharme-7b-4bit-128g-sym-trits-damp0.1

6.585468769073486

51.55869674682617

8.2803316116333

Metharme-7b-4bit-ts-128g-sym-trits

6.756448745727539

51.510311126708984

8.292160987854004

Metharme-7b-4bit-128g-sym-damp0.1

6.379064083099365

52.17233657836914

8.316649436950684

Metharme-7b-4bit-128g-sym-trits

7.056288242340088

48.983768463134766

8.339276313781738

Metharme-7b-4bit-ts-128g

9.475017547607422

52.358829498291016

8.340700149536133

Metharme-7b-4bit-128g-sym

6.9575653076171875

49.356834411621094

8.35644817352295

Metharme-7b-4bit-ts-128g-sym

6.819341659545898

55.28740310668945

8.377721786499023

Metharme-7b-4bit-ts-damp0.1

6.7783050537109375

51.81301498413086

8.621373176574707

Metharme-7b-4bit-ts-trits-damp0.1

6.631694793701172

51.7371711730957

8.656966209411621

Metharme-7b-4bit-damp0.1

6.495014190673828

49.39763641357422

8.68167781829834

Metharme-sym-damp0.1

6.896804332733154

57.4250602722168

8.703770637512207

Metharme-7b-4bit-ts-sym

7.270263671875

54.35262680053711

8.787986755371094

Metharme-7b-4bit-trits

7.832409858703613

55.383026123046875

8.806737899780273

Metharme-trits

7.832409858703613

55.383026123046875

8.806737899780273

Metharme-7b-4bit-ts-sym-damp0.1

6.7517595291137695

54.06147384643555

8.821818351745605

Metharme-7b-4bit-alone

6.997134685516357

58.87525177001953

8.824191093444824

Metharme-7b-4bit-ts-trits

7.2306809425354

66.78710174560547

8.879831314086914

Metharme-7b-4bit-ts-sym-trits-damp0.1

6.886506080627441

64.72743225097656

8.880627632141113

Metharme-7b-4bit-ts

7.735969543457031

62.92238235473633

8.913650512695312

Metharme-sym-trits-damp0.1

7.075908184051514

59.13897705078125

8.919178009033203

Metharme-7b-4bit-ts-sym-trits

7.599876403808594

55.75454330444336

8.932201385498047

Metharme-sym-trits

7.494253635406494

63.320709228515625

8.969240188598633

Metharme-sym

7.585672855377197

61.01168441772461

9.032520294189453

Metharme-7b-4bit-ao-128g

251321.265625

250117.859375

232929.234375

Metharme-7b-4bit-ao-32g

275425.5

267733.25

254506.71875

Benchmarks Sorted by Wikitext2

GPTQ Variation:

Wikitext2

Ptb-New

C4-New

Metharme-7b-16bit

5.7208476066589355

41.61103439331055

7.512405872344971

Metharme-7b-4bit-32g-damp0.1

6.107605934143066

47.91380310058594

8.068695068359375

Metharme-7b-4bit-32g-sym-trits-damp0.1

6.109438896179199

47.35737228393555

8.075060844421387

Metharme-7b-4bit-ts-32g-trits-damp0.1

6.118431568145752

45.67333221435547

8.077078819274902

Metharme-7b-4bit-32g-trits-damp0.1

6.128157138824463

48.04175567626953

8.0708646774292

Metharme-ao-trits-damp0.1

6.163661956787109

46.50249099731445

7.877425193786621

Metharme-ao-ts-trits-damp0.1

6.172733783721924

45.19890213012695

7.872506141662598

Metharme-7b-4bit-ts-128g-damp0.1

6.17774772644043

46.47630310058594

8.20170783996582

Metharme-ao-ts-damp0.1

6.184001445770264

46.17180633544922

7.880400657653809

Metharme-7b-4bit-32g-sym-trits

6.206963539123535

46.88833999633789

8.13610553741455

Metharme-ao-ts-sym-damp0.1

6.210323333740234

47.66789245605469

7.952476978302002

Metharme-7b-4bit-ts-32g-damp0.1

6.219024658203125

45.834869384765625

8.071272850036621

Metharme-7b-4bit-ao-damp0.1

6.220707893371582

47.82929611206055

7.884565353393555

Metharme-7b-4bit-ts-128g-sym-trits-damp0.1

6.225503921508789

53.12746047973633

8.240595817565918

Metharme-ao-ts-sym-trits-damp0.1

6.232576370239258

48.081459045410156

7.95023250579834

Metharme-7b-4bit-128g-damp0.1

6.242006301879883

45.30938720703125

8.14249324798584

Metharme-7b-4bit-128g-trits-damp0.1

6.262309551239014

47.80055618286133

8.192194938659668

Metharme-7b-4bit-ts-32g-sym-trits-damp0.1

6.266031265258789

47.346702575683594

8.068148612976074

Metharme-7b-4bit-ts-32g-sym-damp0.1

6.274552822113037

47.35737228393555

8.06270980834961

Metharme-7b-4bit-ao

6.281311511993408

46.79158401489258

7.906069755554199

Metharme-ao-trits

6.283935546875

46.57590103149414

7.907411575317383

Metharme-7b-4bit-ts-128g-trits-damp0.1

6.289113521575928

46.06787109375

8.122251510620117

Metharme-ao-ts-trits

6.310682773590088

46.4483757019043

7.898126602172852

Metharme-ao-sym-trits-damp0.1

6.329384803771973

48.06882858276367

7.959168910980225

Metharme-ao-ts

6.329496383666992

46.88129806518555

7.910323143005371

Metharme-7b-4bit-128g-sym-damp0.1

6.379064083099365

52.17233657836914

8.316649436950684

Metharme-7b-4bit-ts-32g-sym

6.387663841247559

48.07244110107422

8.173730850219727

Metharme-ao-sym-trits

6.390106678009033

48.15375900268555

7.9804582595825195

Metharme-7b-4bit-ts-32g

6.424218654632568

48.48588943481445

8.089512825012207

Metharme-ao-ts-sym

6.460526943206787

47.190460205078125

7.9732160568237305

Metharme-ao-ts-sym-trits

6.471063137054443

49.650611877441406

7.969552040100098

Metharme-7b-4bit-ao-sym

6.477842807769775

48.53507614135742

7.993765354156494

Metharme-ao-sym

6.477842807769775

48.53507614135742

7.993765354156494

Metharme-7b-4bit-damp0.1

6.495014190673828

49.39763641357422

8.68167781829834

Metharme-7b-4bit-32g-sym

6.518134117126465

49.66925811767578

8.13516616821289

Metharme-7b-4bit-32g-sym-damp0.1

6.566899299621582

48.0670166015625

8.095841407775879

Metharme-7b-4bit-ts-128g-sym-damp0.1

6.584965705871582

55.20026397705078

8.268644332885742

Metharme-7b-4bit-128g-sym-trits-damp0.1

6.585468769073486

51.55869674682617

8.2803316116333

Metharme-7b-4bit-ts-trits-damp0.1

6.631694793701172

51.7371711730957

8.656966209411621

Metharme-7b-4bit-ts-32g-trits

6.632943153381348

47.973228454589844

8.013848304748535

Metharme-7b-4bit-ts-sym-damp0.1

6.7517595291137695

54.06147384643555

8.821818351745605

Metharme-7b-4bit-ts-128g-sym-trits

6.756448745727539

51.510311126708984

8.292160987854004

Metharme-7b-4bit-ts-damp0.1

6.7783050537109375

51.81301498413086

8.621373176574707

Metharme-7b-4bit-ts-128g-sym

6.819341659545898

55.28740310668945

8.377721786499023

Metharme-7b-4bit-ts-32g-sym-trits

6.82415771484375

48.82029724121094

8.090987205505371

Metharme-7b-4bit-ts-sym-trits-damp0.1

6.886506080627441

64.72743225097656

8.880627632141113

Metharme-sym-damp0.1

6.896804332733154

57.4250602722168

8.703770637512207

Metharme-7b-4bit-32g

6.902080535888672

50.237754821777344

8.081602096557617

Metharme-7b-4bit-128g-sym

6.9575653076171875

49.356834411621094

8.35644817352295

Metharme-7b-4bit-alone

6.997134685516357

58.87525177001953

8.824191093444824

Metharme-7b-4bit-32g-trits

7.017086029052734

45.04129409790039

8.074845314025879

Metharme-7b-4bit-128g-sym-trits

7.056288242340088

48.983768463134766

8.339276313781738

Metharme-sym-trits-damp0.1

7.075908184051514

59.13897705078125

8.919178009033203

Metharme-7b-4bit-ts-trits

7.2306809425354

66.78710174560547

8.879831314086914

Metharme-7b-4bit-ts-sym

7.270263671875

54.35262680053711

8.787986755371094

Metharme-7b-4bit-ts-128g-trits

7.350858688354492

44.25314712524414

8.274221420288086

Metharme-sym-trits

7.494253635406494

63.320709228515625

8.969240188598633

Metharme-sym

7.585672855377197

61.01168441772461

9.032520294189453

Metharme-7b-4bit-ts-sym-trits

7.599876403808594

55.75454330444336

8.932201385498047

Metharme-7b-4bit-ts

7.735969543457031

62.92238235473633

8.913650512695312

Metharme-7b-4bit-trits

7.832409858703613

55.383026123046875

8.806737899780273

Metharme-trits

7.832409858703613

55.383026123046875

8.806737899780273

Metharme-7b-4bit-128g-trits

8.68796443939209

49.73833465576172

8.2406587600708

Metharme-7b-4bit-ts-128g

9.475017547607422

52.358829498291016

8.340700149536133

Metharme-7b-4bit-128g

10.206376075744629

49.00401306152344

8.198845863342285

Metharme-7b-4bit-ao-128g

251321.265625

250117.859375

232929.234375

Metharme-7b-4bit-ao-32g

275425.5

267733.25

254506.71875

Benchmarks Sorted by PTB-new Score

GPTQ Variation:

Wikitext2

Ptb-New

C4-New

Metharme-7b-16bit

5.7208476066589355

41.61103439331055

7.512405872344971

Metharme-7b-4bit-ts-128g-trits

7.350858688354492

44.25314712524414

8.274221420288086

Metharme-7b-4bit-32g-trits

7.017086029052734

45.04129409790039

8.074845314025879

Metharme-ao-ts-trits-damp0.1

6.172733783721924

45.19890213012695

7.872506141662598

Metharme-7b-4bit-128g-damp0.1

6.242006301879883

45.30938720703125

8.14249324798584

Metharme-7b-4bit-ts-32g-trits-damp0.1

6.118431568145752

45.67333221435547

8.077078819274902

Metharme-7b-4bit-ts-32g-damp0.1

6.219024658203125

45.834869384765625

8.071272850036621

Metharme-7b-4bit-ts-128g-trits-damp0.1

6.289113521575928

46.06787109375

8.122251510620117

Metharme-ao-ts-damp0.1

6.184001445770264

46.17180633544922

7.880400657653809

Metharme-ao-ts-trits

6.310682773590088

46.4483757019043

7.898126602172852

Metharme-7b-4bit-ts-128g-damp0.1

6.17774772644043

46.47630310058594

8.20170783996582

Metharme-ao-trits-damp0.1

6.163661956787109

46.50249099731445

7.877425193786621

Metharme-ao-trits

6.283935546875

46.57590103149414

7.907411575317383

Metharme-7b-4bit-ao

6.281311511993408

46.79158401489258

7.906069755554199

Metharme-ao-ts

6.329496383666992

46.88129806518555

7.910323143005371

Metharme-7b-4bit-32g-sym-trits

6.206963539123535

46.88833999633789

8.13610553741455

Metharme-ao-ts-sym

6.460526943206787

47.190460205078125

7.9732160568237305

Metharme-7b-4bit-ts-32g-sym-trits-damp0.1

6.266031265258789

47.346702575683594

8.068148612976074

Metharme-7b-4bit-ts-32g-sym-damp0.1

6.274552822113037

47.35737228393555

8.06270980834961

Metharme-7b-4bit-32g-sym-trits-damp0.1

6.109438896179199

47.35737228393555

8.075060844421387

Metharme-ao-ts-sym-damp0.1

6.210323333740234

47.66789245605469

7.952476978302002

Metharme-7b-4bit-128g-trits-damp0.1

6.262309551239014

47.80055618286133

8.192194938659668

Metharme-7b-4bit-ao-damp0.1

6.220707893371582

47.82929611206055

7.884565353393555

Metharme-7b-4bit-32g-damp0.1

6.107605934143066

47.91380310058594

8.068695068359375

Metharme-7b-4bit-ts-32g-trits

6.632943153381348

47.973228454589844

8.013848304748535

Metharme-7b-4bit-32g-trits-damp0.1

6.128157138824463

48.04175567626953

8.0708646774292

Metharme-7b-4bit-32g-sym-damp0.1

6.566899299621582

48.0670166015625

8.095841407775879

Metharme-ao-sym-trits-damp0.1

6.329384803771973

48.06882858276367

7.959168910980225

Metharme-7b-4bit-ts-32g-sym

6.387663841247559

48.07244110107422

8.173730850219727

Metharme-ao-ts-sym-trits-damp0.1

6.232576370239258

48.081459045410156

7.95023250579834

Metharme-ao-sym-trits

6.390106678009033

48.15375900268555

7.9804582595825195

Metharme-7b-4bit-ts-32g

6.424218654632568

48.48588943481445

8.089512825012207

Metharme-7b-4bit-ao-sym

6.477842807769775

48.53507614135742

7.993765354156494

Metharme-ao-sym

6.477842807769775

48.53507614135742

7.993765354156494

Metharme-7b-4bit-ts-32g-sym-trits

6.82415771484375

48.82029724121094

8.090987205505371

Metharme-7b-4bit-128g-sym-trits

7.056288242340088

48.983768463134766

8.339276313781738

Metharme-7b-4bit-128g

10.206376075744629

49.00401306152344

8.198845863342285

Metharme-7b-4bit-128g-sym

6.9575653076171875

49.356834411621094

8.35644817352295

Metharme-7b-4bit-damp0.1

6.495014190673828

49.39763641357422

8.68167781829834

Metharme-ao-ts-sym-trits

6.471063137054443

49.650611877441406

7.969552040100098

Metharme-7b-4bit-32g-sym

6.518134117126465

49.66925811767578

8.13516616821289

Metharme-7b-4bit-128g-trits

8.68796443939209

49.73833465576172

8.2406587600708

Metharme-7b-4bit-32g

6.902080535888672

50.237754821777344

8.081602096557617

Metharme-7b-4bit-ts-128g-sym-trits

6.756448745727539

51.510311126708984

8.292160987854004

Metharme-7b-4bit-128g-sym-trits-damp0.1

6.585468769073486

51.55869674682617

8.2803316116333

Metharme-7b-4bit-ts-trits-damp0.1

6.631694793701172

51.7371711730957

8.656966209411621

Metharme-7b-4bit-ts-damp0.1

6.7783050537109375

51.81301498413086

8.621373176574707

Metharme-7b-4bit-128g-sym-damp0.1

6.379064083099365

52.17233657836914

8.316649436950684

Metharme-7b-4bit-ts-128g

9.475017547607422

52.358829498291016

8.340700149536133

Metharme-7b-4bit-ts-128g-sym-trits-damp0.1

6.225503921508789

53.12746047973633

8.240595817565918

Metharme-7b-4bit-ts-sym-damp0.1

6.7517595291137695

54.06147384643555

8.821818351745605

Metharme-7b-4bit-ts-sym

7.270263671875

54.35262680053711

8.787986755371094

Metharme-7b-4bit-ts-128g-sym-damp0.1

6.584965705871582

55.20026397705078

8.268644332885742

Metharme-7b-4bit-ts-128g-sym

6.819341659545898

55.28740310668945

8.377721786499023

Metharme-7b-4bit-trits

7.832409858703613

55.383026123046875

8.806737899780273

Metharme-trits

7.832409858703613

55.383026123046875

8.806737899780273

Metharme-7b-4bit-ts-sym-trits

7.599876403808594

55.75454330444336

8.932201385498047

Metharme-sym-damp0.1

6.896804332733154

57.4250602722168

8.703770637512207

Metharme-7b-4bit-alone

6.997134685516357

58.87525177001953

8.824191093444824

Metharme-sym-trits-damp0.1

7.075908184051514

59.13897705078125

8.919178009033203

Metharme-sym

7.585672855377197

61.01168441772461

9.032520294189453

Metharme-7b-4bit-ts

7.735969543457031

62.92238235473633

8.913650512695312

Metharme-sym-trits

7.494253635406494

63.320709228515625

8.969240188598633

Metharme-7b-4bit-ts-sym-trits-damp0.1

6.886506080627441

64.72743225097656

8.880627632141113

Metharme-7b-4bit-ts-trits

7.2306809425354

66.78710174560547

8.879831314086914

Metharme-7b-4bit-ao-128g

251321.265625

250117.859375

232929.234375

Metharme-7b-4bit-ao-32g

275425.5

267733.25

254506.71875

Benchmarks Sorted in Alphabetical Order

GPTQ Variation:

Wikitext2

Ptb-New

C4-New

Metharme-7b-16bit

5.7208476066589355

41.61103439331055

7.512405872344971

Metharme-7b-4bit-128g-damp0.1

6.242006301879883

45.30938720703125

8.14249324798584

Metharme-7b-4bit-128g-sym-damp0.1

6.379064083099365

52.17233657836914

8.316649436950684

Metharme-7b-4bit-128g-sym-trits-damp0.1

6.585468769073486

51.55869674682617

8.2803316116333

Metharme-7b-4bit-128g-trits-damp0.1

6.262309551239014

47.80055618286133

8.192194938659668

Metharme-7b-4bit-128g

10.206376075744629

49.00401306152344

8.198845863342285

Metharme-7b-4bit-32g-damp0.1

6.107605934143066

47.91380310058594

8.068695068359375

Metharme-7b-4bit-32g-sym-damp0.1

6.566899299621582

48.0670166015625

8.095841407775879

Metharme-7b-4bit-32g-sym-trits-damp0.1

6.109438896179199

47.35737228393555

8.075060844421387

Metharme-7b-4bit-32g-trits-damp0.1

6.128157138824463

48.04175567626953

8.0708646774292

Metharme-7b-4bit-32g

6.902080535888672

50.237754821777344

8.081602096557617

Metharme-7b-4bit-alone

6.997134685516357

58.87525177001953

8.824191093444824

Metharme-7b-4bit-ao-128g

251321.265625

250117.859375

232929.234375

Metharme-7b-4bit-ao-32g

275425.5

267733.25

254506.71875

Metharme-7b-4bit-ao-damp0.1

6.220707893371582

47.82929611206055

7.884565353393555

Metharme-7b-4bit-ao-sym

6.477842807769775

48.53507614135742

7.993765354156494

Metharme-7b-4bit-ao

6.281311511993408

46.79158401489258

7.906069755554199

Metharme-7b-4bit-damp0.1

6.495014190673828

49.39763641357422

8.68167781829834

Metharme-7b-4bit-trits

7.832409858703613

55.383026123046875

8.806737899780273

Metharme-7b-4bit-ts-128g-damp0.1

6.17774772644043

46.47630310058594

8.20170783996582

Metharme-7b-4bit-ts-128g-sym-damp0.1

6.584965705871582

55.20026397705078

8.268644332885742

Metharme-7b-4bit-ts-128g-sym-trits-damp0.1

6.225503921508789

53.12746047973633

8.240595817565918

Metharme-7b-4bit-ts-128g-trits-damp0.1

6.289113521575928

46.06787109375

8.122251510620117

Metharme-7b-4bit-ts-128g

9.475017547607422

52.358829498291016

8.340700149536133

Metharme-7b-4bit-ts-32g-damp0.1

6.219024658203125

45.834869384765625

8.071272850036621

Metharme-7b-4bit-ts-32g-sym-damp0.1

6.274552822113037

47.35737228393555

8.06270980834961

Metharme-7b-4bit-ts-32g-sym-trits-damp0.1

6.266031265258789

47.346702575683594

8.068148612976074

Metharme-7b-4bit-ts-32g-trits-damp0.1

6.118431568145752

45.67333221435547

8.077078819274902

Metharme-7b-4bit-ts-32g

6.424218654632568

48.48588943481445

8.089512825012207

Metharme-7b-4bit-ts-damp0.1

6.7783050537109375

51.81301498413086

8.621373176574707

Metharme-7b-4bit-ts-sym-damp0.1

6.7517595291137695

54.06147384643555

8.821818351745605

Metharme-7b-4bit-ts-sym-trits-damp0.1

6.886506080627441

64.72743225097656

8.880627632141113

Metharme-7b-4bit-ts-trits-damp0.1

6.631694793701172

51.7371711730957

8.656966209411621

Metharme-7b-4bit-ts

7.735969543457031

62.92238235473633

8.913650512695312

Metharme-7b-4bit-128g-sym-trits

7.056288242340088

48.983768463134766

8.339276313781738

Metharme-7b-4bit-128g-sym

6.9575653076171875

49.356834411621094

8.35644817352295

Metharme-7b-4bit-128g-trits

8.68796443939209

49.73833465576172

8.2406587600708

Metharme-7b-4bit-32g-sym-trits

6.206963539123535

46.88833999633789

8.13610553741455

Metharme-7b-4bit-32g-sym

6.518134117126465

49.66925811767578

8.13516616821289

Metharme-7b-4bit-32g-trits

7.017086029052734

45.04129409790039

8.074845314025879

Metharme-7b-4bit-ts-128g-sym-trits

6.756448745727539

51.510311126708984

8.292160987854004

Metharme-7b-4bit-ts-128g-sym

6.819341659545898

55.28740310668945

8.377721786499023

Metharme-7b-4bit-ts-128g-trits

7.350858688354492

44.25314712524414

8.274221420288086

Metharme-7b-4bit-ts-32g-sym-trits

6.82415771484375

48.82029724121094

8.090987205505371

Metharme-7b-4bit-ts-32g-sym

6.387663841247559

48.07244110107422

8.173730850219727

Metharme-7b-4bit-ts-32g-trits

6.632943153381348

47.973228454589844

8.013848304748535

Metharme-7b-4bit-ts-sym-trits

7.599876403808594

55.75454330444336

8.932201385498047

Metharme-7b-4bit-ts-sym

7.270263671875

54.35262680053711

8.787986755371094

Metharme-7b-4bit-ts-trits

7.2306809425354

66.78710174560547

8.879831314086914

Metharme-ao-sym-trits-damp0.1

6.329384803771973

48.06882858276367

7.959168910980225

Metharme-ao-sym-trits

6.390106678009033

48.15375900268555

7.9804582595825195

Metharme-ao-sym

6.477842807769775

48.53507614135742

7.993765354156494

Metharme-ao-trits-damp0.1

6.163661956787109

46.50249099731445

7.877425193786621

Metharme-ao-trits

6.283935546875

46.57590103149414

7.907411575317383

Metharme-ao-ts-damp0.1

6.184001445770264

46.17180633544922

7.880400657653809

Metharme-ao-ts-sym-damp0.1

6.210323333740234

47.66789245605469

7.952476978302002

Metharme-ao-ts-sym-trits-damp0.1

6.232576370239258

48.081459045410156

7.95023250579834

Metharme-ao-ts-sym-trits

6.471063137054443

49.650611877441406

7.969552040100098

Metharme-ao-ts-sym

6.460526943206787

47.190460205078125

7.9732160568237305

Metharme-ao-ts-trits-damp0.1

6.172733783721924

45.19890213012695

7.872506141662598

Metharme-ao-ts-trits

6.310682773590088

46.4483757019043

7.898126602172852

Metharme-ao-ts

6.329496383666992

46.88129806518555

7.910323143005371

Metharme-sym-damp0.1

6.896804332733154

57.4250602722168

8.703770637512207

Metharme-sym-trits-damp0.1

7.075908184051514

59.13897705078125

8.919178009033203

Metharme-sym-trits

7.494253635406494

63.320709228515625

8.969240188598633

Metharme-sym

7.585672855377197

61.01168441772461

9.032520294189453

Metharme-trits

7.832409858703613

55.383026123046875

8.806737899780273

.hf-sanitized.hf-sanitized-0FrWBjGbVmsncwsf2Zc-R table { border: 1px solid #b3adad; border-collapse: collapse; padding: 5px; } .hf-sanitized.hf-sanitized-0FrWBjGbVmsncwsf2Zc-R table th { border: 1px solid #b3adad; padding: 5px; background: #f0f0f0; color: #313030; } .hf-sanitized.hf-sanitized-0FrWBjGbVmsncwsf2Zc-R table td { border: 1px solid #b3adad; text-align: center; padding: 5px; background: #ffffff; color: #313030; }


This summary was produced with help from an AI and may contain inaccuracies - check out the links to read the original source documents!

Privacy Policy

Terms of Service