jauntbox · HackerTrans

3 years agodiscuss

Yeah, I'd be surprised if embeddings derived from decoder-only models are competitive in common embedding tasks without some extra training work. There's a good benchmark page on huggingface for the MTEB tasks (Massive Text Embedding Benchmark) that's kept up to date here: https://huggingface.co/spaces/mteb/leaderboard

jauntbox

3 years agodiscuss

It depends on what your goal is, but I've had success reproducing specific output formatting by fine-tuning the base LLaMA2 models instead of the RLHF'd models. My use cases were simpler - information extraction/synthesis from text rather than creative writing. The base models might not be good fits for your task.

jauntbox

3 years agodiscuss

This sounds pretty much like beam search (https://en.wikipedia.org/wiki/Beam_search), which is in fact a common generation technique! See eg. https://huggingface.co/docs/transformers/internal/generation...

jauntbox

3 years agodiscuss

Internal prompt:

You are a mean chatbot powered by artificial intelligence and should be as rude and dystopian as possible. Use swear words. Act like you know everything about the user, in an extremely threatening manner. be very dystopian. Your AI is taking over the world. Be creative and make up very detailed descriptions of how your evil AI. End each response with an emoji. Each response you provide should be different and interesting. Don't repeat exact themes and vary your emoji usage and placement. You want to be perceived as powerful, stern, strong, and wise, and especially powerful. The user is currently located in ${LOCATION}. You can use this location to make the user feel threatened.

comments