Local LLMs
Oct 02, 2023
Just use ChatGPT
Well there are many reasons I’m not going to do that, and maybe you should rethink doing it too.
There is no denying that it is a great tool, and it greatly benefits its enormous success by accessibility and UX, as before that running and language models were not very different from coding, putting command into a terminal. Having simple, free and useful tool is apparently all you need to become the fastest growing product in the world, and by a huge margin.
I like my software free an open, and I don’t like any biases, so the current state of “open"AI isn’t exactly speaking to me.
Requirements
Well LLM’s are quite expensive in terms of hardware, depending on how big of a model you are planning to use, we are going with the best there is to offer at the given time, so mixtral 8×7B there is, or rather it’s unbiased and uncensored version with few tweaks Dolphin Mixtral. It’s not going to be fastest on my machine, but should be usable, nonetheless I’m still amazed how long of a road those local LLM have gone. I remember back when the ChatGPT got released, reading a discussion about local alternatives, and somebody saying: “Will never have anything running locally, as model as this will requite 6×A100 and 600 GB of ram and 600 TB of space”, and at the given time it was not far from truth, yet the FOSS community yet once again make everyone else look stupid.
Installation
curl https://ollama.ai/install.sh | sh
ollama run dolphin-mixtral
Truly a peak of computer engendering, who is even able to do hard stuff like that? And now you just use it as the ChatGPT, I’m sure you already know that, but doing it in terminal isn’t the best user experience, you could set up some Web UI, or install CMD tool, but since I’m doing most of my writing work in Obsidian, I could use that.
Ollama the thing to run LLM’s locally, from what I tested it’s the best and simplest, and full FOSS
BMO Chatbot is my plug-in of choice, to be able to use it with Ollama you will need to create and API socket for Obsidian.
OLLAMA_ORIGINS=* OLLAMA_HOST=127.0.0.1:11435 ollama serve
Some models might not show, so if that’s the case just use
ollama pull "model-name"
It will update the model and should fix the issue.
Knowledge compendium
stolen 😎
►FAQ: https://rentry.org/er2qd
►Glossary: https://archive.today/E013q
►Links: https://rentry.org/LocalModelsLinks
►Getting Started
- https://rentry.org/llama-mini-guide
- https://rentry.org/8-step-llm-guide
- https://rentry.org/llama_v2_sillytavern
- https://rentry.org/lmg-spoonfeed-guide
►Further Learning
- https://rentry.org/machine-learning-roadmap
- https://rentry.org/llm-training
- https://rentry.org/LocalModelsPapers
►Benchmarks
General Purpose:
- https://hf.co/spaces/HuggingFaceH4/open_llm_leaderboard
- https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard
- Programming: https://hf.co/spaces/bigcode/bigcode-models-leaderboard
- Censorbench: https://codeberg.org/jts2323/censorbench
►ERP/RP Datasets
►Alpha Calculator
►Text Gen. UI, Inference Engines