Depends on the version you're running.
robber
joined 2 years ago
I see. When I run the inference engine containerized, will the container be able to run its own version of CUDA or use the host's version?
Thank you for taking the time to respond.
I've used vLLM for hosting a smaller model which could fit in two of GPUs, it was very performant especially for multiple requests at the same time. The major drawback for my setup was that it only supports tensor parallelism for 2, 4, 8, etc. GPUs and data paralellism slowed inference down considerably, at least for my cards. exllamav3 is the only engine I'm aware of which support 3-way TP.
But I'm fully with you in that vLLM seems to be the most recommended and battle-tested solution.
I might take a look at how I can safely upgrade the driver until I can afford a fourth card and switch back to vLLM.
I use the the proprietary ones from Nvidia, they're at 535 on oldstable IIRC but there are a lot newer ones.
I use 3xRTX2000e Ada. It's a rather new, quite power efficient GPU manufactured by PNY.
As inference engine I use exllamav3 with tabbyAPI. I like it very much because it supports 3-way tensor paralellism, making it a lot faster for me than llamacpp.
I use the the proprietary ones from Nvidia, they're at 535 on oldstable IIRC but there are a lot newer ones.
That brian typo really gave me a chuckle. Hope you found the movie you were looking for.
The country's official app for COVID immunity certificates or whatever they were called was available on F-Droid at the time.
Too bad they've only been dropping dense models recently. Also kind of interesting since with Mixtral back in the days they were way ahead of time.
A review from earlier this year didn't sound too bad.
Edit: as pointed out, the review seems to be about the previous version of the phone.
I'd add that memory bandwidth is still a relevant factor, so the faster the RAM the faster the inference will be. I think this model would be a perfect fit for the Strix Halo or a >= 64GB Apple Silicon machine, when aiming for CPU-only inference. But mind that llamacpp does not yet support the qwen3-next architecture.
Hey fellow llama enthusiasts! Great to see that not all of lemmy is AI sceptical.
I'm in the process of upgrading my server with a bunch of GPUs. I'm really excited about the new Mistral / Magistral Small 3.2 models and would love to serve them for me and a couple of friends. My research led me to vLLM with which I was able to double inference speed compared to ollama at least for qwen3-32b-awq.
Now sadly, the most common quantization methods (GGUF, EXL, BNB) are either not fully (GGUF) or not at all (EXL) supported in vLLM, or multi-gpu inference thouth tensor parallelism is not supported (BNB). And especially for new models it's hard to find pre-quantized models in different, more broadly supported formats (AWQ, GPTQ).
Does any of you guys face a similar problem? Do you quantize models yourself? Are there any up-to-date guides you would recommend? Or did I completely overlook another, obvious solution?
It feels like when I've researched something yesterday, it's already outdated again today, since the landscape is so rapidly evolving.
Anyways, thank you for reading and sharing your thoughts or experience if you feel like it.
Text: Allows you to determine whether to limit CPUID maximum value. Set this to enabled for legacy operating systems such as Linux or Unix.
Found this in the BIOS of a Gigabyte Z97X-UD3H mobo.
view more: next ›
Given that Google generated more than 250 billion U.S. dollars in ad revenue in 2024, I'd say they must be pretty effective.
Source