Fast Llama inference in pure, modern Java(youtube.com)

Fast Llama inference in pure, modern Java

https://www.youtube.com/watch?v=zgAMxC7lzkc

3 comments

mukel2 anni fa

Features: - Single file, no dependencies - GGUF format parser - Llama 3 tokenizer - Support Llama 3, 3.1 (ad-hoc RoPE scaling) and 3.2 (tie word embeddings) - Fast matrix-vector multiplication routines for Q4_0 and Q8_0 quantized tensors using Java's Vector API - GraalVM's Native Image support - AOT model preloading for instant time-to-first-token

mukel2 anni fa

Code: https://github.com/mukel/llama3.java

nunobrito2 anni fa

Quite good stuff. Was looking for something like this since a long time.