@novastera-oss/llamarn

LlamaRN

A thin, reliable React Native Turbo Module wrapping llama.cpp for on-device LLM inference on iOS and Android.

Features

• Model loading, text completion, and chat completion
• Metal GPU on iOS; OpenCL / Vulkan / Hexagon NPU on Android
• Automatic CPU/GPU detection and optimal GPU layer estimation
• Chat completion with Jinja template support
• Multi-turn KV cache prefix reuse (pass a stable id per message)
• Embeddings generation
• Function / tool calling
• Thinking and reasoning model support (reasoning_budget, reasoning_format)
• Multimodal / Vision — image-in-prompt chat, CLIP-style embeddings, Whisper-style transcription, open-ended vision reasoning, and zero-copy camera frame pipeline

Breaking changes

See BREAKING.md for migration notes (thinking models / reasoning_content, KV cache behavior, GGUF + Jinja + tools pitfalls) and the v0.7.0 release summary.

Installation

npm install @novastera-oss/llamarn

Developer Setup

Prerequisites

1. Clone the repository
2. React Native development environment for your target platform(s)

Initial Setup

npm install
npm run setup-llama-cpp        # init llama.cpp submodule

Android

./scripts/build_android_external.sh   # build native libraries
cd example && npm run android

iOS

cd example/ios && bundle exec pod install
cd .. && npm run ios

Troubleshooting:

• Android: cd android && ./gradlew clean
• iOS: cd example/ios && rm -rf build Podfile.lock && pod install

---

Basic Usage

Simple Completion

import { initLlama } from '@novastera-oss/llamarn';

const context = await initLlama({
  model: '/path/to/model.gguf',
  n_ctx: 2048,
  n_batch: 512,
});

const result = await context.completion({
  prompt: 'What is artificial intelligence?',
  temperature: 0.7,
  top_p: 0.95,
});

console.log(result.text);

Chat Completion

const context = await initLlama({
  model: '/path/to/model.gguf',
  n_ctx: 4096,
  use_jinja: true,
});

const result = await context.completion({
  messages: [
    { role: 'system', content: 'You are a helpful assistant.' },
    { role: 'user',   content: 'Tell me about quantum computing.' },
  ],
  temperature: 0.7,
});

console.log(result.text);
// OpenAI-compatible: result.choices[0].message.content

Multi-Turn Chat with KV Cache Reuse

Assign a stable id to each message. The native layer re-uses KV cache entries for messages whose ID matches the previous turn — only new tokens are encoded.

const history = [
  { role: 'system',    content: 'You are a helpful assistant.', id: 'sys-1'    },
  { role: 'user',      content: 'Hi!',                          id: 'turn-1-u' },
];

const r1 = await context.completion({ messages: history });

history.push({ role: 'assistant', content: r1.text,         id: 'turn-1-a' });
history.push({ role: 'user',      content: 'Tell me more.', id: 'turn-2-u' });

// Only [turn-2-u] is encoded; everything before hits the cache
const r2 = await context.completion({ messages: history });

Rules:

• Messages without an id are always fully re-encoded.
• If you edit a message's content, change its id so the cache is invalidated.
• reset_kv_cache: true forces a full clear.

await context.completion({ messages: history, reset_kv_cache: true });

Completion parameter naming

Completion request keys are strict snake_case to match the native layer (top_p, top_k, min_p, repeat_penalty, frequency_penalty, presence_penalty, reset_kv_cache, etc.). CamelCase aliases are not parsed by the native bridge.

Thinking and Reasoning Models

const context = await initLlama({
  model: '/path/to/reasoning-model.gguf',
  n_ctx: 4096,
  use_jinja: true,
  reasoning_budget: -1,          // -1 = unlimited, 0 = disabled, >0 = token limit
  reasoning_format: 'deepseek',  // 'none' | 'auto' | 'deepseek' | 'deepseek-legacy'
  thinking_forced_open: true,
});

const result = await context.completion({
  messages: [{ role: 'user', content: 'Solve: what is 15% of 240?' }],
});

// result.text may include <think>…</think> depending on the model

See BREAKING.md for multi-turn thinking, KV cache, and tool-calling pitfalls.

Tool Calling

const context = await initLlama({
  model: '/path/to/model.gguf',
  n_ctx: 2048,
  use_jinja: true,          // parse_tool_calls enabled automatically
  parallel_tool_calls: false,
});

const response = await context.completion({
  messages: [
    { role: 'system', content: 'You are a helpful assistant.' },
    { role: 'user',   content: "What's the weather in Paris?" },
  ],
  tools: [{
    type: 'function',
    function: {
      name: 'get_weather',
      description: 'Get the current weather in a location',
      parameters: {
        type: 'object',
        properties: {
          location: { type: 'string' },
          unit:     { type: 'string', enum: ['celsius', 'fahrenheit'] },
        },
        required: ['location'],
      },
    },
  }],
  tool_choice: 'auto',
});

if (response.choices?.[0]?.finish_reason === 'tool_calls') {
  const call = response.choices[0].message.tool_calls[0];
  console.log(call.function.name, call.function.arguments);
}

Embeddings

const context = await initLlama({
  model: '/path/to/embedding-model.gguf',
  embedding: true,
  n_ctx: 2048,
});

const { data } = await context.embedding({ input: 'Text to embed' });
console.log(data[0].embedding); // Float32 array

---

Multimodal / Vision

llamarn supports image and audio input for compatible models (LLaVA, Qwen-VL, Whisper) via a separate multimodal projection model (mmproj .gguf).

Init with capabilities

const model = await initLlama({
  model:        '/path/to/model.gguf',
  mmproj:       '/path/to/mmproj.gguf',
  capabilities: ['vision-chat'],  // 'image-encode' | 'audio-transcribe' | 'vision-reasoning'
  n_ctx: 4096,
  use_jinja: true,
});

const mods = await model.getSupportedModalities();
// { vision: true, audio: false }

const enabled = await model.isMultimodalEnabled();
// true

Supported capabilities:

Value	Description
vision-chat	Image-in-prompt chat completions (LLaVA, Qwen-VL)
image-encode	CLIP-style image embeddings
audio-transcribe	Whisper-style audio → text
vision-reasoning	Open-ended image analysis returning raw model text

Vision Chat

const result = await model.completion({
  messages: [{
    role: 'user',
    content: [
      { type: 'text',      text: 'What is in this image?' },
      { type: 'image_url', image_url: { url: 'file:///path/to/image.jpg' } },
    ],
  }],
  n_predict: 256,
});

console.log(result.text);

Multiple images per message are supported. file:// URIs and data:image/...;base64,... strings are both accepted. KV cache prefix reuse is automatically disabled when messages contain images.

Image Embeddings

const { embedding, n_tokens, n_embd } = await model.embedImage(
  'file:///path/to/image.jpg',
  { normalize: true },
);
// embedding: flat Float32 array, length = n_tokens * n_embd

Audio Transcription

const { text } = await model.transcribeAudio('file:///path/to/audio.wav');

Open-ended Vision Reasoning

const { raw_text } = await model.visionReasoning(
  'file:///path/to/image.jpg',
  { prompt: 'List all objects you see.' },
);

Camera Frame (zero-copy)

Works with VisionCamera NativeBuffer frames — no disk I/O required.

import { useFrameProcessor, runAsync } from 'react-native-vision-camera';

const frameProcessor = useFrameProcessor((frame) => {
  'worklet';
  runAsync(frame, async () => {
    // maxSize: 336 downsamples to ~330 KB during the pixel copy (recommended for LLaVA/CLIP)
    // Omit maxSize (or 0) for full-resolution analysis (OCR, detailed scenes)
    const result = await model.runOnFrame(
      frame, frame.width, frame.height, 'image-encode', { maxSize: 336 });

    if (result) {
      // result.embedding — Float32 array (n_tokens × n_embd)
      console.log('Tokens:', result.n_tokens);
    }
    // null means the encoder was busy with the previous frame — dropped automatically
  });
}, [model]);

Frame dropping is handled automatically in C++ via an atomic flag — no JS-side throttle needed.

Supported model families:

Model	Capability
LLaVA 1.5 / 1.6	vision-chat
Qwen2-VL / Qwen3-VL	vision-chat
LLaMA 4 Scout	vision-chat
MiniCPM-V	vision-chat
Whisper	audio-transcribe
CLIP	image-encode

---

Model Info and Optimal Initialization

loadLlamaModelInfo queries a model's properties without fully loading it. The returned values are designed to be passed directly to initLlama — this is the recommended initialization pattern.

Text-only model

import { loadLlamaModelInfo, initLlama } from '@novastera-oss/llamarn';

const info = await loadLlamaModelInfo('/path/to/model.gguf');

const model = await initLlama({
  model:          '/path/to/model.gguf',
  n_ctx:          4096,
  n_gpu_layers:   info.optimalGpuLayers,   // device-safe GPU layer count
  chunk_size:     info.suggestedChunkSize,  // cooperative ingestion chunk size
  is_cpu_only:    info.isCpuOnly,
  prompt_chunk_gap_ms: 5,                  // GPU minimum inter-chunk gap (thermal pacing)
  use_jinja:      true,
});

Vision model with mmproj

Pass mmprojPath so the VRAM budget is split between the LLM and the projection model before computing optimalGpuLayers:

const info = await loadLlamaModelInfo('/path/to/model.gguf', '/path/to/mmproj.gguf');

const model = await initLlama({
  model:          '/path/to/model.gguf',
  mmproj:         '/path/to/mmproj.gguf',
  capabilities:   ['vision-chat'],
  n_ctx:          4096,
  n_gpu_layers:   info.optimalGpuLayers,   // already accounts for mmproj VRAM
  chunk_size:     info.suggestedChunkSize,
  is_cpu_only:    info.isCpuOnly,
  prompt_chunk_gap_ms: 5,
  use_jinja:      true,
});

console.log(info.mmprojSizeMB);      // MB reserved for the projection model

All returned fields

console.log(info.description);         // human-readable model name + quant
console.log(info.n_params);            // parameter count
console.log(info.n_layers);            // total layer count
console.log(info.model_size_bytes);    // quantized on-disk size in bytes
console.log(info.optimalGpuLayers);    // recommended n_gpu_layers for this device
console.log(info.availableMemoryMB);   // current free device RAM in MB
console.log(info.estimatedVramMB);     // estimated VRAM needed for optimalGpuLayers
console.log(info.architecture);        // e.g. "llama", "qwen2", "mistral"
console.log(info.gpuSupported);        // true if GPU offload is available
console.log(info.suggestedChunkSize);  // 32 (CPU-only) or 128 (GPU) — pass as chunk_size
console.log(info.isCpuOnly);           // true when optimalGpuLayers == 0
console.log(info.mmprojSizeMB);        // present only when mmprojPath was supplied
console.log(info.samplingDefaults);    // GGUF-embedded sampling params (see below)

Sampling defaults from the model

loadLlamaModelInfo also reads any sampling parameters the model author embedded in the GGUF file and returns them as samplingDefaults. Only fields the model actually specifies are present — if a model doesn't embed a value, the key is absent.

const info = await loadLlamaModelInfo('/path/to/model.gguf');
const sd = info.samplingDefaults ?? {};

// Merge: your explicit values > GGUF defaults > nothing
const result = await model.completion({
  messages,
  temperature:    sd.temperature    ?? 0.8,
  top_p:          sd.top_p          ?? 0.9,
  top_k:          sd.top_k          ?? 40,
  min_p:          sd.min_p          ?? 0.05,
  repeat_penalty: sd.repeat_penalty ?? 1.1,
});

Important: When you omit a sampling parameter from a completion() call entirely, the native layer now falls back to the GGUF-embedded defaults (loaded at initLlama time) rather than llama.cpp hardcoded values. This means omitting a field is safe and intentional — you only need to pass a value when you want to override what the model recommends. See BREAKING.md for the full priority chain.

---

Advanced Configuration

GPU and Memory Parameters

Parameter	Default	Description
n_gpu_layers	0	Layers to offload to GPU — use optimalGpuLayers from loadLlamaModelInfo
n_ctx	2048	Context window size
n_batch	512	Prompt processing batch allocation size
use_mmap	true	Memory-mapped loading (faster startup)
use_mlock	false	Lock model in RAM (prevents swapping)

loadLlamaModelInfo applies automatic device-tier VRAM budgeting when computing optimalGpuLayers:

• ≥ 8 GB RAM (flagship — A17 Pro, M-series, Snapdragon 8 Gen 3): 20% of total RAM
• ≥ 6 GB RAM (upper-mid — A15/A16, Snapdragon 8 Gen 2): 18% of total RAM
• < 6 GB RAM (budget / older devices): 15% of total RAM (conservative, avoids GPU fence timeouts)

Pass n_gpu_layers: info.optimalGpuLayers to initLlama and the right budget is applied automatically.

Android arm64-v8a: CPU inference uses KleidiAI v1.24.0 microkernels (SVE-256, SME/SME2, optimized GEMM for Q4_0/Q8_0/Q4_K/Q5_K). The runtime selects the best available kernel based on CPU features — no configuration needed. Older devices fall back to baseline kernels.

Cooperative Ingestion Parameters

These control how the prompt is encoded into the KV cache — smaller chunks with OS yields prevent display fence timeouts on Android and UI starvation on CPU-only devices. Use the values from loadLlamaModelInfo directly.

Parameter	Default	Description
chunk_size	128	Tokens per llama_decode call during prompt encoding (8–512, independent of n_batch)
is_cpu_only	false	true → 2 ms sleep after each chunk
prompt_chunk_gap_ms	5	GPU minimum inter-chunk gap; increase on thermally constrained devices

Completion pacing and cache keys

completion() now supports runtime pacing and cache-key controls:

Parameter	Default	Description
token_rate_cap	30	Max generated tokens/sec (0 = uncapped)
token_buffer_size	4	Stream callback flush cadence in tokens
prompt_id	""	Cache key for prompt/template/tool identity
config_id	""	Cache key for effective completion config identity (include tools + main system prompt identity)

Use prompt_id and config_id together. If either is missing, config-cache reuse is skipped. Changes to completion config are expected to take effect when config_id changes.

const promptSignature = {
  systemPromptVersion: 'support-bot-v3',
  toolNames: tools.map(t => t.function.name).sort(),
  template: 'chatml',
};

const configSignature = {
  model: 'qwen3-8b-q4_k_m',
  temperature: 0.6,
  top_p: 0.95,
  top_k: 40,
  min_p: 0.05,
  repeat_penalty: 1.05,
  repeat_last_n: 64,
  frequency_penalty: 0.0,
  presence_penalty: 0.0,
  tool_choice: 'auto',
  response_format: 'text',
};

// Use any stable JSON + hash utility available in your app.
const stableJson = (value: object) =>
  JSON.stringify(Object.keys(value).sort().reduce((acc, key) => {
    acc[key] = (value as Record<string, unknown>)[key];
    return acc;
  }, {} as Record<string, unknown>));

const prompt_id = `prompt-${sha256Hex(stableJson(promptSignature)).slice(0, 16)}`;
const config_id = `config-${sha256Hex(stableJson(configSignature)).slice(0, 16)}`;

const result = await model.completion({
  messages,
  tools,
  prompt_id,
  config_id,
  token_rate_cap: 30,
  token_buffer_size: 4,
});

When tool-call parsing fails, finish_reason can now be tool_call_parse_error and tool_call_parse_error will contain the parser error message.

Thinking and Reasoning Parameters

Parameter	Default	Description
reasoning_budget	-1	Token budget for thinking: -1 unlimited, 0 disabled, >0 limited
reasoning_format	'auto'	'none' | 'auto' | 'deepseek' | 'deepseek-legacy'
thinking_forced_open	false	Force thinking tags in every response
parse_tool_calls	true	Parse tool call JSON from output (auto-enabled with use_jinja)
parallel_tool_calls	false	Allow multiple tool calls per response

---

Model Path Handling

iOS

• Bundle path (Xcode resource): ${RNFS.MainBundlePath}/model.gguf
• Absolute path: /var/mobile/…/model.gguf

Android

• Cache directory: ${RNFS.CachesDirectoryPath}/model.gguf
• App assets (copied to cache first): RNFS.copyFileAssets('model.gguf', dest)

---

Documentation

• Interface Documentation — Detailed API interfaces
• Breaking Changes — Migration notes
• Example App — Working example with text chat and vision demo
• Contributing Guide

---

About Novastera

LlamaRN is part of the Novastera open-source ecosystem. This library powers on-device LLM inference in Novastera's mobile applications — no data leaves the user's device.

Learn more: https://novastera.com/resources

License

Apache 2.0

Acknowledgments

• mybigday/llama.rn — foundational React Native binding for llama.cpp
• ggml-org/llama.cpp — the core C++ inference library
• react-native-pure-cpp-turbo-module-library — reference for the Android C++ Turbo Module pattern