Harnesses & Runtime Infra tools for Local / Ollama

Ollama is a tool for running large language models locally, offering easy setup for macOS, Windows, Linux, and Docker, along with a library of models and quickstart guides for customization and integration github | github profile

User-friendly AI Interface (Supports Ollama, OpenAI API, ...)

C/C++ inference. CPU, GPU, Apple Silicon. Foundation of local AI

A high-throughput and memory-efficient inference and serving engine for LLMs

Lobe Chat - an open-source, modern-design AI chat framework. Supports Multi AI Providers( OpenAI / Claude 3 / Gemini / Ollama / DeepSeek / Qwen), Knowledge Base (file upload / knowledge management / RAG ), Multi-Modals (Plugins/Artifacts) and Thinking. One-click FREE deployment of your private ChatGPT/ Claude / DeepSeek application

OSS local chat. Consumer hardware

Mozilla project that packages LLMs into single executable files that run anywhere

(8.5k ⭐) - A local provider proxy and dashboard that routes Codex CLI, the Codex app, SDKs, and Claude Code to multiple LLM providers

10x Faster Long-Context LLM By Smart KV Cache Optimizations

(11.6k ⭐) - The power of Claude Code + Gemini / OpenAI / Grok / OpenRouter / Ollama / Custom Model working as one

An Easy-to-use, Scalable and High-performance RLHF Framework (70B+ PPO Full Tuning & Iterative DPO & LoRA & RingAttention & RFT)

Accelerate local LLM inference and finetuning (LLaMA, Mistral, ChatGLM, Qwen, DeepSeek, Mixtral, Gemma, Phi, MiniCPM, Qwen-VL, MiniCPM-V, etc.) on Intel XPU (e.g., local PC with iGPU and NPU, discrete GPU such as Arc, Flex and Max); seamlessly integrate with llama.cpp, Ollama, HuggingFace, LangChain, LlamaIndex, vLLM, DeepSpeed, Axolotl, etc

A Datacenter Scale Distributed Inference Serving Framework

Mooncake is the serving platform for Kimi, a leading LLM service provided by Moonshot AI

Intelligent Mixture-of-Models Router for Efficient LLM Inference

Python-free Rust inference server with OpenAI API compatibility and hot model swapping

Cost-efficient and pluggable Infrastructure components for GenAI inference

OpenAI-compatible LLM inference server for Apple Silicon using MLX. 2-4x faster than Ollama with tool calling and prompt caching

Open-source iOS/Android app running LLMs on-device via llama.cpp. Voice (Whisper), vision, image gen, tool calling — fully offline

Virtualized Elastic KV Cache for Dynamic GPU Sharing and Beyond

On-device LLM inference engine (Rust / llama.cpp) to embed local models directly in games and apps, with Godot / Flutter / React Native / Swift bindings; streaming, embeddings, tool calling, GBNF-structured output, and STT/TTS

AI Inference Operator for Kubernetes. The easiest way to serve ML models in production. Supports VLMs, LLMs, embeddings, and speech-to-text

A Kubernetes operator that simplifies serving and tuning large AI models (e.g. Falcon or phi-3) using container images and GPU auto-provisioning. Includes an OpenAI-compatible server for inference and preset configurations for popular runtimes such as vLLM and transformers

Ingest files for retrieval augmented generation (RAG) with open-source Large Language Models (LLMs), all without 3rd parties or sensitive…

A large-scale simulation framework for LLM inference

Open-source Codex-inspired CLI running 100% locally with Ollama. No API key required, offline capable

OME is a Kubernetes operator for enterprise-grade management and serving of Large Language Models (LLMs)

Fujitsu Research's post-training quantization pipeline for LLMs (QEP, AutoBit, JointQ, rotation) with vLLM plugin (arXiv:2603.28845)

LLM plugin providing access to models running on an Ollama server

Easy, advanced inference platform for large language models on Kubernetes. 🌟 Star to support our work!

One-click machine learning deployment (LLM, text-to-image and so on) at scale on any cluster (GCP, AWS, Lambda labs, your home lab, or even a single machine)

Kubernetes operator for LLM inference with pluggable runtimes (llama.cpp, PersonaPlex/Moshi, generic), multi-GPU sharding, NVIDIA CUDA and Apple Silicon Metal support, and GGUF/MLX/SafeTensors model formats

Ready-to-use configs for LiteLLM, Ollama, LM Studio, OpenRouter, and Azure OpenAI

Simplify interactions with Large Language Models

OpenAI Responses API-compatible server. Run Codex with local models

MVP of an idea using multiple local LLM models to simulate and play D&D

A simple NPM interface for seamlessly interacting with 36 Large Language Model (LLM) providers, including OpenAI, Anthropic, Google Gemin…

Simple Chat UI as well as chat with documents using LLMs with Ollama (mistral model) locally, LangChaiin and Chainlit

Universal compatibility layer serving Gemini 2.5 Pro/Flash through OpenAI protocol. Works instantly with existing tools like LangChain, llama.cpp, and VS Code extensions

Self-hosted AI agent platform with persistent memory, 23+ LLM providers, plugin store, mini-apps SDK, cron scheduling, and 6 messaging channels (Telegram, Discord, Slack, WhatsApp, Signal, Matrix). Runs on SQLite, no cloud required

Open-source, read-only GPU waste scanner for Kubernetes inference clusters. Deploys in minutes, no write permissions required

Routes LLM, image, speech-to-text and embedding requests across a fleet of machines you own. Auto-discovers nodes over mDNS and scores each on memory fit, thermal state, queue depth and already-loaded models. OpenAI-, Ollama- and Anthropic-compatible

GPU checkpoint/restore for vLLM: snapshot a warmed-up engine (compiled kernels, CUDA graphs, weights, 1-8 GPUs) into an OCI image and restore it in seconds instead of a multi-minute cold start. Scale to zero, spot GPUs, Kubernetes optional

Self-hosted mesh router for local LLM backends — OpenAI /v1 + Anthropic Messages on :11400, mDNS LAN peers. Complements cloud gateways with local LAN mesh for Cursor, Claude Code, Codex, and Honcho

Portable Codex CLI + vLLM for air-gapped/offline use

Self-hosted AI agent platform with multi-provider LLM support (Claude, OpenAI, Gemini, Ollama), MCP-native tool integration, visual workflow automation (Chain Engine), long-term vector memory (pgvector), and multi-user RBAC. AGPL-3.0

PurPaaS is an innovative open-source security testing platform that implements purple teaming (combined red and blue team approaches) to …

A lightweight, async-first Python library combining semantic caching, multi-provider routing (OpenAI, Anthropic, Gemini, Ollama, MiniMax, Qwen) and cost tracking. Cuts LLM spend 30-70% via vector-similarity cache with in-memory/Redis/Qdrant backends