Open Source: #inference

Catalog projects marked with #inference. Tags work as dedicated landing pages, so related tools are easier to find and connect.

This collection holds 4 projects with a combined 296,600 GitHub stars. Main languages: Python, C++.

Filters

Reset

Found: 4

Tag: #inference ×

ggml-org/llama.cpp

llama.cpp

llama.cpp runs large language models locally, from command-line inference to an OpenAI-compatible API server.

Stars 115,212 Forks 19,283 Author ggml-org Language C++ License MIT

#ai #llm #cplusplus #inference #cli #local-ai

Open

vllm-project/vllm

vLLM

vLLM is a high-performance engine for LLM inference and serving with an OpenAI-compatible API, batching, and efficient memory management.

Stars 82,414 Forks 17,884 Author vllm-project Language Python License Apache-2.0

#ai #llm #inference #python #serving #machine-learning

Open

meta-llama/llama

Llama

Llama is Meta’s repository with inference code for earlier Llama models and links to newer family repositories.

Stars 59,472 Forks 9,792 Author meta-llama Language Python License NOASSERTION

#ai #llm #python #inference #meta

Open

microsoft/BitNet

BitNet

BitNet is Microsoft’s inference framework for 1-bit LLMs.

Stars 39,502 Forks 3,618 Author microsoft Language Python License MIT

#llm #inference #python #microsoft

Open