Web Framework (Axum)¶

Build production-ready REST APIs for LLM inference using Mullama's integrated Axum web framework support.

Feature Gate

This feature requires the web feature flag, which transitively enables async.

[dependencies]
mullama = { version = "0.3", features = ["web"] }

Overview¶

Mullama provides first-class integration with the Axum web framework, offering:

AppState for shared model and context management
AppStateBuilder pattern for configuration
RouterBuilder for auto-generated REST endpoints
Built-in endpoints for generate, tokenize, health, and metrics
Middleware support (CORS, timeout, rate limiting, logging)
SSE streaming for real-time token delivery
AppError for structured error responses

AppState¶

AppState manages shared resources across all request handlers, including the loaded model, metrics, and configuration.

Node.jsPythonRustCLI

const { createServer } = require('mullama');

const server = await createServer({
  model: 'model.gguf',
  port: 3000,
  streaming: true,
  metrics: true,
  maxConcurrentRequests: 64,
  rateLimit: { requests: 100, windowSecs: 60 }
});

await server.start();

from mullama import create_server

server = await create_server(
    model="model.gguf",
    port=3000,
    streaming=True,
    metrics=True,
    max_concurrent_requests=64,
    rate_limit={"requests": 100, "window_secs": 60}
)

await server.start()

use mullama::{AppState, AsyncModel, ApiMetrics};
use std::sync::Arc;

let model = Arc::new(AsyncModel::load("model.gguf").await?);

let app_state = AppState::new(model)
    .enable_streaming()
    .enable_metrics()
    .max_concurrent_requests(64)
    .rate_limit(100, Duration::from_secs(60))
    .build();

# Start server with default configuration
mullama serve --model model.gguf --port 3000

# With rate limiting and metrics
mullama serve --model model.gguf \
  --port 3000 \
  --max-concurrent 64 \
  --rate-limit 100 \
  --metrics

AppStateBuilder Methods¶

Method	Description	Default
`enable_streaming()`	Enable SSE streaming endpoints	Disabled
`enable_metrics()`	Enable `/metrics` endpoint	Disabled
`max_concurrent_requests(n)`	Limit concurrent requests	32
`rate_limit(requests, window)`	Rate limiting per client	None

Accessing State in Handlers¶

use axum::extract::State;
use std::sync::Arc;

async fn my_handler(
    State(state): State<Arc<AppState>>,
) -> impl IntoResponse {
    let model = state.model();
    let metrics = state.metrics();
    let is_streaming = state.streaming_enabled();
    // ...
}

RouterBuilder¶

The create_router function generates a fully-configured Axum Router with default endpoints.

use mullama::{create_router, AppState};

let app = create_router(app_state);

Default Endpoints¶

Method	Path	Description
`POST`	`/generate`	Text generation
`POST`	`/tokenize`	Text tokenization
`POST`	`/embeddings`	Generate embeddings
`GET`	`/metrics`	Performance metrics
`GET`	`/health`	Health check
`WS`	`/ws`	WebSocket streaming

Adding Custom Endpoints¶

use axum::{routing::{get, post}, Router};

let app = create_router(app_state)
    .route("/custom/summarize", post(summarize_handler))
    .route("/custom/status", get(status_handler));

Request/Response Types¶

GenerateRequest¶

#[derive(Serialize, Deserialize)]
pub struct GenerateRequest {
    pub prompt: String,
    pub max_tokens: Option<usize>,
    pub temperature: Option<f32>,
    pub top_k: Option<u32>,
    pub top_p: Option<f32>,
    pub stop_sequences: Option<Vec<String>>,
    pub stream: Option<bool>,
}

GenerateResponse¶

#[derive(Serialize, Deserialize)]
pub struct GenerateResponse {
    pub text: String,
    pub tokens_generated: usize,
    pub processing_time_ms: u64,
    pub model_info: Option<ModelInfo>,
}

TokenizeRequest / TokenizeResponse¶

#[derive(Serialize, Deserialize)]
pub struct TokenizeRequest {
    pub text: String,
    pub add_bos: Option<bool>,
    pub special_tokens: Option<bool>,
}

#[derive(Serialize, Deserialize)]
pub struct TokenizeResponse {
    pub tokens: Vec<TokenId>,
    pub token_count: usize,
    pub text_length: usize,
}

Middleware Integration¶

Mullama's web layer integrates seamlessly with tower-http middleware for production deployments.

CORS¶

Node.jsPythonRust

const server = await createServer({
  model: 'model.gguf',
  cors: {
    origins: ['https://myapp.com'],
    methods: ['GET', 'POST'],
    headers: ['Content-Type', 'Authorization']
  }
});

server = await create_server(
    model="model.gguf",
    cors={
        "origins": ["https://myapp.com"],
        "methods": ["GET", "POST"],
        "headers": ["Content-Type", "Authorization"]
    }
)

use tower_http::cors::{CorsLayer, Any};

let cors = CorsLayer::new()
    .allow_origin(Any)
    .allow_methods(Any)
    .allow_headers(Any);

let app = create_router(app_state).layer(cors);

Timeout and Rate Limiting¶

use tower_http::timeout::TimeoutLayer;
use std::time::Duration;

let app = create_router(app_state)
    .layer(TimeoutLayer::new(Duration::from_secs(30)));

Combined Middleware Stack¶

use tower_http::{
    cors::CorsLayer,
    trace::TraceLayer,
    compression::CompressionLayer,
    timeout::TimeoutLayer,
};
use std::time::Duration;

let app = create_router(app_state)
    .layer(CorsLayer::permissive())
    .layer(TraceLayer::new_for_http())
    .layer(CompressionLayer::new())
    .layer(TimeoutLayer::new(Duration::from_secs(30)));

ApiMetrics¶

Built-in metrics collection for monitoring API performance.

let metrics: &ApiMetrics = state.metrics();

// Metrics include:
// - Total requests served
// - Average response time
// - Tokens generated per second
// - Active connections
// - Error rate

Prometheus Integration

When using the daemon feature, metrics are automatically exposed in Prometheus format at /metrics.

SSE Streaming¶

Stream generated tokens to clients in real-time using Server-Sent Events.