Chat Completions

POST /chat/completions

Generate responses from a conversation history. Identical to the OpenAI Chat Completions API.

Request

{
  "model": "llama-3-1-8b",
  "messages": [
    {"role": "system", "content": "You are a helpful research assistant."},
    {"role": "user", "content": "Explain self-supervised learning in two paragraphs."}
  ],
  "temperature": 0.7,
  "max_tokens": 512,
  "stream": false
}

Parameters

Parameter	Type	Default	Description
model	string	—	Model ID (see Models)
messages	array	—	Conversation history (system/user/assistant roles)
temperature	float	1.0	Sampling temperature 0–2. Lower = more deterministic
max_tokens	int	model max	Maximum tokens to generate
top_p	float	1.0	Nucleus sampling probability
stream	bool	false	Stream tokens as server-sent events
stop	string/array	—	Stop sequences
seed	int	—	Seed for reproducible outputs

Streaming

stream = client.chat.completions.create(
    model="llama-3-1-8b",
    messages=[{"role": "user", "content": "Write a haiku about gradient descent."}],
    stream=True,
)
for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)

System prompts for research tasks

Good system prompts improve output quality for domain-specific tasks:

RESEARCH_SYSTEM_PROMPT = """You are a scientific research assistant with expertise in
machine learning and computational biology. When answering:
- Cite relevant concepts and methods by name
- Use precise technical language
- Acknowledge uncertainty when present
- Prefer brief, structured responses over lengthy prose"""

response = client.chat.completions.create(
    model="llama-3-1-8b",
    messages=[
        {"role": "system", "content": RESEARCH_SYSTEM_PROMPT},
        {"role": "user", "content": "What are the trade-offs between LoRA and full fine-tuning?"},
    ],
    temperature=0.3,
)

Multi-turn conversations

history = [{"role": "system", "content": "You are a helpful assistant."}]

def chat(user_message: str) -> str:
    history.append({"role": "user", "content": user_message})
    response = client.chat.completions.create(
        model="llama-3-1-8b",
        messages=history,
    )
    reply = response.choices[0].message.content
    history.append({"role": "assistant", "content": reply})
    return reply

print(chat("What is a transformer?"))
print(chat("How does attention differ from convolution?"))

Context length

The Llama 3.1 8B model supports up to 128k tokens of context. For very long documents, consider chunking and using the Embeddings endpoint for retrieval-augmented generation rather than stuffing all content into the context window.