1  ---
  2  title: "TogetherAIGenerator"
  3  id: togetheraigenerator
  4  slug: "/togetheraigenerator"
  5  description: "This component enables text generation using models hosted on Together AI."
  6  ---
  7  
  8  # TogetherAIGenerator
  9  
 10  This component enables text generation using models hosted on Together AI.
 11  
 12  <div className="key-value-table">
 13  
 14  |  |  |
 15  | --- | --- |
 16  | **Most common position in a pipeline** | After a [`PromptBuilder`](../builders/promptbuilder.mdx) |
 17  | **Mandatory init variables** | `api_key`: A Together API key. Can be set with `TOGETHER_API_KEY` env var. |
 18  | **Mandatory run variables** | `prompt`: A string containing the prompt for the LLM |
 19  | **Output variables** | `replies`: A list of strings with all the replies generated by the LLM  <br /> <br />`meta`: A list of dictionaries with the metadata associated with each reply, such as token count, finish reason, and so on |
 20  | **API reference** | [TogetherAI](/reference/integrations-togetherai) |
 21  | **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/togetherai |
 22  
 23  </div>
 24  
 25  ## Overview
 26  
 27  `TogetherAIGenerator` supports models hosted on [Together AI](https://docs.together.ai/intro), such as `meta-llama/Llama-3.3-70B-Instruct-Turbo`. For the full list of supported models, see [Together AI documentation](https://docs.together.ai/docs/chat-models).
 28  
 29  This component needs a prompt string to operate. You can pass any text generation parameters valid for the Together AI chat completion API directly to this component using the `generation_kwargs` parameter in `__init__` or the `generation_kwargs` parameter in `run` method. For more details on the parameters supported by the Together AI API, see [Together AI API documentation](https://docs.together.ai/reference/chat-completions-1).
 30  
 31  You can also provide an optional `system_prompt` to set context or instructions for text generation. If not provided, the system prompt is omitted, and the default system prompt of the model is used.
 32  
 33  To use this integration, you need to have an active TogetherAI subscription with sufficient credits and an API key. You can provide it with:
 34  
 35  - The `TOGETHER_API_KEY` environment variable (recommended)
 36  - The `api_key` init parameter and Haystack [Secret](../../concepts/secret-management.mdx) API: `Secret.from_token("your-api-key-here")`
 37  
 38  By default, the component uses Together AI's OpenAI-compatible base URL `https://api.together.xyz/v1`, which you can override with `api_base_url` if needed.
 39  
 40  ### Streaming
 41  
 42  `TogetherAIGenerator` supports [streaming](guides-to-generators/choosing-the-right-generator.mdx#streaming-support) responses from the LLM, allowing tokens to be emitted as they are generated. To enable streaming, pass a callable to the `streaming_callback` parameter during initialization.
 43  
 44  :::info
 45  This component is designed for text generation, not for chat. If you want to use Together AI LLMs for chat, use [`TogetherAIChatGenerator`](togetheraichatgenerator.mdx) instead.
 46  :::
 47  
 48  ## Usage
 49  
 50  Install the `togetherai-haystack` package to use the `TogetherAIGenerator`:
 51  
 52  ```shell
 53  pip install togetherai-haystack
 54  ```
 55  
 56  ### On its own
 57  
 58  Basic usage:
 59  
 60  ```python
 61  from haystack_integrations.components.generators.togetherai import TogetherAIGenerator
 62  
 63  client = TogetherAIGenerator(model="meta-llama/Llama-3.3-70B-Instruct-Turbo")
 64  response = client.run("What's Natural Language Processing? Be brief.")
 65  print(response)
 66  
 67  >> {'replies': ['Natural Language Processing (NLP) is a branch of artificial intelligence
 68  >> that focuses on enabling computers to understand, interpret, and generate human language
 69  >> in a way that is meaningful and useful.'],
 70  >> 'meta': [{'model': 'meta-llama/Llama-3.3-70B-Instruct-Turbo', 'index': 0,
 71  >> 'finish_reason': 'stop', 'usage': {'prompt_tokens': 15, 'completion_tokens': 36,
 72  >> 'total_tokens': 51}}]}
 73  ```
 74  
 75  With streaming:
 76  
 77  ```python
 78  from haystack_integrations.components.generators.togetherai import TogetherAIGenerator
 79  
 80  client = TogetherAIGenerator(
 81      model="meta-llama/Llama-3.3-70B-Instruct-Turbo",
 82      streaming_callback=lambda chunk: print(chunk.content, end="", flush=True),
 83  )
 84  
 85  response = client.run("What's Natural Language Processing? Be brief.")
 86  print(response)
 87  ```
 88  
 89  With system prompt:
 90  
 91  ```python
 92  from haystack_integrations.components.generators.togetherai import TogetherAIGenerator
 93  
 94  client = TogetherAIGenerator(
 95      model="meta-llama/Llama-3.3-70B-Instruct-Turbo",
 96      system_prompt="You are a helpful assistant that provides concise answers.",
 97  )
 98  
 99  response = client.run("What's Natural Language Processing?")
100  print(response["replies"][0])
101  ```
102  
103  ### In a Pipeline
104  
105  ```python
106  from haystack import Pipeline, Document
107  from haystack.components.retrievers.in_memory import InMemoryBM25Retriever
108  from haystack.components.builders.prompt_builder import PromptBuilder
109  from haystack.document_stores.in_memory import InMemoryDocumentStore
110  from haystack_integrations.components.generators.togetherai import TogetherAIGenerator
111  
112  docstore = InMemoryDocumentStore()
113  docstore.write_documents([
114      Document(content="Rome is the capital of Italy"),
115      Document(content="Paris is the capital of France")
116  ])
117  
118  query = "What is the capital of France?"
119  
120  template = """
121  Given the following information, answer the question.
122  
123  Context:
124  {% for document in documents %}
125      {{ document.content }}
126  {% endfor %}
127  
128  Question: {{ query }}?
129  """
130  
131  pipe = Pipeline()
132  pipe.add_component("retriever", InMemoryBM25Retriever(document_store=docstore))
133  pipe.add_component("prompt_builder", PromptBuilder(template=template))
134  pipe.add_component("llm", TogetherAIGenerator(model="meta-llama/Llama-3.3-70B-Instruct-Turbo"))
135  
136  pipe.connect("retriever", "prompt_builder.documents")
137  pipe.connect("prompt_builder", "llm")
138  
139  result = pipe.run({
140      "prompt_builder": {"query": query},
141      "retriever": {"query": query}
142  })
143  
144  print(result)
145  
146  >> {'llm': {'replies': ['The capital of France is Paris.'],
147  >> 'meta': [{'model': 'meta-llama/Llama-3.3-70B-Instruct-Turbo', ...}]}}
148  ```