1  ---
 2  title: "CSVDocumentCleaner"
 3  id: csvdocumentcleaner
 4  slug: "/csvdocumentcleaner"
 5  description: "Use `CSVDocumentCleaner` to clean CSV documents by removing empty rows and columns while preserving specific ignored rows and columns. It processes CSV content stored in documents and helps standardize data for further analysis."
 6  ---
 7  
 8  # CSVDocumentCleaner
 9  
10  Use `CSVDocumentCleaner` to clean CSV documents by removing empty rows and columns while preserving specific ignored rows and columns. It processes CSV content stored in documents and helps standardize data for further analysis.
11  
12  |                                        |                                                                                                                          |
13  | :------------------------------------- | :----------------------------------------------------------------------------------------------------------------------- |
14  | **Most common position in a pipeline** | In indexing pipelines after [Converters](../converters.mdx) , before [Embedders](../embedders.mdx) or [Writers](../writers/documentwriter.mdx) |
15  | **Mandatory run variables**            | "documents": A list of documents containing CSV content                                                                  |
16  | **Output variables**                   | "documents": A list of cleaned CSV documents                                                                             |
17  | **API reference**                      | [PreProcessors](/reference/preprocessors-api)                                                                                   |
18  | **GitHub link**                        | https://github.com/deepset-ai/haystack/blob/main/haystack/components/preprocessors/csv_document_cleaner.py             |
19  
20  ## Overview
21  
22  `CSVDocumentCleaner` expects a list of `Document` objects as input, each containing CSV-formatted content as text. It cleans the data by removing fully empty rows and columns while allowing users to specify the number of rows and columns to be preserved before cleaning.
23  
24  ### Parameters
25  
26  - `ignore_rows`: Number of rows to ignore from the top of the CSV table before processing. If any columns are removed, the same columns will be dropped from the ignored rows.
27  - `ignore_columns`: Number of columns to ignore from the left of the CSV table before processing. If any rows are removed, the same rows will be dropped from the ignored columns.
28  - `remove_empty_rows`: Whether to remove entirely empty rows.
29  - `remove_empty_columns`: Whether to remove entirely empty columns.
30  - `keep_id`: Whether to retain the original document ID in the output document.
31  
32  ### Cleaning Process
33  
34  The `CSVDocumentCleaner` algorithm follows these steps:
35  
36  1. Reads each document's content as a CSV table using pandas.
37  2. Retains the specified number of `ignore_rows` from the top and `ignore_columns` from the left.
38  3. Drops any rows and columns that are entirely empty (contain only NaN values).
39  4. If columns are dropped, they are also removed from ignored rows.
40  5. If rows are dropped, they are also removed from ignored columns.
41  6. Reattaches the remaining ignored rows and columns to maintain their original positions.
42  7. Returns the cleaned CSV content as a new `Document` object.
43  
44  ## Usage
45  
46  ### On its own
47  
48  You can use `CSVDocumentCleaner` independently to clean up CSV documents:
49  
50  ```python
51  from haystack import Document
52  from haystack.components.preprocessors import CSVDocumentCleaner
53  
54  cleaner = CSVDocumentCleaner(ignore_rows=1, ignore_columns=0)
55  
56  documents = [Document(content="""col1,col2,col3\n,,\na,b,c\n,,""")]
57  cleaned_docs = cleaner.run(documents=documents)
58  ```
59  
60  ### In a pipeline
61  
62  ```python
63  from pathlib import Path
64  from haystack import Pipeline
65  from haystack.document_stores.in_memory import InMemoryDocumentStore
66  from haystack.components.converters import XLSXToDocument
67  from haystack.components.preprocessors import CSVDocumentCleaner
68  from haystack.components.writers import DocumentWriter
69  
70  document_store = InMemoryDocumentStore()
71  p = Pipeline()
72  p.add_component(instance=XLSXToDocument(), name="xlsx_file_converter")
73  p.add_component(
74      instance=CSVDocumentCleaner(ignore_rows=1, ignore_columns=1),
75      name="csv_cleaner",
76  )
77  p.add_component(instance=DocumentWriter(document_store=document_store), name="writer")
78  
79  p.connect("xlsx_file_converter.documents", "csv_cleaner.documents")
80  p.connect("csv_cleaner.documents", "writer.documents")
81  
82  p.run({"xlsx_file_converter": {"sources": [Path("your_xlsx_file.xlsx")]}})
83  ```
84  
85  This ensures that CSV documents are properly cleaned before further processing or storage.