05.2 documents

Documents

Documents are the primary source of knowledge in BindAI.

They contain the information that agents retrieve during execution to answer questions with accurate, domain-specific context.

Documents can represent almost any type of content, from simple Markdown files to large collections of technical documentation.

---

What is a Document?

A document is a unit of information that can be indexed and retrieved.

Examples include:

• Markdown files
• PDF documents
• Word documents
• HTML pages
• API documentation
• Product manuals
• Knowledge base articles
• FAQ pages

Each document becomes part of the knowledge base available to the agent.

---

Why Documents?

Large language models cannot reliably answer questions about information they have never seen.

Documents provide that missing information.

Instead of asking the model to “remember” everything, BindAI retrieves only the documents relevant to the user’s request.

---

Document Lifecycle

A typical document passes through several stages.

Document

↓

Load

↓

Process

↓

Index

↓

Retrieve

↓

Language Model

This pipeline prepares documents for efficient retrieval during execution.

---

Example

Suppose you have a product manual.

product-manual.pdf

A user asks:

How do I reset the device?

BindAI retrieves the section describing the reset procedure and includes it in the prompt sent to the language model.

---

Document Collections

Documents are usually grouped into collections.

Documentation

├── Installation

├── API

├── Tutorials

└── FAQ

Collections make it easier to organize large knowledge bases.

---

Document Metadata

Documents often include metadata.

Examples:

• title
• author
• creation date
• tags
• category
• language
• version

Metadata improves search quality and filtering.

Example:

Title:
API Reference

Category:
Documentation

Version:
2.0

---

Document Chunking

Large documents are usually divided into smaller pieces before indexing.

Instead of retrieving a 200-page manual:

Manual

↓

Chunk 1

Chunk 2

Chunk 3

...

Chunk N

Only the most relevant chunks are sent to the language model.

This improves both speed and accuracy.

---

Supported Formats

A knowledge system may support many document types.

Common examples include:

• Markdown
• Plain text
• PDF
• DOCX
• HTML
• JSON
• CSV

Additional formats can be added through custom loaders.

---

Loading Documents

Documents are typically imported into a knowledge source before use.

Conceptually:

knowledge.add_document(
    document,
)

The implementation depends on the knowledge provider being used.

---

Updating Documents

Knowledge bases evolve over time.

Common operations include:

• adding new documents
• updating existing content
• removing obsolete documents
• rebuilding indexes

Keeping documents current improves response quality.

---

Document Retrieval

When a user asks a question:

Question

↓

Search Documents

↓

Relevant Chunks

↓

Language Model

↓

Answer

Only a small subset of the knowledge base is typically retrieved.

---

Documents vs Memory

Documents contain reference information.

Memory contains conversation history.

Documents	Memory
Static information	Dynamic conversation
Shared knowledge	User-specific context
Manuals and guides	Previous interactions
Updated manually	Updated automatically

Both systems complement one another.

---

Best Practices

• Keep documents focused on one topic.
• Organize documents into logical collections.
• Remove outdated versions.
• Include useful metadata.
• Prefer smaller, well-structured documents over extremely large files.
• Use chunking for long documents.
• Rebuild indexes after significant updates.

---

Summary

Documents are the foundation of BindAI’s knowledge system.

By organizing, processing, and indexing documents before execution, BindAI can retrieve only the information needed to answer a user’s question, producing responses that are more accurate, current, and relevant than relying solely on the language model.