BotIntelli — Pipeline Management User Guide

This guide explains Pipeline Management: what pipelines are, how to create and manage them, connect data sources (Salesforce, HubSpot, Zendesk, databases, APIs, webhooks, file storage), and run ingestion.

What Is Pipeline Management?

Pipeline Management is where you define how data is ingested, processed, and made available to the app. Pipelines can:

Connect to data sources — CRMs (Salesforce, HubSpot, Zendesk), databases, data warehouses, file storage, webhooks, Universal API.
Sync or ingest data — Pull objects, tables, or files into BotIntelli for use in Knowledge Base, search, RAG, and workflows.
Monitor status — See running, paused, failed, or completed pipelines and fix issues.

Where to Find It

Sidebar: Business Context → Pipeline Management.
URL: https://app.botintelli.com/dashboard?view=pipeline-management.

Complete User Journey

Step 1: Open Pipeline Management

You see a list of pipelines and/or data source connections (e.g. by type: Salesforce, HubSpot, Database, API, File storage). Each may show name, type, status (running, paused, failed), last run.

Step 2: Create a New Pipeline or Connection

Click Create pipeline or Add connection (or “Connect” for a specific source type).
Choose type, e.g.:
- Salesforce — Connect org, choose objects, map fields, schedule or run ingestion.
- HubSpot — Connect account, select objects/datasets, sync.
- Zendesk — Connect, select objects, ingest.
- SAP HANA Cloud — Connect, select schemas/tables, ingest.
- Databases — PostgreSQL, MongoDB, etc.: connection details, then tables/queries.
- Data warehouses — Snowflake, BigQuery, Redshift, etc.: connection and dataset selection.
- File storage — S3, Google Drive, etc.: bucket/folder and sync options.
- Webhook — Configure URL and payload; receive data from external systems.
- Universal API — Connect to other REST APIs; define endpoints and ingestion.

Step 3: Configure the Connection

Credentials: Enter required fields (e.g. client ID/secret, connection string, API key). Use BotIntelli-managed keys where the app supports it.
Scope: Choose what to sync (objects, tables, folders, endpoints).
Schedule (if available): Set frequency (e.g. daily, hourly) or trigger manually.
Save. The connection appears in the list; first run may start automatically or you click Run / Sync.

Step 4: Run or Schedule Ingestion

Manual run: Open the pipeline/connection → Run or Sync now.
Scheduled: If you set a schedule, runs happen automatically; check status in the list or in a Pipeline details or Runs view.
Embeddings: For vector/RAG pipelines, embedding jobs may run after ingestion; monitor status in the pipeline or in an Embedding monitor section.

Step 5: Monitor and Fix Issues

Status: Running, Paused, Failed, Success. Click a pipeline for details.
Logs / errors: In the pipeline detail view, check run history and error messages.
Pause / Resume: Use Pause or Resume when the pipeline supports it.
Edit / Delete: Use Edit to change configuration; Delete to remove the pipeline or connection.

Step 6: Use the Data

Ingested data can appear in Knowledge Base (e.g. under a “Pipeline” or “Data sources” tab) for RAG and search.
Workflows and AI features can use this data as configured (e.g. AI Insights, Ask BotIntelli).

Data Source Types and Typical Inputs

Source	Typical inputs	What you do
Salesforce	Org URL, OAuth or credentials, object selection	Connect → authorize → select objects → run ingestion.
HubSpot	OAuth, object/dataset selection	Connect → authorize → select datasets → sync.
Zendesk	Subdomain, credentials, object selection	Connect → enter details → select objects → ingest.
SAP HANA Cloud	Host, port, credentials, schema/table selection	Connect → enter connection details → select tables → ingest.
Database	Host, port, database, user, password, SSL options	Create connection → run wizard → select tables or SQL.
Data warehouse	Account, warehouse, database, schema, credentials	Same idea as database; select datasets/tables.
File storage	Bucket/folder path, credentials or IAM	Configure storage connection → choose path → sync.
Webhook	Webhook URL, optional secret	Create webhook → use URL in external system to send data.
Universal API	Base URL, auth, endpoints	Configure API connection → define ingestion steps (e.g. endpoints, mapping).

Exact fields depend on the in-app forms; use placeholders and validation messages as guidance.

Embedding Monitor

For pipelines that produce vector embeddings (for RAG/search), an Embedding monitor or similar view shows:
- Job status (queued, running, completed, failed).
- Document/chunk counts.
Use it to confirm data is ready for assistants and AI Insights.

Tips and Troubleshooting

Connection failed: Check credentials, network, and firewall; ensure OAuth or API access is granted.
Ingestion stuck or failed: Check pipeline logs and error message; fix mapping or scope and re-run.
Data not in Knowledge Base: Ensure the pipeline run completed and that the app is configured to expose that source in Knowledge Base.

Quick Reference

Goal	Action
Add a data source	Create pipeline / Add connection → choose type → configure → save.
Run ingestion	Open pipeline → Run / Sync now.
Check status	View pipeline list or open pipeline for details and logs.
Pause/Resume	Use Pause / Resume on the pipeline.
Use data in app	After ingestion, use Knowledge Base and AI Insights / workflows.

For document uploads and organization, see the Knowledge Base guide.