aichat-go

Every incoming Telegram message follows this path:

1. Webhook reception — POST /webhook/{tg_api_key} received on the public HTTP server (:8080). The API key in the URL routes to the correct project bundle.
2. Gateway normalization — TelegramUpdate is converted to domain.Message with text, caption, files, and sender info extracted.
3. State loading — Chat state loaded from PostgreSQL using composite key (chat_id, project_id).
4. Command check — /start resets state and returns greeting. /debug toggles debug mode.
5. File collection — Any file attachments are accumulated into state. Files without text get an immediate acknowledgement without LLM calls.
6. Stage dispatch — Based on derived state (CurrentStage()), the message is routed to the appropriate stage handler.
7. State persist — Updated state written back to PostgreSQL.
8. Timer reset — Follow-up timer reset to step 0 for this chat.

cmd/
  bot/         # Main bot binary
    main.go     # Wiring: DB, tiered LLM, per-project bundles, HTTP server
    config.go   # JSON config loading with env var fallback
  crawler/     # Knowledge base indexer CLI
    main.go     # Subcommands: init-project, seed, add-urls, register
  cli/         # Local readline chat interface (testing)
    main.go     # Same agent pipeline, file-based CRM output

internal/
  domain/      types.go        # Stage, Message, ChatState, Lead, SearchResult
  agent/       agent.go        # HandleMessage dispatch
                stages.go       # Stage 1/1.5/2/3 handlers
                commands.go     # /start, /debug
                lead_commands.go # Lead group: /prompt, /stats
  llm/         tiered.go       # TieredClient: routes by tier
                chain.go        # Provider chain with retry + fallback
                openai.go       # OpenAI-compatible provider
                claude.go       # Claude/Anthropic provider
                embedding_adapter.go # Model-specific text formatting
  search/      hybrid.go       # Qdrant DBSF + reranking + neighbors
                reranker.go     # Cohere cross-encoder client
                bm25.go         # BM25Encoder, Tokenize
  db/          postgres.go     # PostgreSQL implementation
  gateway/     telegram.go     # TelegramUpdate normalization
  telegram/    client.go       # HTTP client, retry on 429
  crm/         telegram.go     # TelegramCRM: summary + history + media
  timer/       timer.go        # Scheduler, sequences, LLM classification
  metrics/     metrics.go      # Metric definitions
                llm.go          # InstrumentedLLMClient wrapper
  crawler/     pipeline.go     # Resumable 4-phase pipeline
                chunker.go      # Semantic + simple chunking
                indexer.go      # Qdrant upsert
  i18n/        i18n.go         # T(), Tf(), embedded locales
                locales/    ru.json, en.json

migrations/  001_initial.sql, 002_bm25_index.sql, 003_timers.sql
nix/         module.nix, example-configuration.nix
flake.nix    Makefile

Finished=true is written to the database before the CRM send, and LeadSent=true is written after. This prevents duplicate leads from concurrent webhook requests or timer fires reading stale state. If the bot crashes between the two persists, the timer system retries the CRM send on next fire or on restart.

When a user sends a file without text, the bot immediately acknowledges it ("Accepted! Let me know when you're done.") without calling the LLM. Files are accumulated in the chat state and attached to the lead when it is generated.

Media files in the lead are grouped by type for clean presentation:

• Photos + Videos — sent as a media album (Telegram groups them visually)
• Documents — sent as a document album
• Voice / Video notes — sent individually (Telegram does not support albums for these)

On the first Stage 2 interaction, the bot sends a Telegram request_contact keyboard button alongside its reply. This lets users share their phone number with one tap.

• Sent only once per conversation (tracked via state.UserSettings.ContactRequested)
• Shared contact is stored in chat state (not in history) for privacy
• Contact data is included in the lead summary and extraction prompts
• When a user says "take my phone from Telegram", the LLM can use the actual shared contact data

The bot gently reminds the user to share their city and phone number, but does not block lead generation. If the user provides enough information for a meaningful lead, finish=true is allowed even without all fields. This avoids frustrating users who prefer not to share certain information while still maximizing lead quality.

• Standard BM25 formula: IDF * (tf * (k1 + 1)) / (tf + k1 * (1 - b + b * dl/avgDL))
• Parameters: k1=1.5, b=0.75
• Tokenizer: Unicode-aware (Cyrillic + Latin), lowercased, split on non-letter/digit
• Persistence: vocabulary, IDF, avgDL, numDocs stored as JSONB in PostgreSQL (bm25_indexes table)
• In-memory cache: loaded from DB on first use per project, invalidated on crawler updates

// User message flow
User message
  → #4 rewriteQuery (extract) → search variations
  → hybrid search (DBSF fusion, dense + BM25)
  → reranker (Cohere cross-encoder, if configured)
  → top 5 results → neighbor expansion (position N-1/N+1)
  → #5 determineURL (extract) → pick service or clarify
  → if entering Stage 2: #14 price extraction (extract, one-time, cached)
  → #1 or #2 Stage reply (chat) → answer to user
  → #6 + #7 + #8 extractContact (3x extract, parallel)
  → if finished:
      → #9 generateSummary (extract) → CRM summary
      → #3 Stage 3 closing (chat) → goodbye

// Timer flow
Timer fires
  → #10 classifyStatus (extract) → cold/hot/finished
  → #11 generateFollowUp (extract, conditional)
  → if finished: #12 summary (extract) → CRM lead

// Crawler flow
Crawler
  → #13 semantic boundary detection per page (extract)
  → slice original text at detected boundaries
  → topic prepend at embed time ("Topic: X\n\n...")
  → batch embedding (32 chunks per API call)

The TieredClient routes Complete calls based on req.Tier:

• TierChat → chat chain (customer-facing models)
• TierExtract → extract chain (lightweight models). Falls back to chat chain if not configured.
• Embed → embedding chain (wrapped in AdaptedProvider for model-specific text formatting)

Each chain has independent retry/fallback. Backoff formula: 2^attempt + 10% jitter, capped at 60s. Key pool rotates API keys round-robin.

Embedding Adapters

AdaptedProvider wraps the embedding provider and formats text based on mode (document vs query). Adapter selected automatically by model name. E5-instruct models get Instruct: ...\nQuery: ... prefix for queries. Token limit auto-split: when a chunk exceeds the model's token limit, text is split by sentence, each half embedded recursively, and vectors averaged + L2-normalized.

new-projects/{project-name}/
  1_pages/{urlhash}.json    # single Page per URL
  2_chunks/{urlhash}.json   # []Chunk per URL
  3_embeds/{urlhash}.json   # []EmbeddedChunk per URL
  3_embeds/bm25.json        # BM25 encoder snapshot

Semantic Chunking (default)

One LLM call per page (extract tier, temp 0.1) detects natural topic boundaries. The LLM outputs anchor phrases in the format TOPIC: X | STARTS: Y. The original text is sliced at the detected boundaries, preserving the exact source text without LLM paraphrasing. Each chunk receives a topic label for embedding.

Simple Chunking (--no-llm-chunk)

Paragraph and heading-aware splitting. Text is split on double newlines and markdown headings. Short paragraphs are merged up to maxChars (default 1500). Chunks get numbered topic labels ("Part 1", "Part 2", etc.).

At index time, each chunk's text is prepended with its topic label before embedding:

Before: "Our basic plan starts at $99/month with unlimited support."
After:  "Topic: Pricing plans\n\nOur basic plan starts at $99/month with unlimited support."

The topic acts as a semantic anchor — the embedding now captures the chunk's theme, not just its surface content. A query about "costs" will match closer to a chunk anchored with "Pricing plans" even if the chunk text never mentions "costs".

Search queries do not need the topic prefix — the embedding space naturally aligns.

# Create a new project and seed immediately
bin/aichat-crawler init-project \
  --name myproject \
  --tg-api-key "123456:ABC-DEF" \
  --tg-lead-group -1001234567890 \
  --language en \
  --prompt1 "You are a sales consultant..." \
  --prompt2 "You are helping a customer..." \
  --start-reply "Welcome! How can I help?" \
  --sitemap-url "https://example.com/sitemap.xml"

# Seed a project later (separate from creation)
bin/aichat-crawler seed \
  --project-id "uuid-from-init-output" \
  --project-name myproject \
  --sitemap-url "https://example.com/sitemap.xml"

# Add URLs to an existing project
bin/aichat-crawler add-urls \
  --project-name myproject \
  --urls "https://example.com/new-page1,https://example.com/new-page2"

# Set prompts from files
bin/aichat-crawler set \
  --project-name myproject prompt1 @prompts/prompt1.txt

1. Reset — Every incoming message resets the timer for that chat, starting the sequence from step 0.
2. Fire — Fetches fresh chat state and history. Skips if already finished. Classifies the client via LLM, then acts based on classification.
3. Persist — Timer state is written to PostgreSQL with the next trigger time. On restart, Reload() recovers all active timers, calculating remaining delay and firing overdue ones immediately.
4. Lead retry — If a timer finds Finished=true, LeadSent=false, it retries the CRM send instead of classifying.
5. Cancel — When a user sends a new message, the old timer is cancelled (goroutine killed + DB record updated).

Goroutine safety: Each chat/project pair gets one goroutine. A current == self identity check prevents a superseded goroutine from cleaning up a newer timer's state.

Token savings: Two-step approach (classify first, generate conditionally) skips follow-up generation for finished clients and hot clients who don't need a message yet.

Project bundles (Telegram client, agent, CRM, KB) live in an in-memory map. New projects can be registered at runtime via two paths:

• Admin group — /project_init command creates the project, seeds KB, and registers the bundle instantly. No restart needed.
• Internal API — POST /api/register-project/{id} for CLI or external tools that seed the DB independently.

Step 1: Create a bot in BotFather, copy the token.

Step 2: Send one command in the admin project's lead group:

/project_init myproject sitemap https://example.com/sitemap.xml 123456:AAH...token

The system verifies the token, creates the project, seeds the KB, and replies with progress:

> Project "myproject" created (bot: @myproject_bot). Seeding started...
> Scraping completed: 42 pages
> Chunking completed: 156 chunks
> Embeddings completed: 156 vectors
> Upload completed: 42 pages, 156 chunks indexed
> Project "myproject" init finished!
> Test the bot: https://t.me/myproject_bot
> To create the CRM group, click:
> https://t.me/myproject_bot?startgroup=connect_myproject

Step 3: Click the deep link. Telegram opens the "create group" UI with the bot pre-added. The bot auto-detects the group and connects it as the lead group. No group ID needed — discovered automatically via the /start connect_{name} command.

Lead group commands (per project)

User chat commands

Three migrations in migrations/:

001_initial.sql

CREATE TABLE projects (
  id          UUID PRIMARY KEY,
  name        TEXT UNIQUE,
  tg_lead_group BIGINT,
  tg_api_key  TEXT UNIQUE,
  prompts     JSONB,
  language    TEXT DEFAULT 'ru'
);

CREATE TABLE chat_history (
  id          UUID PRIMARY KEY,
  chat_id     BIGINT,
  project_id  UUID REFERENCES projects(id),
  question    TEXT,
  reply       TEXT,
  dbinfo      JSONB,
  created_at  TIMESTAMPTZ
);

CREATE TABLE chat_states (
  chat_id     BIGINT,
  project_id  UUID REFERENCES projects(id),
  state       JSONB,
  PRIMARY KEY (chat_id, project_id)
);

002_bm25_index.sql

CREATE TABLE bm25_indexes (
  project_id  UUID PRIMARY KEY REFERENCES projects(id),
  vocab       JSONB,
  idf         JSONB,
  avg_dl      FLOAT,
  num_docs    INT,
  updated_at  TIMESTAMPTZ
);

003_timers.sql

CREATE TABLE timers (
  chat_id     BIGINT,
  project_id  UUID REFERENCES projects(id),
  step        INT,
  trigger_at  TIMESTAMPTZ,
  PRIMARY KEY (chat_id, project_id)
);

Key indexes

• idx_chat_history_chat_id on (chat_id, created_at) — history lookup
• idx_chat_states_project on (project_id) — project-level queries
• idx_chat_states_state_gin GIN on (state) — JSONB queries on state

type CRM interface {
    SendLead(ctx context.Context, lead domain.Lead) (string, error)
    Name() string
}

Current implementation: TelegramCRM. The interface is designed for future CRM backends (Bitrix, AmoCRM) via the MultiCRM dispatcher pattern.

services.aichat = {
  enable = true;
  configFile = "/run/secrets/aichat-config.json";
  listenAddr = ":8080";

  # Database
  enablePostgres = true;
  dbName = "aichat";
  postgresPort = 5432;

  # Qdrant
  enableQdrant = true;
  qdrantPort = 6333;
  qdrantDataDir = "/var/lib/qdrant";

  # Backup
  enableBackup = true;
  backupDir = "/var/backup/aichat";
  backupRetentionDays = 14;
};

Backup

• Daily systemd timer with RandomizedDelaySec=1h
• pg_dump + Qdrant snapshots
• 14-day retention with automatic cleanup

Secrets (sops-nix)

• Encrypted in secrets/production.yaml using age keys
• Decrypted at boot on the target machine using its SSH host key
• Keys: aichat_config (full JSON config), cloudflared_creds (tunnel credentials)

# Deploy code changes to production
make prod-deploy

# SSH tunnels for crawler access to prod DB + Qdrant
make prod-tunnel
# PostgreSQL on localhost:15432, Qdrant on localhost:16333

# Seed a project on prod (while tunnel is open)
./bin/aichat-crawler --config /tmp/config.prod-tunnel.json init-project \
  --name myproject --tg-api-key "TOKEN" --language ru \
  --no-llm-chunk --workers 1 --urls "URL1,URL2,..."

# Fetch trace files from prod
ssh root@server "cat /var/lib/aichat/traces/{chatID}/{dialogID}.log"

After the bot is running and accessible via HTTPS, register the webhook with Telegram:

curl -X POST "https://api.telegram.org/bot{TG_API_KEY}/setWebhook" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://your-domain.com/webhook/{TG_API_KEY}"}'

The webhook URL must exactly match https://{host}/webhook/{tg_api_key} where tg_api_key is the bot token from the projects table. The API key in the URL routes incoming updates to the correct project bundle.

Prerequisites: Go 1.22+, PostgreSQL 16, Qdrant

# Build all binaries
make build
# produces bin/aichat-bot, bin/aichat-crawler, bin/aichat-cli

# Run migrations
make migrate DATABASE_URL="postgres://user:pass@localhost/aichat"

# Run the bot with JSON config
bin/aichat-bot --config config.prod.json

# Or with environment variables
DATABASE_URL="postgres://user:pass@localhost/aichat" \
OPENAI_API_KEYS="sk-xxx" \
bin/aichat-bot

Production uses Cloudflare Tunnel to route HTTPS traffic to the bot without exposing ports. The tunnel routes aichat.example.com to localhost:8080 on the server.

• Tunnel credentials managed via sops-nix (encrypted at rest)
• Configured in nix/production.nix
• No need for TLS certificates or reverse proxy configuration
• Telegram webhooks point to the tunnel URL

• Group LLM metrics by provider to compare provider performance and error rates
• Track aichat_agent_messages_total by stage to see funnel conversion rates (general → service_specific → final)
• Watch aichat_timer_active_sequences as a proxy for total active conversations
• Use aichat_webhook_request_duration_seconds p95 to spot slow responses (typical: 2-5s including LLM + search)
• Monitor aichat_crm_leads_total by project to track lead generation per tenant

Model identifiers use "provider/model" format. The provider name is split on the first / character and matched against the providers map.

Extract model constraint: Must return content in choices[].message.content field. Thinking/reasoning models that use a reasoning field will not work for the extract tier.

# Build all three binaries
make build        # produces bin/aichat-bot, bin/aichat-crawler, bin/aichat-cli

# Run tests
make test         # go test ./...

# Lint
make lint         # go vet ./...

# Local development services (PostgreSQL + Qdrant via Docker)
make dev-services

# Run the CLI interface for testing (no Telegram needed)
bin/aichat-cli --config config.dev.json --project myproject