Local LLM Server Manager

Appearance

AI Chat Assistant ​

The AI Chat Assistant provides an interactive AI Assist directly inside Local LLM Server Manager. You can control local engines, inspect hardware metrics, and troubleshoot generation errors through plain-language dialogue.

Open the AI Assistant ​

Follow these steps to access the assistant interface:

  1. Launch Local LLM Server Manager.
  2. Click the AI Assistant tab in the top navigation bar.
  3. Verify that the chat view loads with the header toolbar, suggestion chips, and composer bar.

Interface Component Breakdown ​

Interface ComponentLocationOperational Function
Model SelectorHeader BarSelects active model with capability icons (Vision 👁️, Tool Calling ⚡, Context 1M).
Setup & EndpointTop RightOpens connection settings dialog for LiteLLM URL and API key.
Pop Out ButtonTop RightDetaches AI Assist into a magnetic companion window snapped to the right flank.
Quick Action ChipsAbove ComposerOne-click prompts for common operations (VRAM check, hardware fitting, image creation).
Image AttachmentComposer BarAttaches PNG/JPG screenshots via file picker or clipboard paste (Ctrl+V).
Send ButtonComposer BarSubmits prompt and streams tokens back into chat history.

Configure the Connection Gateway ​

The assistant requires a connection to an external gateway, such as a LiteLLM proxy or Vertex AI endpoint.

Follow these steps to configure your connection:

  1. Click ⚙️ Setup & Endpoint in the upper-right toolbar.
  2. Enter your gateway URL in the OpenAI-Compatible Endpoint URL field (for example, http://127.0.0.1:4000/v1).
  3. Enter your secret credential in the API Key field. Leave this field blank if your local LiteLLM server does not require authentication.
  4. Enter the default model name in the Model Identifier field (for example, vertex_ai/gemini-2.5-flash).
  5. Click ⚡ Test Connection. The application validates network latency and queries available models.
  6. Click 💾 Save Settings to store your configuration in settings.json.

TIP

Run a local LiteLLM container on port 4000. LiteLLM translates OpenAI format requests into Google Vertex AI Gemini Flash calls with zero local GPU usage.

Select Models & Inspect Capability Badges ​

The composer bar features a dynamic model selector dropdown. The discovery engine inspects remote model metadata and renders compact capability badges.

Capability Badge Reference ​

Each badge informs you about the features supported by that specific model:

Badge IconCapability NameMeaning
👁️VisionThe model accepts image attachments alongside text prompts.
⚡Function CallingThe model supports autonomous tool calling and application control.
1M / 128kContext WindowThe maximum input token capacity (1M = 1,000,000 tokens, 128k = 128,000 tokens).
[provider]Provider TagIdentifies the hosting backend (e.g. [vertex_ai], [openai], [anthropic]).

Common Model Examples ​

  • 👁️ ⚡ 1M [vertex_ai] (vertex_ai/gemini-2.5-flash): Accepts images, executes tools, and supports massive 1-million-token contexts.
  • 👁️ ⚡ 128k [openai] (gpt-4o): Accepts images and executes tools with a 128,000-token context.
  • ⚡ 128k [anthropic] (claude-3-5-haiku): Fast text reasoning and tool calling without image input.

Local Model Exclusion Filter ​

By default, the assistant excludes local LLMs (such as models hosted directly in Ollama) from the assistant dropdown.

IMPORTANT

The assistant excludes local LLMs intentionally. Running assistant queries on local GPUs consumes memory needed for diffusion rendering and audio synthesis. External models guarantee zero VRAM impact on your creative engines.

Attach Multimodal Images ​

You can supply visual context to vision-enabled models. Attach screenshots of ComfyUI errors, generation outputs, or reference pictures.

Add Attachments ​

Choose one of the following methods to attach an image:

  • File Browser: Click the attachment button (📎) next to the prompt text box. Select a PNG, JPG, WEBP, or GIF image.
  • Clipboard Paste: Capture a screenshot with your operating system shortcut. Click inside the prompt text box and press Ctrl+V.

Manage the Staged Image Tray ​

Staged images appear in a preview tray directly above the text box:

  1. Review the generated thumbnail and filename.
  2. Click the remove button (✕) on any card to delete unwanted attachments before sending.
  3. Type your question (for example: "Review this error message and suggest the correct resolution").
  4. Click 🚀 Send or press Enter.

Tool Execution & Safety Controls ​

When you ask the assistant to perform an action, the model executes native C# tools autonomously.

Inline Execution Cards ​

During tool calling, the assistant displays an execution card inside the chat response bubble:

  • Tool Name: Displays the executed function (e.g., calculate_hardware_fit).
  • Execution Time: Shows tool duration in milliseconds (e.g., 12 ms).
  • Arguments: Details the parameter values sent to the tool.
  • Result: Displays the response data returned to the model.
Execution Card FieldExample ContentPurpose
Tool Header⚡ Tool: calculate_hardware_fitIdentifies the executed C# tool.
Execution Latency12 msReports internal tool execution duration.
Input Arguments{"modelName": "llama3.3:70b", "quantization": "Q4_K_M"}Shows parameters passed by the LLM.
Output Result{"verdict": "PERFECT FIT", "vramRequiredMb": 41200}Shows structured payload passed back to the model.

Safety and Approval Boundaries ​

The assistant operates under strict safety policies defined in Prompts/app-control.md:

  • The assistant cannot modify sensitive system directories or arbitrary executable paths.
  • Sensitive values like API keys appear masked in tool outputs.
  • You can cancel any running operation immediately by clicking ⏹️ Stop.

WARNING

Do not share your screen or logs while displaying unmasked API tokens. Always keep secret credentials in your private configuration files.

Toolbar Actions & Quick Chips ​

Suggestion Chips ​

Click any suggestion chip above the composer to trigger pre-built diagnostic tasks:

  • ⚡ Check live VRAM and GPU telemetry: Queries active VRAM allocation.
  • 🦙 Can I run Llama 3.3 70B: Runs a memory fit calculation.
  • 🎨 Generate a scenic landscape: Submits a prompt to your default image engine.
  • 🛑 Unload all models from VRAM: Frees GPU memory immediately.

Header Controls ​

  • 🔄 Reload Prompts: Flushes cached markdown prompt files from memory immediately.
  • 🧹 Clear Chat: Deletes conversation history and resets token context.

Floating Pop-Out Mode & Magnetic Docking ​

The AI Assistant can detach from the tab row into a floating companion window (AiAssistWindow).

Detach and Dock ​

  1. Click the pop-out button (⧉ Pop Out) in the AI Assistant header bar.
  2. The assistant window pops out and docks to the right flank of the main window.
  3. The WindowSnapManager binds the companion window in lockstep with the main window.
  4. Dragging the main window moves the AI Assistant companion window automatically.
  5. Click 🧲 Attached in the companion window title bar to detach the window.
  6. Drag the companion window within 32 pixels of the right flank to re-snap automatically.
  7. Click the close button (✕) in the companion title bar to return the assistant to the main tab layout.

TIP

Read the complete Magnetic Companion Windows Guide to learn about multi-monitor workflows and proximity thresholds.

Released under the MIT License.

Copyright © Spelech