Edge AI and local Agents are being adopted at an increasingly rapid pace, while NPU inference performance across various edge devices has also improved significantly. However, many developers still face substantial barriers when deploying real-world applications: manually downloading model weights, repeatedly debugging model conversion, and writing NPU inference code from scratch.
Even after the environment is finally configured, practical issues may remain, including reduced model accuracy, underutilized hardware performance, and unstable long-term operation. The entire process is time-consuming and difficult to get started with, slowing down the rapid validation of edge large-model and Agent projects.
To address these challenges, the Firefly AI team developed LlamaPi, an edge large-model deployment tool designed to package and simplify complex deployment processes. It provides developers with an out-of-the-box foundation for offline, private Agent inference.
Launch Local Large-Model Inference with One Command
Deploying a large model on an edge device traditionally requires a series of manual steps, including downloading the model, converting its format, and configuring the inference environment. With LlamaPi, the entire process can be simplified to a single command, eliminating the need to handle complicated intermediate steps.
For example, to run the Qwen3.5-4B model, simply execute the following command in the device terminal:
llamapi run qwen3.5:4b
As shown in the video demonstration, after the command is executed, LlamaPi automatically downloads the model to the local device. With NPU hardware acceleration provided by AIBOX PRO, it quickly delivers high-performance local large-model inference services.
For users who prefer not to work with the command line, LlamaPi also provides a desktop client with a visual graphical interface. Models can be launched and conversations started through simple point-and-click operations, significantly lowering the barrier to entry.
Convenient Local Model Library Management
During edge AI development, developers often need to test several different large models. LlamaPi provides comprehensive model management capabilities, helping users search for, download, and remove models without manually maintaining scattered model-weight files.
Users can browse the model library directly in the LlamaPi desktop client. With a few clicks, they can download models, view locally installed models, and delete models that are no longer needed.
LlamaPi also provides a lightweight and straightforward command-line interface for fast debugging and validation:
llamapi listLists all models that have already been downloaded to the local device.
llamapi list --onlineDisplays all models available for the current device, including models that have not yet been downloaded.
llamapi pull <model_name>Downloads the specified model to the local device.
llamapi rm <model_name>Removes the specified model from the device.
Model Service Scheduling and Control
After downloading a model, developers also need flexible control over its operating status. This includes loading and starting the model, stopping it to release resources, checking which models are currently running, and configuring models to load automatically when the device starts.
LlamaPi provides complete deployment management capabilities, allowing developers to schedule and control local large-model services with ease.
In the desktop client, users can directly load or unload downloaded models, enable automatic startup, and view the current operating status of each model.
All deployment and scheduling operations can also be completed through simple command-line commands:
llamapi psDisplays the status of models currently deployed and running on the device.
llamapi load <model_name>Loads and starts the specified model that has already been downloaded locally.
llamapi unload <model_name>Unloads a running model and releases the hardware resources it occupies.
llamapi enable <model_name>Enables persistent model deployment so that the model loads automatically after the device restarts.
Third-Party Application Integration
After a model has been deployed on the device, LlamaPi automatically provides an OpenAI-compatible API. No additional development is required to connect local edge inference capabilities to various third-party AI applications, making it possible to quickly build complete offline and private AI workflows.
Third-Party Application Configuration
-
Select OpenAI or an OpenAI-compatible service provider in the third-party application.
-
Find the custom service address setting, usually labeled Base URL or API Endpoint, and enter:
http://<device-IP>:9265/v1- Enter the corresponding model ID. The model ID can be found by running the following command:
llamapi psIt can also be viewed on the deployment management page in the LlamaPi desktop client.
The following video uses OpenCode as an example to demonstrate how LlamaPi can be connected to a third-party application.
LlamaPi Architecture Overview
Third-party applications send inference requests to LlamaPi through the OpenAI-compatible API.
Internally, LlamaPi provides unified model management and service scheduling. It then sends inference tasks to the NPU hardware of devices such as AIBOX PRO for accelerated computing. Once inference is complete, the results are returned to the upper-layer application through the same path.
The following diagram provides a complete overview of the system components and their interactions.
For complete specifications and additional practical examples, visit the Firefly community documentation:
LlamaPi Wiki | Firefly Community
Experience LlamaPi
LlamaPi is designed to lower the barrier to deploying edge large models and local Agents by integrating model downloading, format conversion, NPU inference, and service APIs into one streamlined solution.
Whether you are rapidly validating a technical prototype or building an offline, privately deployed AI service, LlamaPi can help shorten the development cycle and simplify the path from model to application.
