What happened
Researchers from Korea released HILIPS, an open-source software designed to streamline the creation of training data for object detection in industrial environments. The tool addresses 'visual-semantic ambiguity,' a problem where objects, such as identical buttons on a CNC panel, are visually indistinguishable but serve different functions. HILIPS uses a three-stage workflow: it first uses the Google Gemini 2.5 Pro API and SAM2 to generate initial labels, then trains a lightweight YOLOv8 model on those human-reviewed labels, and finally uses that model to auto-approve future detections, reducing manual effort. The software is available under an MIT license.
The core challenge addressed by HILIPS is visual-semantic ambiguity, where objects appear visually similar but have distinct functions. Unlike standard fine-grained recognition tasks that rely on subtle visual differences, industrial settings often require contextual cues like embedded text or spatial relationships to distinguish objects. The researchers developed a three-stage workflow to handle this: Cold-start Labeling, Label Transfer, and Iterative Refinement.
In the Cold-start stage, the system pairs the Google Gemini 2.5 Pro API with SAM2. The language model provides bounding boxes and class labels based on user prompts, which are then used to generate precise segmentation masks locally. In the Label Transfer stage, these human-reviewed annotations are used to train a lightweight YOLOv8 detection model. Finally, in Iterative Refinement, the trained model is deployed back into the to auto-approve high-confidence detections, with lower-confidence items routed to human reviewers.
The tool is implemented as a web application using Next.js 15, React 19, and FastAPI. It allows users to mix manual polygon drawing, point-based SAM2 labeling, and prompt-based auto-labeling within a single image. The system is modular, allowing the MLLM, segmentation, and detection components to be swapped independently. The code is released under an MIT license, making it accessible to teams without specialized ML infrastructure.
A user study with ten participants demonstrated significant efficiency gains. Time per image dropped from 37.5 seconds for manual labeling to 5.1 seconds for auto-labeling. Perceived workload and mental demand decreased substantially, with usability scores rising from 'Not Acceptable' for manual methods to 'Excellent' for assisted methods. The resulting YOLOv8 detector achieved 99.12% precision and 94.03% recall on a held-out test set of 100 images.
Source details: bioengineer.org
Why it matters
Image labeling is a major bottleneck in deploying AI for industrial automation, particularly for collaborative robots that need to operate control panels or inspect equipment. Traditional manual labeling is slow and prone to error when domain experts lack annotation skills or workers lack domain knowledge. By integrating multimodal large language models with human-in-the-loop verification, HILIPS significantly reduces the time and cognitive load required to create high-quality datasets. This makes it easier for research teams and industrial operators without dedicated machine learning staff to build reliable detection systems for complex, real-world manufacturing scenarios.
Industrial automation increasingly relies on collaborative robots for tasks like control panel operation and equipment inspection. However, building high-quality training data for these robots is difficult because standard annotation tools do not account for the specific expertise mismatch between domain experts and annotation workers. HILIPS bridges this gap by using AI to handle the initial semantic interpretation, allowing humans to focus on verification rather than from-scratch labeling.
The use of multimodal large language models (MLLMs) like Gemini 2.5 Pro is critical because they can interpret complex visual scenes from natural language descriptions, which is essential in cold-start scenarios where no prior training data exists. By combining this with local SAM2 segmentation and a lightweight YOLOv8 detector, the balances the high reasoning capability of MLLMs with the speed and efficiency required for practical deployment.
The open-source nature of HILIPS under the MIT license is significant for the broader AI community. It provides a reproducible framework for addressing visual-semantic ambiguity, a problem that extends beyond industrial control panels to any domain where visual similarity does not equate to functional identity. This could accelerate the development of more robust object detection systems in manufacturing and other industrial sectors.
Interactive Mechanism: How It Actually Works
Explore the underlying technology behind this development interactively.
Which component of an AI application is the machine-learning model itself?
What to watch next
The primary limitation is that the current evaluation is restricted to industrial control panels. Future work will need to validate the 's effectiveness in other domains involving visual-semantic ambiguity. Additionally, the team plans to conduct broader user studies to confirm the workload reductions observed in the initial ten-participant trial and to explore the integration of active learning methods to further optimize the annotation process.
The current evaluation is limited to industrial control panels. The authors explicitly state that applying the to other domains involving visual-semantic ambiguity remains to be validated. Future research will likely test the tool in different industrial settings, such as equipment inspection or maintenance support, to determine its generalizability.
The user study involved a small sample of ten participants. While the results showed statistically significant improvements in workload and usability, a broader study is planned to confirm these findings with a larger and more diverse group of users. This will help establish the robustness of the workflow in real-world industrial environments.
The team plans to integrate active learning methods to further optimize the annotation process. This could involve the system automatically selecting the most informative images for human review, thereby maximizing the efficiency of the labeling effort. Additionally, the authors note that adding image-preprocessing modules could improve the 's robustness to varying environmental conditions.