Ui.Vision

Task and UI test automation with Computer Vision/OCR. Ui.Vision combines browser automation and desktop automation.

New Dec 5, 2024 Update: Anthropic Claude Computer Use Integration Open-Source Ui.Vision has consistently been at the forefront of visual web automation. With Claude’s integration, we’re taking the next step forward. The aiComputerUse command allows you to automate complex tasks with a single line of code that would traditionally require hundreds of lines of classic Ui.Vision commands (such as XClick, OCRExtractScreenshot, If/then statements, and more). For example, you can teach Ui.Vision to play TicTacToe with just one short "Play this game..." prompt. -- Ui.Vision is an open-source automation RPA software that combines classic browser automation with modern computer vision and OCR: (1) **Visual Browser Automation** Ui.Vision's visual UI testing commands assist web designers and developers in checking and ensuring the accuracy of website layouts and canvas elements. It can identify and read images and text within canvas elements, images, and videos. (2) **Visual Desktop Automation for Windows, Mac, and Linux** Beyond web browser automation, Ui.Vision uses image and text recognition (OCR) to automate browser extensions and desktop environments as well. It can interpret images and text on the desktop, executing actions like clicking, moving, dragging and dropping the mouse, and simulating keyboard inputs. This desktop automation requires installing the free Ui.Vision XModules, available for Windows, Mac, and Linux. These modules provide Ui.Vision with the necessary capabilities for desktop interaction. (3) **Selenium IDE compatible commands** Ui.Vision includes Selenium-style commands for web automation, testing, form filling, and web scraping. Learning Ui.Vision also means learning Selenium IDE, and vice versa. However, Ui.Vision differs in philosophy from the classic Selenium IDE. Ui.Vision offers features not found in the classic Selenium IDE, including computer vision for UI testing, image comparison, file download automation, OCR screen scraping, PDF testing, and capturing full web page and desktop screenshots. **Command Line API** Ui.Vision provides a detailed command line API for integration with other applications, often used with Jenkins, CI/CD tools, or the Windows task scheduler. It can be automated and controlled using any programming or scripting language, such as Python or PowerShell. **Open-Source (AGPL license)** The Ui.Vision extension source code is available on Github. This makes Ui.Vision a good open-source Selenium IDE alternative and iMacros alternative. **100% Local Software** Free and Open-Source. No cloud and no subscription. No recurring payments. The software does not send any data back to us or any other place. Everything, including image recognition and OCR processing, is done locally on your machine. The only exception to the "all data is processed locally" rule is if you select an optional online OCR engine or the AI Computer Use commands. All cloud-features are disabled by default. Only when you explicitly enable them in the settings does Ui.Vision send screenshots to cloud services. The default OCR options are Javascript OCR or XModule OCR, which both run 100% locally on the machine. **Happy Automating!** For questions and suggestions, please visit the Ui.Vision community forum at https://forum.ui.vision.