Home > Enhancing Scripts > Improving Object Recognition by using UI Automation Technology

Improving Object Recognition by using UI Automation Technology

The UI Automation Technology has been developed by Microsoft to provide a unified object model that enables all user interface (UI) frameworks to expose complex and rich functionality in an accessible and easily automated manner.

UI Automation is designed to provide robust functionality for automated testing, it is an accessibility framework that enables Microsoft Windows applications to provide and consume programmatic information about user interfaces (UIs). It provides programmatic access to most UI elements on the desktop.

WinTask x64 supports the UI Automation technology and can use UI Automation to
obtain information on UI elements of the application to automatize. Properties of buttons, check boxes, menus and menu items, data grids, list boxes and other controls can be retrieved using UI Automation technology. The UI Automation technology works only on applications which have implemented the UI Automation framework. Obviously, Microsoft applications such as Office 2010 or 2013 and WPF applications do support UI Automation.

WinTask x64 addresses any desktop element exposed to UI Automation framework by its UIA descriptor. The UIA descriptor identifies uniquely one element displayed on desktop. The UIA descriptor is not generated automatically by Recording mode and you need to identify manually a UI element using Spy tool (if Spy tool finds that the object is addresses by UI Automation framework). Then you can use the pasted UIA descriptor as parameter of any WinTask x64 UIA functions.

In this version, three functions are available:
ClickUIA, clicks the object specified by its UIA descriptor
WriteUIA, writes a text into the field specified by its UIA descriptor
GetUIAProperty, retrieves the content of the specified property for the object specified by its UIA descriptor. 

UIA Descriptor

The UIA Descriptor is used to uniquely identify UI elements within a desktop. It is a string where each node of the UI Automation tree is represented between brackets. The tree starts at the top window to which the element belongs (it does not start at the desktop level). The first part of the UIA descriptor uses the application name and class name of the top window to identify it. Wildcards ? and * can be used in the class name. If the top window has a title, the title of the window is added. If no title is available, the UIA property called AUTOID is then used. For example the first part of an UIA descriptor can be:

"[APP='iexplore.exe',CLASS='DirectUIHWND',AUTOID='IENotificationBar']"     'In Internet Explorer, many elements outside the page itself belongs to the window whose CLASS is DirectUIHWND and so AutoID tells that it belongs to the small notification bar at the bottom.
"[APP='Application.exe',CLASS='WindowsForms10.Window*',AUTOID='dataGridView1']   'In a WinForms application, the spyeded element belongs to a grid whose AUTOID property value is dataGridView1
"[APP='Application.exe',CLASS='WindowsForms10.BUTTON*',TITLE='Delete']"  'In a WinForms application, there is a Delete button, this button is in itself a top window with the Delete TITLE word
Note that Spy tool in its Window Information part shows the APP, CLASS and TITLE parameters.

The other nodes are specified using some of their UI Automation properties, for example:
"[APP='iexplore.exe',CLASS='DirectUIHWND',AUTOID='IENotificationBar'][NAME='Save']"   'Button named Save in the IE Dowload window (class DirectUIHWND)
"[APP='Application.exe',CLASS='WindowsForms10.Window*',AUTOID='dataGridView1'][NAME='Gender Row 1']   'Cell in the row 1 for the column called Gender belonging to a grid in a WinForms application

Always build the UIA descriptor of an element using WinTask x64 Spy tool.