The new "Human-Eye" Engine (Preview for up coming features)

Started by Theo Gottwald, May 23, 2021, 11:02:13 AM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

Theo Gottwald

In the past when we worked with Window-Structures and Accessibility, we had several Tasks that could not be done - especially on Java or PHP-Framworks,
or in unsupported Browsers like Google's Chrome. NOW that is History!

The newest SPR-Recognition Engine is already silently been addded to the Autoupdate.
Yes, its working and its very fast. With typical tasks the Engine recognizes Patterns on WEB-Sites faster then a human User.
So it can click faster then a human user.

In the past, using Accessibility and other technolgies it was often a slow task to automate stuff on a WEB-Site.

Now, using then brand-new "Human Eye Engine" the Robot can recognize anything on any GUI-Surface.
In the past we had problems to recognize GUI-Elements in JAVA or PHP-Environment.
This gone. Now the Robot can recognize it, and he does not need any underlying "AddOn" in the Browser or wherever.
He will just "do it like a Humanoid User".  The how the Engine got its name.

So recognizing any Pattern on any GUI is now possible. Recording is done with the new "Human Eye-Recorder"-Tool (see picture1).

Technical details:
Doing the development my calculation was like this: 150% Browser Zoom to 100% should be equal like resizing a Pattern from 12x12 to 8x8.
This is theoretically the case.
Now what happens really if you record the Automation at 150% (Browser-Zoom-Setting) and then you want it also to work on other Scales on other Computers?

Due to the fact that the underlying 4-Level Fuzzy-Logic can recognize similar Patterns, it looks like this:

Assume you record a Pattern in 12x12 Size. This Pattern will automatically stored in 5 Sizes, from 8x8 to 12x12.

Now my tests show that the 150% Pattern will also work on 144% and 133% Browser Zoom with only few changes.
Just need to set the Color-Tolerance from zero to 100 and increase Pixel-Hits.

The automatically generated 11x11 Pattern will work fine at a Browser Setting of 125% Zoom, also with a Color-Tolerance of 100 and maybe 118 Pixel-Hits.
The automatically generated 10x10 Pattern will work fine at a Browser Setting of 125% and 110% also with a Color-Tolerance of 100 and maybe 91 Pixel-Hits.

The automatically generated 9x9 Pattern will work fine at a Browser Setting of 90%,100% and 110% also with a Color-Tolerance between 80  (for 110%) to 100 and maybe 63 to 74 Pixel-Hits. The automatically generated 8x8 Pattern will work fine at a Browser Setting of 75% to 80% taking into account that the recording was done with 150% this is a very strong result.
And this is funny because I did not really expect that. I thought the 8x8 would work fine at a 100% Zoom Level which is not the case.

So, it works already. Its lightning fast. It can click your Programs faster then you could do it manually.
It also supports Scaling to some degree already, but this is where we will get improvements before the final real release.
So this was the Preview.