awt - Java search for on-screen text field

Question

I am trying to create a program that automatically searches for a text field on the screen and types a word repetitively into that text field. Is there any class that can find a text field? Or is there any way in which a text field can be found? Because I know that the Robot class can type text, I just need to either get the cursor onto the text field and use the mousePress() and mouseRelease() methods.

Thanks

Answer 1

I can't directly give you a solution, but I messed around with some code and may be able to point you in the right direction.

Java, as you probably know, runs in the JVM. This allows it to execute in any operating environment. Each operating environment (windows, mac, etc) has its own system for handling edit boxes and setting focus to the right window and whatnot. The following example code is designed for use on windows only, which does not follow the spirit of the Java language. As Adriaan pointed out, there are other languages for this sort of thing, but it IS possible (to an extent) to accomplish with Java alone.

In windows, you must understand how all of the active windows are managed and that everything you see (including edit boxes) are considering a "window" by the Windows OS. I don't truly understand how things work under the hood, so I can't provide much more information than that. In a native language such as C++, there are a few functions provided by the Windows OS API that would be used to accomplish your goal. Namely, EnumWindows(), EnumChildWindows(), GetClassName(), and SetForegroundWindow(). You can find documentation on how to use these functions within a native language by searching the MSDN documentation library.

So with that said, you NEED to be able to call these functions from Java. Under normal circumstances, calling these native methods is not possible. However, there is a library available to help you out: the JNA library. JNA stands for Java Native Access and lets you work with the shiny new functions I mentioned earlier.

So, to accomplish your goal in a native language, normally one would begin with a call toEnumWindows() to return a list of all Parent windows that the OS is aware of. This list will contain window handles of parent windows - windows titled "MSN", "Eclipse", "Microsoft Office", etc. Each of these windows, as a Parent, has children. It is in this list of children that you will find the "control" that you are looking for: an Edit control. Now, many applications use different libraries and non-standard things for text boxes - i.e Pidgin, a messaging application I tested some relevant code with, has every control named "gdkWindowChild" which doesn't exactly tell us which control is actually an EditBox or otherwise a place that allows us to enter text. That's the main problem with your idea; you can't always tell exactly what control you wish to have focus of so that you may enter text. Regardless of that, we'll continue:

After finding the relevant Parent window with EnumWindows(), a call to EnumChildWindows() will give us all of the sub-windows and other "controls" (including potential edit-boxes) that belong to the Parent. EnumChildWindows() calls a callback function for each sub-window it finds, so it's pretty easy to "search" through the list of child windows - using GetClassName() to find the name of a control - to potentially find the HWND (window handle) of the control you want.

Once you have found the correct HWND of the edit box (that, of course, being the difficult part given the general scope of your question) a simple call to SetForegroundWindow(targetHWND) ought to bring the control to the front and set your cursor in a ready-to-type edit box.

Here is some working example code I've written to get you started. This code will iterate through all of the active windows using EnumWindows() and then call EnumChildWindows() on each parent, printing out all of the controls that it finds. Note that this code requires the JNA library to run.

import com.sun.jna.Native;
import com.sun.jna.Pointer;
import com.sun.jna.examples.win32.W32API.HWND;
import com.sun.jna.examples.win32.W32API.LPARAM;
import com.sun.jna.win32.StdCallLibrary;
import com.sun.jna.win32.W32APIOptions;


public class IterateChildWindows {
    public interface User32 extends StdCallLibrary {
        User32 INSTANCE = (User32) Native.loadLibrary("user32", User32.class, W32APIOptions.DEFAULT_OPTIONS);

        HWND FindWindow(String lpClassName, String lpWindowName);
        int GetWindowRect(HWND handle, int[] rect);
        int SendMessage(HWND hWnd, int msg, int wParam, byte[] lParam); 
        HWND FindWindowEx(HWND parent, HWND child, String className, String window);

        boolean EnumWindows(WNDENUMPROC lpEnumFunc, Pointer arg);
        boolean EnumChildWindows(HWND parent, WNDENUMPROC callback, LPARAM info);

        interface WNDENUMPROC extends StdCallCallback {
            boolean callback(HWND hWnd, Pointer arg);
        }

        int GetWindowTextA(HWND hWnd, byte[] lpString, int nMaxCount);
        long GetWindowLong(HWND hWnd, int index);
        boolean SetForegroundWindow(HWND in);
        int GetClassNameA(HWND in, byte[] lpString, int size);
    }

    public static void main(String[] args) {        
        User32.INSTANCE.EnumWindows(new User32.WNDENUMPROC() {
            public boolean callback(HWND hWnd, Pointer userData) { // this will be called for each parent window found by EnumWindows(). the hWnd parameter is the HWND of the window that was found.
                byte[] textBuffer = new byte[512];
                User32.INSTANCE.GetWindowTextA(hWnd, textBuffer, 512);
                String wText = Native.toString(textBuffer);
                System.out.println("Window found: " + wText);

                // now call EnumChildWindows() giving the previously found parent window as the first parameter
                User32.INSTANCE.EnumChildWindows(hWnd, new User32.WNDENUMPROC() {
                    public boolean callback(HWND hWnd, Pointer userData) { // this is called for each child window that EnumChildWindows() finds - just like before with EnumWindows().
                        byte[] textBuffer = new byte[512];
                        User32.INSTANCE.GetClassNameA(hWnd, textBuffer, 512);
                        System.out.println(" - Found sub window / control class: " + new String(textBuffer).trim());
                        return true;
                    }
                }, null);
                return true;
            }
        }, null);
    }
}

Here is an excerpt of output provided by this code:

Window found: Pidgin
 - Found sub window / control class: gdkWindowChild
 - Found sub window / control class: gdkWindowChild
 - Found sub window / control class: gdkWindowChild
 - Found sub window / control class: gdkWindowChild
Window found: Malwarebytes Anti-Malware
 - Found sub window / control class: Static
 - Found sub window / control class: Static
 - Found sub window / control class: Button
 - Found sub window / control class: Button
 - Found sub window / control class: Button

Sending messages directly to the HWND of controls via PostMessage() and SendMessage(), for example to the MalwareBytes Button class, will trigger a button press in the program itself, very similarl to how SetForegroundWindow() should bring an edit-box style control to the front giving you the ability to type. Fun stuff to play with :)

If you wish to visualize what I mean when I am saying "Parent" and "children" and "control", you may find this program helpful: Control Viewer. It can show you each control and highlight it within an applications window and much more - very useful tool.

Sorry if this post left the comfort-zone that java provides, but there's really no other way to accomplish your goal in such a general scope.

I hope I have at least shown you what is necessary to accomplish your goal and pointed you in the right direction. I am no god when it comes to native windows API's, so I may be wrong in some place, however the code does work. Good luck :)