AndroidNode
InterfaceAccessibility tree nodes for UI element interaction.
AndroidNode represents a node in the Android accessibility tree. Each node corresponds to a UI element on screen and contains properties like text, bounds, and state. Use methods like screenContent() to get the root node and traverse the tree.
// Get the accessibility tree
const screen = await agent.actions.screenContent();
// Find elements
const button = screen.findTextOne("Submit");
const allInputs = screen.filterAdvanced(f => f.isEditText());
// Interact with elements
if (button) {
const { left, top, right, bottom } = button.boundsInScreen;
await agent.actions.tap((left + right) / 2, (top + bottom) / 2);
}Identification
| Name | Type | Description |
|---|---|---|
falseId | string | UUID unique per request (changes each time screen content is fetched) |
viewId? | string | Android view ID (e.g., 'com.example:id/button') |
className | string | null | Android class name (e.g., 'android.widget.Button') |
packageName | string | Package name of the app owning this node |
Layout
| Name | Type | Description |
|---|---|---|
boundsInScreen | {left, top, right, bottom} | Screen coordinates of the node's bounding box |
index | number | Index within parent's children |
Text Content
| Name | Type | Description |
|---|---|---|
text? | string | Text content of the node |
description? | string | Content description (accessibility label) |
hintText | string | null | Hint text for input fields |
maxTextLength | number | Maximum text length for input fields |
inputType | string | Input type for text fields |
isShowingHintText | boolean | Whether hint text is currently shown |
tooltipText? | string | null | Tooltip textsince Android 9+ |
paneTitle? | string | null | Pane title for accessibilitysince Android 9+ |
containerTitle? | string | null | Container titlesince Android 14+ |
State
| Name | Type | Description |
|---|---|---|
isEnabled | boolean | Whether the node is enabled |
clickable | boolean | Whether the node is clickable |
isLongClickable | boolean | Whether the node responds to long click |
isContextClickable | boolean | Whether the node responds to context click |
isFocusable | boolean | Whether the node can receive focus |
isFocused | boolean | null | Whether the node currently has focus |
isScrollable | boolean | Whether the node is scrollable |
isSelected | boolean | Whether the node is selected |
isChecked | boolean | null | Checked state (for checkboxes, switches) |
isEditable | boolean | Whether text can be edited |
isDismissable | boolean | Whether the node can be dismissed |
isPassword | boolean | Whether this is a password field |
isMultiLine | boolean | Whether this is a multi-line text field |
isVisibleToUser | boolean | Whether the node is visible on screen |
isImportantForAccessibility | boolean | Whether important for accessibility |
isHeading? | boolean | Whether this node is a headingsince Android 9+ |
isTextSelectable? | boolean | Whether text can be selectedsince Android 13+ |
isTextEntryKey? | boolean | Whether this is a keyboard keysince Android 10+ |
Collection Info
| Name | Type | Description |
|---|---|---|
collectionInfo? | object | Info when node is a collection (ListView, RecyclerView). Contains rowCount, columnCount, itemCount, hierarchical, selectionMode |
collectionItemInfo? | object | Info when node is a collection item. Contains rowIndex, columnIndex, rowSpan, columnSpan, isSelected |
Hierarchy
| Name | Type | Description |
|---|---|---|
children | AndroidNode[] | Child nodes |
parent | AndroidNode | null | Parent node (null for root) |
Actions
| Name | Type | Description |
|---|---|---|
actions | number[] | Array of supported action IDs (use with agent.constants.ACTION_*) |
actionLabels | {id: number, label?: string}[] | Action labels with optional custom text |
Methods
allNodes
allNodes(): AndroidNode[]Returns all nodes in the subtree as a flat array, including this node and all descendants.
Returns AndroidNode[] — Flat array of all nodes
const screen = await agent.actions.screenContent();
const allNodes = screen.allNodes();
console.log("Total nodes:", allNodes.length);
// Find all visible text
const textNodes = allNodes.filter(n => n.text && n.isVisibleToUser);findById
findById(id: string): AndroidNode[]Finds all nodes with the matching viewId.
| Parameter | Type | Description |
|---|---|---|
id | string | View ID to search for (e.g., 'com.example:id/button') |
Returns AndroidNode[] — Array of matching nodes
const buttons = screen.findById("com.example:id/submit_button");findByIdOne
findByIdOne(id: string): AndroidNode | nullFinds the first node with the matching viewId.
| Parameter | Type | Description |
|---|---|---|
id | string | View ID to search for |
Returns AndroidNode | null — First matching node or null
const button = screen.findByIdOne("com.example:id/submit");
if (button) {
await button.performAction(agent.constants.ACTION_CLICK);
}findText
findText(textOrRegex: string | RegExp): AndroidNode[]Finds all nodes containing the text in description, text, or hintText (case-insensitive). Supports strings and regular expressions.
| Parameter | Type | Description |
|---|---|---|
textOrRegex | string | RegExp | Text to search for or regex pattern |
Returns AndroidNode[] — Array of matching nodes
const nodes = screen.findText("Submit");const priceNodes = screen.findText(/\$\d+\.\d{2}/);findTextOne
findTextOne(textOrRegex: string | RegExp): AndroidNode | nullFinds the first node containing the text (case-insensitive).
| Parameter | Type | Description |
|---|---|---|
textOrRegex | string | RegExp | Text to search for or regex pattern |
Returns AndroidNode | null — First matching node or null
const submit = screen.findTextOne("Submit");
if (submit) {
const { left, top, right, bottom } = submit.boundsInScreen;
await agent.actions.tap((left + right) / 2, (top + bottom) / 2);
}find
find(predicate: (node: AndroidNode) => boolean): AndroidNode | nullFinds the first node matching a custom predicate function.
| Parameter | Type | Description |
|---|---|---|
predicate | (node: AndroidNode) => boolean | Function to test each node |
Returns AndroidNode | null — First matching node or null
// Find first clickable button with specific text
const btn = screen.find(node =>
node.className === "android.widget.Button" &&
node.clickable &&
node.text === "Submit"
);
// Find first visible input field
const input = screen.find(node =>
node.isEditable && node.isVisibleToUser
);filter
filter(predicate: (node: AndroidNode) => boolean): AndroidNode[]Filters all nodes matching a custom predicate function.
| Parameter | Type | Description |
|---|---|---|
predicate | (node: AndroidNode) => boolean | Function to test each node |
Returns AndroidNode[] — Array of matching nodes
// Find all buttons
const buttons = screen.filter(node =>
node.className === "android.widget.Button"
);
// Find all clickable, visible nodes
const clickables = screen.filter(node =>
node.clickable && node.isVisibleToUser
);
// Find nodes with text containing dollar amounts
const prices = screen.filter(node =>
node.text?.includes("$")
);findAdvanced
findAdvanced(filterBuilder: (f: AndroidNodeFilter) => AndroidNodeFilter): AndroidNode | nullFinds the first node using the AndroidNodeFilter builder pattern.
| Parameter | Type | Description |
|---|---|---|
filterBuilder | (f: AndroidNodeFilter) => AndroidNodeFilter | Function to build the filter |
Returns AndroidNode | null — First matching node or null
// Find the submit button
const submit = screen.findAdvanced(f => f.isButton().hasText("Submit"));
// Find first editable text field
const input = screen.findAdvanced(f => f.isEditText().isEditable());
// Find by view ID
const header = screen.findAdvanced(f => f.hasId("com.example:id/header"));filterAdvanced
filterAdvanced(filterBuilder: (f: AndroidNodeFilter) => AndroidNodeFilter): AndroidNode[]Filters all nodes using the AndroidNodeFilter builder pattern.
| Parameter | Type | Description |
|---|---|---|
filterBuilder | (f: AndroidNodeFilter) => AndroidNodeFilter | Function to build the filter |
Returns AndroidNode[] — Array of matching nodes
// Find all clickable buttons
const buttons = screen.filterAdvanced(f => f.isButton().isClickable());
// Find text nodes with specific text
const labels = screen.filterAdvanced(f => f.isText().hasText("Welcome"));
// Find nodes in a specific list
const listItems = screen.filterAdvanced(f =>
f.anyParent(p => p.hasId("com.example:id/list"))
);matches
matches(filterBuilder: (f: AndroidNodeFilter) => AndroidNodeFilter): booleanChecks if this node matches the given filter.
| Parameter | Type | Description |
|---|---|---|
filterBuilder | (f: AndroidNodeFilter) => AndroidNodeFilter | Function to build the filter |
Returns boolean — true if node matches the filter
// Check if a node is a clickable button
if (node.matches(f => f.isButton().isClickable())) {
await node.performAction(agent.constants.ACTION_CLICK);
}
// Filter children manually
const clickableChildren = node.children.filter(child =>
child.matches(f => f.isClickable())
);toJSON
toJSON(): objectConverts the node to a plain JSON object. Useful for debugging or serialization.
Returns object — Plain object representation
console.log(JSON.stringify(node.toJSON(), null, 2));Action Methods
These methods allow you to perform actions directly on nodes. Available since app version 2.119.
performAction
since 2.119performAction(actionInt: number, data?: object, fieldsToIgnore?: string[]): Promise<{ actionPerformed: boolean }>Performs an accessibility action on this node. This is the preferred way to interact with UI elements as it uses the accessibility system.
| Parameter | Type | Description |
|---|---|---|
actionInt | number | Action constant (use agent.constants.ACTION_*) |
data? | object | Additional action data (used for ACTION_SET_TEXT etc.) |
fieldsToIgnore? | string[] | Fields to ignore when matching node |
Returns Promise<{ actionPerformed: boolean }> — Whether the action was performed successfully
const button = screen.findTextOne("Submit");
if (button) {
await button.performAction(agent.constants.ACTION_CLICK);
}const input = screen.findAdvanced(f => f.isEditText());
if (input) {
await input.performAction(agent.constants.ACTION_SET_TEXT, {
[agent.constants.ACTION_ARGUMENT_SET_TEXT_CHARSEQUENCE]: "Hello World"
});
}const scrollView = screen.findAdvanced(f => f.isScrollable());
if (scrollView) {
await scrollView.performAction(agent.constants.ACTION_SCROLL_FORWARD);
}adbPerformAction
since 2.119adbPerformAction(actionInt: number, data?: object, fieldsToIgnore?: string[]): Promise<{ actionPerformed: boolean }>Performs an accessibility action on this node via ADB. The ADB equivalent of performAction(). Uses UiAutomation through the ADB touch server instead of the accessibility service.
| Parameter | Type | Description |
|---|---|---|
actionInt | number | Action constant (use agent.constants.ACTION_*) |
data? | object | Additional action data (used for ACTION_SET_TEXT etc.) |
fieldsToIgnore? | string[] | Fields to ignore when matching node |
Returns Promise<{ actionPerformed: boolean }> — Whether the action was successfully performed
const button = screen.findTextOne("Submit");
if (button) {
await button.adbPerformAction(agent.constants.ACTION_CLICK);
}const input = screen.findAdvanced(f => f.isEditText());
if (input) {
await input.adbPerformAction(agent.constants.ACTION_SET_TEXT, {
[agent.constants.ACTION_ARGUMENT_SET_TEXT_CHARSEQUENCE]: "Hello World"
});
}randomClick
since 2.119randomClick(): voidPerforms a tap at a random position within this node's bounds. Uses boundsInScreen to determine the click area. Useful for more natural-looking automation.
const button = screen.findTextOne("Submit");
if (button) {
button.randomClick();
}const items = screen.filterAdvanced(f => f.isClickable().hasText(/Item \d+/));
for (const item of items) {
item.randomClick();
await sleep(500);
}randomSwipe
since 2.119randomSwipe(direction: "up" | "down" | "left" | "right"): voidPerforms a swipe gesture within this node's bounds. The swipe starts and ends at random positions within the node for more natural-looking automation.
| Parameter | Type | Description |
|---|---|---|
direction | "up" | "down" | "left" | "right" | Direction to swipe |
const listView = screen.findAdvanced(f => f.isScrollable());
if (listView) {
listView.randomSwipe("up"); // Swipe up to scroll down
}const carousel = screen.findAdvanced(f => f.hasId("carousel"));
if (carousel) {
carousel.randomSwipe("left"); // Swipe left to see next item
}randomClickAdb
since 2.141 (153)randomClickAdb(): voidPerforms a tap at a random position within this node's bounds using ADB shell input instead of the accessibility service. Fire-and-forget like randomClick().
const button = screen.findTextOne("Submit");
if (button) {
button.randomClickAdb();
}randomSwipeAdb
since 2.141 (153)randomSwipeAdb(direction: "up" | "down" | "left" | "right"): voidPerforms a swipe gesture within this node's bounds using ADB shell input instead of the accessibility service. Fire-and-forget like randomSwipe().
| Parameter | Type | Description |
|---|---|---|
direction | "up" | "down" | "left" | "right" | Direction to swipe |
const listView = screen.findAdvanced(f => f.isScrollable());
if (listView) {
listView.randomSwipeAdb("up"); // Swipe up to scroll down
}randomClickAdb() and randomSwipeAdb() require app version 2.141 (153)+. They use ADB shell commands instead of the accessibility service, which can be more reliable in some scenarios.Common Patterns
const button = screen.findTextOne("Submit");
if (button) {
button.randomClick();
}const button = screen.findAdvanced(f => f.isButton().hasText("OK"));
if (button) {
await button.performAction(agent.constants.ACTION_CLICK);
}const input = screen.findAdvanced(f => f.isEditText().isEditable());
if (input) {
await input.performAction(agent.constants.ACTION_CLICK);
await agent.actions.writeText("Hello World");
}const scrollView = screen.findAdvanced(f => f.isScrollable());
if (scrollView) {
// Using accessibility action
await scrollView.performAction(agent.constants.ACTION_SCROLL_FORWARD);
// Or using random swipe for more natural scrolling
scrollView.randomSwipe("up");
}