agent.actions.screen

Actions

Screen content, screenshots, and node interactions.

Access these methods through agent.actions. Get screen content, take screenshots, and interact with UI nodes.

screenContent()

TypeScript

screenContent(): Promise<AndroidNode>

Gets the accessibility tree of the currently focused window. Returns an AndroidNode representing the root of the UI hierarchy.

Returns

Promise<AndroidNode>Root node of the accessibility tree.

Examples

TypeScript

const screen = await agent.actions.screenContent();

// Find elements
const button = screen.findTextOne("Submit");
const allButtons = screen.filterAdvanced(f => f.isButton());
const input = screen.findAdvanced(f => f.isEditText().isEditable());

allScreensContent()

TypeScript

allScreensContent(): Promise<AndroidNode[]>

Gets the accessibility trees from all visible windows (useful for dialogs, overlays).

Returns

Promise<AndroidNode[]>Array of root nodes for each window.

Examples

TypeScript

const screens = await agent.actions.allScreensContent();
for (const screen of screens) {
  const dialog = screen.findTextOne("OK");
  if (dialog) break;
}

screenshot()

TypeScript

screenshot(maxWidth: number, maxHeight: number, quality: number, cropX1?: number, cropY1?: number, cropX2?: number, cropY2?: number): Promise<{screenshot: string | null, compressedWidth: number, compressedHeight: number, originalWidth: number, originalHeight: number}>

Takes a screenshot with optional scaling and cropping.

Parameters

Name	Type	Description
`maxWidth`	number	Maximum width to scale to.
`maxHeight`	number	Maximum height to scale to.
`quality`	number	JPEG quality (1-100).
`cropX1?`	number	Crop region left.
`cropY1?`	number	Crop region top.
`cropX2?`	number	Crop region right.
`cropY2?`	number	Crop region bottom.

Returns

{screenshot, compressedWidth, compressedHeight, originalWidth, originalHeight}Screenshot data as base64 string with dimensions.

Examples

Full screenshot

TypeScript

const result = await agent.actions.screenshot(1080, 1920, 80);

Cropped screenshot

TypeScript

const result = await agent.actions.screenshot(500, 500, 90, 100, 100, 600, 600);

nodeAction()

TypeScript

nodeAction(node: AndroidNode | object, actionInt: number, data?: object, fieldsToIgnore?: string[]): Promise<{actionPerformed: boolean}>

Performs an accessibility action on a node.

Parameters

Name	Type	Description
`node`	AndroidNode \| object	The node to perform action on.
`actionInt`	number	Action constant (use agent.constants.ACTION_*).
`data?`	object	Additional action data.
`fieldsToIgnore?`	string[]	Node fields to ignore when matching.

Returns

{actionPerformed: boolean}Whether the action was successfully performed.

Examples

Using node.performAction (preferred)

TypeScript

const screen = await agent.actions.screenContent();
const button = screen.findTextOne("Submit");

if (button) {
  const result = await button.performAction(agent.constants.ACTION_CLICK);
  console.log("Clicked:", result.actionPerformed);
}

Using agent.actions.nodeAction (legacy)

TypeScript

const result = await agent.actions.nodeAction(
  button,
  agent.constants.ACTION_CLICK
);

showNotification()

TypeScript

showNotification(title: string, message: string): Promise<void>

Shows a system notification.

Parameters

Name	Type	Description
`title`	string	Notification title.
`message`	string	Notification message.

Returns

Promise<void>Resolves when notification is shown.

Examples

TypeScript

await agent.actions.showNotification("Task Complete", "Your automation has finished.");