移动开发 \ Android \ Android抓取文字、文字位置的分析

Android抓取文字、文字位置的分析

总点击674
简介:引文: 因为我弃用原来ATX框架中的uiautomator的东西,所以现在要把UiSelector().text(\"XXX\")这部分的功能给重新实现下。

引文:

因为我弃用原来ATX框架中的uiautomator的东西,所以现在要把 UiSelector().text("XXX")这部分的功能给重新实现下。


所以这篇文章介绍的是抓取到页面中的文字还有文字的位置的方法及其分析。

现有的方法

1,UiSelector.text /自动化测试框架

大多数测试框架使用的方法。好像需要在手机上安装一个测试的app,没动手实践

2,Layout Inspector

android studio->Tools->android->Layout Inspector

显示很友好,没找到开源部分的代码

3,uiautomator dump

adb shell uiautomator dump

dump出来的东西会自动保存在/sdcard/window_dump.xml中,内容大概是这样

<?xml version="1.0" encoding="utf-8"?>

<hierarchy rotation="0">

...

<node index="1" text=""

resource-id="com.android.contacts:id/contacts_unavailable_view"

class="android.widget.FrameLayout"

package="com.android.contacts"

content-desc="" checkable="false" checked="false"

clickable="false" enabled="true" focusable="false"

focused="false" scrollable="false" long-clickable="false"

password="false" selected="false"

bounds="[0,408][1080,1776]">

<node index="0" text=""

resource-id="com.android.contacts:id/contacts_unavailable_container" class="android.widget.FrameLayout"

package="com.android.contacts"

content-desc="" checkable="false" checked="false"

clickable="false" enabled="true" focusable="false"

focused="false" scrollable="false" long-clickable="false"

password="false" selected="false"

bounds="[0,1776]">

<node index="0" text=""

resource-id="com.android.contacts:id/floating_action_button"

class="android.widget.ImageButton"

package="com.android.contacts"

content-desc="添加新联系人" checkable="false"

checked="false" clickable="true"

enabled="true" focusable="true" focused="false"

scrollable="false" long-clickable="false"

password="false" selected="false"

bounds="[864,1560][1032,1728]"/>

...

</node>

</node>

...

</hierarchy>

数据类型都是类似的,都是node节点,bounds就是各个view的边界了,举个例子

添加新联系人


bounds= 864,1560 1032,1728

所以重心就是 (948,1644)

adb shell input tap 948 1644

就点击到文字了。

ok,下面也是谈实现细节。dump调用链是这样的

DumpCommand:run


-> automationWrapper.getUiAutomation().getRootInActiveWindow()


-> AccessibilityInteractionClient:findAccessibilityNodeInfoByAccessibilityId


-> binder


-> ViewRootImpl.findAccessibilityNodeInfoByAccessibilityId

-> AccessibilityNodeInfoDumper.dumpWindowToFile

序列化的节点node的数据存在这里

AccessibilityNodeInfo.java

可以看到,它是一颗树的节点

public class AccessibilityNodeInfo implements Parcelable {

...

private static final int MAX_POOL_SIZE = 50;

private static final SynchronizedPool<AccessibilityNodeInfo> sPool =

new SynchronizedPool<>(MAX_POOL_SIZE);

...

}

节点是这样建的

view.java

public AccessibilityNodeInfo createAccessibilityNodeInfoInternal() {

AccessibilityNodeProvider provider = getAccessibilityNodeProvider();

if (provider != null) {

return provider.createAccessibilityNodeInfo(AccessibilityNodeProvider.HOST_VIEW_ID);

} else {

AccessibilityNodeInfo info = AccessibilityNodeInfo.obtain(this);

onInitializeAccessibilityNodeInfo(info);

return info;

}

}

public void onInitializeAccessibilityNodeInfoInternal(AccessibilityNodeInfo info) {

...

getDrawingRect(bounds);

info.setBoundsInParent(bounds);

}

在TextView

Override

public void onInitializeAccessibilityNodeInfoInternal(AccessibilityNodeInfo info) {

super.onInitializeAccessibilityNodeInfoInternal(info);

final boolean isPassword = hasPasswordTransformationMethod();

info.setPassword(isPassword);

if (!isPassword || shouldSpeakPasswordsForAccessibility()) {

info.setText(getTextForAccessibility());

}

...

}

所以AccessibilityNodeInfo更像是在View中存了一个副本,这个副本可以用于辅助操作。

// TODO 更细节的分析

4,hierarchyviewer1

这个是在源码中找到的工具,发现它抓取的信息出乎意料的很全,比3中的信息还要多许多,是这样用的

.out/host/linux-x86/bin/hierarchyviewer1

是一个gui工具,打开之后,选中某个元素,可以看到Property中有absolute_x、absolute_y、getHeight、getWidth、mText。更加这些信息我们可以判断找的这个元素是不是在屏幕内,在的话,具体是哪个位置。

/sdk/hierachyviewer/ com.android.hierarchyviewer.scene.ViewHierarchyLoader

package com.android.hierarchyviewer.scene;

public class ViewHierarchyLoader {

public static ViewHierarchyScene loadScene(IDevice device,Window window) {

...

System.out.println("==> Starting client");

socket = new Socket();

socket.connect(new InetSocketAddress("127.0.0.1",

DeviceBridge.getDeviceLocalPort(device)));

out = new BufferedWriter(new OutputStreamWriter(socket.getOutputStream()));

in = new BufferedReader(new InputStreamReader(socket.getInputStream(),"utf-8"));

System.out.println("==> DUMP");

out.write("DUMP " + window.encode());

out.newLine();

out.flush();

Stack<ViewNode> stack = new Stack<ViewNode>();

boolean setRoot = true;

ViewNode lastNode = null;

int lastWhitespaceCount = Integer.MAX_VALUE;

while ((line = in.readLine()) != null) {

// debug by yeshen:

// System.out.println(line);

if ("DONE.".equalsIgnoreCase(line)) {

break;

}

...

}

...

}

}

看样子是自己建了一个socket取连本地的socket server,在代码里面打log,发现line的数据是类似这样的:

android.widget.Button@eb7e4a6

text:mCurTextColor=9,-13224394text:mGravity=2,17

text:mText=6,创建新联系人

getEllipsize()=4,null

text:getScaledTextSize()=4,14.0

text:getSelectionEnd()=2,-1

text:getSelectionStart()=2,-1

text:getTextSize()=4,42.0

text:getTypefaceStyle()=6,NORMAL

layout:mBottom=3,144

theme:com.android.contacts:style/PeopleTheme()=6,forced

theme:android:style/Theme.DeviceDefault()=6,forced

fg_=4,null

mID=24,id/create_contact_button

drawing:mLayerType=4,NONE

layout:mLeft=1,0

measurement:mMeasuredHeight=3,144

measurement:mMeasuredWidth=3,324

measurement:mMinHeight=3,144

measurement:mMinWidth=3,264

padding:mPaddingBottom=2,30

padding:mPaddingLeft=2,36

padding:mPaddingRight=2,36

padding:mPaddingTop=2,30

mPrivateFlags_DRAWN=4,0x20

mPrivateFlags=9,0x1028830

layout:mRight=3,324

scrolling:mScrollX=1,0

scrolling:mScrollY=1,0

mSystemUiVisibility_SYSTEM_UI_FLAG_VISIBLE=3,0x0

mSystemUiVisibility=3,0x0

layout:mTop=1,0

padding:mUserPaddingBottom=2,30

padding:mUserPaddingEnd=11,-2147483648

padding:mUserPaddingLeft=2,36

padding:mUserPaddingRight=2,36

padding:mUserPaddingStart=11,-2147483648

mViewFlags=10,0x18004001

drawing:getAlpha()=3,1.0

layout:getBaseline()=2,88

accessibility:getContentDescription()=4,null

drawing:getElevation()=3,6.0

getFilterTouchesWhenObscured()=5,false

getFitsSystemWindows()=5,false

layout:getHeight()=3,144

accessibility:getImportantForAccessibility()=3,yes

accessibility:getLabelFor()=2,-1

layout:getLayoutDirection()=22,RESOLVED_DIRECTION_LTR

layout:layout_gravity=4,NONE

layout:layout_weight=3,0.0

layout:layout_bottomMargin=2,45

layout:layout_endMargin=11,-2147483648

layout:layout_leftMargin=1,0

layout:layout_mMarginFlags_LEFT_MARGIN_UNDEFINED_MASK=3,0x4

layout:layout_mMarginFlags_RIGHT_MARGIN_UNDEFINED_MASK=3,0x8

layout:layout_mMarginFlags=4,0x0C

layout:layout_rightMargin=1,0

layout:layout_startMargin=11,-2147483648

layout:layout_topMargin=1,0

layout:layout_height=12,WRAP_CONTENT

layout:layout_width=12,MATCH_PARENT

layout:getLocationOnScreen_x()=3,378

layout:getLocationOnScreen_y()=3,757

measurement:getMeasuredHeightAndState()=3,144

measurement:getMeasuredWidthAndState()=3,324

drawing:getPivotX()=5,162.0

drawing:getPivotY()=4,72.0

layout:getRawLayoutDirection()=7,INHERIT

text:getRawTextAlignment()=7,GRAVITY

text:getRawTextDirection()=7,INHERIT

drawing:getRotation()=3,0.0

drawing:getRotationX()=3,0.0

drawing:getRotationY()=3,0.0

drawing:getScaleX()=3,1.0

drawing:getScaleY()=3,1.0

getScrollBarStyle()=14,INSIDE_OVERLAY

drawing:getSolidColor()=1,0

getTag()=4,null

text:getTextAlignment()=7,GRAVITY

text:getTextDirection()=12,FIRST_STRONG

drawing:getTransitionAlpha()=3,1.0

getTransitionName()=4,null

drawing:getTranslationX()=3,0.0

drawing:getTranslationY()=3,0.0

drawing:getTranslationZ()=3,0.0

getVisibility()=7,VISIBLE

layout:getWidth()=3,324

drawing:getX()=3,0.0

drawing:getY()=3,0.0

drawing:getZ()=3,6.0

focus:hasFocus()=5,false

drawing:hasOverlappingRendering()=4,true

drawing:hasShadow()=4,true

layout:hasTransientState()=5,false

isActivated()=5,false

isClickable()=4,true

drawing:isDrawingCacheEnabled()=5,false

isEnabled()=4,true

focus:isFocusable()=4,true

isFocusableInTouchMode()=5,false

focus:isFocused()=5,false

isHapticFeedbackEnabled()=4,true

drawing:isHardwareAccelerated()=4,true

isHovered()=5,false

isInTouchMode()=4,true

layout:isLayoutRtl()=5,false

drawing:isOpaque()=5,false

isPressed()=5,false

isSelected()=5,false

isSoundEffectsEnabled()=4,true

drawing:willNotCacheDrawing()=5,false

drawing:willNotDraw()=5,false

不过这里缺了 absolute_x absolute_y,找了一下代码

com.android.hierarchyviewer.ui.model.PropertiesTableModel

private void loadPrivateProperties(ViewNode node) {

int x = node.left;

int y = node.top;

ViewNode p = node.parent;

while (p != null) {

x += p.left - p.scrollX;

y += p.top - p.scrollY;

p = p.parent;

}

ViewNode.Property property = new ViewNode.Property();

property.name = "absolute_x";

property.value = String.valueOf(x);

privateProperties.add(property);

property = new ViewNode.Property();

property.name = "absolute_y";

property.value = String.valueOf(y);

privateProperties.add(property);

}

所以我们已经知道了,上面抓到的数据是一个ViewNode,然后递归解析,算出绝对位置。


所以找到文字的位置就要,递归算到自己在根视图的绝对位置,然后再修正下layout:getWidth(),layout:getHeight(),找到view的重点,最后再执行点击。

ok,下面谈实现细节

hierarchyviewer其实是在android上开了一个ViewServer,然后把ViewServer的端口转发到本地,然后在连本地的sockect,用sockect与ViewServer进行通讯。


可以看到,dump出来的信息很全,而且是在用户进程中才有的信息。所以ViewServer会通过mWindowToken中的打印。调用链是这样的:

WMS:viewServerWindowCommand


-> ViewRootImpl:executeCommand


-> ViewDebug:dispatchCommand


-> ViewDebug:dumpViewProperties

打印到目标属性用了注解+反射

private static void dumpViewProperties(Context context,Object view,

BufferedWriter out,String prefix) throws IOException {

if (view == null) {

out.write(prefix + "=4,null ");

return;

}

Class<?> klass = view.getClass();

do {

exportFields(context,view,out,klass,prefix);

exportMethods(context,prefix);

klass = klass.getSuperclass();

} while (klass != Object.class);

}

在每个view中,如果允许dump的话,就加入注解,比如说

@ViewDebug.ExportedProperty(flagMapping = {

@ViewDebug.FlagToString(mask = SYSTEM_UI_FLAG_LOW_PROFILE,

equals = SYSTEM_UI_FLAG_LOW_PROFILE,

name = "SYSTEM_UI_FLAG_LOW_PROFILE",

outputIf = true),

@ViewDebug.FlagToString(mask = SYSTEM_UI_FLAG_HIDE_NAVIGATION,

equals = SYSTEM_UI_FLAG_HIDE_NAVIGATION,

name = "SYSTEM_UI_FLAG_HIDE_NAVIGATION",

@ViewDebug.FlagToString(mask = PUBLIC_STATUS_BAR_VISIBILITY_MASK,

equals = SYSTEM_UI_FLAG_VISIBLE,

name = "SYSTEM_UI_FLAG_VISIBLE",

outputIf = true)

},formatToHexString = true)

int mSystemUiVisibility;

hierarchyviewer小结:


hierarchyviewer 就是通过在wms中开启一个viewServer,把server的端口forward到本地端口上。socket连本地端口,发送 DUMP $(WindowHash),通过binder调用到具体的某个view,递归打印出有加注解的信息,回信息给socket,hierarchyviewer(gui)再对信息进行整理,获得绝对位置等信息。

小结

本文提供了四种获取当前显示屏幕的文字信息的方法,简单分析了下原理。

接下来准备拓展下uiautomator,就可以支持文字查找与点击了。实现看这篇文章:

Android抓取文字、文字位置的实现

如果对精确度有高要求,对速度有高要求,可以考虑下移植下hierarchyviewer部分的实现。这部分完全重写需要不少时间,暂时偷个懒吧,不实现了:)

尾声

用文字匹配抢微信红包,查资料的时候发现网上不少这样的文章。下面只是针对这个场景的个人猜想,没有实际调研。

攻击的话,看大多数的文章都提到用Accessibility,其实就是用第三种方法(uiautomator)。可以考虑用第四种方法(hierarchyviewer)。就不会受到Accessibility的限制了。


防守的话,新消息那一栏,用自定义view、非规则view,那么有新红包这个文字信息就抓不到。

扫码关注,实时互动

Android抓取文字、文字位置的分析

意见反馈 常见问题 官方微信 返回顶部