RetroShare/build_scripts/Android

= RetroShare development on Android

// SPDX-FileCopyrightText: RetroShare Team <contact@retroshare.cc>
// SPDX-License-Identifier: CC-BY-SA-4.0


Compiling an application for Android is not as easy as one would imagine,
expecially one like RetroShare that has a big codebase and is not well
documented. This document is aimed to empower the reader so she can hopefully
succed or at least have a significant help in compiling her own RetroShare APK
installable on Android.


== Preparing The Environement

First of all setup your Qt for Android development environement following the
guide on the link:http://doc.qt.io/qt-5/androidgs.html[Qt for android web site].
At this point you should have Android SDK, Android NDK, and Qt for Android
working fine, and you should be capable of executing on an Android emulator or
on your Android phone Qt for Android examples.

But RetroShare is not as simple to compile as those examples. The good news is
that Android NDK ships all the necessary to build a custom toolchain that is
suitable to build RetroShare.
In order to build the toolchain with needed libraries RetroShare provides the
+android-prepare-toolchain.sh+ script; before you execute it you should define
some variables the script cannot determine in an easy and reliable manner by
itself in your terminal.

[source,bash]
-------------------------------------------------------------------------------
## The path where Android NDK is installed in your system
export ANDROID_NDK_PATH="/opt/android-ndk/"

## The path where your fresh compiled toolchain will be installed, make sure
## the parent exists
export NATIVE_LIBS_TOOLCHAIN_PATH="${HOME}/Builds/android-toolchains/retroshare-android/"

## The CPU architecture of the Android device you want to target
export ANDROID_NDK_ARCH="arm"

## The Android API level the Android device you want to target
export ANDROID_PLATFORM_VER="16"

## The number of core that yout host CPU have (just to speed up compilation) set
## it to 1 if unsure
export HOST_NUM_CPU=1

./android-prepare-toolchain.sh
-------------------------------------------------------------------------------


== Preparing Qt Creator

Now that your environement is set up you should configure Qt Creator for Android
following the
link:http://doc.qt.io/qtcreator/creator-developing-android.html[official guide].
At the end of this step your Qt Creator should recognize the Android compiler
and the Qt for Android kit.

Your Kit is now ready to use. Now you can open RetroShare as a Qt Creator
project and in the Projects left menu add the newly created kit if not already
present, so you can select it on the build type selection button down on the
left.

Now you need to set properly a few options like `JSONAPI_GENERATOR_EXE` and
disable some of RetroShare modules like `retroshare-gui` that are not available
on Android so you will have to go to

_Qt Creator left pane -> Projects -> Build and Run -> Android SOMESTUFF kit ->
Build Steps -> qmake -> Additional arguments_  +

and add the following configurations (change `Your_Path` according to your
deployment)

[source,makefile]
-------------------------------------------------------------------------------
CONFIG+=retroshare_service CONFIG+=rs_jsonapi CONFIG+=rs_deep_search
RS_UPNP_LIB=miniupnpc
JSONAPI_GENERATOR_EXE=Your_Path/jsonapi-generator/src/jsonapi-generator
NATIVE_LIBS_TOOLCHAIN_PATH=Your_Path/retroshare-android-16-arm/
CONFIG+=no_retroshare_gui CONFIG+=no_rs_service_webui_terminal_password
CONFIG+=no_rs_service_terminal_login
-------------------------------------------------------------------------------

TIP: Some versions of QtCreator try to find the Android SDK in
`/opt/android/sdk`. A workaround to this is to make a symbolic link there
pointing to your SDK installation path, like
+mkdir -p /opt/android/sdk && ln -s /home/user/android-sdk-linux
/opt/android/sdk+


== Quircks

=== Protected Apps

On some Android devices like +Huawei ALE-L21+ background applications are
killed when screen is turned off unless they are in the _protected app_ list.
At moment seems apps developers don't have a way to have the application
_protected_ by default, unless the phone vendor decide the app is _popular_ so
the user have to enable _protection_ for RetroShare manually on those mobile
phones. +

{empty} +
To enable enable _protection_: +Android menu -> Settings -> Privacy & security 
-> Protected apps -> RetroShare+ +
{empty} +

Other devices may offer similar _features_ please report them.


=== APK signature mismatch

If you try to install a RetroShare APK that comes from a different source
(eg: if you try to upgrade from F-Droid when you originally installed an APK
build by yourself) Android will prevent that from happening. In that case the
only solution is to uninstall the app and then install the new APK but if you do
it also the application data and your precious cryptographic keys, friend list
etc. will be lost forever.
To avoid that you can attempt to manually backup and then restore from the
command-line (`adb backup` seems not working either) to change the app source
without erasing the appliation data.

CAUTION: Following steps require root access on your Android device

.Backup RetroShare Android application data
[source,bash]
--------------------------------------------------------------------------------
export ORIG_DIR="/data/data/org.retroshare.android.qml_app"
export BACKUP_DIR="org.retroshare.android.qml_app.backup"

adb root

adb shell am force-stop org.retroshare.android.qml_app
sleep 1s

mkdir ${BACKUP_DIR}

# Avoid adb pull failing
adb shell rm ${ORIG_DIR}/files/.retroshare/libresapi.sock
adb pull ${ORIG_DIR}/files/ ${BACKUP_DIR}/files/
--------------------------------------------------------------------------------

After this you should be able to uninstall the old APK with your preferred
method, as example from the command-line.

.Uninstall RetroShare Android from the command-line
[source,bash]
--------------------------------------------------------------------------------
adb uninstall org.retroshare.android.qml_app
--------------------------------------------------------------------------------

Now you can install a different signature APK and then restore the application
data with the following commands.

[source,bash]
--------------------------------------------------------------------------------
export ORIG_DIR="/data/data/org.retroshare.android.qml_app"
export BACKUP_DIR="org.retroshare.android.qml_app.backup"

adb root

## Launch the app to make sure the parent directory exists and has proper owner
adb shell monkey -p org.retroshare.android.qml_app -c android.intent.category.LAUNCHER 1
sleep 1s

adb shell am force-stop org.retroshare.android.qml_app
sleep 1s


APP_OWNER="$(adb shell busybox ls -lnd ${ORIG_DIR} | awk '{print $3":"$4}')"
adb shell rm -rf ${ORIG_DIR}/files
adb push ${BACKUP_DIR}/files/ ${ORIG_DIR}/files/
adb shell busybox chown -R ${APP_OWNER} ${ORIG_DIR}/files/
--------------------------------------------------------------------------------

Opening RetroShare android app now should show your old profile.


== Debugging with GDB

If building RetroShare Android package seems tricky, setting up a functional
debugging environement for it feels like black magic. This section is meant to
help you doing it with less headache and hopefully in a reproducible way.

Unfortunately at the time of the last update to this guide, Qt build system
strips debugging symbols from the package and from the buildroot also if you
compile with debugging enabled. Fiddling with `qmake` configurations and
variables like `CONFIG+=debug`, `CONFIG+=force_debug_info` or `QMAKE_STRIP`
either as commandline arguments or inside retroshare `.pro` and `.pri` files is
uneffective. IMHO Qt should handle this on itself so it is probably worth
reporting a bug to them. So to workaround this problem you need to fiddle a bit
with the Android NDK. In my case I always keep +Debug+ or +Release+ suffix in
my build directory name depending on what kind of build it is, so I use this
information and modify `llvm-strip` in a way that it will strip only if stripped
file path doesn't contain +Debug+.

.Modify llvm-strip inside Android NDK
--------------------------------------------------------------------------------
## Set ANDROID_NDK_PATH to your Android NDK installation path
export ANDROID_NDK_PATH="/opt/android-ndk/"

## Define a convenience variable with llvm-strip path
export ANDROID_NDK_LLVM_STRIP="${ANDROID_NDK_PATH}/toolchains/llvm/prebuilt/linux-x86_64/bin/llvm-strip"

## If not existing yer create a backup of the original llvm-strip
[ -f "${ANDROID_NDK_LLVM_STRIP}.back" ] ||
	cp "${ANDROID_NDK_LLVM_STRIP}" "${ANDROID_NDK_LLVM_STRIP}.back"

## Create a new llvm-strip that does nothing if the argument path contains Debug
cat > "${ANDROID_NDK_LLVM_STRIP}" << LLVMSTRIPTRICK
#!/bin/bash

echo "\${2}" | grep -q Debug ||
	"${ANDROID_NDK_LLVM_STRIP}.back" --strip-all "\${2}"

LLVMSTRIPTRICK

## Eventually you can revert back simply by running
# `mv "${ANDROID_NDK_LLVM_STRIP}.back" "${ANDROID_NDK_LLVM_STRIP}"`
--------------------------------------------------------------------------------


To attach to the `retroshare-service` running on Android you need also to pull a
debugging sysroot out of your device first, RetroShare sources provides an
helper script to do that.

.Prepare debugging sysroot with helper script
[source,bash]
--------------------------------------------------------------------------------
## Set RetroShare source path
export RS_SOURCE_DIR="${HOME}/Development/rs-develop"

## Optionally set your device ID first available will be used, you can run
## `adb devices` to list available devices.
#export ANDROID_SERIAL="YT013PSPGK"

## Optionally set a path where to save the debugging sysroot otherwise default
## is used.
#export DEBUG_SYSROOT="${HOME}/Builds/debug_sysroot/${ANDROID_SERIAL}/"

## Run the helper script
${RS_SOURCE_DIR}/build_scripts/Android/pull_sysroot.sh
--------------------------------------------------------------------------------

.Prepare Android NDK GDB configurations
[source,bash]
--------------------------------------------------------------------------------
## Optionally set Qt version variable consistently with your installation
export QT_VERSION="5.12.7"

## Optionally set Qt architecture variable consistently with Android device
export QT_ARCH="arm64_v8a"

## Optionally set Qt path variable consistently with your installation
export QT_DIR="/opt/Qt-${QT_VERSION}/${QT_VERSION}/"

## Optionally set RetroShare buildroot path
export RS_BUILD_DIR="${HOME}/Builds/RetroShare-Android_for_${QT_ARCH}_Clang_Qt_${QT_VERSION//./_}_android_${QT_ARCH}-Debug/"

## Optionally set gdb config file path
export GDB_CONFIGS_FILE="${HOME}/Builds/gdb_configs_${QT_ARCH}"

## Generate Android NDK GDB configuration
${RS_SOURCE_DIR}/build_scripts/Android/generate_gdb_init_commands.sh
--------------------------------------------------------------------------------


You will need to run the following steps everytime you want to debug
`retroshare-service` on Android.

Make sure `retroshare-service` is running on your connected Android device.

.Run GDB server on your Android device from your host console
[source,bash]
--------------------------------------------------------------------------------
${RS_SOURCE_DIR}/build_scripts/Android/start_gdbserver.sh
--------------------------------------------------------------------------------


.Run Android NDK GDB on your workstation and attach
[source,bash]
--------------------------------------------------------------------------------
## Start NDK gdb
${ANDROID_NDK_PATH}/prebuilt/linux-x86_64/bin/gdb

## Instruct GDB how and where to find debugging symbols
(gdb) source $GDB_CONFIGS_FILE

## Connect to the gdbserver running on the phone
(gdb) target remote 127.0.01:5039

## Have fun debugging
(gdb)
--------------------------------------------------------------------------------


== Debugging with Qt Creator

WARNING: As of the last update to this guide, debugging retroshare-service
running on Android via Qt creator doesn't wrok even with all the trickery
explained in this section, you better learn how to debug with GDB reading
carefully previous section.

Qt Creator actually support debugging only for the foreground activity, so to
debug what's happening in the core extra trickery is needed.

- Run the App in Debug mode from QtCreator "Start Debugging" button
- Enable QtCreator debugger console view Menu->Window->Debugger Log
- Run +show solib-search-path+ in the QtCreator GDB console
- Take note of the output you get in the right pane of the console
- Thanks https://stackoverflow.com/a/31164313 for the idea

TIP: QtCreator GDB console seems a bit buggy and easly trigger timeout
message when a command is run, in that case attempt to run the command while the
debugging is paused or at breakpoint, or with other similar tricks.

CAUTION: Following steps require root access on your Android device

Now on your phone yuo need to authorize root access to applications, then once
you plug your sacrifical Android phone run this commands

.Run gdbserver as root on Android phone
[source,bash]
--------------------------------------------------------------------------------
## Open a shell from your workstation on the connected Android phone
adb shell

## take the note of the phone IP address
ip address show

## Gain root permissions on the shell
su

## Attach with gdbserver and listen on one TCP port
gdbserver :4567 --attach $(pgrep org.retroshare.android.qml_app:rs)
--------------------------------------------------------------------------------


.Prepare and run Android NDK GDB on your workstation
[source,bash]
--------------------------------------------------------------------------------
## Setup some convenience variables
NDK_GDB="${ANDROID_NDK_PATH}/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/arm-linux-androideabi-gdb"
RS_BUILD_PATH="${HOME}/Builds/RetroShare-Android_for_armeabi_v7a_GCC_4_9_Qt_5_9_2_android_armv7-Debug/"

## Start Android NDK GDB of your phone architecture passing the executable
$NDK_GDB $RS_BUILD_PATH/retroshare-android-service/src/libretroshare-android-service.so

## Instruct GDB how and where to find debugging symbols
(gdb) set auto-solib-add on
(gdb) set solib-search-path THE:BIG:LIST:OF:DIRECTORIES:YOU:TAKE:NOTE:BEFORE

## Connect to the gdbserver running on the phone
(gdb) target remote $PHONE_IP:4567

## Have fun debugging
(gdb)
--------------------------------------------------------------------------------

TIP: Some time WiFi power saving on Android mess with the GDB connection,
to prevent that from appening open another +adb shell+ and live +ping+ toward
your work-station running

== Embedding into other Android packages

As showed by https://elrepo.io/[elRepo.io] developers it is possible and quite
easy to embed `retroshare-service` into other Android packages see description

https://gitlab.com/elRepo.io/elRepo.io-android/-/blob/master/README.adoc

And implementation details

https://gitlab.com/elRepo.io/elRepo.io-android/-/blob/master/android/app/build.gradle


== Furter Readings

- link:http://doc.qt.io/qt-5/android-support.html[]
- link:https://developer.android.com/ndk/guides/libs.html[]
- link:retroshare://forum?name=Compiling%20nogui%20for%20android&id=8fd22bd8f99754461e7ba1ca8a727995&msgid=4e0f92330600bba9cf978f384f4b7b2f2ca64eff[]
- link:retroshare://file?name=Android%20Native%20Development%20Kit%20Cookbook.pdf&size=29214468&hash=0123361c1b14366ce36118e82b90faf7c7b1b136[]
- link:https://groups.google.com/forum/#!topic/android-developers/srATPaL0aRU[]
- link:https://stackoverflow.com/questions/31638986/protected-apps-setting-on-huawei-phones-and-how-to-handle-it[]
- link:https://tthtlc.wordpress.com/2012/09/19/how-to-do-remote-debugging-via-gdbserver-running-inside-the-android-phone/[]
- link:https://source.android.com/devices/tech/debug/[]
- link:https://source.android.com/devices/tech/debug/gdb[]
- link:https://fw4spl-org.github.io/fw4spl-blog/2015/07/27/Native-debugging-on-Android-with-QtCreator.html[]
- link:https://fragglet.livejournal.com/19646.html[]
- link:https://github.com/android-ndk/ndk/issues/773[How to build without using standalone toolchain?]

== License

Copyright (C) 2016-2020  Gioacchino Mazzurco <gio@eigenlab.org> +
Copyright (C) 2020       Asociación Civil Altermundi <info@altermundi.net> +

This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License.

image::https://i.creativecommons.org/l/by-sa/4.0/88x31.png[Creative Commons License, link=http://creativecommons.org/licenses/by-sa/4.0/]