RetroShare/jsonapi-generator/README.adoc

// SPDX-FileCopyrightText: (C) 2004-2019 Retroshare Team <contact@retroshare.cc>
// SPDX-License-Identifier: CC0-1.0

RetroShare JSON API
===================

:Cxx: C&#43;&#43;

== How to use RetroShare JSON API

Look for methods marked with +@jsonapi+ doxygen custom command into
+libretroshare/src/retroshare+. The method path is composed by service instance
pointer name like +rsGxsChannels+ for +RsGxsChannels+, and the method name like
+createGroup+ and pass the input paramethers as a JSON object.

.Service instance pointer in rsgxschannels.h
[source,cpp]
--------------------------------------------------------------------------------
/**
 * Pointer to global instance of RsGxsChannels service implementation
 * @jsonapi{development}
 */
extern RsGxsChannels* rsGxsChannels;
--------------------------------------------------------------------------------

.Method declaration in rsgxschannels.h
[source,cpp]
--------------------------------------------------------------------------------
	/**
	 * @brief Request channel creation.
	 * The action is performed asyncronously, so it could fail in a subsequent
	 * phase even after returning true.
	 * @jsonapi{development}
	 * @param[out] token Storage for RsTokenService token to track request
	 * status.
	 * @param[in] group Channel data (name, description...)
	 * @return false on error, true otherwise
	 */
	virtual bool createGroup(uint32_t& token, RsGxsChannelGroup& group) = 0;
--------------------------------------------------------------------------------

.paramethers.json
[source,json]
--------------------------------------------------------------------------------
{
    "group":{
        "mMeta":{
            "mGroupName":"JSON test group",
            "mGroupFlags":4,
            "mSignFlags":520
        },
        "mDescription":"JSON test group description"
    },
    "caller_data":"Here can go any kind of JSON data (even objects) that the caller want to get back together with the response"
}
--------------------------------------------------------------------------------

.Calling the JSON API with curl on the terminal
[source,bash]
--------------------------------------------------------------------------------
curl -u $API_USER --data @paramethers.json http://127.0.0.1:9092/rsGxsChannels/createGroup
--------------------------------------------------------------------------------

.JSON API call result
[source,json]
--------------------------------------------------------------------------------
{
    "caller_data": "Here can go any kind of JSON data (even objects) that the caller want to get back together with the response",
    "retval": true,
    "token": 3
}
--------------------------------------------------------------------------------

Even if it is less efficient because of URL encoding HTTP +GET+ method is
supported too, so in cases where the client cannot use +POST+ she can still use
+GET+ taking care of encoding the JSON data. With +curl+ this can be done at
least in two different ways.

.Calling the JSON API with GET method with curl on the terminal
[source,bash]
--------------------------------------------------------------------------------
curl -u $API_USER --get --data-urlencode jsonData@paramethers.json \
	http://127.0.0.1:9092/rsGxsChannels/createGroup
--------------------------------------------------------------------------------

Letting +curl+ do the encoding is much more elegant but it is semantically
equivalent to the following.

.Calling the JSON API with GET method and pre-encoded data with curl on the terminal
--------------------------------------------------------------------------------
curl -u $API_USER http://127.0.0.1:9092/rsGxsChannels/createGroup?jsonData=%7B%0A%20%20%20%20%22group%22%3A%7B%0A%20%20%20%20%20%20%20%20%22mMeta%22%3A%7B%0A%20%20%20%20%20%20%20%20%20%20%20%20%22mGroupName%22%3A%22JSON%20test%20group%22%2C%0A%20%20%20%20%20%20%20%20%20%20%20%20%22mGroupFlags%22%3A4%2C%0A%20%20%20%20%20%20%20%20%20%20%20%20%22mSignFlags%22%3A520%0A%20%20%20%20%20%20%20%20%7D%2C%0A%20%20%20%20%20%20%20%20%22mDescription%22%3A%22JSON%20test%20group%20description%22%0A%20%20%20%20%7D%2C%0A%20%20%20%20%22caller_data%22%3A%22Here%20can%20go%20any%20kind%20of%20JSON%20data%20%28even%20objects%29%20that%20the%20caller%20want%20to%20get%20back%20together%20with%20the%20response%22%0A%7D
--------------------------------------------------------------------------------

Note that using +GET+ method +?jsonData=+ and then the JSON data URL encoded are
added after the path in the HTTP request.


== JSON API authentication

Most of JSON API methods require authentication as they give access to
RetroShare user data, and we don't want any application running on the system
eventually by other users be able to access private data indiscriminately.
JSON API support HTTP Basic as authentication scheme, this is enough as JSON API
server is intented for usage on the same system (127.0.0.1) not over an
untrusted network.
If you need to use JSON API over an untrusted network consider using a reverse
proxy with HTTPS such as NGINX in front of JSON API server.
If RetroShare login has been effectuated through the JSON API you can use your
location SSLID as username and your PGP password as credential for the JSON API,
but we suggests you use specific meaningful and human readable credentials for
each JSON API client so the human user can have better control over which client
can access the JSON API.

.NewToken.json
[source,json]
--------------------------------------------------------------------------------
{
	"token": "myNewUser:myNewPassword"
}
--------------------------------------------------------------------------------

.An authenticated client can authorize new tokens like this
--------------------------------------------------------------------------------
curl -u $API_USER --data @NewToken.json http://127.0.0.1:9092/jsonApiServer/authorizeToken
--------------------------------------------------------------------------------

.An unauthenticated JSON API client can request access with
--------------------------------------------------------------------------------
curl --data @NewToken.json http://127.0.0.1:9092/jsonApiServer/requestNewTokenAutorization
--------------------------------------------------------------------------------

When an unauthenticated client request his token to be authorized, JSON API
server will try to ask confirmation to the human user if possible through
+mNewAccessRequestCallback+, if it is not possible or the user didn't authorized
the token +false+ is returned.


== Offer new RetroShare services through JSON API

To offer a retroshare service through the JSON API, first of all one need find
the global pointer to the service instance and document it in doxygen syntax,
plus marking with the custom doxygen command +@jsonapi{RS_VERSION}+ where
+RS_VERSION+ is the retroshare version in which this service became available
with the current semantic (major changes to the service semantic, changes the
meaning of the service itself, so the version should be updated in the
documentation in that case).

.Service instance pointer in rsgxschannels.h
[source,cpp]
--------------------------------------------------------------------------------
/**
 * Pointer to global instance of RsGxsChannels service implementation
 * @jsonapi{development}
 */
extern RsGxsChannels* rsGxsChannels;
--------------------------------------------------------------------------------


Once the service instance itself is known to the JSON API you need to document
in doxygen syntax and mark with the custom doxygen command
+@jsonapi{RS_VERSION}+ the methods of the service that you want to make
available through JSON API.

.Offering RsGxsChannels::getChannelDownloadDirectory in rsgxschannels.h
[source,cpp]
--------------------------------------------------------------------------------
	/**
	 * Get download directory for the given channel
	 * @jsonapi{development}
	 * @param[in] channelId id of the channel
	 * @param[out] directory reference to string where to store the path
	 * @return false on error, true otherwise
	 */
	virtual bool getChannelDownloadDirectory( const RsGxsGroupId& channelId,
	                                          std::string& directory ) = 0;
--------------------------------------------------------------------------------

For each paramether you must specify if it is used as input +@param[in]+ as
output +@param[out]+ or both +@param[inout]+. Paramethers and return value
types must be of a type supported by +RsTypeSerializer+ which already support
most basic types (+bool+, +std::string+...), +RsSerializable+ and containers of
them like +std::vector<std::string>+. Paramethers passed by value and by
reference of those types are both supported, while passing by pointer is not
supported. If your paramether or return +class+/+struct+ type is not supported
yet by +RsTypeSerializer+ most convenient approach is to make it derive from
+RsSerializable+ and implement +serial_process+ method like I did with
+RsGxsChannelGroup+.

.Deriving RsGxsChannelGroup from RsSerializable in rsgxschannels.h
[source,cpp]
--------------------------------------------------------------------------------
struct RsGxsChannelGroup : RsSerializable
{
	RsGroupMetaData mMeta;
	std::string mDescription;
	RsGxsImage mImage;

	bool mAutoDownload;

	/// @see RsSerializable
	virtual void serial_process( RsGenericSerializer::SerializeJob j,
	                             RsGenericSerializer::SerializeContext& ctx )
	{
		RS_SERIAL_PROCESS(mMeta);
		RS_SERIAL_PROCESS(mDescription);
		RS_SERIAL_PROCESS(mImage);
		RS_SERIAL_PROCESS(mAutoDownload);
	}
};
--------------------------------------------------------------------------------

You can do the same recursively for any member of your +struct+ that is not yet
supported by +RsTypeSerializer+.

Some Retroshare {Cxx} API functions are asyncronous, historically RetroShare
didn't follow a policy on how to expose asyncronous API so differents services
and some times even differents method of the same service follow differents
asyncronous patterns, thus making automatic generation of JSON API wrappers for
those methods impractical. Instead of dealing with all those differents patterns
I have chosed to support only one new pattern taking advantage of modern {Cxx}11
and restbed features. On the {Cxx}11 side lambdas and +std::function+s are used,
on the restbed side Server Side Events are used to send asyncronous results.

Lets see an example so it will be much esier to understand.

.RsGxsChannels::turtleSearchRequest asyncronous API
[source,cpp]
--------------------------------------------------------------------------------
	/**
	 * @brief Request remote channels search
	 * @jsonapi{development}
	 * @param[in] matchString string to look for in the search
	 * @param multiCallback function that will be called each time a search
	 * result is received
	 * @param[in] maxWait maximum wait time in seconds for search results
	 * @return false on error, true otherwise
	 */
	virtual bool turtleSearchRequest(
	        const std::string& matchString,
	        const std::function<void (const RsGxsGroupSummary& result)>& multiCallback,
	        std::time_t maxWait = 300 ) = 0;
--------------------------------------------------------------------------------

+RsGxsChannels::turtleSearchRequest(...)+ is an asyncronous method because it
send a channel search request on turtle network and then everytime a result is
received from the network +multiCallback+ is called and the result is passed as
parameter. To be supported by the automatic JSON API wrappers generator an
asyncronous method need a parameter of type +std::function<void (...)>+ called
+callback+ if the callback will be called only once or +multiCallback+ if the
callback is expected to be called more then once like in this case.
A second mandatory parameter is +maxWait+ of type +std::time_t+ it indicates the
maximum amount of time in seconds for which the caller is willing to wait for
results, in case the timeout is reached the callback will not be called anymore.

[IMPORTANT]
================================================================================
+callback+ and +multiCallback+ parameters documentation must *not* specify
+[in]+, +[out]+, +[inout]+, in Doxygen documentation as this would fool the
automatic wrapper generator, and ultimately break the compilation.
================================================================================

.RsFiles::turtleSearchRequest asyncronous JSON API usage example
[source,bash]
--------------------------------------------------------------------------------
$ cat turtle_search.json
{
    "matchString":"linux"
}
$ curl --data @turtle_search.json http://127.0.0.1:9092/rsFiles/turtleSearchRequest
data: {"retval":true}

data: {"results":[{"size":157631,"hash":"69709b4d01025584a8def5cd78ebbd1a3cf3fd05","name":"kill_bill_linux_1024x768.jpg"},{"size":192560,"hash":"000000000000000000009a93e5be8486c496f46c","name":"coffee_box_linux2.jpg"},{"size":455087,"hash":"9a93e5be8486c496f46c00000000000000000000","name":"Linux.png"},{"size":182004,"hash":"e8845280912ebf3779e400000000000000000000","name":"Linux_2_6.png"}]}

data: {"results":[{"size":668,"hash":"e8845280912ebf3779e400000000000000000000","name":"linux.png"},{"size":70,"hash":"e8845280912ebf3779e400000000000000000000","name":"kali-linux-2016.2-amd64.txt.sha1sum"},{"size":3076767744,"hash":"e8845280912ebf3779e400000000000000000000","name":"kali-linux-2016.2-amd64.iso"},{"size":2780872,"hash":"e8845280912ebf3779e400000000000000000000","name":"openwrt-ar71xx-generic-vmlinux.bin"},{"size":917504,"hash":"e8845280912ebf3779e400000000000000000000","name":"openwrt-ar71xx-generic-vmlinux.lzma"},{"size":2278404096,"hash":"e8845280912ebf3779e400000000000000000000","name":"gentoo-linux-livedvd-amd64-multilib-20160704.iso"},{"size":151770333,"hash":"e8845280912ebf3779e400000000000000000000","name":"flashtool-0.9.23.0-linux.tar.7z"},{"size":2847372,"hash":"e8845280912ebf3779e400000000000000000000","name":"openwrt-ar71xx-generic-vmlinux.elf"},{"size":1310720,"hash":"e8845280912ebf3779e400000000000000000000","name":"openwrt-ar71xx-generic-vmlinux.gz"},{"size":987809,"hash":"e8845280912ebf3779e400000000000000000000","name":"openwrt-ar71xx-generic-vmlinux-lzma.elf"}]}

--------------------------------------------------------------------------------

By default JSON API methods requires client authentication and their wrappers
are automatically generated by +json-api-generator+.
In some cases methods need do be accessible without authentication such as
+rsLoginHelper/getLocations+ so in the doxygen documentaion they have the custom
command +@jsonapi{RS_VERSION,unauthenticated}+.
Other methods such as +/rsControl/rsGlobalShutDown+ need special care so they
are marked with the custom doxygen command +@jsonapi{RS_VERSION,manualwrapper}+
and their wrappers are not automatically generated but written manually into
+JsonApiServer::JsonApiServer(...)+.

== Quirks

=== 64 bits integers handling

While JSON doesn't have problems representing 64 bits integers JavaScript, Dart
and other languages are not capable to handle those numbers natively.
To overcome this limitation JSON API output 64 bit integers as an object with
two keys, one as proper integer and one as string representation.

.JSON API 64 bit integer output example
[source,json]
--------------------------------------------------------------------------------
"lobby_id": { "xint64": 6215642878098695544, "xstr64": "6215642878098695544" }
--------------------------------------------------------------------------------

So from languages that have proper 64bit integers support like Python or C++ one
better read from `xint64` which is represented as a JSON integer, from languages
where there is no proper 64bit integers support like JavaScriptone can read from
`xstr64` which is represented as JSON string (note that the first is not wrapped
in "" while the latter is).

When one input a 64bit integer into the JSON API it first try to parse it as if
it was sent the old way for retrocompatibility.

.JSON API 64 bit integer deprecated format input example
[source,json]
--------------------------------------------------------------------------------
"lobby_id":6215642878098695544
--------------------------------------------------------------------------------

This way is *DEPRECATED* and may disappear in the future, it is TEMPORALLY kept
only for retrocompatibiliy with old clients.

If retrocompatible parsing attempt fail then it try to parse with the new way
with proper JSON integer format.

.JSON API 64 bit integer new proper integer format input example
[source,json]
--------------------------------------------------------------------------------
lobby_id": { "xint64": 6215642878098695544 }
--------------------------------------------------------------------------------

If this fails then it try to parse with the new way with JSON string format.

.JSON API 64 bit integer new string format input example
[source,json]
--------------------------------------------------------------------------------
"lobby_id": { "xstr64": "6215642878098695544" }
--------------------------------------------------------------------------------

[WARNING]
================================================================================
Clients written in languages without proper 64bit integers support must
use *ONLY* the string format otherwise they will send approximated values and
get unexpected results from the JSON API, because parsing will success but the
value will not be exactly the one you believe you sent.
================================================================================


== A bit of history

=== First writings about this

The previous attempt of exposing a RetroShare JSON API is called +libresapi+ and
unfortunatley it requires a bunch of boilerplate code when we want to expose
something present in the {Cxx} API in the JSON API.

As an example here you can see the libresapi that exposes part of the retroshare
chat {Cxx} API and lot of boilerplate code just to convert {Cxx} objects to JSON

https://github.com/RetroShare/RetroShare/blob/v0.6.4/libresapi/src/api/ChatHandler.cpp#L44

To avoid the {Cxx} to JSON and back conversion boilerplate code I have worked out
an extension to our {Cxx} serialization code so it is capable to serialize and
deserialize to JSON you can see it in this pull request

https://github.com/RetroShare/RetroShare/pull/1155

So first step toward having a good API is to take advantage of the fact that RS
is now capable of converting C++ objects from and to JSON.

The current API is accessible via HTTP and unix socket, there is no
authentication in both of them, so anyone having access to the HTTP server or to
the unix socket can access the API without extra restrictions.
Expecially for the HTTP API this is a big risk because also if the http server
listen on 127.0.0.1 every application on the machine (even rogue javascript
running on your web browser) can access that and for example on android it is
not safe at all (because of that I implemented the unix socket access so at
least in android API was reasonably safe) because of this.

A second step to improve the API would be to implement some kind of API
authentication mechanism (it would be nice that the mechanism is handled at API
level and not at transport level so we can use it for any API trasport not just
HTTP for example)

The HTTP server used by libresapi is libmicrohttpd server that is very minimal,
it doesn't provide HTTPS nor modern HTTP goodies, like server notifications,
websockets etc. because the lack of support we have a token polling mechanism in
libresapi to avoid polling for every thing but it is still ugly, so if we can
completely get rid of polling in the API that would be really nice.
I have done a crawl to look for a replacement and briefly looked at

- https://www.gnu.org/software/libmicrohttpd/
- http://wolkykim.github.io/libasyncd/
- https://github.com/corvusoft/restbed
- https://code.facebook.com/posts/1503205539947302/introducing-proxygen-facebook-s-c-http-framework/
- https://github.com/cmouse/yahttp

taking in account a few metrics like modern HTTP goodies support, license,
platform support, external dependencies and documentation it seemed to me that
restbed is the more appropriate.

Another source of boilerplate code into libresapi is the mapping between JSON
API requests and C++ API methods as an example you can look at this

https://github.com/RetroShare/RetroShare/blob/v0.6.4/libresapi/src/api/ChatHandler.cpp#L158

and this

https://github.com/RetroShare/RetroShare/blob/v0.6.4/libresapi/src/api/ApiServer.cpp#L253

The abstract logic of this thing is, when libreasapi get a request like
+/chat/initiate_distant_chat+ then call
+ChatHandler::handleInitiateDistantChatConnexion+ which in turn is just a
wrapper of +RsMsgs::initiateDistantChatConnexion+ all this process is basically
implemented as boilerplate code and would be unnecessary in a smarter design of
the API because almost all the information needed is already present in the
C++ API +libretroshare/src/retroshare+.

So a third step to improve the JSON API would be to remove this source of
boilerplate code by automatizing the mapping between C++ and JSON API call.

This may result a little tricky as language parsing or other adevanced things
may be required.

Hope this dive is useful for you +
Cheers +
G10h4ck

=== Second writings about this

I have been investigating a bit more about:
[verse, G10h4ck]
________________________________________________________________________________
So a third step to improve the JSON API would be to remove this source of
boilerplate code by automatizing the mapping between C++ and JSON API call
________________________________________________________________________________

After spending some hours investigating this topic the most reasonable approach
seems to:

1. Properly document headers in +libretroshare/src/retroshare/+ in doxygen syntax
specifying wihich params are input and/or output (doxygen sysntax for this is
+@param[in/out/inout]+) this will be the API documentation too.

2. At compile time use doxygen to generate XML description of the headers and use
the XML to generate the JSON api server stub.
http://www.stack.nl/~dimitri/doxygen/manual/customize.html#xmlgenerator

3. Enjoy