mirror of
https://github.com/RetroShare/RetroShare.git
synced 2024-12-27 00:19:25 -05:00
448 lines
22 KiB
Plaintext
448 lines
22 KiB
Plaintext
// SPDX-FileCopyrightText: (C) 2004-2019 Retroshare Team <contact@retroshare.cc>
|
|
// SPDX-License-Identifier: CC0-1.0
|
|
|
|
RetroShare JSON API
|
|
===================
|
|
|
|
:Cxx: C++
|
|
|
|
== How to use RetroShare JSON API
|
|
|
|
Look for methods marked with +@jsonapi+ doxygen custom command into
|
|
+libretroshare/src/retroshare+. The method path is composed by service instance
|
|
pointer name like +rsGxsChannels+ for +RsGxsChannels+, and the method name like
|
|
+createGroup+ and pass the input paramethers as a JSON object.
|
|
|
|
.Service instance pointer in rsgxschannels.h
|
|
[source,cpp]
|
|
--------------------------------------------------------------------------------
|
|
/**
|
|
* Pointer to global instance of RsGxsChannels service implementation
|
|
* @jsonapi{development}
|
|
*/
|
|
extern RsGxsChannels* rsGxsChannels;
|
|
--------------------------------------------------------------------------------
|
|
|
|
.Method declaration in rsgxschannels.h
|
|
[source,cpp]
|
|
--------------------------------------------------------------------------------
|
|
/**
|
|
* @brief Request channel creation.
|
|
* The action is performed asyncronously, so it could fail in a subsequent
|
|
* phase even after returning true.
|
|
* @jsonapi{development}
|
|
* @param[out] token Storage for RsTokenService token to track request
|
|
* status.
|
|
* @param[in] group Channel data (name, description...)
|
|
* @return false on error, true otherwise
|
|
*/
|
|
virtual bool createGroup(uint32_t& token, RsGxsChannelGroup& group) = 0;
|
|
--------------------------------------------------------------------------------
|
|
|
|
.paramethers.json
|
|
[source,json]
|
|
--------------------------------------------------------------------------------
|
|
{
|
|
"group":{
|
|
"mMeta":{
|
|
"mGroupName":"JSON test group",
|
|
"mGroupFlags":4,
|
|
"mSignFlags":520
|
|
},
|
|
"mDescription":"JSON test group description"
|
|
},
|
|
"caller_data":"Here can go any kind of JSON data (even objects) that the caller want to get back together with the response"
|
|
}
|
|
--------------------------------------------------------------------------------
|
|
|
|
.Calling the JSON API with curl on the terminal
|
|
[source,bash]
|
|
--------------------------------------------------------------------------------
|
|
curl -u $API_USER --data @paramethers.json http://127.0.0.1:9092/rsGxsChannels/createGroup
|
|
--------------------------------------------------------------------------------
|
|
|
|
.JSON API call result
|
|
[source,json]
|
|
--------------------------------------------------------------------------------
|
|
{
|
|
"caller_data": "Here can go any kind of JSON data (even objects) that the caller want to get back together with the response",
|
|
"retval": true,
|
|
"token": 3
|
|
}
|
|
--------------------------------------------------------------------------------
|
|
|
|
Even if it is less efficient because of URL encoding HTTP +GET+ method is
|
|
supported too, so in cases where the client cannot use +POST+ she can still use
|
|
+GET+ taking care of encoding the JSON data. With +curl+ this can be done at
|
|
least in two different ways.
|
|
|
|
.Calling the JSON API with GET method with curl on the terminal
|
|
[source,bash]
|
|
--------------------------------------------------------------------------------
|
|
curl -u $API_USER --get --data-urlencode jsonData@paramethers.json \
|
|
http://127.0.0.1:9092/rsGxsChannels/createGroup
|
|
--------------------------------------------------------------------------------
|
|
|
|
Letting +curl+ do the encoding is much more elegant but it is semantically
|
|
equivalent to the following.
|
|
|
|
.Calling the JSON API with GET method and pre-encoded data with curl on the terminal
|
|
--------------------------------------------------------------------------------
|
|
curl -u $API_USER http://127.0.0.1:9092/rsGxsChannels/createGroup?jsonData=%7B%0A%20%20%20%20%22group%22%3A%7B%0A%20%20%20%20%20%20%20%20%22mMeta%22%3A%7B%0A%20%20%20%20%20%20%20%20%20%20%20%20%22mGroupName%22%3A%22JSON%20test%20group%22%2C%0A%20%20%20%20%20%20%20%20%20%20%20%20%22mGroupFlags%22%3A4%2C%0A%20%20%20%20%20%20%20%20%20%20%20%20%22mSignFlags%22%3A520%0A%20%20%20%20%20%20%20%20%7D%2C%0A%20%20%20%20%20%20%20%20%22mDescription%22%3A%22JSON%20test%20group%20description%22%0A%20%20%20%20%7D%2C%0A%20%20%20%20%22caller_data%22%3A%22Here%20can%20go%20any%20kind%20of%20JSON%20data%20%28even%20objects%29%20that%20the%20caller%20want%20to%20get%20back%20together%20with%20the%20response%22%0A%7D
|
|
--------------------------------------------------------------------------------
|
|
|
|
Note that using +GET+ method +?jsonData=+ and then the JSON data URL encoded are
|
|
added after the path in the HTTP request.
|
|
|
|
|
|
== JSON API authentication
|
|
|
|
Most of JSON API methods require authentication as they give access to
|
|
RetroShare user data, and we don't want any application running on the system
|
|
eventually by other users be able to access private data indiscriminately.
|
|
JSON API support HTTP Basic as authentication scheme, this is enough as JSON API
|
|
server is intented for usage on the same system (127.0.0.1) not over an
|
|
untrusted network.
|
|
If you need to use JSON API over an untrusted network consider using a reverse
|
|
proxy with HTTPS such as NGINX in front of JSON API server.
|
|
If RetroShare login has been effectuated through the JSON API you can use your
|
|
location SSLID as username and your PGP password as credential for the JSON API,
|
|
but we suggests you use specific meaningful and human readable credentials for
|
|
each JSON API client so the human user can have better control over which client
|
|
can access the JSON API.
|
|
|
|
.NewToken.json
|
|
[source,json]
|
|
--------------------------------------------------------------------------------
|
|
{
|
|
"token": "myNewUser:myNewPassword"
|
|
}
|
|
--------------------------------------------------------------------------------
|
|
|
|
.An authenticated client can authorize new tokens like this
|
|
--------------------------------------------------------------------------------
|
|
curl -u $API_USER --data @NewToken.json http://127.0.0.1:9092/jsonApiServer/authorizeToken
|
|
--------------------------------------------------------------------------------
|
|
|
|
.An unauthenticated JSON API client can request access with
|
|
--------------------------------------------------------------------------------
|
|
curl --data @NewToken.json http://127.0.0.1:9092/jsonApiServer/requestNewTokenAutorization
|
|
--------------------------------------------------------------------------------
|
|
|
|
When an unauthenticated client request his token to be authorized, JSON API
|
|
server will try to ask confirmation to the human user if possible through
|
|
+mNewAccessRequestCallback+, if it is not possible or the user didn't authorized
|
|
the token +false+ is returned.
|
|
|
|
|
|
== Offer new RetroShare services through JSON API
|
|
|
|
To offer a retroshare service through the JSON API, first of all one need find
|
|
the global pointer to the service instance and document it in doxygen syntax,
|
|
plus marking with the custom doxygen command +@jsonapi{RS_VERSION}+ where
|
|
+RS_VERSION+ is the retroshare version in which this service became available
|
|
with the current semantic (major changes to the service semantic, changes the
|
|
meaning of the service itself, so the version should be updated in the
|
|
documentation in that case).
|
|
|
|
.Service instance pointer in rsgxschannels.h
|
|
[source,cpp]
|
|
--------------------------------------------------------------------------------
|
|
/**
|
|
* Pointer to global instance of RsGxsChannels service implementation
|
|
* @jsonapi{development}
|
|
*/
|
|
extern RsGxsChannels* rsGxsChannels;
|
|
--------------------------------------------------------------------------------
|
|
|
|
|
|
Once the service instance itself is known to the JSON API you need to document
|
|
in doxygen syntax and mark with the custom doxygen command
|
|
+@jsonapi{RS_VERSION}+ the methods of the service that you want to make
|
|
available through JSON API.
|
|
|
|
.Offering RsGxsChannels::getChannelDownloadDirectory in rsgxschannels.h
|
|
[source,cpp]
|
|
--------------------------------------------------------------------------------
|
|
/**
|
|
* Get download directory for the given channel
|
|
* @jsonapi{development}
|
|
* @param[in] channelId id of the channel
|
|
* @param[out] directory reference to string where to store the path
|
|
* @return false on error, true otherwise
|
|
*/
|
|
virtual bool getChannelDownloadDirectory( const RsGxsGroupId& channelId,
|
|
std::string& directory ) = 0;
|
|
--------------------------------------------------------------------------------
|
|
|
|
For each paramether you must specify if it is used as input +@param[in]+ as
|
|
output +@param[out]+ or both +@param[inout]+. Paramethers and return value
|
|
types must be of a type supported by +RsTypeSerializer+ which already support
|
|
most basic types (+bool+, +std::string+...), +RsSerializable+ and containers of
|
|
them like +std::vector<std::string>+. Paramethers passed by value and by
|
|
reference of those types are both supported, while passing by pointer is not
|
|
supported. If your paramether or return +class+/+struct+ type is not supported
|
|
yet by +RsTypeSerializer+ most convenient approach is to make it derive from
|
|
+RsSerializable+ and implement +serial_process+ method like I did with
|
|
+RsGxsChannelGroup+.
|
|
|
|
.Deriving RsGxsChannelGroup from RsSerializable in rsgxschannels.h
|
|
[source,cpp]
|
|
--------------------------------------------------------------------------------
|
|
struct RsGxsChannelGroup : RsSerializable
|
|
{
|
|
RsGroupMetaData mMeta;
|
|
std::string mDescription;
|
|
RsGxsImage mImage;
|
|
|
|
bool mAutoDownload;
|
|
|
|
/// @see RsSerializable
|
|
virtual void serial_process( RsGenericSerializer::SerializeJob j,
|
|
RsGenericSerializer::SerializeContext& ctx )
|
|
{
|
|
RS_SERIAL_PROCESS(mMeta);
|
|
RS_SERIAL_PROCESS(mDescription);
|
|
RS_SERIAL_PROCESS(mImage);
|
|
RS_SERIAL_PROCESS(mAutoDownload);
|
|
}
|
|
};
|
|
--------------------------------------------------------------------------------
|
|
|
|
You can do the same recursively for any member of your +struct+ that is not yet
|
|
supported by +RsTypeSerializer+.
|
|
|
|
Some Retroshare {Cxx} API functions are asyncronous, historically RetroShare
|
|
didn't follow a policy on how to expose asyncronous API so differents services
|
|
and some times even differents method of the same service follow differents
|
|
asyncronous patterns, thus making automatic generation of JSON API wrappers for
|
|
those methods impractical. Instead of dealing with all those differents patterns
|
|
I have chosed to support only one new pattern taking advantage of modern {Cxx}11
|
|
and restbed features. On the {Cxx}11 side lambdas and +std::function+s are used,
|
|
on the restbed side Server Side Events are used to send asyncronous results.
|
|
|
|
Lets see an example so it will be much esier to understand.
|
|
|
|
.RsGxsChannels::turtleSearchRequest asyncronous API
|
|
[source,cpp]
|
|
--------------------------------------------------------------------------------
|
|
/**
|
|
* @brief Request remote channels search
|
|
* @jsonapi{development}
|
|
* @param[in] matchString string to look for in the search
|
|
* @param multiCallback function that will be called each time a search
|
|
* result is received
|
|
* @param[in] maxWait maximum wait time in seconds for search results
|
|
* @return false on error, true otherwise
|
|
*/
|
|
virtual bool turtleSearchRequest(
|
|
const std::string& matchString,
|
|
const std::function<void (const RsGxsGroupSummary& result)>& multiCallback,
|
|
std::time_t maxWait = 300 ) = 0;
|
|
--------------------------------------------------------------------------------
|
|
|
|
+RsGxsChannels::turtleSearchRequest(...)+ is an asyncronous method because it
|
|
send a channel search request on turtle network and then everytime a result is
|
|
received from the network +multiCallback+ is called and the result is passed as
|
|
parameter. To be supported by the automatic JSON API wrappers generator an
|
|
asyncronous method need a parameter of type +std::function<void (...)>+ called
|
|
+callback+ if the callback will be called only once or +multiCallback+ if the
|
|
callback is expected to be called more then once like in this case.
|
|
A second mandatory parameter is +maxWait+ of type +std::time_t+ it indicates the
|
|
maximum amount of time in seconds for which the caller is willing to wait for
|
|
results, in case the timeout is reached the callback will not be called anymore.
|
|
|
|
[IMPORTANT]
|
|
================================================================================
|
|
+callback+ and +multiCallback+ parameters documentation must *not* specify
|
|
+[in]+, +[out]+, +[inout]+, in Doxygen documentation as this would fool the
|
|
automatic wrapper generator, and ultimately break the compilation.
|
|
================================================================================
|
|
|
|
.RsFiles::turtleSearchRequest asyncronous JSON API usage example
|
|
[source,bash]
|
|
--------------------------------------------------------------------------------
|
|
$ cat turtle_search.json
|
|
{
|
|
"matchString":"linux"
|
|
}
|
|
$ curl --data @turtle_search.json http://127.0.0.1:9092/rsFiles/turtleSearchRequest
|
|
data: {"retval":true}
|
|
|
|
data: {"results":[{"size":157631,"hash":"69709b4d01025584a8def5cd78ebbd1a3cf3fd05","name":"kill_bill_linux_1024x768.jpg"},{"size":192560,"hash":"000000000000000000009a93e5be8486c496f46c","name":"coffee_box_linux2.jpg"},{"size":455087,"hash":"9a93e5be8486c496f46c00000000000000000000","name":"Linux.png"},{"size":182004,"hash":"e8845280912ebf3779e400000000000000000000","name":"Linux_2_6.png"}]}
|
|
|
|
data: {"results":[{"size":668,"hash":"e8845280912ebf3779e400000000000000000000","name":"linux.png"},{"size":70,"hash":"e8845280912ebf3779e400000000000000000000","name":"kali-linux-2016.2-amd64.txt.sha1sum"},{"size":3076767744,"hash":"e8845280912ebf3779e400000000000000000000","name":"kali-linux-2016.2-amd64.iso"},{"size":2780872,"hash":"e8845280912ebf3779e400000000000000000000","name":"openwrt-ar71xx-generic-vmlinux.bin"},{"size":917504,"hash":"e8845280912ebf3779e400000000000000000000","name":"openwrt-ar71xx-generic-vmlinux.lzma"},{"size":2278404096,"hash":"e8845280912ebf3779e400000000000000000000","name":"gentoo-linux-livedvd-amd64-multilib-20160704.iso"},{"size":151770333,"hash":"e8845280912ebf3779e400000000000000000000","name":"flashtool-0.9.23.0-linux.tar.7z"},{"size":2847372,"hash":"e8845280912ebf3779e400000000000000000000","name":"openwrt-ar71xx-generic-vmlinux.elf"},{"size":1310720,"hash":"e8845280912ebf3779e400000000000000000000","name":"openwrt-ar71xx-generic-vmlinux.gz"},{"size":987809,"hash":"e8845280912ebf3779e400000000000000000000","name":"openwrt-ar71xx-generic-vmlinux-lzma.elf"}]}
|
|
|
|
--------------------------------------------------------------------------------
|
|
|
|
By default JSON API methods requires client authentication and their wrappers
|
|
are automatically generated by +json-api-generator+.
|
|
In some cases methods need do be accessible without authentication such as
|
|
+rsLoginHelper/getLocations+ so in the doxygen documentaion they have the custom
|
|
command +@jsonapi{RS_VERSION,unauthenticated}+.
|
|
Other methods such as +/rsControl/rsGlobalShutDown+ need special care so they
|
|
are marked with the custom doxygen command +@jsonapi{RS_VERSION,manualwrapper}+
|
|
and their wrappers are not automatically generated but written manually into
|
|
+JsonApiServer::JsonApiServer(...)+.
|
|
|
|
== Quirks
|
|
|
|
=== 64 bits integers handling
|
|
|
|
While JSON doesn't have problems representing 64 bits integers JavaScript, Dart
|
|
and other languages are not capable to handle those numbers natively.
|
|
To overcome this limitation JSON API output 64 bit integers as an object with
|
|
two keys, one as proper integer and one as string representation.
|
|
|
|
.JSON API 64 bit integer output example
|
|
[source,json]
|
|
--------------------------------------------------------------------------------
|
|
"lobby_id": { "xint64": 6215642878098695544, "xstr64": "6215642878098695544" }
|
|
--------------------------------------------------------------------------------
|
|
|
|
So from languages that have proper 64bit integers support like Python or C++ one
|
|
better read from `xint64` which is represented as a JSON integer, from languages
|
|
where there is no proper 64bit integers support like JavaScript one can read from
|
|
`xstr64` which is represented as JSON string (note that the first is not wrapped
|
|
in "" while the latter is).
|
|
|
|
When one input a 64bit integer into the JSON API it first try to parse it as if
|
|
it was sent the old way for retrocompatibility.
|
|
|
|
.JSON API 64 bit integer deprecated format input example
|
|
[source,json]
|
|
--------------------------------------------------------------------------------
|
|
"lobby_id":6215642878098695544
|
|
--------------------------------------------------------------------------------
|
|
|
|
This way is *DEPRECATED* and may disappear in the future, it is TEMPORALLY kept
|
|
only for retrocompatibiliy with old clients.
|
|
|
|
If retrocompatible parsing attempt fail then it try to parse with the new way
|
|
with proper JSON integer format.
|
|
|
|
.JSON API 64 bit integer new proper integer format input example
|
|
[source,json]
|
|
--------------------------------------------------------------------------------
|
|
lobby_id": { "xint64": 6215642878098695544 }
|
|
--------------------------------------------------------------------------------
|
|
|
|
If this fails then it try to parse with the new way with JSON string format.
|
|
|
|
.JSON API 64 bit integer new string format input example
|
|
[source,json]
|
|
--------------------------------------------------------------------------------
|
|
"lobby_id": { "xstr64": "6215642878098695544" }
|
|
--------------------------------------------------------------------------------
|
|
|
|
[WARNING]
|
|
================================================================================
|
|
Clients written in languages without proper 64bit integers support must
|
|
use *ONLY* the string format otherwise they will send approximated values and
|
|
get unexpected results from the JSON API, because parsing will success but the
|
|
value will not be exactly the one you believe you sent.
|
|
================================================================================
|
|
|
|
|
|
== A bit of history
|
|
|
|
=== First writings about this
|
|
|
|
The previous attempt of exposing a RetroShare JSON API is called +libresapi+ and
|
|
unfortunatley it requires a bunch of boilerplate code when we want to expose
|
|
something present in the {Cxx} API in the JSON API.
|
|
|
|
As an example here you can see the libresapi that exposes part of the retroshare
|
|
chat {Cxx} API and lot of boilerplate code just to convert {Cxx} objects to JSON
|
|
|
|
https://github.com/RetroShare/RetroShare/blob/v0.6.4/libresapi/src/api/ChatHandler.cpp#L44
|
|
|
|
To avoid the {Cxx} to JSON and back conversion boilerplate code I have worked out
|
|
an extension to our {Cxx} serialization code so it is capable to serialize and
|
|
deserialize to JSON you can see it in this pull request
|
|
|
|
https://github.com/RetroShare/RetroShare/pull/1155
|
|
|
|
So first step toward having a good API is to take advantage of the fact that RS
|
|
is now capable of converting C++ objects from and to JSON.
|
|
|
|
The current API is accessible via HTTP and unix socket, there is no
|
|
authentication in both of them, so anyone having access to the HTTP server or to
|
|
the unix socket can access the API without extra restrictions.
|
|
Expecially for the HTTP API this is a big risk because also if the http server
|
|
listen on 127.0.0.1 every application on the machine (even rogue javascript
|
|
running on your web browser) can access that and for example on android it is
|
|
not safe at all (because of that I implemented the unix socket access so at
|
|
least in android API was reasonably safe) because of this.
|
|
|
|
A second step to improve the API would be to implement some kind of API
|
|
authentication mechanism (it would be nice that the mechanism is handled at API
|
|
level and not at transport level so we can use it for any API trasport not just
|
|
HTTP for example)
|
|
|
|
The HTTP server used by libresapi is libmicrohttpd server that is very minimal,
|
|
it doesn't provide HTTPS nor modern HTTP goodies, like server notifications,
|
|
websockets etc. because the lack of support we have a token polling mechanism in
|
|
libresapi to avoid polling for every thing but it is still ugly, so if we can
|
|
completely get rid of polling in the API that would be really nice.
|
|
I have done a crawl to look for a replacement and briefly looked at
|
|
|
|
- https://www.gnu.org/software/libmicrohttpd/
|
|
- http://wolkykim.github.io/libasyncd/
|
|
- https://github.com/corvusoft/restbed
|
|
- https://code.facebook.com/posts/1503205539947302/introducing-proxygen-facebook-s-c-http-framework/
|
|
- https://github.com/cmouse/yahttp
|
|
|
|
taking in account a few metrics like modern HTTP goodies support, license,
|
|
platform support, external dependencies and documentation it seemed to me that
|
|
restbed is the more appropriate.
|
|
|
|
Another source of boilerplate code into libresapi is the mapping between JSON
|
|
API requests and C++ API methods as an example you can look at this
|
|
|
|
https://github.com/RetroShare/RetroShare/blob/v0.6.4/libresapi/src/api/ChatHandler.cpp#L158
|
|
|
|
and this
|
|
|
|
https://github.com/RetroShare/RetroShare/blob/v0.6.4/libresapi/src/api/ApiServer.cpp#L253
|
|
|
|
The abstract logic of this thing is, when libreasapi get a request like
|
|
+/chat/initiate_distant_chat+ then call
|
|
+ChatHandler::handleInitiateDistantChatConnexion+ which in turn is just a
|
|
wrapper of +RsMsgs::initiateDistantChatConnexion+ all this process is basically
|
|
implemented as boilerplate code and would be unnecessary in a smarter design of
|
|
the API because almost all the information needed is already present in the
|
|
C++ API +libretroshare/src/retroshare+.
|
|
|
|
So a third step to improve the JSON API would be to remove this source of
|
|
boilerplate code by automatizing the mapping between C++ and JSON API call.
|
|
|
|
This may result a little tricky as language parsing or other adevanced things
|
|
may be required.
|
|
|
|
Hope this dive is useful for you +
|
|
Cheers +
|
|
G10h4ck
|
|
|
|
=== Second writings about this
|
|
|
|
I have been investigating a bit more about:
|
|
[verse, G10h4ck]
|
|
________________________________________________________________________________
|
|
So a third step to improve the JSON API would be to remove this source of
|
|
boilerplate code by automatizing the mapping between C++ and JSON API call
|
|
________________________________________________________________________________
|
|
|
|
After spending some hours investigating this topic the most reasonable approach
|
|
seems to:
|
|
|
|
1. Properly document headers in +libretroshare/src/retroshare/+ in doxygen syntax
|
|
specifying wihich params are input and/or output (doxygen sysntax for this is
|
|
+@param[in/out/inout]+) this will be the API documentation too.
|
|
|
|
2. At compile time use doxygen to generate XML description of the headers and use
|
|
the XML to generate the JSON api server stub.
|
|
http://www.stack.nl/~dimitri/doxygen/manual/customize.html#xmlgenerator
|
|
|
|
3. Enjoy
|