Commit Graph

70 Commits

Author SHA1 Message Date
Gioacchino Mazzurco
3a26ccf6a5
Implement deep indexing for files through Xapian
ATM it support extracting metadata only from OGG files.
The system has been designed to be easly extensible to more file formats
  registering more indexer functions which just need to extract metadata
  from a certain type of file and feed it to Xapian.
The system has been integrated into existent file search system to
  through generric search requests and results, it keep a good level of
  retro-compatibility due to some tricks.
The indexing system is released under AGPLv3  so when libretroshare is compiled
  with deep search enabled AGPLv3 must be honored instead of LGPLv3-or-later.
Cleaned up the debian copyright file using non-deprecated license
  code-names.
2019-10-26 15:57:38 +02:00
Phenom
b46e07191f Fix clang warning for
In file included from ../../../trunk/libretroshare/src/turtle/
p3turtle.cc:30:
In file included from ../../../trunk/libretroshare/src/rsserver/p3face.h:
38:
In file included from ../../../trunk/libretroshare/src/chat/
p3chatservice.h:30:
In file included from ../../../trunk/libretroshare/src/rsitems/
rsmsgitems.h:34:
In file included from ../../../trunk/libretroshare/src/grouter/
grouteritems.h:32:
In file included from ../../../trunk/libretroshare/src/grouter/
groutermatrix.h:29:
In file included from ../../../trunk/libretroshare/src/grouter/
groutertypes.h:28:
../../../trunk/libretroshare/src/turtle/p3turtle.h:447:8: warning:
private field '_force_digg_new_tunnels' is not used [-Wunused-private-
field]
                bool _force_digg_new_tunnels ;
                     ^
2019-04-04 21:06:01 +02:00
Gioacchino Mazzurco
329050a9c2
Use safer rstime_t instead of time_t
Avoid problems to serialization on different platforms, without breaking
nested STL containers serialization.

The conversion have been made with sed, and checked with grep, plus
kdiff3 visual ispection, plus rutime tests, so it should be fine.
2018-10-07 13:07:06 +02:00
csoler
2bab688dec
fixed merge with upstream/master 2018-07-19 23:22:24 +02:00
csoler
98f8e4da0a
added per-item type limit for turtle search results 2018-07-05 17:40:06 +02:00
csoler
c67084b7de
fixed mutex problem in turtle-GXS search 2018-06-17 21:23:16 +02:00
csoler
84194b6234
sereral fixes to GXS distant search 2018-06-16 22:39:35 +02:00
csoler
7caf06b57d
added structures for generic turtle search and access functions in Gxs objects 2018-06-06 23:15:29 +02:00
csoler
6cf2090149
relicensed tcponudp, turtle, unused, upnp, and part of util/ 2018-05-30 21:19:13 +02:00
csoler
58aa2413b3
finished moving tunnel encryption into turtle 2018-03-23 22:46:04 +01:00
csoler
00a6bd5b73
started to move FT encryption into p3turtle 2018-03-22 14:41:50 +01:00
csoler
6fe14136ae
added limiter in number of responses forwarded back by turtle search, which should avoid bursts of bandwidth for popular requests 2018-02-17 23:37:25 +01:00
csoler
ba7cf4995b
Merge pull request #1037 from RetroPooh/searchspy
Searchspy
2017-11-14 20:53:24 +01:00
Pooh
c47e6f620d Merge branch 'master' into tunnel-names 2017-09-25 21:43:41 +03:00
RetroPooh
41251fdd9e search spy 2017-09-15 15:03:06 +03:00
csoler
782c02744f added names for turtle sub items 2017-04-22 16:00:01 +02:00
RetroPooh
6a2181ca14 transfers - add peer names for turtle routed sources 2017-03-10 09:27:46 +03:00
csoler
84341f26b3 changed _outgoing_file_hashes into a new map that connects services to tunnel ID instead of file hash, hence avoiding to remove a hash on server side when a tunnel is closed (bug found by Jolavillette) 2016-09-29 22:30:42 +02:00
csoler
b62b66a24c replace std::vector for std::set to represent _hashes_to_remove, so as to avoid duplicates (suggested by Jo) 2016-09-29 22:30:42 +02:00
mr-alice
a2e34f9cc6 moved regular expression classes into a separate namespace 2016-09-13 12:05:22 +02:00
csoler
697ef4061c added test against arbitrary large items in turtle search requests. 2016-02-17 09:39:08 -05:00
csoler
67c5ef9c8a increased responsivness of distant chat by forcing tunnel re-digging when no activity is detected. WARNING: the new timestamps make the communication break will peers not up to date, since they will not send keep alive packets on the same delay basis.
git-svn-id: http://svn.code.sf.net/p/retroshare/code/trunk@8023 b45a01b8-16f6-495d-af2f-9b41ad6348cc
2015-03-15 13:23:03 +00:00
csoler
817aa793fc added non aggressive mode option to turtle router to make distant chat happy
git-svn-id: http://svn.code.sf.net/p/retroshare/code/trunk@7856 b45a01b8-16f6-495d-af2f-9b41ad6348cc
2015-01-22 20:25:39 +00:00
csoler
11e370c56d fixed non working tunneling system, due to wrong service id used to query online peers
git-svn-id: http://svn.code.sf.net/p/retroshare/code/trunk@7283 b45a01b8-16f6-495d-af2f-9b41ad6348cc
2014-04-19 15:52:06 +00:00
csoler
d340440998 - added debug ifdef to GRouter code
- switched turtle router and grouter to use service control


git-svn-id: http://svn.code.sf.net/p/retroshare/code/trunk@7216 b45a01b8-16f6-495d-af2f-9b41ad6348cc
2014-03-29 23:03:04 +00:00
drbob
83a78bcaee Added ServiceControl + ServiceInfo. Basics are working, but still a lot to do!
git-svn-id: http://svn.code.sf.net/p/retroshare/code/trunk@7196 b45a01b8-16f6-495d-af2f-9b41ad6348cc
2014-03-22 03:53:44 +00:00
chrisparker126
0f29d28b1b merge of branch v0.6-idclean 7180
git-svn-id: http://svn.code.sf.net/p/retroshare/code/trunk@7187 b45a01b8-16f6-495d-af2f-9b41ad6348cc
2014-03-17 20:56:06 +00:00
csoler
6780ff2cff fixed bug that would cause virtual peers list for outgoing files to never get cleanred up. Also improved cost of updating the status of virtual peers. Many thanks to Jolavillette for finding this out!
git-svn-id: http://svn.code.sf.net/p/retroshare/code/trunk@6858 b45a01b8-16f6-495d-af2f-9b41ad6348cc
2013-10-21 09:05:51 +00:00
csoler
81a4523434 - Enabled multi-tunneling by consistently perturbating partial tunnel id in TRs.
This causes tunnels with a different route to have different ids and therefore
  coexist as different sources for a given transfer.
  This should cause file transfers to have more tunnels and therefore a better balance
  of bandwidth over intermediate peers. This needs to be evaluated on a large scale network.

- Cleaned some unused code.



git-svn-id: http://svn.code.sf.net/p/retroshare/code/trunk@6740 b45a01b8-16f6-495d-af2f-9b41ad6348cc
2013-09-15 19:05:04 +00:00
csoler
4868a90d81 patch from Henry morgan to fix soem typos (e.g misspelled connexion into connection)
git-svn-id: http://svn.code.sf.net/p/retroshare/code/trunk@6460 b45a01b8-16f6-495d-af2f-9b41ad6348cc
2013-06-24 21:23:50 +00:00
csoler
5e319b1261 fixed autoWash() method in p3turtle.
git-svn-id: http://svn.code.sf.net/p/retroshare/code/trunk@6447 b45a01b8-16f6-495d-af2f-9b41ad6348cc
2013-06-20 19:12:33 +00:00
csoler
1c3ff8c54b - lots of debugging.
- added missing bits to turtle router for better handling symmetric traffic 


git-svn-id: http://svn.code.sf.net/p/retroshare/code/branches/v0.5-GenericTunneling@6321 b45a01b8-16f6-495d-af2f-9b41ad6348cc
2013-04-20 18:56:06 +00:00
csoler
f12ad9583d - added functions to add/remove virtual peers in turtle client service
- removed all explicit connections between turtle and FT
- added tunnel and invitation management in p3ChatServer (still needs PGP encryption)



git-svn-id: http://svn.code.sf.net/p/retroshare/code/branches/v0.5-GenericTunneling@6300 b45a01b8-16f6-495d-af2f-9b41ad6348cc
2013-04-10 08:52:52 +00:00
csoler
5e62f6c0c5 added serialisation/deserialisation and handling of generic tunnel data items
git-svn-id: http://svn.code.sf.net/p/retroshare/code/branches/v0.5-GenericTunneling@6295 b45a01b8-16f6-495d-af2f-9b41ad6348cc
2013-04-06 13:58:27 +00:00
csoler
0abb21d962 Added ability for turtle router to take client services
- new class RsTurtleClientService, from which should inherit the services
- turtle serialisation takes client services for deserialising
- added a client to p3turtle::monitorTunnels()
- added a generic turtle item to pass on arbitrary data

Moved ftServer as a subclass of RsTurtleClientService. 
- moved file transfer items that inherited from RsTurtleGenericTunnelItem into a separate file
- moved deserialisation triage into ftServer
- moved sending/receiving methods in ftServer (much less code, much more consistent!)

One bug remains: the TR cannot work as a server.

Changes should be 100% backward compatible.



git-svn-id: http://svn.code.sf.net/p/retroshare/code/branches/v0.5-GenericTunneling@6293 b45a01b8-16f6-495d-af2f-9b41ad6348cc
2013-04-06 09:21:01 +00:00
csoler
270abcdc94 - removed _sharing_strategy and the methods that go with it, as its behavior is handled by the sessionEnabled() method
- added generic tunnel data item, along with item priority consts. Serialization still missing.
- added new class for turtle client services, and entry point methods.
- removed file name and file size ffrom _outgoing_file_hashes and _incoming_file_hashes. They are not needed.
- changed some names to more generic ones.



git-svn-id: http://svn.code.sf.net/p/retroshare/code/branches/v0.5-GenericTunneling@6285 b45a01b8-16f6-495d-af2f-9b41ad6348cc
2013-04-01 21:18:58 +00:00
drbob
90415627a2 * Added second (temporary) switch to turtle.
* Use session switch for OperatingMode.
 * made OperatingMode temporary - not saved.



git-svn-id: http://svn.code.sf.net/p/retroshare/code/trunk@5895 b45a01b8-16f6-495d-af2f-9b41ad6348cc
2012-11-25 22:49:00 +00:00
csoler
fd723bcea7 added global switch to enable/disable anonymous routing. This will be used for connexion profiles with very limited bandwidth
git-svn-id: http://svn.code.sf.net/p/retroshare/code/trunk@5886 b45a01b8-16f6-495d-af2f-9b41ad6348cc
2012-11-25 14:26:32 +00:00
csoler
dc82cee700 First shot of file sharing permissions. Compiles, but needs some testing/debugging.
- added type-safe flags in retroshare/rsflags.h. This should be used to make new flags types in order to prevent mixing flags up in function prototypes.
- group handling is left to rsPeers. We'll move it to rsCircles later.



git-svn-id: http://svn.code.sf.net/p/retroshare/code/branches/v0.5-FileSharingPermissions@5754 b45a01b8-16f6-495d-af2f-9b41ad6348cc
2012-11-01 10:06:12 +00:00
csoler
32b893ef92 made one method virtual to help simulating
git-svn-id: http://svn.code.sf.net/p/retroshare/code/trunk@5628 b45a01b8-16f6-495d-af2f-9b41ad6348cc
2012-10-01 11:50:34 +00:00
csoler
1c3d6e7c16 Improvements to turtle router:
- made faster tunnels stay longer by increasing tunnel campain time. Should favor fast tunnels. 
- desactivated tunnel shortenning since it cannot produce consistent tunnels every time
- increase cache TTL for search requests to 240 secs and tunnel requests to 120 secs.
  Will remove some zombie tunnels and search requests.



git-svn-id: http://svn.code.sf.net/p/retroshare/code/trunk@5589 b45a01b8-16f6-495d-af2f-9b41ad6348cc
2012-09-23 19:58:08 +00:00
csoler
889a2b2433 Added deferred check of chunks during file transfer. Chunk sha1 sums are requested to the sources and checked for downloaded data.
Validated chunks are shared to other peers. Force check is now very simple since it just turns all chunks into "needs checking" mode
and sums are asked to sources. Sources maintain a temporary cache of chunks. Since sums are requested sparsely, this should not 
affect the sources in terms of performance. We can still imagine precomputing and saving sha1 of chunks while hashing them.

For backward compatibility reasons, the following has been setup *temporarily* in this version:
- unvalidated chunks are still considered as already obtained, and are shared and saved
- force check has been disabled
- final file check is maintained
- in case of file fail, the old checking mode will be used.

All changes for next version are kept in the define 'USE_NEW_CHUNK_CHECKING_CODE' that will be made the default in a few weeks.
At start, I expect most chunk to stya yellow during download, until most sources are able to provide chunk hashs.




git-svn-id: http://svn.code.sf.net/p/retroshare/code/trunk@5019 b45a01b8-16f6-495d-af2f-9b41ad6348cc
2012-03-15 19:55:43 +00:00
csoler
fd83e5950f added control over TR forward rate, so that we can experiment with it on large bandwidth servers
git-svn-id: http://svn.code.sf.net/p/retroshare/code/trunk@4918 b45a01b8-16f6-495d-af2f-9b41ad6348cc
2012-02-10 22:23:06 +00:00
thunder2
c6beee44ec Removed unused parameter warnings in libretroshare.
git-svn-id: http://svn.code.sf.net/p/retroshare/code/trunk@4558 b45a01b8-16f6-495d-af2f-9b41ad6348cc
2011-08-12 13:42:30 +00:00
drbob
f6f721a492 Removed p3ConnectMgr. Replaced it with p3LinkMgr, p3NetMgr and p3PeerMgr.
Updated references in other classes.
 * Most classes need p3LinkMgr, and only use two functions: OwnId() and getOnlineList()




git-svn-id: http://svn.code.sf.net/p/retroshare/code/branches/v0.5-netupgrade@4416 b45a01b8-16f6-495d-af2f-9b41ad6348cc
2011-07-09 18:39:34 +00:00
csoler
4d846513e2 added additional statistics from turtle router. Needs full recompilation.
git-svn-id: http://svn.code.sf.net/p/retroshare/code/trunk@4221 b45a01b8-16f6-495d-af2f-9b41ad6348cc
2011-05-26 22:11:06 +00:00
csoler
3f21835114 added a display of turtle search/tunnel requests over time, to help spotting potential spammers
git-svn-id: http://svn.code.sf.net/p/retroshare/code/trunk@4216 b45a01b8-16f6-495d-af2f-9b41ad6348cc
2011-05-23 21:45:25 +00:00
csoler
4ea647b78c implemented a verification for tunnel results. When the network gets heavily loaded, some tunnel results happen to get back twice. This stops the tunnel result sibblings
git-svn-id: http://svn.code.sf.net/p/retroshare/code/trunk@4152 b45a01b8-16f6-495d-af2f-9b41ad6348cc
2011-04-10 19:22:15 +00:00
csoler
8e6962808d - added random bias to search and tunnel request maximum depth
- added random (and biased) re-routing of tunnel request to improve tunnel length.


git-svn-id: http://svn.code.sf.net/p/retroshare/code/trunk@4130 b45a01b8-16f6-495d-af2f-9b41ad6348cc
2011-04-07 20:16:43 +00:00
csoler
9e2be4b5be implemented a better management of tunnel requests
git-svn-id: http://svn.code.sf.net/p/retroshare/code/trunk@4093 b45a01b8-16f6-495d-af2f-9b41ad6348cc
2011-03-15 23:09:31 +00:00