Commit Graph

47 Commits

Author SHA1 Message Date
csoler
a1b36cd530 added method in ftServer to allow retrieving data from shared/downloaded files (patch from electron, modified)
git-svn-id: http://svn.code.sf.net/p/retroshare/code/trunk@7999 b45a01b8-16f6-495d-af2f-9b41ad6348cc
2015-03-08 13:46:07 +00:00
chrisparker126
0f29d28b1b merge of branch v0.6-idclean 7180
git-svn-id: http://svn.code.sf.net/p/retroshare/code/trunk@7187 b45a01b8-16f6-495d-af2f-9b41ad6348cc
2014-03-17 20:56:06 +00:00
csoler
162ef94b70 extended max chunk TTL to 1 hour. A short TTL is not anymore needed since chuns are shared between sources
git-svn-id: http://svn.code.sf.net/p/retroshare/code/trunk@6935 b45a01b8-16f6-495d-af2f-9b41ad6348cc
2013-12-09 13:35:06 +00:00
csoler
4e4e81fa8e fixed uninitialized memory read in debug statement
git-svn-id: http://svn.code.sf.net/p/retroshare/code/trunk@6203 b45a01b8-16f6-495d-af2f-9b41ad6348cc
2013-03-10 22:29:25 +00:00
csoler
4c3905e257 simplified the chunk drawing code a little bit
git-svn-id: http://svn.code.sf.net/p/retroshare/code/trunk@6177 b45a01b8-16f6-495d-af2f-9b41ad6348cc
2013-03-01 23:41:42 +00:00
csoler
1fb605cfa9 fixed stupid bug causing streaming mode to get stalled
git-svn-id: http://svn.code.sf.net/p/retroshare/code/trunk@6176 b45a01b8-16f6-495d-af2f-9b41ad6348cc
2013-03-01 23:16:09 +00:00
csoler
4f1dda2ffb added new DL mode: Progressive. This is a random mode that ensures no more than 50MB of free space will be written in the file at once. This should effectively remove the lag when initiating a DL on a big file, while keeping enough entropy to ensure a good dissemination of chunks
git-svn-id: http://svn.code.sf.net/p/retroshare/code/trunk@6164 b45a01b8-16f6-495d-af2f-9b41ad6348cc
2013-02-28 20:42:01 +00:00
csoler
b2214fcd46 removed #ifdef USE_NEW_CHUNK_CHECKING_CODE, since it is enabled by default now. Also removed the use of assume_availablility to put chunks or not in checking mode. This was inconsistent. As a side effect, cache files of size > 1MB get chunk-checked as well, which is good
git-svn-id: http://svn.code.sf.net/p/retroshare/code/trunk@6050 b45a01b8-16f6-495d-af2f-9b41ad6348cc
2012-12-27 21:16:33 +00:00
csoler
d3a3efd492 added additional checking of incoming compressed chunk map, in case the data is currupted, which might happens with a corrupted ft_transfer.cfg file.
git-svn-id: http://svn.code.sf.net/p/retroshare/code/trunk@5816 b45a01b8-16f6-495d-af2f-9b41ad6348cc
2012-11-13 20:13:09 +00:00
csoler
24f8ee6222 Added dynamic choose of sources for chunk crc requests. Fixes the bug that would let
a transfer not finish if the original surce for a crc is not here anymore.

If you have a unfinished transfer do a force-check after restart to get back chunks that 
where still on verificaiton stage.



git-svn-id: http://svn.code.sf.net/p/retroshare/code/trunk@5315 b45a01b8-16f6-495d-af2f-9b41ad6348cc
2012-07-19 20:52:04 +00:00
csoler
0d3d1ebc18 fixed bug preventing files < 1MB to terminate properly
git-svn-id: http://svn.code.sf.net/p/retroshare/code/trunk@5240 b45a01b8-16f6-495d-af2f-9b41ad6348cc
2012-06-21 19:24:14 +00:00
csoler
36198b7e6a Enabled full chunk checking code:
- unchecked chunks are not made available to swarming sources, not saved as done.
- force check now uses the simple method to put all chunks in checking mode
- force checked files can be cancelled (finally!)
- improved display (use red for active chunks, yellow for checking)
- cache file are not using chunk checking (assume_availability=true)
The code still contains the #ifdef. It should be removed soon if everything works ok.



git-svn-id: http://svn.code.sf.net/p/retroshare/code/trunk@5235 b45a01b8-16f6-495d-af2f-9b41ad6348cc
2012-06-19 21:36:10 +00:00
csoler
f5c345659a Improved multi-source file transfer in two ways:
- tunnels can borrow data chunks from only slower tunnels, to prevent blocking fast tunnels
- tunnels can handle up to 50 data chunks at once instead of 5
Multi-source transfers are generaly faster and more stable.


git-svn-id: http://svn.code.sf.net/p/retroshare/code/trunk@5156 b45a01b8-16f6-495d-af2f-9b41ad6348cc
2012-05-09 21:31:05 +00:00
csoler
08a52dc369 fixed bug causing some transfers to never finish
git-svn-id: http://svn.code.sf.net/p/retroshare/code/trunk@5063 b45a01b8-16f6-495d-af2f-9b41ad6348cc
2012-03-30 19:33:58 +00:00
csoler
8f69251334 small improvements to the chunk checking algorithm
git-svn-id: http://svn.code.sf.net/p/retroshare/code/trunk@5047 b45a01b8-16f6-495d-af2f-9b41ad6348cc
2012-03-24 14:45:33 +00:00
csoler
889a2b2433 Added deferred check of chunks during file transfer. Chunk sha1 sums are requested to the sources and checked for downloaded data.
Validated chunks are shared to other peers. Force check is now very simple since it just turns all chunks into "needs checking" mode
and sums are asked to sources. Sources maintain a temporary cache of chunks. Since sums are requested sparsely, this should not 
affect the sources in terms of performance. We can still imagine precomputing and saving sha1 of chunks while hashing them.

For backward compatibility reasons, the following has been setup *temporarily* in this version:
- unvalidated chunks are still considered as already obtained, and are shared and saved
- force check has been disabled
- final file check is maintained
- in case of file fail, the old checking mode will be used.

All changes for next version are kept in the define 'USE_NEW_CHUNK_CHECKING_CODE' that will be made the default in a few weeks.
At start, I expect most chunk to stya yellow during download, until most sources are able to provide chunk hashs.




git-svn-id: http://svn.code.sf.net/p/retroshare/code/trunk@5019 b45a01b8-16f6-495d-af2f-9b41ad6348cc
2012-03-15 19:55:43 +00:00
csoler
2edb61758b Improvements to file transfer:
- corrected 2 potential flaws of swarming causing request of unavailable data (This should normally be taken care of on the server side!)
- improved swarming test code with fault simulation
- only ask CRC32 maps to peers that have a complete file.
 


git-svn-id: http://svn.code.sf.net/p/retroshare/code/trunk@4947 b45a01b8-16f6-495d-af2f-9b41ad6348cc
2012-02-16 21:10:46 +00:00
csoler
f441f4a407 set max chunk inactivity period to 5 mins instead of 1 mins. Shoudl favor slow DLs. This is nw possible since missing bits will be re-asked regularly
git-svn-id: http://svn.code.sf.net/p/retroshare/code/trunk@4925 b45a01b8-16f6-495d-af2f-9b41ad6348cc
2012-02-11 15:26:04 +00:00
csoler
fa690de1d6 small bug fixes about CRC32 verification when the file is actually complete.
git-svn-id: http://svn.code.sf.net/p/retroshare/code/trunk@4662 b45a01b8-16f6-495d-af2f-9b41ad6348cc
2011-11-04 21:29:51 +00:00
csoler
1fc8c5ef70 implemented slice sharing between peers.
git-svn-id: http://svn.code.sf.net/p/retroshare/code/trunk@4650 b45a01b8-16f6-495d-af2f-9b41ad6348cc
2011-10-23 18:43:58 +00:00
csoler
f5bfc8c92a changed a bit the file transfer strategy. To be tested further.
git-svn-id: http://svn.code.sf.net/p/retroshare/code/trunk@3898 b45a01b8-16f6-495d-af2f-9b41ad6348cc
2010-12-07 17:09:21 +00:00
csoler
34a3fc4ea8 removed useless include
git-svn-id: http://svn.code.sf.net/p/retroshare/code/trunk@3854 b45a01b8-16f6-495d-af2f-9b41ad6348cc
2010-11-22 21:01:54 +00:00
csoler
b94e4376a4 only assume availability for cache transfers. All other transfers now use ChunkMaps
git-svn-id: http://svn.code.sf.net/p/retroshare/code/trunk@3853 b45a01b8-16f6-495d-af2f-9b41ad6348cc
2010-11-22 20:57:53 +00:00
csoler
c00e11ed2b switched random chunk strategy to true random
git-svn-id: http://svn.code.sf.net/p/retroshare/code/trunk@3777 b45a01b8-16f6-495d-af2f-9b41ad6348cc
2010-11-10 22:05:34 +00:00
csoler
f2406d4e57 improved random chunk strategy, to limit cluster effects on files partially downloaded in streaming
git-svn-id: http://svn.code.sf.net/p/retroshare/code/trunk@3764 b45a01b8-16f6-495d-af2f-9b41ad6348cc
2010-11-08 22:09:43 +00:00
csoler
8369d42600 corrected a bug in chunkmaps that affected file creators being providers at the same time, which could corrupt data. Added some debug info to chunk maps.
git-svn-id: http://svn.code.sf.net/p/retroshare/code/trunk@3673 b45a01b8-16f6-495d-af2f-9b41ad6348cc
2010-10-15 22:12:29 +00:00
csoler
d71dd2889d added safety check to ftChunkMap call to p3turtle, mainly for tests to run
git-svn-id: http://svn.code.sf.net/p/retroshare/code/trunk@3582 b45a01b8-16f6-495d-af2f-9b41ad6348cc
2010-10-01 20:07:29 +00:00
leander-256
4bb4fc11e2 This intimidating patch renames libretroshare/src/rsiface/ to
libretroshare/src/retroshare/

All the relevant headers have been modified to reflect that change.

This allows installation of libretroshare on a system, headers will
be put in $WHEREVER/retroshare/ and we keep the ability to compile
against them, be it on the system or in the SVN tree.



git-svn-id: http://svn.code.sf.net/p/retroshare/code/trunk@3342 b45a01b8-16f6-495d-af2f-9b41ad6348cc
2010-08-06 09:40:23 +00:00
csoler
8bfc74485a First implementation of per-chunk CRC32 check. This is triggered
- by the right-click+Force Check on files.
- when a global hash on a downloaded file does not match the announced hash.

When a CRC map check is ordered, the CRC map is requested to one of the sources for the current file download.
When received, all downloaded chunks are checked w.r.t the reference CRC and marked as not done if the CRCs do not match.
The exchange of CRC32 map and requests has been tested, as well as CRC map checking during download (force check).

To be implemented soon:
    - caching of CRC32 maps (although these are fast to compute)
    - CRC32 map packets for normal downloads. For now these work only for turtle tunnels.
    - handling of errors if the CRC never comes. For now, the download will stay stuck in "Checking..." mode.

So, don't play too much with the force check feature for now...





git-svn-id: http://svn.code.sf.net/p/retroshare/code/trunk@3310 b45a01b8-16f6-495d-af2f-9b41ad6348cc
2010-07-21 23:14:10 +00:00
csoler
c8fc75c75c added some safety checks to setting chunk strategy against inconsistent values.
git-svn-id: http://svn.code.sf.net/p/retroshare/code/trunk@3236 b45a01b8-16f6-495d-af2f-9b41ad6348cc
2010-06-29 20:51:16 +00:00
csoler
5f7486f160 added proper dynamic suppression of sources from chunkmap, and updated the gui
git-svn-id: http://svn.code.sf.net/p/retroshare/code/trunk@2498 b45a01b8-16f6-495d-af2f-9b41ad6348cc
2010-03-07 14:42:13 +00:00
csoler
e3f2f66702 suppressed bug in file transfer while removing inactive chunks.
git-svn-id: http://svn.code.sf.net/p/retroshare/code/trunk@2468 b45a01b8-16f6-495d-af2f-9b41ad6348cc
2010-03-01 20:46:29 +00:00
csoler
851e9069f6 corrected compled completion and file progress issues
git-svn-id: http://svn.code.sf.net/p/retroshare/code/trunk@2356 b45a01b8-16f6-495d-af2f-9b41ad6348cc
2010-02-17 22:10:12 +00:00
csoler
5926c79822 corrected bug in file completion
git-svn-id: http://svn.code.sf.net/p/retroshare/code/trunk@2301 b45a01b8-16f6-495d-af2f-9b41ad6348cc
2010-02-13 20:42:49 +00:00
csoler
e08d13397a fixed a completion issue in FT
git-svn-id: http://svn.code.sf.net/p/retroshare/code/trunk@2174 b45a01b8-16f6-495d-af2f-9b41ad6348cc
2010-02-02 22:15:42 +00:00
csoler
a70b106005 - added NETWORK_WIDE flag to remote dir model download
- added file priority strategy based on which files are requested first. This provides:
	- equal file speed for files with equal (source,priority)
	- effective priority speed for file of same source but different priority
- removed state variable load/save from turtle, as it's not needed anymore (FileRequest re-opens tunnels as needed)
- manage availability per peer instead of per file type: direct peer ids always assume file availability, while turtle tunnels don't

I still need to make the download queue work, and code this gui for it.



git-svn-id: http://svn.code.sf.net/p/retroshare/code/trunk@2133 b45a01b8-16f6-495d-af2f-9b41ad6348cc
2010-01-26 20:40:21 +00:00
csoler
0e7f9df638 made contextual menus in FT more contextual, added info about transfer type and chunking strategy, load/save of transfer flags
git-svn-id: http://svn.code.sf.net/p/retroshare/code/trunk@2114 b45a01b8-16f6-495d-af2f-9b41ad6348cc
2010-01-22 23:23:37 +00:00
csoler
e2b807aad7 added regular cleaning of inactive chunks and slices, so that they can be treated by a different peer
git-svn-id: http://svn.code.sf.net/p/retroshare/code/trunk@2102 b45a01b8-16f6-495d-af2f-9b41ad6348cc
2010-01-21 12:31:00 +00:00
joss17
f481dbef59 add include for latest gcc. Comment usleep fonction in tou_net.h
git-svn-id: http://svn.code.sf.net/p/retroshare/code/trunk@2101 b45a01b8-16f6-495d-af2f-9b41ad6348cc
2010-01-20 22:02:43 +00:00
csoler
96147a9734 added flags to file request: RS_FILE_HINTS_ASSUME_AVAILABILITY to not wait/ask for chunk maps. Used for cache transfer and also whenever the turtle traffic is not involved, such as chat file exchange, and RS_FILE_HINTS_NETWORK_WIDE to force using turtle routing for traffic. Also moved turtle tunnel start/stop orders from ftServer to ftController
git-svn-id: http://svn.code.sf.net/p/retroshare/code/trunk@1995 b45a01b8-16f6-495d-af2f-9b41ad6348cc
2010-01-11 22:38:18 +00:00
csoler
f02b009d7b fixed win32 compilation
git-svn-id: http://svn.code.sf.net/p/retroshare/code/trunk@1994 b45a01b8-16f6-495d-af2f-9b41ad6348cc
2010-01-11 20:10:04 +00:00
csoler
cfaaec31c7 - Implemented chunk-based file transfer from partial sources. This in particular means:
- exchange of chunk availability maps from different peers
    - correct handling of what is available to which source before asking the data
    - correct display of chunks in the progress bars
    - generalised the use of compressed chunk maps
    - removed the size parameters from the hash search functions
   
- In addition:
    - suppressed a number of per-value transfers of std::string
    - improved the FileTransferInfo Widget, to show some additional info

Still to be done:
    - chunk map exchange for non anonymous traffic (easy)
    - improve accuracy of completion for uploads (for now it's a integer number of chunks)
    - check compilation on windows




git-svn-id: http://svn.code.sf.net/p/retroshare/code/trunk@1993 b45a01b8-16f6-495d-af2f-9b41ad6348cc
2010-01-11 16:00:42 +00:00
csoler
f4a2eaecce - implemented a visualisation for currently handled chunks, availability maps, and transfer info
- implemented transfer protocol for chunk availability maps between peers (not enabled yet though)
- suppressed rsiface directory from retroshare-gui
- moved notifyqt.{h,cpp} to gui/

next moves: 
	- send availability maps to clients; 
	- connect turtle search to unfinished files;
	- test multisource download with unfinished files.



git-svn-id: http://svn.code.sf.net/p/retroshare/code/trunk@1939 b45a01b8-16f6-495d-af2f-9b41ad6348cc
2009-12-28 21:11:00 +00:00
csoler
b6c048a5fd Implemented load/save of chunk maps for current downloads.
The side effect is that even when stopped, downloads show the correct downloaded size (up to the size of one chunk).



git-svn-id: http://svn.code.sf.net/p/retroshare/code/trunk@1865 b45a01b8-16f6-495d-af2f-9b41ad6348cc
2009-12-10 22:55:27 +00:00
csoler
4d190963cf windows compilation fix
git-svn-id: http://svn.code.sf.net/p/retroshare/code/trunk@1864 b45a01b8-16f6-495d-af2f-9b41ad6348cc
2009-12-09 12:42:31 +00:00
csoler
25a09900e9 - Added a ChunkMap class responsible for allocating new chunks to be downloaded, according to
- a given chunk strategy
	- the availablility map of each source 
- Integrated this into ftFileCreator
- added gui menu in file transfer+right click to change the chunk strategy: streaming vs. random

Next step: 
	- loading/saving file downloading state and availability map
	- displaying chunk details in the selected transfer tab (e.g. list of currently worked chunks, and their current downloading completion)



git-svn-id: http://svn.code.sf.net/p/retroshare/code/trunk@1863 b45a01b8-16f6-495d-af2f-9b41ad6348cc
2009-12-08 22:29:52 +00:00
csoler
426fc06938 forgot to add these two files
git-svn-id: http://svn.code.sf.net/p/retroshare/code/trunk@1839 b45a01b8-16f6-495d-af2f-9b41ad6348cc
2009-11-19 08:35:11 +00:00