Git-Mirrors/brozzler
1
0
Fork 0
mirror of https://github.com/internetarchive/brozzler.git synced 2025-04-20 23:56:34 -04:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity

some thoughts on distributed database

Browse Source
This commit is contained in:
Noah Levitt 2015-08-11 18:06:58 +00:00
parent ce154fc3db
commit 3d70776ce3
1 changed files with 7 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

8
hq-notes.txt
View File

@ -50,4 +50,10 @@ brozzler-worker
- reads urls from brozzler.{site-id}.crawl_urls
- after each(?) (every n?) urls, feeds brozzler.{site_id}.completed_urls
=== considering distributed database ===
preferred database requirements:
- secondary index (so we can look up by url or priority)
- good performance on updates since we will be doing many updates
- good performance of secondary index on updates that change the value of secondarily indexed field
- ideally strong consistency, for multiple instances of brozzler-hq will
- redundancy, fault tolerance