summaryrefslogtreecommitdiff
AgeCommit message (Collapse)AuthorFilesLines
2013-07-08share: Insert hashed files into database + various bug fixesdevYorhel5-9/+13
2013-07-07share/hash: Use separate thread pool + kill idle threads after timeoutYorhel3-4/+33
2013-07-07share/hash: Add 'Hasher" object to the API treeYorhel4-2/+15
Not that it does anything yet.
2013-07-07share/hash: Process new queue items as soon as job slots are availableYorhel1-1/+8
2013-07-07share/hash: Implemented the actual hashing and TTH merging functionsYorhel2-3/+135
I've only done a few basic tests, but no doubt there'll be some bugs left. Changing HASH_READ_BUF_SIZE will no doubt find one. I've not actually measured the performance yet. The code *should* be very fast, but I can't make any performance claims at this point yet.
2013-07-01share/hash: Queue (still empty) hash jobs + remove 'cancelled' flagYorhel1-9/+44
I had already planned to use fn->fl == NULL to indicate cancelled state, which kinda make sense because in most cases that pointer would be dangling after cancellation. The only situation where this isn't the case is when hashing is "cancelled" after a hash job failed with an error, but it's handled pretty much the same.
2013-06-30share/hash: Add basic framework for cancelling a file hashYorhel1-9/+27
2013-06-30share/hash: Asynchronously open() files to be hashedYorhel1-7/+39
2013-06-30share/hash: Perform database check before hashingYorhel2-7/+42
2013-06-30share/hash: Add share-local (de)initializationYorhel3-2/+30
2013-06-30Fix incorrect use of important function calls within assert()Yorhel7-12/+10
I feel like a noob for making such an elementary mistake...
2013-06-30share/db: Add share_db_{exists,addfile} functionsYorhel4-1/+176
These will be needed by share/hash.c. Totally untested code for now, as usual.
2013-06-29deps,main: Update to latest evtpYorhel2-2/+1
2013-06-28share/hash: Implemented the hash index + update share sizeYorhel1-4/+23
2013-06-28util/list: Re-add hlist_* macrosYorhel2-6/+31
This is a partial revert of 38d80827ba30059a9a513c45793f853ed59f067f. I also added the respective hp* macros, although I won't be using these for now. I realized why I had implemented the single-pointer list anchor case; It will halve the memory usage of the hash index. With the hlist_ macros, that index is simply a hash set of share_fl_t pointers. Without, it'll be a hash set of struct { share_fl_t *head, *tail; }, i.e. twice as large. The tail pointer really isn't necessary.
2013-06-28share,api: Started on hash queue + index + multithreaded hasherYorhel9-16/+327
Very much unfinished, early stages, etc. Doesn't do anything useful yet, other than queueing up files to hash and calculating some fun scheduling information.
2013-06-27share: Add share_share_fl_path() functionYorhel2-0/+28
Will be needed for the hashing code to figure out what file to hash. (And later on likely for other purposes, too)
2013-06-27share: Rename share_share_t -> share_t + embed config into share_t objYorhel8-89/+88
The new share_t structure has the same or a slightly longer lifetime than the share_conf_t structure, so embedding makes sense.
2013-06-27util/list: Added prefix macrosYorhel1-14/+26
I'll be needing them now
2013-06-22util/list: Remove hlist_* + generalize list_insert() to support appendYorhel3-29/+14
I wanted the linked list abstraction to be convenient in use, yet result in exactly the same kind of (generated) code as if each list was hand-written. That's the reason for the hlist_*, but I'm starting to doubt whether that optimization is really worth the effort of maintaining two separate APIs. I'm keeping the idea for an slist_ abstraction open, however; Saving memory on pointers in list nodes likely *is* worth the effort. I also figured that, with a tiny change, list_insert_before() could be used as a list_insert_after(), too, so I fixed that and renamed it to a single powerful list_insert().
2013-06-16share/fl: Use separate 'is directory' flag instead of special size valueYorhel6-27/+50
This allows directories themselves to have a size, too. I initially preferred using bit fields, but bit fields are not specified to work on 64 bit integers. The current solution is more portable. Note that the size field isn't updated for directories yet. I'll get to that once files are actually being hashed. Unhashed files should be treated as if they don't exist in the list yet, atfer all.
2013-06-16share,api: Add + implement basic browse interfaceYorhel4-3/+100
2013-06-15Add share_share object to manage actions to the file listYorhel11-31/+178
This is an extra level of indirection and complicates the structure of the code a bit. However, this does allow us to schedule multiple simultaneous actions on the file list without running into data races or memory management issues.
2013-06-09doc/api: Some documentation on the sharep interfaceYorhel1-2/+49
2013-06-09share/scan: Honor ExcludeHidden settingYorhel1-2/+9
2013-06-09share/scan: Honor FollowSymlinks settingYorhel1-2/+3
2013-06-09share/scan: Use casestr_cmp() for removing duplicate filesYorhel1-5/+3
2013-06-09util/casestr: casestr_eq() -> casestr_cmp() + define some orderYorhel3-17/+35
This makes it suitable for more use cases.
2013-06-09share/scan: Log scanning timeYorhel1-0/+15
A quick benchmark indicates that Globster scans a little more than twice as slow as 'ncdu -q0o- >/dev/null'. I haven't looked at where the overhead goes to exactly (I do have several ideas), but I suppose it's fast enough for now. Scanning 30k files/dirs in 0.2s isn't so bad, after all; More than fast enough to have disk I/O as the bottleneck in the majority of cases.
2013-06-09share/conf: Don't allow sharing the filesystem rootYorhel1-0/+10
2013-06-09share/scan: Only report dir-open errors if all dirs failed + bug fixYorhel1-9/+18
Another bug related to the special handling of the root path. Sigh.
2013-06-09share: Fix bug with scanning nested vpaths + remove some debug codeYorhel2-5/+2
The special handling required for the root directory '/' compared to any subdirectory '/blah' is a minefield...
2013-06-08share: Add recursive directory scanning + various bug fixesYorhel7-34/+145
The scanning code is now mostly functional. There's still some details missing regarding progress/error reporting, handling of filename encodings and directories that don't exist and some configuration-dependent features such as symlink following and excludes patterns.
2013-06-08util/path: Add path_isancestor() functionYorhel2-1/+27
Despite its simplicity, I keep finding it hard to get such string comparison functions right. This function makes it significantly harder to make any mistakes in path matching, and also keeps the intent of the code clear.
2013-06-07share/scan: Ensure that virtual mount paths exist as dir entriesYorhel1-17/+48
Note that due to the way this is implemented, virtual mount paths will *always* exist, even if there exists a file with the same name in a higher-precedence path. I'm not really sure what behaviour is the most intuitive in that situation. In terms of "correctness", one would expect a lower-precedence path to be ignored if its mount path clashes with a higher-precedence file. On the other hand, what point is there in configuring a path if it's going to be ignored? (Again, code in this commit is totally untested. What the hell am I doing, really?)
2013-06-07share/scan: Add directory merging codeYorhel1-7/+66
Totally untested, as usual.
2013-06-07util/casestr: Add casestr_eq() for "fast" case-sensitive comparisonYorhel3-9/+39
2013-06-05Update deps + make use of new yuri libraryYorhel2-2/+6
2013-06-04Increase default log level to 3Yorhel3-5/+5
2013-06-04Update deps for new ylog + Make use of ytrace()Yorhel4-7/+8
The 'trace' log level is for very verbose debugging information that you really don't want to have enabled for anything except the specific sub-component that you happen to be debugging.
2013-06-04db: Unlink the lock file on shutdownYorhel1-0/+5
The file isn't necessary anymore and will be recreated when a new globster instance is started.
2013-06-04db: Accept a formatted argument in db_get()Yorhel2-7/+13
2013-06-04util/{adc,logfile}: Use kvsprintf() now that it existsYorhel4-13/+24
2013-06-04util/adc: Use utf8proc to validate UTF-8Yorhel2-27/+5
Now that a UTF-8 processing library is being used, let's make actually make use of it.
2013-06-03share/scan: Fix memory leak and an if condition bugYorhel1-1/+2
2013-06-03share/scan: Add function to scan multiple filesystem pathsYorhel1-2/+84
Still untested, and the code is still never called.
2013-06-02share: Made a start on the file system scanning codeYorhel4-2/+248
Totally untested code. It is never called, either.
2013-06-02casestr: Add casestr_len() function + remove printf() from test/pathYorhel4-3/+27
2013-05-13db: Use a separate SQLite database file for the hash dataYorhel2-20/+45
Startup time more than doubles when the tth leaves are stored in the sharefiles table, as was my previous idea. I don't expect storing the leaves in a separate table in the same file is going to help that much, since the data is not going to be stored in a sequential chunk in the file. Luckily, SQLite's ATTACH functionality makes this splitting quite easy. The downside of splitting the hash data into a separate table is that garbage collection (that is, removing unused hash data) will require some extra memory to keep a hash table of all used TTH hashes in memory. For those people who have been experimenting with this branch; Please either delete your globster.sqlite3 to have the tth_leaves column removed, the db.c update code won't do it. (Of course, stable globster releases will have a better updating process)
2013-05-13share/db: Use sqlasync_custom() to load the file list in the DB threadYorhel1-38/+54
Improves performance a bit on single-core systems. Not as significant as I had hoped.
generated by cgit v1.2.3 (git 2.25.1) at 2024-04-19 07:54:27 +0000