path: root/lib
AgeCommit message (Collapse)AuthorFilesLines
2015-10-25DB::Discussions: Minor improvement to post <-> board filter query2.25Yorhel1-1/+1
I don't know why I didn't apply this one before, I did make this change when benchmarking the fulltext search queries and with the introduction of the bb_tsvector() function this change pretty much always improves performance.
2015-10-25Staff: Add error msg when removing used alias + fix bug in alias editingYorhel2-3/+9
The new database schema doesn't allow an alias to be removed when it is still linked to a VN.
2015-10-24DB::Staff: Fix 'aid' filter, used when editing VN staff infoYorhel1-1/+1
This broken filter would cause all staff info to be deleted from a VN upon edit. Not so nice.
2015-10-24SQL: Throwing around some indicesYorhel1-2/+2
These indices provide a significant speed-up of /v+ and /u+ pages, and improve some other stuff as well.
2015-10-24Improve several discussion board SQL queriesYorhel1-8/+14
An index on was necessary to speed up some very common "recent posts" queries on both the homepage and the thread index. Postgres thought that the same index could be used to speed up the full-text search (because it's ordered by date, after all), but that completely killed performance. That was solved with a bb_tsvector() wrapper to tell the query planner that not using the full-text index is incredibly show, which in turn improved the search performance beyond what it was. Many thread-related queries are still somewhat slow, but that seems to be a limitation in the schema. I'll just keep monitoring to see if that's worth fixing in the future. Interestingly, dbThreadCount() needs to use a sequential scan, but it's still remarkably fast.
2015-10-21Improve SQL in dbRevisionGet()Yorhel3-49/+35
Two main improvements: - Filtering on (non)hidden items now doesn't join any of the item tables, instead it looks up the latest revision from the changes table itself, using the index on (type,itemid,rev). It's still not super fast, but a pretty large improvement nonetheless. - The item titles/names are obtained in a separate query. I tried to modify the main query in various ways, but couldn't make it as fast as I'd have liked. I also removed the 'what' flag while I was at it, all uses of the method request all information anyway.
2015-10-21SQL: Fix Multi to use the new DB schemaYorhel5-163/+145
That should be the last thing to convert to the new schema.
2015-10-21SQL: Fix editing + func.sql + triggers.sql + autocreate editing funcsYorhel10-52/+51
This changes quite a bit to the way the editing functions work. Because these functions are very repetitive and it's easy to keep things out of sync, I created a script to generate them automatically. I had to rename a few function and table names for consistency to make this work. Since database entries don't have a 'latest' column anymore, and since the order in which tables are updated doesn't have to be fixed, I dropped many of the SQL triggers and replaced them with a edit_committed() function which is called from edit_*_commit() and checks for stuff to be done. Don't forget to run 'make' before importing the update script.
2015-10-17SQL: Fix all browsing queries to use the new schemaYorhel19-436/+540
This basically makes VNDB browsable again, but editing entries is still broken. I split off the get-old-revision functionality from the db*Get() methods into db*GetRev(). This split makes sense even with the old SQL schema: db*Get() had to special-case some joins/filters when fetching an older revision, and none of the other filters would work in that case. This split does cause some code duplication in that all db*GetRev() methods look very much alike, and that the columns they fetch is almost identical to the db*Get() methods. Not sure yet how to avoid the duplication elegantly. I didn't do a whole lot of query optimization yet (most issues require extra indices, I'll investigate later which indices will make a big difference), but I did fix some low hanging fruit whenever I encountered something. I don't think I've worsened anything, performance-wise.
2015-10-12Notifications: Allow max 500 notifies per user + add SQL index on uidYorhel1-0/+2
Turns out that fetching whether or not you have unread notifications (done on every pageview if you're logged in) was pretty slow. The index speeds up both that query and the "my notifications" view. The extra purge for old notifications for users with more than 500 notifications ensures that the index stays effective for the unread notifications count. Otherwise it'll have to read half of the notifications table anyway to check the 'unread' filter.
2015-10-11VNDBUtil: Add some tildes to VN search normalizationYorhel1-1/+1
2015-10-03formValidate: Add json_(maxitems|unique|sort) options to json templateYorhel2-38/+13
Adds slightly more strict validation and simplifies further processing.
2015-10-03js: Minor fixesYorhel1-1/+1
2015-10-03Handle JSON data natively when processing form dataYorhel5-37/+45
No more need for extra json_encode/json_decode calls, and the form_compare() function is more lenient w.r.t. integer/string comparison. This is the improvement I described in commit ed86cfd12b0bed7352e2be525b8e63cb4d6d5448
2015-10-03Rewrote screenshot uploader to support multiple files + use jsonYorhel2-28/+24
This might have broken the screenshot uploader on some crappy browsers, but it's much cleaner than the old iframe hack. The ability to upload multiple files in one go is also very convenient.
2015-10-01VNEdit: Give error on duplicate alias + improved msg on id/page errorYorhel2-9/+10
2015-10-01VN search: Add some more quote characters + & to normalizationYorhel1-1/+2
As suggested by
2015-09-20formValidate(): Let's just allow a '0' id - fix more errorsYorhel2-5/+5
Looks like 0 is actually used often to indicate some special value. Affects basically all 'check all' boxes (had to modify some of those boxes because some used -1, but that wasn't a problem).
2015-09-20Handler::Staff: Fix validation error on primary alias IDYorhel1-1/+1
It can be 0 when creating a new alias as primary name.
2015-09-20Fix handling of empty seiyuu/credits fieldsYorhel1-2/+2
2015-09-20formValidate: Add json template and remove json_validate() functionYorhel3-51/+23
This is less convenient than I had expected, because all the form handling code is designed to work with plain strings rather than any scalar. This means the json data has to be encoded again to get into $frm (not doing this means that, if the form didn't validate, the field won't be filled out correctly). And then decoded for validation, and then encoded again for comparison. I suspect the better solution is to fix the form handling code to handle arbitrary data structures: comparison can be done by deep comparison rather than a simple string compare, and the form generator can auto-encode-to-json if it sees a complex object. Another advantage of this solution is that the comparison function can be less strict with respect to number formatting. In the current scheme you have to be very careful that numbers are not automatically coerced into string format, otherwise the comparison will fail. Either way, that's an idea for the future...
2015-09-20formValidate: Created templates for gtin and editsum fieldsYorhel6-14/+13
2015-09-20Use plural version of reqPostsYorhel1-1/+1
Surprisingly, this was the only change I needed to make to comply with the TUWF reqGet-style function split.
2015-09-20Use the new reqQuery() + update usage of reqPath() for TUWF 1.0Yorhel2-3/+2
2015-09-20Update usage kv_validate() to upcoming TUWF 1.0Yorhel13-81/+73
And added new 'page' and 'id' templates for more strict validation.
2015-09-17Handler::Discussions: Add search box to non-item thread listsYorhel1-22/+31
As requested at
2015-09-17Handler::VNEdit: Use json_validate() for the VN staff/cast dataYorhel1-36/+38
This (only very slightly) simplified further processing of the data. It does add more validation than was present: Previously it was possible to send invalid roles (would give a 500) or invalid numeric IDs (would throw some perl warnings). These issues are now handled earlier on. This change also puts a maximum length on the notes field, but nobody has passed the 300 characters so far.
2015-09-17Rename jsonEncode/Decode to C-style namingYorhel3-13/+13
CamelCase is used for methods, C-style names for functions. I'm just nitpicking now.
2015-09-17Created json_validate() for JSON form data, used for Staff alias editorYorhel3-25/+44
The intention is to move more JS editing forms to use JSON, but manually verifying JSON objects is both painful and likely to introduce errors or vulnerabilities. json_validate() is a bit of a hack, but has the advantage that its validation syntax is the same as for normal forms, and it automatically strips whitespace. I intent to give kv_validate() an upgrade to be more flexible/modular so it can do more custom normalization. But that's for later. I've been meaning to rewrite the JS forms anyway together with the large JS rewrite, but I'm rather lazy. This is one small step in the right direction anyway. Note that json_validate() assumes that the JS code will provide user-friendly messages on bad input, but the staff alias editor doesn't quite do this yet.
2015-09-08Multi::API: Set keepalive on sockets to detect dead TCP connectionsYorhel1-0/+1
Hopefully prevents issues like
2015-09-07Handler::Discussions: Fix two bugs introduced in last commitYorhel2-2/+2
2015-09-07Handler::Discussions: Use ts_headline() to format search resultsYorhel2-5/+33
And also fix strip_bb_tags() to be case-insensitive and fix a bug in converting the query into a tsquery.
2015-09-07Handler::Discussions: Remove formcode from search formYorhel2-4/+7
It's not verified and only uglifies the URLs.
2015-09-07Implement discussion board search functionYorhel3-13/+130
Inspired by wakaranai's implementation at This version is different in a number of aspects: - Separate search functions for title search and fulltext post search. Perhaps not the most convenient option, but the downside of a combined search is that if the query matches the threads' title, then all of the posts in that thread will show up in the results. This didn't seem very useful. - Sorting is based purely on post date. Rank-based sort is slow without a separate caching column, and in my opinion not all that useful. Implementation differences: - Integrated in the existing DB::Discussions functions, so less code to maintain and more code reuse. - No separate caching column for the tsvector, a functional index is used instead. This is a bit slower (index results need to be re-checked against the actual messages, hence the slowdown), but has the advantage of smaller database dumps and less complexity in updating the cache. Things to fix or look at: - Highlighting of the search query in message contents. - Allow or-style query matching
2015-09-06SQL: Convert threads_board.type to ENUMYorhel1-2/+2
The char(2) solution is both inefficient and ugly. Also needed to be careful with handling the extra space that Postgres would automatically add to single-character types.
2015-08-31Handler::VNPage: Increase limit on number of visible releasesYorhel1-2/+2
From 50 to 200.
2015-08-17js: Clean up misc.jsYorhel2-6/+8
- Individual blocks don't leak variables into the parent scope anymore. Previously some blocks would re-use variables from other blocks, creating (almost invisible) dependencies between te blocks. - More consistent code for ulist-change-dropdowns, and all of them have a ref= argument now. - Use 'hidden' class instead of style.display wherever that makes sense. - Remove dead 'advselect' code. Hasn't been used since the addition of search filters. - lang_select doesn't rely on the position of the language class in className anymore (seriously that stuff is fragile...)
2015-08-17js: Add L10N strings to all relevant varsYorhel1-6/+1
This simplifies the JS code in some places and removes a whole number of L10N strings from the "l10n_str" var, thus shrinking the JS size a bit (uncompressed about 1500 bytes, in fact. 500 bytes after gzip).
2015-08-10Split script.js into multiple smaller filesYorhel1-0/+2
First part of a Javascript cleanup.
2015-08-08Char page: Dynamically show/hide trait groups depending on visible traitsYorhel1-7/+1
2015-08-08Hide sexual traits by default + Add profile option to change defaultYorhel2-5/+7
2015-08-08Add sexual trait toggle to character informationYorhel3-29/+24
It's not a preference yet and the sexual traits are still visible by default. I'll fix that later.
2015-07-27Handler::VNPage: Bunch of fixes for release page comparison rewriteYorhel1-5/+4
- Remove the obsolete 'use's - Re-add max-width style when requested - Fix comparison in displaying media - Fix sorting on animation column
2015-07-26Handler::VNPage: Rewrite release comparison pageYorhel1-408/+227
The previous code was using experimental perl features (switch / smartmatch) that weren't really needed, and the information about individual columns was spread around in multiple functions. This rewrite makes the code consistent with the rest of VNDB, and has *all* of the column-specific information in one data structure. I did not replicate the similar-cell-merging feature, partly because the code for it is definitely not trivial and partly because it doesn't make the table look any less cluttered. In fact, I feel that it only makes the table harder to interpret because it looks messy. This is a matter opinion, of course, so I might reimplement the feature if people who actually use this comparison page want to. Overall, I'm still undecided on whether this comparison page should exist at all in its current form - it's not very user-friendly and often looks cluttered. I'm keeping it because it does have some use-cases where it avoids opening every release page to do a manual comparison, but I'd love a more friendly-looking alternative.
2015-07-25Handler::Misc: Fix minor calculation error in determining "admin" userYorhel1-1/+1
2015-07-22Util::BrowseHTML: Fix XSS in "order by" link URLsYorhel1-2/+2
Reported by dim0k at
2015-07-21Add profile option for the default spoiler settingYorhel6-31/+30
This fixes the unexpected behaviour that changing the spoiler setting on one page will change it for all pages. All manual spoiler changing options are temporary now.
2015-07-21L10N: Combine spoiler setting stringsYorhel4-19/+18
2015-07-21Add profile option for default visible tag categories on VN pagesYorhel2-3/+9
The name of the profile setting isn't very clear. Not sure what to do with it.
2015-07-21Add profile option to show or summarize tags on VN pages by defaultYorhel2-4/+6