summaryrefslogtreecommitdiff
path: root/util
AgeCommit message (Collapse)AuthorFilesLines
2015-10-01VN search: Add some more quote characters + & to normalizationYorhel1-0/+4
As suggested by https://vndb.org/t2520.168
2015-09-20formValidate(): Let's just allow a '0' id - fix more errorsYorhel1-1/+1
Looks like 0 is actually used often to indicate some special value. Affects basically all 'check all' boxes (had to modify some of those boxes because some used -1, but that wasn't a problem).
2015-09-20Fix handling of empty seiyuu/credits fieldsYorhel1-1/+1
2015-09-20formValidate: Add json template and remove json_validate() functionYorhel1-1/+24
This is less convenient than I had expected, because all the form handling code is designed to work with plain strings rather than any scalar. This means the json data has to be encoded again to get into $frm (not doing this means that, if the form didn't validate, the field won't be filled out correctly). And then decoded for validation, and then encoded again for comparison. I suspect the better solution is to fix the form handling code to handle arbitrary data structures: comparison can be done by deep comparison rather than a simple string compare, and the form generator can auto-encode-to-json if it sees a complex object. Another advantage of this solution is that the comparison function can be less strict with respect to number formatting. In the current scheme you have to be very careful that numbers are not automatically coerced into string format, otherwise the comparison will fail. Either way, that's an idea for the future...
2015-09-20formValidate: Created templates for gtin and editsum fieldsYorhel1-0/+3
2015-09-20Update usage kv_validate() to upcoming TUWF 1.0Yorhel1-6/+3
And added new 'page' and 'id' templates for more strict validation.
2015-09-07Handler::Discussions: Use ts_headline() to format search resultsYorhel2-2/+14
And also fix strip_bb_tags() to be case-insensitive and fix a bug in converting the query into a tsquery.
2015-09-07Implement discussion board search functionYorhel3-0/+18
Inspired by wakaranai's implementation at https://github.com/morkt/vndb/commit/b852c87ad145fdaaa09c79b6378dd819b46f7e87 This version is different in a number of aspects: - Separate search functions for title search and fulltext post search. Perhaps not the most convenient option, but the downside of a combined search is that if the query matches the threads' title, then all of the posts in that thread will show up in the results. This didn't seem very useful. - Sorting is based purely on post date. Rank-based sort is slow without a separate caching column, and in my opinion not all that useful. Implementation differences: - Integrated in the existing DB::Discussions functions, so less code to maintain and more code reuse. - No separate caching column for the tsvector, a functional index is used instead. This is a bit slower (index results need to be re-checked against the actual messages, hence the slowdown), but has the advantage of smaller database dumps and less complexity in updating the cache. Things to fix or look at: - Highlighting of the search query in message contents. - Allow or-style query matching
2015-09-06SQL: Convert threads_board.type to ENUMYorhel3-1/+7
The char(2) solution is both inefficient and ugly. Also needed to be careful with handling the extra space that Postgres would automatically add to single-character types.
2015-08-17spritegen.pl: Add pngcrush/slow options + force png32 + atomic replaceYorhel1-6/+13
A recent version of imagemagick creates 16 bit depth PNG images by default for some reason. This results in an unnecessarily large file size increase and pngcrush doesn't do much to counter it (and its -bit_depth option has been deprecated, too). The atomic replace is quite handy to avoid people seeing any wierd intermediate images while the slow+pngcrush options are being used.
2015-08-17jsgen: Support external command for JS compression (like uglifyjs)Yorhel1-7/+30
Tends to compress a bit better than JavaScript::Minifier::JS. But is also a lot slower, so not really useful when devving. Stats for en.js: raw gzip uglifyjs 68199 19446 JS::Minifier::XS 79862 21624 Uncompressed 107662 28663 On an unrelated note, I like how jQuery boasts about being "Only 32kB minified and gzipped.". That's quite a bit more than all of VNDB's Javascript combined. For a damn library.
2015-08-17js: Add L10N strings to all relevant varsYorhel1-8/+8
This simplifies the JS code in some places and removes a whole number of L10N strings from the "l10n_str" var, thus shrinking the JS size a bit (uncompressed about 1500 bytes, in fact. 500 bytes after gzip).
2015-08-15js: Let jsgen.pl preprocess L10N strings + add L10N strings to some varsYorhel1-18/+48
This simplifies the JS version of mt() a bit and makes the whole internationalization framework a bit more robust. I also changed the VARS.{rlist_status,age_ratings,languages,platforms,char_roles} arrays to include the L10N string. This simplifies the JS code and reduces the JS size. There's a few more of such lists that can be transformed in the same way, I'll get to that later.
2015-08-15js: Wrap included files in anonymous functionYorhel1-1/+1
This removes the need to indent all files and add the anonymous function manually, and it also provides clean and consistent semantics. I already rewrote the library-like files earlier on to add their public interfaces to the window object, so everything should keep working after this change. It's still possible that some files use use a function from another non-library file. Those will break, but I'm sure such cases will be found soon enough, if they exist.
2015-08-10js: Move generated variables into global VARS structureYorhel1-46/+44
This removes one source of namespace polution, and makes it more clear which code is using the variables.
2015-08-10Split script.js into multiple smaller filesYorhel1-3/+13
First part of a Javascript cleanup.
2015-08-08Hide sexual traits by default + Add profile option to change defaultYorhel2-1/+2
2015-07-21Add profile option for the default spoiler settingYorhel2-1/+2
This fixes the unexpected behaviour that changing the spoiler setting on one page will change it for all pages. All manual spoiler changing options are temporary now.
2015-07-21Add profile option for default visible tag categories on VN pagesYorhel2-1/+2
The name of the profile setting isn't very clear. Not sure what to do with it.
2015-07-21Add profile option to show or summarize tags on VN pages by defaultYorhel2-1/+6
2015-07-19Remove the MSIE browser check & warning pageYorhel1-3/+0
It's a relic of the past. IE 6 & 7 are very rarely used nowadays, and people still using it will quickly realize why things don't quite work - they'll be used to it.
2015-05-13Add staff statistic to main menuYorhel2-0/+10
2015-05-13I18N: Add cleanup routine to lang.pl + remove English text from tlsYorhel1-0/+44
2015-05-13Change order of staff rolesYorhel2-1/+10
https://vndb.org/t6138.226 - https://vndb.org/t6048.132
2015-05-11New language: Catalan (/Valencian)Yorhel2-1/+3
2015-05-11Three new platforms: FM Towns, PC Engine, X68000Yorhel2-1/+7
2015-05-11Generate icons.png and associated CSS automaticallyYorhel2-0/+151
It became a bit of a hassle to keep updating that file manually in Gimp. This script performs surprisingly well for our set of icons.
2015-05-03Remove "script" role from vn<->staff linksYorhel1-0/+12
Note that it's still in the postgresql ENUM type. Removing that is possible, but not very trivial.
2015-01-29staff: Added 'scenario' credit typeYorhel2-1/+3
2015-01-28sql: Merge staff changes into /util/sql/ and rename staff.sql2.24Yorhel3-2/+64
So that the /util/sql/ files are in sync with the actual DB again.
2015-01-28Merge branch 'staff' into masterYorhel1-1/+1
Conflicts: lib/VNDB/DB/VN.pm lib/VNDB/Handler/VNPage.pm
2015-01-17staff: Fix search to ignore spaces in Japanese namesmorkt1-0/+1
2015-01-12staff: Add notifications for staff edits/deletesYorhel2-2/+12
2015-01-12staff: Remove some remnants of the staff imagesYorhel2-4/+3
2015-01-04staff: Add some link fields and guidelinesmorkt2-4/+7
2015-01-02staff: Fix deleting of staff + use JSON to pass data + minor fixesmorkt1-0/+2
2014-12-28More progress on the staff + cast DBmorkt2-9/+12
2014-12-24Further progress on the staff databasemorkt1-1/+2
- Moves staff<->vn linking form to the main VN edit form - Fixes a bug with linking staff aliases to VNs - Adds staff changes to the VN revisions - And some misc. improvements
2014-12-22Initial implementation of a staff/seiyuu databasemorkt3-2/+112
2014-12-02Completely drop l_vnn column from VN infoYorhel3-3/+6
Used to link to a visual-novels.net review or something. Links have been hidden and dead since ages. No need to keep the column around.
2014-10-16Completely get rid of the old charedit perm flagYorhel1-0/+4
2014-10-16Hash session tokens with SHA-1 when storing in DBYorhel1-0/+6
This ensures that, if an attacker evers gets read access to the database, he will not be able to compromise any accounts. SHA-1 suffices here, because the data being hashed is a random 20 byte string. The search space is so damn large that you can't sanely brute force it, nor are rainbow tables any use at that scale. They're not salted. The password reset tokens are also hashed in the database and do include salt, but I've no idea why we did that.
2014-10-15Use scrypt for new password hashesYorhel1-1/+7
I increased the N parameter to approximate about 500ms to generate the hash. This is quite a paranoid setting for a website, but login attempts are throttled so there's not much of a DoS factor. (Alright, password changing feature isn't throttled so the DoS factor still exists. But really, there's some pages with longer page generation times anyway.) I did lower the size of the salt a bit (Crypt::ScryptKDF uses 256 bits by default), because 64 bits of randomness should have low enough chance of collision with only ~100k users (even with a million users, seriously).
2014-10-15SQL: Merge users.(passwd|salt) in one column + document valuesYorhel2-1/+15
It doesn't make a whole lot to separate the hashed password and the salt from each other, you need both to do anything with them, and from the database perspective they're both completely opaque strings only usable for direct comparison with other hashed strings. This change is mostly as preparation for switching to a proper key derivation function (sha256 isn't...) and to add support for longer and/or binary salt. Because the passwd field now needs to be interpreted in Perl, it's being passed around as a binary string rather than a hex-encoded value. API login is broken in this commit. I'll get to that.
2014-10-13SQL: Use enum to represent platformsYorhel4-4/+15
I believe I didn't do this conversion earlier (back when I converted the language types) because PostgreSQL didn't support dynamically adding new values to an existing enum back then, and modifying an enum was a huge pain. Recent versions do support this, so there's no reason to keep it as a string. ...I just felt like adding some churn to the code base.
2014-10-11SQL: Convert login_throttle.timeout to a timestamptzYorhel2-1/+5
Easier to work with in custom queries.
2014-09-15Change normalization factor in popularity ranking calculationYorhel1-1/+1
Suggested by Hinoe, quoting his reasoning: In popularity rankings, change the normalization from "sqrt(LowerVoteCount)" == "LowerVoteCount^0.5" to something that grows somewhat more slowly. Details: Natural logarithm itself (ln(LowerVoteCount+1)) is too slow; at the current VN count (15403), it returns 9.64; however, sqrt(15402) is just above 124.1, which I feel is already too high. After experimenting with the exponents a bit, I decided that the best point likely lies between 0.3, which returns a bit above 18.0, and 0.4, which returns a bit above 47.3. Thus, I suggest that the new function be LowerVoteCount^0.36788; the exponent is a 5-digit approximation of e^-1, just because it's a nice number in the specified area and works well, returning circa 34.7.
2014-09-01Disallow ( and ) in email regexYorhel1-1/+1
2014-08-29Throttle failed login attempts (10/day)Yorhel2-0/+12
2014-08-29Sync language type util/sql/all.sql with recent updateYorhel1-1/+1