Age | Commit message (Collapse) | Author | Files | Lines |
|
My testing, deployment and backup scripts were getting more complicated
with files from various stages being lumped into a single directory
structure, so all generated files (= anything touched by 'make') and
runtime files (= anything touched by the web backend or multi) have now
been moved into separate directories. These directories are also
configurable with $VNDB_GEN and $VNDB_VAR, making it possible to manage
multiple instances from a single source checkout.
I also got rid of *data/* and *dl/* while I was at it, and moved
*static/st/* (that is, the screenshot thumbnails) to */sf.t/*, to be
consistent with the newer and more flexible image directory naming
scheme.
This commit breaks all existing installations and upgrading requires
manual action. General upgrade instructions:
# BEFORE doing a checkout of this commit
make clean
# AFTER
make
mkdir -p var/static var/log
mv static/st var/static/sf.t
mv static/{sf,ch,cv}{,.orig} var/static/
mv data/conf.pl var/
mv data/log var/
mv data/hibp var/
mv dl var/
util/setup-var.sh
Use `git status` find leftover files to clean up or move.
Don't forget to update conf.pl and web server configuration to make sure
they access the new paths.
|
|
My goals, in particular, were to:
- Make use of a more efficient jpeg encoder; jpegli in this case.
- Sandbox the image handling process to protect against codec
vulnerabilities; implemented with seccomp.
This isn't easy to do with software packaged in typical Linux repos, so
I had to improvise a bit. I switched to libvips as that is easier to
sandbox than ImageMagick and provided a Makefile to fetch and build a
custom libvips that uses jpegli. None of this is very portable, so to
simplify testing setups a simpler "imgproc-portable" binary is built
instead.
Also experimented with different resizing and encoding options,
but the defaults were mostly okay. See https://vndb.org/t21692
(This breaks the Docker setup, I'll fix that later. I'll also see if I
can drop the ImageMagick dependency that's still used for pngsprite.pl,
as it seems a little redundant)
|
|
So that they can be recompressed or resized to new formats, in case we
ever add support for alternative sizes or formats.
|
|
Couldn't get fragment identifiers to work satisfyingly, so ended up
going back to old-fashioned CSS background-position. Oh well.
|
|
This change also increases browser version requirements by quite a bit,
though it should still be fair. Pale Moon is an easy "old enough" target
to test.
This new organization is, in a way, pretty similar to what we had before
the switch to Elm, except with more experience I might be able to do
this in a more manageable way.
I have to admit, I like arrow functions more than I had expected.
|
|
These aren't new, but for some reason I thought they weren't interesting
enough to keep in git.
|
|
Currently has 851 million password hashes, taking about 8G of space with
the current approach. It's simple and fast, so should be worth it.
inb4 complains about "why can't I use my password anymore!?"
|
|
Had been planning to use a more powerful preprocessor for CSS for a
while, so that I can also reorganize and clean up the CSS a bit. The
cleanup will come later, this is the first step to reorganize the build
system a bit and remove skingen.pl.
I moved all generated static assets to static/g/ (for _g_enerated),
including icons.png and js files. This simplifies management of
static/f/ and static/s/, which are fully in git.
Skins are now defined as sass files in css/skins/ with their images in
static/s/ using plain directory structure.
|
|
I was waiting for a new stable release so we can get Postgres 13, which
I've already been using in my own dev & prod environments. I don't think
Postgres 13 specific features are being used yet, but that will no doubt
change.
Image::Magick removed as it isn't used anymore. The image is somewhat
more bloated, though, as postgresql-dev pulls in llvm+clang now.
|
|
This means the feeds are now generated on demand rather than every 15
minutes. Main reason for originally implementing this into Multi was
because RSS feeds tended to get requested a *lot* and I didn't want
those requests to impact site performance, but now that RSS is almost
dead it doesn't really matter that much anymore. A caching layer can
still be added anyway.
Be sure to restart Multi and delete www/feeds/ after this change.
|
|
The trace_log showed that the vast majority of pageviews do not need any
Elm modules at all, so we can save a lot by not loading them on the page.
Some pages (most notably: VN pages for not-logged-in visitors) do need
JS but don't load any Elm modules. I've added a 'js' flag to
HTML::framework_() to indicate that and ensure that plain.js is still
loaded.
|
|
I had already rambled on the current composite type solution in
583ae868dfd3c882a8d2dd40b5d5ed099170c1c2 and I had already explored a
few alternatives. This was the one alternative I hadn't yet explored
because I wasn't sure the operational complexity was going to be worth
it, but after seeing how bad PostgreSQL was at optimizing queries with
composite types, I figured I might as well just go with this approach.
It improves performance of some queries by a *lot* (especially the image
selection query) and it's pretty elegant and convenient to work with.
Only downside is the complexity of compiling, installing and maintaining
a vndbid.so library for PostgreSQL.
|
|
Been wanting to do this for a while...
I've kept util/sql as a symlink for compatibility with the devdump, old
update scripts and other code I may have forgotten. I'll remove it
later.
|
|
Most of this is copied from v3. I did improve on a few aspects:
- db_edit() and db_entry() use VNDB::Schema rather than dynamically
querying the DB. This has the minor advantage of a faster startup.
- The Elm code generator now writes to multiple files, this avoids
the namespace pollution seen in v3's Lib.Gen and makes the dependency
graph a bit more lean (i.e. faster incremental builds).
- The Elm code generator doesn't update the timestamp of files that
haven't been modified. This also speeds up incremental builds, the elm
compiler can now skip rebuilding unmodified files.
- The Elm API response generator code now uses plain functions rather
than code references and all possible responses are now defined in
Elm.pm. Turns out most API responses were used from more than a single
place, so it makes sense to have them centrally defined.
The doc page preview function is also much nicer; I'd like to apply this
to all BBCode textareas as well.
(Elm.pm itself is ugly as hell though. And we will prolly need some HTML
form generation functions in Elm to make that part less verbose)
|
|
This gets rid of global.pl, config.pl and config3.pl and uses the
cleaner config3.pl format for the config file. The config is easily
accessible from anywhere by importing the new VNDB::Config module; The
global $VNDB::S,O,M,ROOT variables have been removed.
Sorry for all the churn...
|
|
I think this is the last one. 'make' in a development environment is now
beautifully fast and 'make prod' generates nicely small assets.
(arguably we could have an even faster dev setup by not generating an
icons.png in the first place, but then we'd need more code to
differentiate between dev & prod, which is also a pain)
This change does remove the "slow" option that would use the compressed
image size in the optimization algorithm, but I hadn't used that option
for a while anyway, it takes an hour and only saves about 100 bytes.
|
|
Similar to previous commit.
|
|
This rather significantly speeds up development builds. Also simplifies
skingen.pl and its config a bit.
The new compressed files are written to /s/*/style.min.css{,.gz}, it is
up to the web server to serve those instead of /s/*/style.css.
|
|
I was getting tired of having to keep two branches up-to-date with the
latest developments, so decided to throw v3 into the same branch - just
different files (...which will get mostly rewritten again soon). The two
versions aren't very different in terms of dependencies, build system
and support code, so they can now properly share files. Added a section
to the README to avoid confusion.
This merge also makes it easier to quickly switch between the different
versions, which is handy for development. It's even possible to run both
at the same time, but my scripts use the same port so that needs a
workaround.
And it's amazing how often I break the Docker scripts.
|
|
I initially wanted to move the static/.. files onto the Docker volume,
so that all dynamic site data is stored in a single place. But that
turned out to be impossible to do without some really ugly hacks. So
instead I went with the opposite approach: get rid of the 'vndb-data'
volume and instead store everything in the source directory. This also
requires running PostgreSQL as the 'devuser', but that's fine for a
development setup. All of this makes it more obvious what is going on
and simplifies the init script.
|
|
This affects the following:
- API login with a weak password is disallowed, affected users will have
to change their password through the website to continue using the API.
- Registration, password reset or password change forms require the new
password to not be in the dictionary.
- Attempting to log in to the website with a weak password will
force-redirect to a password change form, allowing a new password to
be set (using the weak-but-still-valid password as check).
|
|
It's been a while since I had static/f/ in git, so I had to adjust
.gitignore a bit.
The CSS changes are purely opinion, but it does integrate better with
the existing layout.
Everything else are bug fixes.
|
|
|
|
This changes quite a bit to the way the editing functions work. Because
these functions are very repetitive and it's easy to keep things out of
sync, I created a script to generate them automatically. I had to rename
a few function and table names for consistency to make this work.
Since database entries don't have a 'latest' column anymore, and since
the order in which tables are updated doesn't have to be fixed, I
dropped many of the SQL triggers and replaced them with a
edit_committed() function which is called from edit_*_commit() and
checks for stuff to be done.
Don't forget to run 'make' before importing the update script.
|
|
It became a bit of a hassle to keep updating that file manually in Gimp.
This script performs surprisingly well for our set of icons.
|
|
This isn't documented yet.
|
|
|
|
This greatly reduces the size of the Javascript file. The compressed
size has been reduced with about 9kB, and is now a total of 14kB for
en.js. A nice property of this is that more translations can be added
without increasing the JS size.
While I was at it, I made jsgen.pl also replace mt() function calls in
cases where an exact TL string was requested without any additional
arguments and/or formatting codes. This helped reduce the compressed
size by about 1kB.
My aim is to keep *all* the JS code of VNDB smaller than the jQuery core
library, as a general "fuck you" towards users of large and bloated JS
libraries. We must keep the VNDB page loading times lower than that of
other sites, after all!
|
|
TODO: add links to these feeds from the site
|
|
This page is pretty specific to the main VNDB, and each VNDB would probably
have it's own d8.
|
|
|
|
While this 'processing' is currently limited to minifying the file if
JavaScript::Minifier::XS is available, this change would make it a lot
easier to make the strings in the JS code translatable.
|
|
The graphs are now stored in the DB in SVG format, the static/rg/
directory can be removed (not used anymore).
SVG data is stored using the xml data type, so now I can say for
sure you'd need at least PostgreSQL 8.3.
This feature still needs some tweaking, though. Current state isn't
perfect.
|
|
How it works:
Create new directory in static/s/
Create a 'conf' file (see the test skin for a template)
Run skingen.pl, which will generate a style.css and boxbg.png
This process will probably be automated using a simple web interface
or something...
There's no skin selector yet, so Util/LayoutHTML.pm has to be modified
to view the generated skin.
|
|
...this is basically everything we're going to rewrite
|
|
|