| Age | Commit message (Collapse) | Author | Files | Lines |
|---|
|
Pretty much a copy of TagEdit with some minor differences. Might make
sense to merge these into a single file.
|
|
This is a lot more robust than that silly pid file.
|
|
|
|
My testing, deployment and backup scripts were getting more complicated
with files from various stages being lumped into a single directory
structure, so all generated files (= anything touched by 'make') and
runtime files (= anything touched by the web backend or multi) have now
been moved into separate directories. These directories are also
configurable with $VNDB_GEN and $VNDB_VAR, making it possible to manage
multiple instances from a single source checkout.
I also got rid of *data/* and *dl/* while I was at it, and moved
*static/st/* (that is, the screenshot thumbnails) to */sf.t/*, to be
consistent with the newer and more flexible image directory naming
scheme.
This commit breaks all existing installations and upgrading requires
manual action. General upgrade instructions:
# BEFORE doing a checkout of this commit
make clean
# AFTER
make
mkdir -p var/static var/log
mv static/st var/static/sf.t
mv static/{sf,ch,cv}{,.orig} var/static/
mv data/conf.pl var/
mv data/log var/
mv data/hibp var/
mv dl var/
util/setup-var.sh
Use `git status` find leftover files to clean up or move.
Don't forget to update conf.pl and web server configuration to make sure
they access the new paths.
|
|
Not particularly useful, but I always enjoy removing dependencies.
|
|
Wanted to upgrade to Alpine 3.19 while I was at it, but getting
postgresql15-dev and libpq-dev installed at the same time didn't go
well. Upgrading to Postgres 16 also didn't go well, I'll probably need
to fix vndbfuncs for that. Later.
|
|
My goals, in particular, were to:
- Make use of a more efficient jpeg encoder; jpegli in this case.
- Sandbox the image handling process to protect against codec
vulnerabilities; implemented with seccomp.
This isn't easy to do with software packaged in typical Linux repos, so
I had to improvise a bit. I switched to libvips as that is easier to
sandbox than ImageMagick and provided a Makefile to fetch and build a
custom libvips that uses jpegli. None of this is very portable, so to
simplify testing setups a simpler "imgproc-portable" binary is built
instead.
Also experimented with different resizing and encoding options,
but the defaults were mostly okay. See https://vndb.org/t21692
(This breaks the Docker setup, I'll fix that later. I'll also see if I
can drop the ImageMagick dependency that's still used for pngsprite.pl,
as it seems a little redundant)
|
|
|
|
|
|
|
|
|
|
This change also increases browser version requirements by quite a bit,
though it should still be fair. Pale Moon is an easy "old enough" target
to test.
This new organization is, in a way, pretty similar to what we had before
the switch to Elm, except with more experience I might be able to do
this in a more manageable way.
I have to admit, I like arrow functions more than I had expected.
|
|
|
|
Had a little fight with ConnMan and network setup stuff on my laptop,
but eventually managed to get docker to run over there. What a mess.
|
|
The previous TRUNCATE + INSERT approach is still faster, but has the
major downside that it acquires an exclusive lock on the cache tables
and hence brings down the site for 10 seconds. This MERGE approach is a
bit slower, but avoids unnecessary writes and the site should hopefully
remain responsive. Both implementations perform equally well for
single-VN updates.
If this approach turns out to work, I'll also fix traits_chars_calc(),
which is responsible for taking down the site for 20 seconds every day.
If this doesn't work, new table + rename is prolly also a viable option.
|
|
function
The HASHES and MERGES options enable some pretty important join
strategies, amazing how I haven't stumbled upon major performance issues
before without them.
The equalimage function enables btree key deduplication in Postgres 13+.
Won't have a major impact on performance, but still a nice to have.
I don't have a migration script for these changes, it'll involve
updating all existing columns and indices that use the vndbid type. I'll
instead work on a script to dump all data and use the sql/ files to
re-import everything. Been wanting such a script for a while, anyway.
|
|
Been meaning to add a section like this for a while. This section
totally isn't complete, but at least it sets the right expectations.
|
|
Yay!
There are no more request handlers in the VNDB::* namespace and no more
Javascript in data/js/. This cleans up a lot of old legacy code that
wasn't fun to maintain.
|
|
Had been planning to use a more powerful preprocessor for CSS for a
while, so that I can also reorganize and clean up the CSS a bit. The
cleanup will come later, this is the first step to reorganize the build
system a bit and remove skingen.pl.
I moved all generated static assets to static/g/ (for _g_enerated),
including icons.png and js files. This simplifies management of
static/f/ and static/s/, which are fully in git.
Skins are now defined as sass files in css/skins/ with their images in
static/s/ using plain directory structure.
|
|
Just pointless cleanup because I'm not in the mood to do meaningful
work...
|
|
This has three potential advantages:
- Memory used for image processing is released immediately, thus
resulting in lower average memory usage. I've yet to test how
significant this is in practice... I'm not entirely sure how VNDB's
memory use is distributed (not that it's really a problem, but if I
see something that could be optimized I try to optimize it. First
world engineering problems)
- The 'convert' command can be better sandboxed for increased security.
- Removes a dependency on a Perl module... except spritegen.pl still
uses it, but that can be rewritten as well if necessary.
In the process I found out that the old code's attempt to sharpen the
image after resizing was buggy and the sharpening was never applied.
This new code *does* apply the sharpen filter, so newly uploaded images
will be visually different (if resized).
I also found that imagemagick's built-in ratio-preserving resize
sometimes ends up with different image dimensions than my imgsize(). I
think that's a bug in my imgsize function, but it does result in a
discrepancy that should prolly be resolved as it may result in
screenshot thumbnails being displayed incorrectly.
|
|
|
|
I had already rambled on the current composite type solution in
583ae868dfd3c882a8d2dd40b5d5ed099170c1c2 and I had already explored a
few alternatives. This was the one alternative I hadn't yet explored
because I wasn't sure the operational complexity was going to be worth
it, but after seeing how bad PostgreSQL was at optimizing queries with
composite types, I figured I might as well just go with this approach.
It improves performance of some queries by a *lot* (especially the image
selection query) and it's pretty elegant and convenient to work with.
Only downside is the complexity of compiling, installing and maintaining
a vndbid.so library for PostgreSQL.
|
|
Been wanting to do this for a while...
I've kept util/sql as a symlink for compatibility with the devdump, old
update scripts and other code I may have forgotten. I'll remove it
later.
|
|
The producers.rgraph column still exists and the old graphs are still
being generated - that will be removed if this new approach works out.
|
|
This adds a method to signal that the docker image should be rebuilt
after a change or update to the dependencies - which is much better than
getting weird hard-to-debug errors because an older version of TUWF
happens to be missing a function or something.
|
|
|
|
|
|
|
|
Most of this is copied from v3. I did improve on a few aspects:
- db_edit() and db_entry() use VNDB::Schema rather than dynamically
querying the DB. This has the minor advantage of a faster startup.
- The Elm code generator now writes to multiple files, this avoids
the namespace pollution seen in v3's Lib.Gen and makes the dependency
graph a bit more lean (i.e. faster incremental builds).
- The Elm code generator doesn't update the timestamp of files that
haven't been modified. This also speeds up incremental builds, the elm
compiler can now skip rebuilding unmodified files.
- The Elm API response generator code now uses plain functions rather
than code references and all possible responses are now defined in
Elm.pm. Turns out most API responses were used from more than a single
place, so it makes sense to have them centrally defined.
The doc page preview function is also much nicer; I'd like to apply this
to all BBCode textareas as well.
(Elm.pm itself is ugly as hell though. And we will prolly need some HTML
form generation functions in Elm to make that part less verbose)
|
|
The FCGI module is only required when running in FastCGI mode, which
isn't how the container is configured. The AnyEvent::HTTP module, on the
other hand, is required for many of the new Multi::* modules. They're
not enabled by default but are still a significant part of Multi, so
it's good to have the dependencies available.
|
|
This bumps the minimum Perl version to 5.26 in order to make use of
lexical subroutines - a feature I've been wanting for a while. This
should be the last version bump, 5.26 is the highest version in Ubuntu
LTS at the moment. Not that I use Ubuntu, but it's used by the Docker
container and it's a sensible reference.
I merged the 'maintabs' and 'hiddenmsg' features into the primary
framework_ function; It fits quite well there, removes a little bit
of boilerplate from the DB entry page code and reduces the reliance on
common "dbSomethingGet()" methods.
I was hoping I'd be able to reduce the boilerplate required for defining
revisions, but I don't think that's going to happen. What I did do was
reimplement the diffing to handle item and text diffs separately, with
sensible defaults for the old split/join/diff options. Diffing is now
performed on the raw structured data rather than on formatted HTML,
which, combined with the db_entry() functions, ought to be less brittle.
|
|
I copied and modified VN3::DB to VNWeb::DB, but it isn't used yet. Now
that I think about it, that module isn't "web" specific at all and
should perhaps be in the VNDB:: namespace. Oh well.
|
|
I love it when I can get rid of a dependency!
I realized in the process that I had already made Perl 2.24+ a
requirement a bit earlier on by using postfix deref. It's a useful
feature and 2.24 isn't *that* new, so let's make that official.
|
|
This gets rid of global.pl, config.pl and config3.pl and uses the
cleaner config3.pl format for the config file. The config is easily
accessible from anywhere by importing the new VNDB::Config module; The
global $VNDB::S,O,M,ROOT variables have been removed.
Sorry for all the churn...
|
|
|
|
|
|
I was getting tired of having to keep two branches up-to-date with the
latest developments, so decided to throw v3 into the same branch - just
different files (...which will get mostly rewritten again soon). The two
versions aren't very different in terms of dependencies, build system
and support code, so they can now properly share files. Added a section
to the README to avoid confusion.
This merge also makes it easier to quickly switch between the different
versions, which is handy for development. It's even possible to run both
at the same time, but my scripts use the same port so that needs a
workaround.
And it's amazing how often I break the Docker scripts.
|
|
I initially wanted to move the static/.. files onto the Docker volume,
so that all dynamic site data is stored in a single place. But that
turned out to be impossible to do without some really ugly hacks. So
instead I went with the opposite approach: get rid of the 'vndb-data'
volume and instead store everything in the source directory. This also
requires running PostgreSQL as the 'devuser', but that's fine for a
development setup. All of this makes it more obvious what is going on
and simplifies the init script.
|
|
Postgres
Update README with basic information on Multi
(cherry picked from commit 01188a82ab736a8975c73ac5ec12621426bf6bf2)
|
|
https://vndb.org/t11296
|
|
|
